1.NaN(空值)与None(缺失值)
Missing data can take a few different forms:
- In Python, the None
keyword and type indicates no value. - The Pandas library uses NaN
, which stands for "not a number", to indicate a missing value.
In general terms, both NaN and None can be called null values.
2.判断缺失值/空字符：pandas.isnull(XXX)
If we want to see which values are NaN, we can use the pandas.isnull() function which takes a pandas series and returns a series of True and False
values, the same way that NumPy did when we compared arrays.
input
age = titanic_survival["age"]
print(age.loc[10:20])
age_is_null = pandas.isnull(age) # 如果是NaN或者None，返回True；否则，返回False
age_null_true = age[age_is_null] #
age_null_count = len(age_null_true)
print(age_null_count)
output
10 47.0
11 18.0
12 24.0
13 26.0
14 80.0
15 NaN
16 24.0
17 50.0
18 32.0
19 36.0
20 37.0
Name: age, dtype: float64
264
3.有null值时做加减乘除法
#计算有null值下的平均年龄
age_is_null = pd.isnull(titanic_survival["age"])
good_ages = titanic_survival['age'][age_is_null == False]
correct_mean_age1 = sum(good_ages) / len(good_ages)
#使用Series.mean()
correct_mean_age2 = titanic_survival["age"].mean()
4.用词典统计不同等级船舱的票价问题
input
passenger_classes = [1, 2, 3] #泰坦尼克的船舱等级分为1，2，3
fares_by_class = {} #创建一个空字典
for this_class in passenger_classes:
pclass_rows = titanic_survival[titanic_survival['pclass'] == this_class] # X等舱的所有数据
mean_fares = pclass_rows['fare'].mean() # X等舱的船票均值
fares_by_class[this_class] = mean_fares # 构建词典用于统计
print(fares_by_class)
output
{1: 87.508991640866881, 2: 21.179196389891697, 3: 13.302888700564973}
5.使用Dataframe.pivot_table()
Pivot tables provide an easy way to subset by one column and then apply a calculation like a sum or a mean.
刚才第4点的问题，可以用Dataframe.pivot_table()
The first parameter of the method, index
tells the method which column to group by.The second parameter values
is the column that we want to apply the calculation to.aggfunc
specifies the calculation we want to perform. The default for the aggfunc
parameter is actually the mean
input1
passenger_class_fares = titanic_survival.pivot_table(index="pclass", values="fare", aggfunc=numpy.mean)
print(passenger_class_fares)
output1
pclass
1.0 87.508992
2.0 21.179196
3.0 13.302889
Name: fare, dtype: float64
input2
passenger_age = titanic_survival.pivot_table(index="pclass", values="age",aggfunc=numpy.mean)
print(passenger_age)
output2
pclass
1.0 39.159918
2.0 29.506705
3.0 24.816367
Name: age, dtype: float64
input3
import numpy as np
port_stats = titanic_survival.pivot_table(index='embarked', values=["fare", "survived"], aggfunc=numpy.sum)
print(port_stats)
output3
fare survivedembarked C 16830.7922 150.0Q 1526.3085 44.0S 25033.3862 304.0
6.剔除缺失值：DataFrame.dropna()
The methodDataFrame.dropna()
will drop any rows that contain missing values.
drop_na_rows = titanic_survival.dropna(axis=0) # 剔除所有含缺失值的行
drop_na_columns = titanic_survival.dropna(axis=1) # 剔除所有含缺失值的列
new_titanic_survival = titanic_survival.dropna(axis=0,subset=["age", "sex"]) # 剔除所有在‘age’和‘sex’中，有缺失值的行
7.Dataframe.loc[4]与Dataframe.iloc[4]
input
# We have already sorted new_titanic_survival by age
first_five_rows_1 = new_titanic_survival.iloc[5] # 定位到按顺序第5的对象
first_five_rows_2 = new_titanic_survival.loc[5] # 定位到索引值为5的对象
row_index_25_survived = new_titanic_survival.loc[25, 'survived'] # 定位到索引值为5，且列名为'survived'的对象
print(first_five_rows_1)
print('------------------------------------------')
print(first_five_rows_2)
output
pclass 3survived 0name Connors, Mr. Patricksex maleage 70.5sibsp 0parch 0ticket 370369fare 7.75cabin NaNembarked Qboat NaNbody 171home.dest NaNName: 727, dtype: object------------------------------------------pclass 1survived 1name Anderson, Mr. Harrysex maleage 48sibsp 0parch 0ticket 19952fare 26.55cabin E12embarked Sboat 3body NaNhome.dest New York, NYName: 5, dtype: object
8.重新整理索引值：Dataframe.reset_index(drop=True)
input
titanic_reindexed = new_titanic_survival.reset_index(drop=True)
print(titanic_reindexed.iloc[0:5,0:3])
output
pclass survived name0 1.0 1.0 Barkworth, Mr. Algernon Henry Wilson1 1.0 1.0 Cavendish, Mrs. Tyrell William (Julia Florence...2 3.0 0.0 Svensson, Mr. Johan3 1.0 0.0 Goldschmidt, Mr. George B4 1.0 0.0 Artagaveytia, Mr. Ramon
9.Apply Functions Over a DataFrame
DataFrame.apply() will iterate through each column in a DataFrame, and perform on each function. When we create our function, we give it one parameter, apply() method passes each column to the parameter as a pandas series.
DataFrame可以调用apply函数对每一列（行）应用一个函数
input
def not_null_count(column):
columns_null = pandas.isnull(column) #
null = column[column_null]
return len(null)
column_null_count = titanic_survival.apply(not_null_count)
print(column_null_count)
output
pclass 1survived 1name 1sex 1age 264sibsp 1parch 1ticket 1fare 2cabin 1015embarked 3boat 824body 1189home.dest 565dtype: int64
10.Applying a Function to a Row
input
def age_label(row):
age = row['age']
if pandas.isnull(age):
return 'unknown'
elif age < 18:
return 'minor'
else:
return 'adult'
age_labels = titanic_survival.apply(age_label, axis=1) # use axis=1
so that the apply()
method applies your function over the rows
print(age_labels[0:5])
output
0 adult1 minor2 minor3 adult4 adultdtype: object
11.Calculating Survival Percentage by Age Group
Now that we have age labels for everyone, let's make a pivot table to find the probability of survival for each age group.
We have added an "age_labels"
column to the dataframe containing the age_labels
variable from the previous step.
input
age_group_survival = titanic_survival.pivot_table(index="age_labels", values="survived")
print(age_group_survival)
output
age_labelsadult 0.387892minor 0.525974unknown 0.277567Name: survived, dtype: float64
