缺失值处理
在Pandas中使用浮点值NaN表示数组中的缺失数据
- 使用
reindex()方法可以对指定轴上的索引进行改变/增加/删除操作，这将返回原始数据的一个拷贝。
In [88]: df1 = df.reindex(index=dates[0:4],columns=list(df.columns)+['E'])
In [89]: df1
Out[89]:
A B C D E
2013-01-01 -2.130791 0.903688 0.645726 -0.776207 NaN
2013-01-02 -0.622650 0.499566 -0.022492 1.326563 NaN
2013-01-03 2.140337 0.605600 -1.312784 1.059143 NaN
2013-01-04 -1.125467 -0.200313 -0.082067 -0.523501 NaN
In [90]: df1.loc[dates[0]:dates[1],'E']=1
In [91]: df1
Out[91]:
A B C D E
2013-01-01 -2.130791 0.903688 0.645726 -0.776207 1
2013-01-02 -0.622650 0.499566 -0.022492 1.326563 1
2013-01-03 2.140337 0.605600 -1.312784 1.059143 NaN
2013-01-04 -1.125467 -0.200313 -0.082067 -0.523501 NaN
- 去掉包含缺失值的行，不改变原来的值
dropna
In [92]: df1.dropna(how='any')
Out[92]:
A B C D E
2013-01-01 -2.130791 0.903688 0.645726 -0.776207 1
2013-01-02 -0.622650 0.499566 -0.022492 1.326563 1
- 对缺失值进行填充
fillna
In [94]: df1.fillna(value=5)
Out[94]:
A B C D E
2013-01-01 -2.130791 0.903688 0.645726 -0.776207 1
2013-01-02 -0.622650 0.499566 -0.022492 1.326563 1
2013-01-03 2.140337 0.605600 -1.312784 1.059143 5
2013-01-04 -1.125467 -0.200313 -0.082067 -0.523501 5
- 对数据进行布尔填充
isnullnotnull
In [95]: pd.isnull(df1)
Out[95]:
A B C D E
2013-01-01 False False False False False
2013-01-02 False False False False False
2013-01-03 False False False False True
2013-01-04 False False False False True
In [96]: df1.isnull()
Out[96]:
A B C D E
2013-01-01 False False False False False
2013-01-02 False False False False False
2013-01-03 False False False False True
2013-01-04 False False False False True
统计操作
- 统计操作
执行描述性统计默认x轴
In [98]: df.mean()
Out[98]:
A -0.234022
B 0.433988
C -0.224383
D 0.164193
dtype: float64
在其他轴上进行相同的操作
In [99]: df.mean(1)
Out[99]:
2013-01-01 -0.339396
2013-01-02 0.295247
2013-01-03 0.623074
2013-01-04 -0.482837
2013-01-05 0.580012
2013-01-06 -0.466437
Freq: D, dtype: float64
对不拥有不同维度，需要对齐的对象进行操作pandas会自动的沿着指定的维度进行广播 shift
In [101]: s
Out[101]:
2013-01-01 1
2013-01-02 3
2013-01-03 5
2013-01-04 NaN
2013-01-05 5
2013-01-06 8
Freq: D, dtype: float64
In [102]: s=pd.Series([1,3,5,np.nan,5,8],index=dates).shift(1)
In [103]: s
Out[103]:
2013-01-01 NaN
2013-01-02 1
2013-01-03 3
2013-01-04 5
2013-01-05 NaN
2013-01-06 5
Freq: D, dtype: float64
-
Apply对数据也应用函数
In [104]: df.apply(np.cumsum)
Out[104]:
A B C D
2013-01-01 -2.130791 0.903688 0.645726 -0.776207
2013-01-02 -2.753441 1.403254 0.623234 0.550356
2013-01-03 -0.613104 2.008854 -0.689550 1.609499
2013-01-04 -1.738571 1.808542 -0.771617 1.085997
2013-01-05 -1.206559 3.211595 -0.650754 1.350118
2013-01-06 -1.404133 2.603928 -1.346298 0.985156
In [107]: df.apply(lambda x:x.max()-x.min())
Out[107]:
A 4.271128
B 2.010720
C 1.958509
D 2.102770
dtype: float64
- 直方图
value_counts()
In [108]: s=pd.Series(np.random.randint(0,7,size=10))
In [109]: s
Out[109]:
0 4
1 6
2 1
3 1
4 2
5 6
6 6
7 4
8 5
9 1
dtype: int32
In [111]: s.value_counts()
Out[111]:
6 3
1 3
4 2
5 1
2 1
dtype: int64
- 字符串方法
In [112]: s=pd.Series(['A','B','C','Aaba','Baca',np.nan,'CABA','dog','cat'])
In [113]: s
Out[113]:
0 A
1 B
2 C
3 Aaba
4 Baca
5 NaN
6 CABA
7 dog
8 cat
dtype: object
In [114]: s.str.lower()
Out[114]:
0 a
1 b
2 c
3 aaba
4 baca
5 NaN
6 caba
7 dog
8 cat
dtype: object
In [115]: s.str.upper()
Out[115]:
0 A
1 B
2 C
3 AABA
4 BACA
5 NaN
6 CABA
7 DOG
8 CAT
dtype: object
