28 Pandas的Categorical数据类型可以降低数据存储提升计算速度
1、读取数据
import pandas as pd
df = pd.read_csv("./datas/movielens-1m/users.dat",
sep="::",
engine="python",
header=None,
names="UserID::Gender::Age::Occupation::Zip-code".split("::"))
df.head()
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
<pre><code>.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</code></pre>
|
UserID |
Gender |
Age |
Occupation |
Zip-code |
| 0 |
1 |
F |
1 |
10 |
48067 |
| 1 |
2 |
M |
56 |
16 |
70072 |
| 2 |
3 |
M |
25 |
15 |
55117 |
| 3 |
4 |
M |
45 |
7 |
02460 |
| 4 |
5 |
M |
25 |
20 |
55455 |
df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 6040 entries, 0 to 6039
Data columns (total 5 columns):
UserID 6040 non-null int64
Gender 6040 non-null object
Age 6040 non-null int64
Occupation 6040 non-null int64
Zip-code 6040 non-null object
dtypes: int64(3), object(2)
memory usage: 236.1+ KB
df.info(memory_usage="deep")
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 6040 entries, 0 to 6039
Data columns (total 5 columns):
UserID 6040 non-null int64
Gender 6040 non-null object
Age 6040 non-null int64
Occupation 6040 non-null int64
Zip-code 6040 non-null object
dtypes: int64(3), object(2)
memory usage: 873.4 KB
df_cat = df.copy()
df_cat.head()
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
<pre><code>.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</code></pre>
|
UserID |
Gender |
Age |
Occupation |
Zip-code |
| 0 |
1 |
F |
1 |
10 |
48067 |
| 1 |
2 |
M |
56 |
16 |
70072 |
| 2 |
3 |
M |
25 |
15 |
55117 |
| 3 |
4 |
M |
45 |
7 |
02460 |
| 4 |
5 |
M |
25 |
20 |
55455 |
2、使用categorical类型降低存储量
df_cat["Gender"] = df_cat["Gender"].astype("category")
df_cat.info(memory_usage="deep")
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 6040 entries, 0 to 6039
Data columns (total 5 columns):
UserID 6040 non-null int64
Gender 6040 non-null category
Age 6040 non-null int64
Occupation 6040 non-null int64
Zip-code 6040 non-null object
dtypes: category(1), int64(3), object(1)
memory usage: 513.8 KB
df_cat.head()
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
<pre><code>.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</code></pre>
|
UserID |
Gender |
Age |
Occupation |
Zip-code |
| 0 |
1 |
F |
1 |
10 |
48067 |
| 1 |
2 |
M |
56 |
16 |
70072 |
| 2 |
3 |
M |
25 |
15 |
55117 |
| 3 |
4 |
M |
45 |
7 |
02460 |
| 4 |
5 |
M |
25 |
20 |
55455 |
df_cat["Gender"].value_counts()
M 4331
F 1709
Name: Gender, dtype: int64
3、提升运算速度
%timeit df.groupby("Gender").size()
564 µs ± 10.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit df_cat.groupby("Gender").size()
324 µs ± 5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
本文使用 文章同步助手 同步
