28 Pandas的Categorical数据类型可以降低数据存储提升计算速度

1、读取数据

import pandas as pd

df = pd.read_csv("./datas/movielens-1m/users.dat",
                 sep="::",
                 engine="python",
                 header=None,
                 names="UserID::Gender::Age::Occupation::Zip-code".split("::"))

df.head()

df.info()

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 6040 entries, 0 to 6039
    Data columns (total 5 columns):
    UserID        6040 non-null int64
    Gender        6040 non-null object
    Age           6040 non-null int64
    Occupation    6040 non-null int64
    Zip-code      6040 non-null object
    dtypes: int64(3), object(2)
    memory usage: 236.1+ KB
    


df.info(memory_usage="deep")


    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 6040 entries, 0 to 6039
    Data columns (total 5 columns):
    UserID        6040 non-null int64
    Gender        6040 non-null object
    Age           6040 non-null int64
    Occupation    6040 non-null int64
    Zip-code      6040 non-null object
    dtypes: int64(3), object(2)
    memory usage: 873.4 KB
    

df_cat = df.copy()
df_cat.head()

2、使用categorical类型降低存储量

df_cat["Gender"] = df_cat["Gender"].astype("category")

df_cat.info(memory_usage="deep")


    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 6040 entries, 0 to 6039
    Data columns (total 5 columns):
    UserID        6040 non-null int64
    Gender        6040 non-null category
    Age           6040 non-null int64
    Occupation    6040 non-null int64
    Zip-code      6040 non-null object
    dtypes: category(1), int64(3), object(1)
    memory usage: 513.8 KB
    


df_cat.head()

df_cat["Gender"].value_counts()



    M    4331
    F    1709
    Name: Gender, dtype: int64

3、提升运算速度

%timeit df.groupby("Gender").size()


    564 µs ± 10.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
    


%timeit df_cat.groupby("Gender").size()


    324 µs ± 5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
    

本文使用 文章同步助手 同步