这篇文章原文出自kaggle，文中给出了reduce_mem_usage方法可以用来自动缩减dataframe占用空间

这篇notebook展示了通过使用更合理的数据类型来减少dataframe的内存使用量

方法如下：

迭代每一个column
检查column是否为数字型
检查column是否可以用integer表示
找出column下的最大值和最小值
选择适用于数据范围的最合适的数据类型
通过以上步骤处理后将一份测试数据从1.3GB减少到466MB

 

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)def reduce_mem_usage(props):start_mem_usg = props.memory_usage().sum() / 1024**2 print("Memory usage of properties dataframe is :",start_mem_usg," MB")NAlist = [] # Keeps track of columns that have missing values filled in. for col in props.columns:if props[col].dtype != object:  # Exclude strings# Print current column typeprint("******************************")print("Column: ",col)print("dtype before: ",props[col].dtype)# make variables for Int, max and minIsInt = Falsemx = props[col].max()mn = props[col].min()# Integer does not support NA, therefore, NA needs to be filledif not np.isfinite(props[col]).all(): NAlist.append(col)props[col].fillna(mn-1,inplace=True)  # test if column can be converted to an integerasint = props[col].fillna(0).astype(np.int64)result = (props[col] - asint)result = result.sum()if result > -0.01 and result < 0.01:IsInt = True# Make Integer/unsigned Integer datatypesif IsInt:if mn >= 0:if mx < 255:props[col] = props[col].astype(np.uint8)elif mx < 65535:props[col] = props[col].astype(np.uint16)elif mx < 4294967295:props[col] = props[col].astype(np.uint32)else:props[col] = props[col].astype(np.uint64)else:if mn > np.iinfo(np.int8).min and mx < np.iinfo(np.int8).max:props[col] = props[col].astype(np.int8)elif mn > np.iinfo(np.int16).min and mx < np.iinfo(np.int16).max:props[col] = props[col].astype(np.int16)elif mn > np.iinfo(np.int32).min and mx < np.iinfo(np.int32).max:props[col] = props[col].astype(np.int32)elif mn > np.iinfo(np.int64).min and mx < np.iinfo(np.int64).max:props[col] = props[col].astype(np.int64)    # Make float datatypes 32 bitelse:props[col] = props[col].astype(np.float32)# Print new column typeprint("dtype after: ",props[col].dtype)print("******************************")# Print final resultprint("___MEMORY USAGE AFTER COMPLETION:___")mem_usg = props.memory_usage().sum() / 1024**2 print("Memory usage is: ",mem_usg," MB")print("This is ",100*mem_usg/start_mem_usg,"% of the initial size")return props, NAlist

原文链接