1、缺失数据的处理
1.1、判断是否为nan
isnull： 是nan为true
notnull： 不是nan为true
import pandas as pd
import numpy as npt3 = pd.DataFrame(np.arange(12).reshape(3,4), index=list("abc"), columns=list("wxyz"))
print(t3)
"""w x y z
a 0 1 2 3
b 4 5 6 7
c 8 9 10 11
"""
t3.iloc[1:,:2] = np.nan
print(t3)
"""w x y z
a 0.0 1.0 2 3
b NaN NaN 6 7
c NaN NaN 10 11
"""
print(pd.isnull(t3))
"""w x y z
a False False False False
b True True False False
c True True False False
"""
print(pd.notnull(t3))
"""w x y z
a True True True True
b False False True True
c False False True True
"""
print(t3[pd.notnull(t3["w"])])
"""w x y z
a 0.0 1.0 2 3
"""
1.2、删除为nan的行 或 列
https://www.runoob.com/pandas/pandas-cleaning.html
dropna(axis=0,how=“any”, thresh=None,subset=None，inplace=False)
dropna 删除为nan 的 行或者 列 ;
axis=0 删除行 axis=1 删除列
how=“all” 全为nan 的删除， how=“any” 有一个为nan的就删除
inplace=True 就地修改，修改后直接会修改原数据，默认为 False，类似于 t3 = t3.dropna(axis=0, how=“any”)
print(t3.dropna(axis=0, how="all"))
"""w x y z
a 0.0 1.0 2 3
b NaN NaN 6 7
c NaN NaN 10 11
"""
print(t3.dropna(axis=0, how="any"))
"""w x y z
a 0.0 1.0 2 3
"""
1.3、为nan 填充数据
print(t3)
"""w x y z
a 0.0 1.0 2 3
b NaN NaN 6 7
c NaN NaN 10 11
"""
# 填充固定数字
print(t3.fillna(10))
"""w x y z
a 0.0 1.0 2 3
b 10.0 10.0 6 7
c 10.0 10.0 10 11
"""
# 填充均值
print(t3.fillna(t3.mean()))
"""w x y z
a 0.0 1.0 2 3
b 0.0 1.0 6 7
c 0.0 1.0 10 11
"""
# 只对某一列 填充均值
print(t3["x"].fillna(t3["x"].mean()))
"""
a 1.0
b 1.0
c 1.0
Name: x, dtype: float64
"""
t3["x"] = t3["x"].fillna(t3["x"].mean())
print(t3)
"""w x y z
a 0.0 1.0 2 3
b NaN 1.0 6 7
c NaN 1.0 10 11
"""
"""
算均值时 pandas 不会把 nan 的算进去
但是 nunpy 会把nan 的算进去
"""
print(np.array([[1,6,5,np.nan],[1,6,5,np.nan]]).mean()) # nan
print(np.array([[1,1,1,1],[2,2,2,2]]).mean()) # 1.5
不要随便把nan赋值为0，不然计算均值 等值时，pandas会把它计算在内
算均值时 pandas 不会把 nan 的算进去，但是 nunpy 会把nan 的算进去
1.4、清洗重复数据
https://www.runoob.com/pandas/pandas-cleaning.html
import pandas as pdperson = {"name": ['Google', 'Runoob', 'Runoob', 'Taobao'],"age": [50, 40, 40, 23]
}
df = pd.DataFrame(person)print(df.duplicated())
"""
0 False
1 False
2 True
3 False
dtype: bool
"""
删除重复数据，可以直接使用drop_duplicates() 方法。
import pandas as pdpersons = {"name": ['Google', 'Runoob', 'Runoob', 'Taobao'],"age": [50, 40, 40, 23]
}df = pd.DataFrame(persons)df.drop_duplicates(inplace = True)
print(df)
"""name age
0 Google 50
1 Runoob 40
3 Taobao 23
"""
视频顺序错位 30–》29–》42
https://www.kaggle.com/damianpanek/sunday-eda/data
https://www.bilibili.com/video/BV1hx411d7jb?p=32
https://www.bilibili.com/video/BV1hx411d7jb?p=28
https://www.bilibili.com/video/BV1hx411d7jb?p=30
https://www.bilibili.com/video/BV1hx411d7jb?p=29
https://www.bilibili.com/video/BV1hx411d7jb?p=42
https://www.bilibili.com/video/BV1hx411d7jb?p=31
