4.6高级处理-缺失值处理

Pandas高级处理缺失值处理数据离散化合并交叉表与透视表分组与聚合综合案例4.6 高级处理-缺失值处理1）如何进行缺失值处理两种思路：1）删除含有缺失值的样本2）替换/插补4.6.1 如何处理nan1）判断数据中是否存在NaNpd.isnull(df)pd.notnull(df)2）删除含有缺失值的样本df.dropna(inplace=False)替换/插补df.fillna(value, inplace=False)4.6.2 不是缺失值nan，有默认标记的1）替换 ？-> np.nandf.replace(to_replace="?", value=np.nan)2）处理np.nan缺失值的步骤2）缺失值处理实例
4.7 高级处理-数据离散化性别 年龄
A    1   23
B    2   30
C    1   18物种 毛发
A    1
B    2
C    3男 女 年龄
A   1  0  23
B   0  1  30
C   1  0  18狗  猪  老鼠 毛发
A   1   0   0   2
B   0   1   0   1
C   0   0   1   1
one-hot编码&哑变量
4.7.1 什么是数据的离散化原始的身高数据：165，174，160，180，159，163，192，184
4.7.2 为什么要离散化
4.7.3 如何实现数据的离散化1）分组自动分组sr=pd.qcut(data, bins)自定义分组sr=pd.cut(data, [])2）将分组好的结果转换成one-hot编码pd.get_dummies(sr, prefix=)
4.8 高级处理-合并numpynp.concatnate((a, b), axis=)水平拼接np.hstack()竖直拼接np.vstack()1）按方向拼接pd.concat([data1, data2], axis=1)2）按索引拼接pd.merge实现合并pd.merge(left, right, how="inner", on=[索引])
4.9 高级处理-交叉表与透视表找到、探索两个变量之间的关系4.9.1 交叉表与透视表什么作用4.9.2 使用crosstab(交叉表)实现pd.crosstab(value1, value2)4.9.3 pivot_table
4.10 高级处理-分组与聚合4.10.1 什么是分组与聚合4.10.2 分组与聚合APIdataframesr

4.6.1如何处理nan

import pandas as pd movie = pd.read_csv("./datas/IMDB-Movie-Data.csv")
movie

1000 rows × 12 columns

# 1. 判断是否存在NaN类型的缺失值,为True的就是缺失值
movie.isnull()

1000 rows × 12 columns

import numpy as np# any() 只要有一个True就会返回True
# 返回结果为True，说明数据中存在缺失值
np.any(movie.isnull())

True

# 为False的就是缺失值
pd.notnull(movie)

1000 rows × 12 columns

# all()只要有一个False就返回False
# 返回结果为False，说明数据中存在缺失值
np.all(pd.notnull(movie))

False

pd.isnull(movie).any()

Rank                  False
Title                 False
Genre                 False
Description           False
Director              False
Actors                False
Year                  False
Runtime (Minutes)     False
Rating                False
Votes                 False
Revenue (Millions)     True
Metascore              True
dtype: bool

pd.notnull(movie).all()

Rank                   True
Title                  True
Genre                  True
Description            True
Director               True
Actors                 True
Year                   True
Runtime (Minutes)      True
Rating                 True
Votes                  True
Revenue (Millions)    False
Metascore             False
dtype: bool

# 缺失值处理
# 方法1： 删除含有缺失值的样本
movie_full = movie.dropna()

movie_full.isnull().any()

Rank                  False
Title                 False
Genre                 False
Description           False
Director              False
Actors                False
Year                  False
Runtime (Minutes)     False
Rating                False
Votes                 False
Revenue (Millions)    False
Metascore             False
dtype: bool

# 方法2： 替换
movie.head()

movie["Revenue (Millions)"].mean()

82.95637614678897

# 含有缺失值的字段
# Revenue (Millions)    False
# Metascore             False
movie["Revenue (Millions)"].fillna(movie["Revenue (Millions)"].mean(),inplace=True)

movie["Revenue (Millions)"].isnull().any()

False

# inplace=True ,直接在原数据上进行填充
movie["Metascore"].fillna(movie["Metascore"].mean(),inplace=True)

movie["Metascore"].isnull().any()

False

movie.isnull().any() # 缺失值已经处理完毕

Rank                  False
Title                 False
Genre                 False
Description           False
Director              False
Actors                False
Year                  False
Runtime (Minutes)     False
Rating                False
Votes                 False
Revenue (Millions)    False
Metascore             False
dtype: bool

不是缺失值nan，有默认标记的处理方法

data = pd.read_csv("./datas/GBvideos.csv",encoding="GBK")

data

1600 rows × 11 columns

# 1. 将 ！ 替换为np.nan
new_data = data.replace(to_replace="?",value=np.nan)

new_data

1600 rows × 11 columns

new_data.isnull().any() # 说明dislikes列中的？已经替换成了NaN

video_id          False
title             False
channel_title     False
category_id       False
tags              False
views             False
likes             False
dislikes           True
comment_total     False
thumbnail_link    False
date              False
dtype: bool

new_data.dropna(inplace=True)

new_data.isnull().any()

video_id          False
title             False
channel_title     False
category_id       False
tags              False
views             False
likes             False
dislikes          False
comment_total     False
thumbnail_link    False
date              False
dtype: bool

4.7 高级处理-数据离散化

import pandas as pd # 准备数据
data = pd.Series([165,174,160,180,159,163,192,184],index=["No1:165","No2:174","No3:160","No4:180","No5:159","No6:163","No7:192","No8:184"])
data

No1:165    165
No2:174    174
No3:160    160
No4:180    180
No5:159    159
No6:163    163
No7:192    192
No8:184    184
dtype: int64

自动分组

# 1. 分组# 自动分组
#qcut(data,组数)
sr = pd.qcut(data,3)
sr

No1:165      (163.667, 178.0]
No2:174      (163.667, 178.0]
No3:160    (158.999, 163.667]
No4:180        (178.0, 192.0]
No5:159    (158.999, 163.667]
No6:163    (158.999, 163.667]
No7:192        (178.0, 192.0]
No8:184        (178.0, 192.0]
dtype: category
Categories (3, interval[float64]): [(158.999, 163.667] < (163.667, 178.0] < (178.0, 192.0]]

# 查看分组情况
sr.value_counts()

(178.0, 192.0]        3
(158.999, 163.667]    3
(163.667, 178.0]      2
dtype: int64

type(sr)

pandas.core.series.Series

# 2. 将分组好的结果转换成独热编码
# prefix,设置列名的前缀
pd.get_dummies(sr,prefix="height")

自定义分组

# 自定义分组
# pd.cut(data,包含全部分界值的列表)
sr = pd.cut(data,[150,165,180,195])
sr

No1:165    (150, 165]
No2:174    (165, 180]
No3:160    (150, 165]
No4:180    (165, 180]
No5:159    (150, 165]
No6:163    (150, 165]
No7:192    (180, 195]
No8:184    (180, 195]
dtype: category
Categories (3, interval[int64]): [(150, 165] < (165, 180] < (180, 195]]

sr.value_counts()

(150, 165]    4
(180, 195]    2
(165, 180]    2
dtype: int64

pd.get_dummies(sr,prefix="身高")

4.8 高级处理-合并

4.8.1 pd.concat实现合并（按方向拼接）

data1 = np.arange(0,20,1).reshape(4,5)
data1 = pd.DataFrame(data1)
data1

data2 = np.arange(100,120,1).reshape(4,5)
data2 = pd.DataFrame(data2)
data2

# 将data1 和 data2 进行水平拼接
data_concat = pd.concat([data1,data2],axis=1)

data_concat

data2.T

# 将data1 和 data2 进行竖直拼接
data_concat1 = pd.concat([data1,data2.T],axis=0)

data_concat1

4.8.2 pd.merge实现合并（按索引拼接）

left=pd.DataFrame({'key1':['K0','K0','K1','K2'],
'key2':['K0','K1','K0','K1'],
'A':['A0','A1','A2','A3'],
'B':['B0','B1','B2','B3']})
left

right=pd.DataFrame({'key1':['K0','K1','K1','K2'], 'key2':['K0','K0','K0','K0'], 'C':['Co','C1','C2','C3'],'D':['DO','D1','D2','D3']})
right

# 默认内连接inner
# inner 保留共有的key
result = pd.merge(left,right,on=['key1','key2'],how="inner")
result

# left ,左连接
# 左表中所有的key都保留，以左表为主进行合并
result_left = pd.merge(left,right,on=['key1','key2'],how="left")
result_left

# right ,右连接
# 右表中所有的key都保留，以右表为主进行合并
result_right = pd.merge(left,right,on=['key1','key2'],how="right")
result_right

# outer ,外连接
# 左右两表中所有的key都保留，进行合并
result_outer = pd.merge(left,right,on=['key1','key2'],how="outer")
result_outer

4.9 高级处理-交叉表与透视表

• 用来探索两个变量之间的关系

4.9.2 使用crosstab（交叉表）实现

data = pd.read_excel("./datas/szfj_baoan.xls")
data

1251 rows × 9 columns

time = "2020-06-23"
# pandas日期类型
date = pd.to_datetime(time)
date

Timestamp('2020-06-23 00:00:00')

type(date)

pandas._libs.tslibs.timestamps.Timestamp

date.year

2020

date.month

6

data["week"] = date.weekday

data.drop("week",axis=1,inplace=True)

data

1251 rows × 9 columns

data["feature"] = np.where(data["per_price"] > 5.0000,1,0)

data

1251 rows × 10 columns

# 交叉表# 查看楼层 和 每平方米单价是否>50000的关系
# 返回值为每个楼层中，为0的个数和为1的个数
data0 = pd.crosstab(data["floor_num"],data["feature"])
data0

data0.sum(axis=1) # 按行求和

floor_num
1      14
3       1
4      10
6      10
7      41
8      51
9      13
10     13
11     19
12      4
13     24
14      5
15     41
16     28
17     41
18     52
19     16
20      6
21      7
22      1
23     12
24     36
25     41
26     66
27     43
28     41
29     94
30    108
31    155
32    147
33     54
34      6
35      3
36      4
37      2
38      1
39     15
40      4
43      1
44      6
45      7
47      1
50      1
51      3
52      2
53      1
dtype: int64

data0.div(data0.sum(axis=1),axis=0) # 按行做除法

data_percent = data0.div(data0.sum(axis=1),axis=0)
data_percent

# stacked=True 是否重叠显示
data_percent.plot(kind="bar",stacked=True)

<matplotlib.axes._subplots.AxesSubplot at 0x24719dd7488>

data_percent = data0.div(data0.sum(axis=1),axis=0)
data_percent

<tr><th>50</th><td>0.000000</td><td>1.000000</td>
</tr>
<tr><th>51</th><td>0.000000</td><td>1.000000</td>
</tr>
<tr><th>52</th><td>0.000000</td><td>1.000000</td>
</tr>
<tr><th>53</th><td>0.000000</td><td>1.000000</td>
</tr>

4.9.3使用pivot_table（透视表）实现

# 通过透视表，整个过程会变得更加简单些
# 结果直接就是值为1的百分比
data.pivot_table(["feature"],index=["floor_num"])

...

4.10 高级处理-分组与聚合

4.10.2 分组与聚合API

col = pd.DataFrame({'color':['white','red','green','red','green'],'object':["pen","pencil","pencil","ashtray","pen"],'price1':[4.56,4.20,1.30,0.56,2.75],'price2':[4.75,4.12,1.68,0.75,3.15]})
col

#  进行分组，对颜色进行分组，对价格price1进行聚合
# 用DataFrame的方法进行分组
col.groupby(by="color")["price1"].max()

color
green    2.75
red      4.20
white    4.56
Name: price1, dtype: float64

# 用Series的方法进行分组
col['price1'].groupby(col["color"])

<pandas.core.groupby.generic.SeriesGroupBy object at 0x000002471D178D08>

col['price1'].groupby(col["color"]).max()

color
green    2.75
red      4.20
white    4.56
Name: price1, dtype: float64

4.11 综合案例

# 1. 准备数据
movie = pd.read_csv("./datas/IMDB-Movie-Data.csv")
movie

1000 rows × 12 columns

#问题1：我们想知道这些电影数据中评分的平均分，导演的人数等信息，
# 我们应该怎么获取？
movie["Rating"].mean()

6.723200000000003

movie["Director"]

0                James Gunn
1              Ridley Scott
2        M. Night Shyamalan
3      Christophe Lourdelet
4                David Ayer...         
995               Billy Ray
996                Eli Roth
997              Jon M. Chu
998          Scot Armstrong
999        Barry Sonnenfeld
Name: Director, Length: 1000, dtype: object

# np.unique()去重，因为导演可能是多个电影的导演
np.unique(movie["Director"])

array(['Aamir Khan', 'Abdellatif Kechiche', 'Adam Leon', 'Adam McKay','Adam Shankman', 'Adam Wingard', 'Afonso Poyart', 'Aisling Walsh','Akan Satayev', 'Akiva Schaffer', 'Alan Taylor', 'Albert Hughes','Alejandro Amenábar', 'Alejandro González Iñárritu',...'Tomas Alfredson', 'Tony Gilroy', 'Tony Scott', 'Travis Knight','Tyler Shields', 'Wally Pfister', 'Walt Dohrn', 'Walter Hill','Warren Beatty', 'Werner Herzog', 'Wes Anderson', 'Wes Ball','Wes Craven', 'Whit Stillman', 'Will Gluck', 'Will Slocombe','William Brent Bell', 'William Oldroyd', 'Woody Allen','Xavier Dolan', 'Yimou Zhang', 'Yorgos Lanthimos', 'Zack Snyder','Zackary Adler'], dtype=object)

# 导演的人数
np.unique(movie["Director"]).size

644

# 问题2 ： 对于这一组电影数据，如果我们先rating,runtime的分布情况，应该如何呈现数据？
movie["Rating"].plot(kind="hist",figsize=(20,8),fontsize=40)

<matplotlib.axes._subplots.AxesSubplot at 0x2471ce18708>

import matplotlib.pyplot as plt# 1. 创建画布
plt.figure(figsize=(20,8),dpi=100)# 2. 绘制直方图
plt.hist(movie["Rating"],20)# 修改刻度
plt.xticks(np.linspace(movie["Rating"].min(),movie["Rating"].max(),21))# 添加网格
plt.grid(linestyle="--",alpha=0.5)# 3. 显示图像
plt.show()

movie["Rating"]

0      8.1
1      7.0
2      7.3
3      7.2
4      6.2... 
995    6.2
996    5.5
997    6.2
998    5.6
999    5.3
Name: Rating, Length: 1000, dtype: float64

# 问题3：对于这一组电影数据，如果我们希望统计电影分类（genre）的情况，应该如何处理数据？# 先统计电影类别有哪些
movie_genre = [i.split(",") for i in movie["Genre"]]
movie_genre

[['Action', 'Adventure', 'Sci-Fi'],['Adventure', 'Mystery', 'Sci-Fi'],['Horror', 'Thriller'],['Animation', 'Comedy', 'Family'],['Action', 'Adventure', 'Fantasy'],...['Horror'],['Drama', 'Music', 'Romance'],['Adventure', 'Comedy'],['Comedy', 'Family', 'Fantasy']]

[j for i in movie_genre for j in i]

['Action','Adventure','Sci-Fi','Adventure','Mystery','Sci-Fi',
...'Animation','Action','Adventure','Action','Adventure','Drama',...]

movie_class = np.unique([j for i in movie_genre for j in i])

movie_class

array(['Action', 'Adventure', 'Animation', 'Biography', 'Comedy', 'Crime','Drama', 'Family', 'Fantasy', 'History', 'Horror', 'Music','Musical', 'Mystery', 'Romance', 'Sci-Fi', 'Sport', 'Thriller','War', 'Western'], dtype='<U9')

len(movie_class) # 20 个电影类别

20

# 统计每个类别有几个电影# 先创建一个空的DataFrame表
count = pd.DataFrame(np.zeros(shape=[1000,20],dtype="int32"),columns=movie_class)

count.head()

count.loc[0,movie_genre[0]]

Action       0
Adventure    0
Sci-Fi       0
Name: 0, dtype: int32

movie_genre[0]

['Action', 'Adventure', 'Sci-Fi']

# 计数填表
for i in range(1000):count.loc[i,movie_genre[i]] = 1

count

1000 rows × 20 columns

# 按列求和
count.sum(axis=0)

Action       303
Adventure    259
Animation     49
Biography     81
Comedy       279
Crime        150
Drama        513
Family        51
Fantasy      101
History       29
Horror       119
Music         16
Musical        5
Mystery      106
Romance      141
Sci-Fi       120
Sport         18
Thriller     195
War           13
Western        7
dtype: int64

count.sum(axis=0).sort_values(ascending=False)

Drama        513
Action       303
Comedy       279
Adventure    259
Thriller     195
Crime        150
Romance      141
Sci-Fi       120
Horror       119
Mystery      106
Fantasy      101
Biography     81
Family        51
Animation     49
History       29
Sport         18
Music         16
War           13
Western        7
Musical        5
dtype: int64

count.sum(axis=0).sort_values(ascending=False).plot(kind="bar",fontsize=20,figsize=(20,9),colormap="cool")

<matplotlib.axes._subplots.AxesSubplot at 0x2472450c1c8>