使用requests模块简单获取数据

一、使用ruquests的基本步骤：

指定url
发起请求
获取响应对象中的数据
持久化存储

1 #1
2 url = 'https://www.sogou.com/'
3 #2.
4 response = requests.get(url=url)
5 #3.
6 page_text = response.text
7 #4.
8 with open('./sogou.html','w',encoding='utf-8') as fp:
9     fp.write(page_text)

二、爬取搜狗指定搜索

 1 import requests
 2 url = "'https://www.sogou.com/web"
 3 wd = input("请输入搜索关键字")
 4 param = {
 5     'query':wd
 6 }
 7 
 8 response = requests.get(url=url,params=param).content
 9 filename = wd+'.html'
10 with open(filename,'w',encoding='utf8') as f1:
11         f1.write(response)

三、Ajax请求

通过抓包，获取请求携带的参数，

例如获取分页显示的数据，当点击下一页时，发送ajax请求，对此时的url请求可以动，这里我们定义好请求参数param，动态的指定页码和每页显示的数据，通过ajax请求，返回一组json数据

存储每页获取的数据的id，编辑new_url，获取详情的信息

 1 import requests
 2 url = 'http://125.35.6.84:81/xk/itownet/portalAction.do?method=getXkzsList'
 3 headers = {
 4     'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 SE 2.X MetaSr 1.0'
 5 }
 6 param = {
 7    "on":"true",
 8     "page":1,
 9     "pageSize":"15",
10     "productName":"",
11     "conditionType":"1",
12     "applyname":"",
13     "applysn":"",
14 }
15 id_list = []
16 json_object = requests.post(url=url,headers=headers,params=param).json()
17 print(json_object['list'])
18 for i in json_object['list']:
19     id_list.append(i['ID'])
20     
21 new_url = 'http://125.35.6.84:81/xk/itownet/portalAction.do?method=getXkzsById'
22 filename = 'yaojians.text'
23 f = open(filename,'w',encoding='utf8')
24 for id in id_list:
25     param = {
26         'id':id
27     }
28     content_json = requests.post(url=new_url,params=param,headers=headers).json()
29     f.write(str(content_json)+'\n')
30     
31     
32

四、使用BeautifullSoup爬取数据

bs4解析：
　　pip install bs4
　　pip install lxml

解析原理
　　1、将要进行解析的源码加载到bs对象

　　2、调用bs对象中相关的方法或属性进行源码中的相关标签的定位

　　3、将定位到的标签之间存在的文本或属性值获取

 1 import requests
 2 from bs4 import BeautifulSoup
 3 
 4 url = 'http://www.shicimingju.com/book/sanguoyanyi.html'
 5 headers = {
 6     "User-Agent":'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36'
 7 }
 8 res = requests.get(url=url, headers=headers).text
 9 soup = BeautifulSoup(res, 'lxml')
10 a_tags_list = soup.select('.book-mulu > ul > li > a')
11 filename = 'snaguo.text'
12 fp = open(filename, 'w', encoding='utf-8')
13 for a_tag in a_list:
14     title = a_tag.string
15     detail_url = "http://www.shicimingju.com"+a_tag["href"] 
16     detail_content = requests.get(url=detail_url, headers=headers).text
17     soup = BeautifulSoup(detail_content, "lxml")
18     detail_text = soup.find('div', class_="chapter_content").text
19     fp.write(title+'\n'+detail_text)
20     print(title, '下载完毕')
21 print('over')
22 fp.close()
23

五、简单使正则爬取图片

 1 url = 'https://www.qiushibaike.com/pic/page/%d/?s=5170552'
 2 start_page = int(input("请输入起始页:"))
 3 end_page = int(input("请输入结束页:"))
 4 headers = {
 5     "User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 SE 2.X MetaSr 1.0"
 6 }
 7 for page in range(start_page,end_page+1):
 8     new_url = format(url%page)
 9     response = requests.get(url=new_url, headers=headers).text
10     # 每一页的图片url
11     images_url = re.findall('<div class="thumb">.*?<img src="(.*?)" alt=.*?</div>',response,re.S)
12 os.mkdir('qiutu')
13     
14     for image_url in images_url:
15         detail_url = 'http:'+image_url
16         # 获取到当前图片的二进制流
17         content = requests.get(url=detail_url,headers=headers).content
18         # 切割 把图片路径最后的字符作为图片名
19         image_name = image_url.split('/')[-1]
20         with open('./qiutu/'+image_name,'wb')as f1:
21             f1.write(content)
22 print('over')