python之requests爬虫Boss数据
需要用到的库：reqeusts、lxml
没有的可以用直接下载
pip install requests
pip install lxm
这里以python岗位，地点北京为例
爬取的数据就是岗位名称、薪资、地点
首先导入需要用到的模块
import requests
from lxml import etree
岗位可以通过input提前输入好，传参给url
job = input('输入职位')
将需要访问的url赋给一个变量
url = 'https://www.zhipin.com/job_detail/?query=%s&city=101010100&industry=&position='%job
query=%s(%s是一个占位)在引号后面的%job就是占的值
访问这个页面需要加一个头部(headers)降低被识别爬虫的概率
在当前页面按f12点击Network，如果没有东西的话可以刷新一下页面
这里只用到两个参数，一个user-agent，一个cookie
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36',
'cookie': '_uab_collina=157853739340991408682799; Hm_lvt_194df3105ad7148dcf2b98a91b5e727a=1578537393,1578554153; __c=1578554153; __g=-; Hm_lpvt_194df3105ad7148dcf2b98a91b5e727a=1578554168; __zp_stoken__=d0e7eq77rh2ql3R%2F5VwP4mPjHKu%2BjYVQMbIFSPnpEWipSXfKaWf%2FM%2FxBRat22vE%2FR4PdiD%2BDhDiSNaW%2FTjVMpYOEMTTUmxg7WSFqYfpdWi5SSIMEcHuwoKbmd%2B6tlv5ONmSF; __l=l=%2Fwww.zhipin.com%2Fjob_detail%2F%3Fquery%3D%25E4%25BA%25BA%25E5%25B7%25A5%25E6%2599%25BA%25E8%2583%25BD%26city%3D101010100%26industry%3D%26position%3D&r=&friend_source=0&friend_source=0; __a=32343010.1578537387.1578537387.1578554153.9.2.2.9'
}
请求数据并且返回值
res = requests.get(url,headers=headers).text
利用etree中的HTML解析数据
html = etree.HTML(res)
#岗位名称
job_name = html.xpath('//*[@id="main"]/div/div[2]/ul/li/div/div[1]/h3/a/div[1]/text()')
#薪资
salary = html.xpath('//*[@id="main"]//ul/li//h3/a/span/text()')
#地点、工作经验、学历
site = html.xpath('//*[@id="main"]/div/div[2]/ul/li/div/div[1]/p')
print('工作岗位:',job)
print('薪资:',salary)
print('地点：',site)
输出的数据如下
整体代码
import requests
from lxml import etree
import json
job = input('输入职位')
url = 'https://www.zhipin.com/job_detail/?query=%s&city=101010100&industry=&position='%job
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36',
'cookie': '_uab_collina=157853739340991408682799; Hm_lvt_194df3105ad7148dcf2b98a91b5e727a=1578537393,1578554153; __c=1578554153; __g=-; __l=l=%2Fwww.zhipin.com%2Fjob_detail%2F%3Fquery%3D%25E4%25BA%25BA%25E5%25B7%25A5%25E6%2599%25BA%25E8%2583%25BD%26city%3D101010100%26industry%3D%26position%3D&r=&friend_source=0&friend_source=0; lastCity=101010100; Hm_lpvt_194df3105ad7148dcf2b98a91b5e727a=1578556532; __zp_stoken__=d0e7eq77rh2ql3R%2F5VwP4mPjHOT%2BY0u%2F2GMG6hriOPZlx6iA6NPb%2FycP1M1RRJxkLq%2FdiD%2BDhDiSNaW%2FTjVMpYOEMScFTSjVVO31G%2B8%2Bwf%2Bxs7gEcHuwoKbmd%2B6tlv5ONmSF; __a=32343010.1578537387.1578537387.1578554153.29.2.22.29'
}
res = requests.get(url,headers=headers).text
html = etree.HTML(res)
job_name = html.xpath('//*[@id="main"]/div/div[2]/ul/li/div/div[1]/h3/a/div[1]/text()')
salary = html.xpath('//*[@id="main"]//ul/li//h3/a/span/text()')
site = html.xpath('//*[@id="main"]/div/div[2]/ul/li/div/div[1]/p/text()')
print('工作岗位:',job_name)
print('薪资:',salary)
print('地点：',site)
最后需要注意网站中的cookie是实时更新，如果数据没有出来再去网页中查看cookie值
