这个教程使用BeautifulSoup库爬取指定贴吧的帖子信息。
本教程的代码托管于github: https://github.com/w392807287/spider_baidu_bar
数据分析部分请移步：
python版本：3.5.2
使用BeautifulSoup库获取网页信息
引入相关库：
from bs4 import BeautifulSoup
from urllib.request import urlopen
from urllib.error import HTTPError
这里使用python吧为例子，python吧的主页为：http://tieba.baidu.com/f?ie=utf-8&kw=python&fr=search，精简一点http://tieba.baidu.com/f?kw=python
获取BeautifulSoup对象：
url = "http://tieba.baidu.com/f?kw=python"
html = urlopen(url).read()
bsObj = BeautifulSoup(html, "lxml")
这里将这一段进行封装，封装成一个传入url返回bs对象的函数：
def get_bsObj(url):'''返回给定url的beautifulsoup对象:param url:目标网址:return:beautifulsoup对象'''try:html = urlopen(url).read()bsObj = BeautifulSoup(html, "lxml")return bsObjexcept HTTPError as e:print(e)return None
这个函数传入一个url，返回beautifulsoup对象，如果发生错误则打印出错误并返回空值。
贴吧主页的处理
在贴吧主页中，包含了这个贴吧的大概信息，比如：关注量，主题数量，帖子数量。我们将这些信息汇总到一个文件。
获取主页bs对象：
bsObj_mainpage = get_bsObj(url)
获取总页数：
last_page = int(bsObj_mainpage.find("a",{"class":"last pagination-item "})['href'].split("=")[-1])
获取最后一页的页数是为了后面爬去帖子时越界。其中使用了beautifulsoup的find方法。
获取需要的信息并写入文件：
red_text = bsObj_mainpage.findAll("span", {"class": "red_text"})subject_sum = red_text[0].get_text() # 主题数post_sum = red_text[1].get_text() # 帖子数follow_sum = red_text[2].get_text() # 关注量with open('main_info.txt','w+') as f:f.writelines("统计时间: "+str(datetime.datetime.now())+"\n")f.writelines("主题数: "+subject_sum+"\n")f.writelines("帖子数: "+post_sum+"\n")f.writelines("关注量: "+follow_sum+"\n")
最后将这些步骤封装成一个函数，传入主页面的url，写入信息，返回最后一页的页码：
def del_mainPage(url):'''对主页面进行处理，返回最后页码:param url:目标主页地址:return:返回最后页码,int'''bsObj_mainpage = get_bsObj(url)last_page = int(bsObj_mainpage.find("a",{"class":"last pagination-item "})['href'].split("=")[-1])try:red_text = bsObj_mainpage.findAll("span", {"class": "red_text"})subject_sum = red_text[0].get_text() # 主题数post_sum = red_text[1].get_text() # 帖子数follow_sum = red_text[2].get_text() # 关注量except AttributeError as e:print("发生错误:" + e + "时间:"+str(datetime.datetime.now()))return Nonewith open('main_info.txt','w+') as f:f.writelines("统计时间: "+str(datetime.datetime.now())+"\n")f.writelines("主题数: "+subject_sum+"\n")f.writelines("帖子数: "+post_sum+"\n")f.writelines("关注量: "+follow_sum+"\n")return last_page
得到的结果：
统计时间: 2016-10-07 15:14:19.642933 主题数: 25083 帖子数: 414831 关注量: 76511
从页面中获取帖子地址
我们想要获取每个帖子的详细信息，就需要进入这个帖子，所以需要这个帖子的地址。比如：http://tieba.baidu.com/p/4700788764
这个url中，http://tieba.baidu.com是服务器地址，/p应该是路由中帖子（post)的对应,/4700788764即帖子的id。
我们观察贴吧首页会发现，每个帖子都是在一个块中。在浏览器中按Ｆ12观察其代码会发现，一个帖子所对应一个<li>标签，每个页面中有50个帖子（首页有广告贴可能不一样）。
其中<li>标签大概长这样：
<li class=" j_thread_list clearfix" data-field="{"id":4700788764,"author_name":"\u6768\u5175507","first_post_id":95008842757,"reply_num":2671,"is_bakan":null,"vid":"","is_good":null,"is_top":null,"is_protal":null,"is_membertop":null,"frs_tpoint":null}"><div class="t_con cleafix"><div class="col2_left j_threadlist_li_left"><span class="threadlist_rep_num center_text" title="回复">2671</span></div><div class="col2_right j_threadlist_li_right "><div class="threadlist_lz clearfix"><div class="threadlist_title pull_left j_th_tit "><a href="/p/4700788764" title="恭喜《从零开始学Python》进入百度阅读平台【首页】新书推荐榜单" target="_blank" class="j_th_tit ">恭喜《从零开始学Python》进入百度阅读平台【首页】新书推荐榜单</a>
</div><div class="threadlist_author pull_right"><span class="tb_icon_author " title="主题作者: 杨兵507" data-field="{"user_id":625543823}"><i class="icon_author"></i><span class="frs-author-name-wrap"><a data-field="{"un":"\u6768\u5175507"}" class="frs-author-name j_user_card " href="/home/main/?un=%E6%9D%A8%E5%85%B5507&ie=utf-8&fr=frs" target="_blank">杨兵507</a></span><span class="icon_wrap icon_wrap_theme1 frs_bright_icons "></span> </span><span class="pull-right is_show_create_time" title="创建时间">7-29</span>
</div></div><div class="threadlist_detail clearfix"><div class="threadlist_text pull_left"><div class="threadlist_abs threadlist_abs_onlyline ">http://yuedu.baidu.com/ebook/ec1aa9f7b90d6c85ec3ac6d7?fr=index</div></div><div class="threadlist_author pull_right"><span class="tb_icon_author_rely j_replyer" title="最后回复人: 杨兵小童鞋"><i class="icon_replyer"></i><a data-field="{"un":"\u6768\u5175\u5c0f\u7ae5\u978b"}" class="frs-author-name j_user_card " href="/home/main/?un=%E6%9D%A8%E5%85%B5%E5%B0%8F%E7%AB%A5%E9%9E%8B&ie=utf-8&fr=frs" target="_blank">杨兵小童鞋</a> </span><span class="threadlist_reply_date pull_right j_reply_data" title="最后回复时间">14:17 </span>
</div></div></div></div>
</li>
我们可以观察到<li>标签中的class=" j_thread_list clearfix" data-field="{"id":4700788764,"author_name":"\u6768\u5175507","first_post_id":95008842757,"reply_num":2671,"is_bakan":null,"vid":"","is_good":null,"is_top":null,"is_protal":null,"is_membertop":null,"frs_tpoint":null}"
根据class属性我们可以找到单个页面中所有的帖子：
posts = bsObj_page.findAll("li", {"class": "j_thread_list"})
在data-field属性中我们可以得到：帖子ＩＤ，作者名称，回复数量，是否精品等信息。根据帖子ＩＤ我们可以得到帖子对应的url，不过下面<a>标签中直接给了。
我们获取链接并将其放入数组中：
post_info = post.find("a", {"class": "j_th_tit "})
urls.append("http://tieba.baidu.com" + post_info.attrs["href"])
将上述代码打包，给出单页链接，返回此链接中所有帖子的url：
def get_url_from_page(page_url):'''对给定页面进行处理返回页面中帖子的url:param page_url: 页面链接:return: 页面中所有帖子的url'''bsObj_page = get_bsObj(page_url)urls = []try:posts = bsObj_page.findAll("li", {"class": "j_thread_list"})except AttributeError as e:print("发生错误:" + e + "时间:" + str(datetime.datetime.now()))for post in posts:post_info = post.find("a", {"class": "j_th_tit "})urls.append("http://tieba.baidu.com" + post_info.attrs["href"])return urls
处理每页的信息
上面我们得到了每页的地址，接下来我们处理每个帖子中的信息。我们需要在这页面中找到一些对我们有用的信息并将其存入csv文件中。
同样用这个地址举例：http://tieba.baidu.com/p/4700788764
首先，当打开这个链接是我们观察到的信息就是：帖子的标题，楼主名称，发帖时间，回复量等。
让我们观察一下这个页面的代码：
<div class="l_post j_l_post l_post_bright noborder " data-field="{"author":{"user_id":625543823,"user_name":"\u6768\u5175507","name_u":"%E6%9D%A8%E5%85%B5507&ie=utf-8","user_sex":2,"portrait":"8f0ae69da8e585b53530374925","is_like":1,"level_id":7,"level_name":"\u8d21\u58eb","cur_score":445,"bawu":0,"props":null},"content":{"post_id":95008842757,"is_anonym":false,"open_id":"tieba","open_type":"","date":"2016-07-29 19:10","vote_crypt":"","post_no":1,"type":"0","comment_num":0,"ptype":"0","is_saveface":false,"props":null,"post_index":0,"pb_tpoint":null}}">
同样两个属性 class 和data-field，在data-field中包含了这个帖子的大部分信息：帖人id,发帖人昵称，性别，等级id，等级昵称，open_id,open_type,发帖日期等。
首先我们创建一个帖子对象，其中属性为帖子的信息，方法为将信息写入对应的csv文件：
class PostInfo:def __init__(self,post_id,post_title,post_url,reply_num,post_date,open_id,open_type,user_name,user_sex,level_id,level_name):self.post_id = post_idself.post_title = post_titleself.post_url = post_urlself.reply_num = reply_numself.post_date = post_dateself.open_id = open_idself.open_type = open_typeself.user_name = user_nameself.user_sex = user_sexself.level_id = level_idself.level_name = level_namedef dump_to_csv(self,filename):csvFile = open(filename, "a+")try:writer = csv.writer(csvFile)writer.writerow((self.post_id,self.post_title,self.post_url,self.reply_num,self.post_date,self.open_id,self.open_type,self.user_name,self.user_sex,self.level_id,self.level_name))finally:csvFile.close()
然后我们通过find方法找到对应信息：
obj1 = json.loads(
bsObj.find("div", attrs={"class": "l_post j_l_post l_post_bright noborder "}).attrs['data-field'])
reply_num = bsObj.find("li", attrs={"class": "l_reply_num"}).span.get_text()
post_title = bsObj.find("h1", attrs={"class": "core_title_txt"}).get_text()post_id = obj1.get('content').get('post_id')
post_url = url
post_date = obj1.get('content').get('date')
open_id = obj1.get('content').get('open_id')
open_type = obj1.get('content').get('open_type')
user_name = obj1.get('author').get('user_name')
user_sex = obj1.get('author').get('user_sex')
level_id = obj1.get('author').get('level_id')
level_name = obj1.get('author').get('level_name')
创建对象，将其保存：
postinfo = PostInfo(post_id, post_title, post_url, reply_num, post_date, open_id, open_type, user_name,user_sex, level_id, level_name)
postinfo.dump_to_csv('post_info2.csv')
其实不用通过对象保存，这只是个人想法。
将上面代码封装成处理每个帖子的函数：
def del_post(urls):'''处理传入url的帖子:param url::return:'''for url in urls:bsObj = get_bsObj(url)try:obj1 = json.loads(bsObj.find("div", attrs={"class": "l_post j_l_post l_post_bright noborder "}).attrs['data-field'])reply_num = bsObj.find("li", attrs={"class": "l_reply_num"}).span.get_text()post_title = bsObj.find("h1", attrs={"class": "core_title_txt"}).get_text()except:print("发生错误:" + "---" + "时间:" + str(datetime.datetime.now()) + url)with open('error.txt', 'a+') as f:f.writelines("发生错误:" + "---" + "时间:" + str(datetime.datetime.now()) + url)return Nonepost_id = obj1.get('content').get('post_id')post_url = urlpost_date = obj1.get('content').get('date')open_id = obj1.get('content').get('open_id')open_type = obj1.get('content').get('open_type')user_name = obj1.get('author').get('user_name')user_sex = obj1.get('author').get('user_sex')level_id = obj1.get('author').get('level_id')level_name = obj1.get('author').get('level_name')postinfo = PostInfo(post_id, post_title, post_url, reply_num, post_date, open_id, open_type, user_name,user_sex, level_id, level_name)postinfo.dump_to_csv('post_info2.csv')del postinfo
得到的结果类似于：
98773024983,【轰动Python界】的学习速成高效大法,http://tieba.baidu.com/p/4811129571,2,2016-10-06 20:32,tieba,,openlabczx,0,7,贡士
组合使用上面的函数
首先，我们让用户输入需要爬去的贴吧的主页：
home_page_url = input("请输入要处理贴吧的主页链接")
处理url：
bar_name = home_page_url.split("=")[1].split("&")[0]
pre_page_url = "http://tieba.baidu.com/f?kw=" + bar_name + "&ie=utf-8&pn=" #page_url 不包含页数的前缀
处理主页：
all_post_num = del_mainPage(home_page_url) #贴吧一共有多少条帖子
让用户输入需要爬去的帖子数量：
del_post_num = int(input("请输入需要处理前多少条帖子：")) #指定需要处理的帖子数目
最后：
if del_post_num > all_post_num:print("需要处理的帖子数大于贴吧帖子总数！")
else:for page in range(0,del_post_num,50):print("It's processing page : " + str(page))page_url = pre_page_url+str(page)urls = get_url_from_page(page_url)t = threading.Thread(target=del_post,args=(urls,))t.start()
主函数代码：
if __name__ == '__main__':#home_page_url = input("请输入要处理贴吧的主页链接")home_page_url = test_urlbar_name = home_page_url.split("=")[1].split("&")[0]pre_page_url = "http://tieba.baidu.com/f?kw=" + bar_name + "&ie=utf-8&pn=" #page_url 不包含页数的前缀all_post_num = del_mainPage(home_page_url) #贴吧一共有多少条帖子del_post_num = int(input("请输入需要处理前多少条帖子：")) #指定需要处理的帖子数目if del_post_num > all_post_num:print("需要处理的帖子数大于贴吧帖子总数！")else:for page in range(0,del_post_num,50):print("It's processing page : " + str(page))page_url = pre_page_url+str(page)urls = get_url_from_page(page_url)t = threading.Thread(target=del_post,args=(urls,))t.start()
全部代码：
from bs4 import BeautifulSoup
from urllib.request import urlopen
from urllib.error import HTTPError
import json
import datetime
import csv
import threadingclass PostInfo:def __init__(self,post_id,post_title,post_url,reply_num,post_date,open_id,open_type,user_name,user_sex,level_id,level_name):self.post_id = post_idself.post_title = post_titleself.post_url = post_urlself.reply_num = reply_numself.post_date = post_dateself.open_id = open_idself.open_type = open_typeself.user_name = user_nameself.user_sex = user_sexself.level_id = level_idself.level_name = level_namedef dump_to_csv(self,filename):csvFile = open(filename, "a+")try:writer = csv.writer(csvFile)writer.writerow((self.post_id,self.post_title,self.post_url,self.reply_num,self.post_date,self.open_id,self.open_type,self.user_name,self.user_sex,self.level_id,self.level_name))finally:csvFile.close()def get_bsObj(url):'''返回给定url的beautifulsoup对象:param url:目标网址:return:beautifulsoup对象'''try:html = urlopen(url).read()bsObj = BeautifulSoup(html, "lxml")return bsObjexcept HTTPError as e:print(e)return Nonedef del_mainPage(url):'''对主页面进行处理，返回最后页码:param url:目标主页地址:return:返回最后页码,int'''bsObj_mainpage = get_bsObj(url)last_page = int(bsObj_mainpage.find("a",{"class":"last pagination-item "})['href'].split("=")[-1])try:red_text = bsObj_mainpage.findAll("span", {"class": "red_text"})subject_sum = red_text[0].get_text() # 主题数post_sum = red_text[1].get_text() # 帖子数follow_sum = red_text[2].get_text() # 关注量except AttributeError as e:print("发生错误:" + e + "时间:"+str(datetime.datetime.now()))return Nonewith open('main_info.txt','w+') as f:f.writelines("统计时间: "+str(datetime.datetime.now())+"\n")f.writelines("主题数: "+subject_sum+"\n")f.writelines("帖子数: "+post_sum+"\n")f.writelines("关注量: "+follow_sum+"\n")return last_pagedef get_url_from_page(page_url):'''对给定页面进行处理返回页面中帖子的url:param page_url: 页面链接:return: 页面中所有帖子的url'''bsObj_page = get_bsObj(page_url)urls = []try:posts = bsObj_page.findAll("li", {"class": "j_thread_list"})except AttributeError as e:print("发生错误:" + e + "时间:" + str(datetime.datetime.now()))for post in posts:post_info = post.find("a", {"class": "j_th_tit "})urls.append("http://tieba.baidu.com" + post_info.attrs["href"])return urlsdef del_post(urls):'''处理传入url的帖子:param url::return:'''for url in urls:bsObj = get_bsObj(url)try:obj1 = json.loads(bsObj.find("div", attrs={"class": "l_post j_l_post l_post_bright noborder "}).attrs['data-field'])reply_num = bsObj.find("li", attrs={"class": "l_reply_num"}).span.get_text()post_title = bsObj.find("h1", attrs={"class": "core_title_txt"}).get_text()except:print("发生错误:" + "---" + "时间:" + str(datetime.datetime.now()) + url)with open('error.txt', 'a+') as f:f.writelines("发生错误:" + "---" + "时间:" + str(datetime.datetime.now()) + url)return Nonepost_id = obj1.get('content').get('post_id')post_url = urlpost_date = obj1.get('content').get('date')open_id = obj1.get('content').get('open_id')open_type = obj1.get('content').get('open_type')user_name = obj1.get('author').get('user_name')user_sex = obj1.get('author').get('user_sex')level_id = obj1.get('author').get('level_id')level_name = obj1.get('author').get('level_name')postinfo = PostInfo(post_id, post_title, post_url, reply_num, post_date, open_id, open_type, user_name,user_sex, level_id, level_name)postinfo.dump_to_csv('post_info2.csv')# t = threading.Thread(target=postinfo.dump_to_csv,args=('post_info2.csv',))# t.start()del postinfotest_url = "http://tieba.baidu.com/f?kw=python&ie=utf-8"if __name__ == '__main__':#home_page_url = input("请输入要处理贴吧的主页链接")home_page_url = test_urlbar_name = home_page_url.split("=")[1].split("&")[0]pre_page_url = "http://tieba.baidu.com/f?kw=" + bar_name + "&ie=utf-8&pn=" #page_url 不包含页数的前缀all_post_num = del_mainPage(home_page_url) #贴吧一共有多少条帖子del_post_num = int(input("请输入需要处理前多少条帖子：")) #指定需要处理的帖子数目if del_post_num > all_post_num:print("需要处理的帖子数大于贴吧帖子总数！")else:for page in range(0,del_post_num,50):print("It's processing page : " + str(page))page_url = pre_page_url+str(page)urls = get_url_from_page(page_url)t = threading.Thread(target=del_post,args=(urls,))t.start()t.join()#del_post(urls)
以上
欢迎多来访问博客：http://liqiongyu.com/blog
微信公众号：
