Python爬虫：XPath解析爬取豆瓣电影Top250示例

一、示例的函数说明：

函数processing()：用于处理字符串中的空白字符，并拼接字符串。

主函数程序入口：每页显示25部影片，实现循环，共10页。通过format方法替换切换的页码的url地址。然后调用实现爬虫程序的函数get_movie_info()，获取电影信息。

二、示例代码：

from lxml import etree
import time
import random
import requestsheaders = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) ''Chrome/117.0.0.0 Safari/537.36'
}def processing(strs):s = ''for n in strs:n = ''.join(n.split())s = s + nreturn sdef get_movie_info(url):response = requests.get(url, headers=headers)html = etree.HTML(response.text)div_all = html.xpath('//div[@class="info"]')for div in div_all:names = div.xpath('./div[@class="hd"]/a//span/text()')name = processing(names)infos = div.xpath('./div[@class="bd"]/p/text()')info = processing(infos)score = div.xpath('./div[@class="bd"]/div/span[2]/text()')evaluation = div.xpath('./div[@class="bd"]/div/span[4]/text()')summary = div.xpath('./div[@class="bd"]/p[@class="quote"]/span/text()')print('电影名称：', name)print('导演与演员：', info)print('影片评分：', score)print('评价人数：', evaluation)print('影片总结：', summary)print('-------分割线-------')if __name__ == '__main__':for i in range(0, 250, 25):url = 'https://movie.douban.com/top250?start={page}&filter='.format(page=i)get_movie_info(url)time.sleep(random.randint(1, 3))