python爬虫进阶篇：Scrapy中使用Selenium+Firefox浏览器爬取沪深A股股票行情

一、前言

上篇记录了Scrapy搭配selenium的使用方法，有了基本的了解后我们可以将这项技术落实到实际需求中。目前很多股票网站的行情信息都是动态数据，我们可以用Scrapy+selenium对股票进行实时采集并持久化，再进行数据分析、邮件通知等操作。

二、环境搭建

详情请看上篇笔记

三、代码实现

items

class StockSpiderItem(scrapy.Item):# define the fields for your item here like:# name = scrapy.Field()# 股票代码stock_code = scrapy.Field()# 股票名称stock_name = scrapy.Field()# 最新价last_price = scrapy.Field()# 涨跌幅rise_fall_rate = scrapy.Field()# 涨跌额rise_fall_price = scrapy.Field()

middlewares

	def __init__(self):# ----------------firefox的设置------------------------------- #self.options = firefox_options()def spider_opened(self, spider):spider.logger.info('Spider opened: %s' % spider.name)spider.driver = webdriver.Firefox(options=self.options)  # 指定使用的浏览器def process_request(self, request, spider):# Called for each request that goes through the downloader# middleware.# Must either:# - return None: continue processing this request# - or return a Response object# - or return a Request object# - or raise IgnoreRequest: process_exception() methods of#   installed downloader middleware will be calledspider.driver.get("https://quote.eastmoney.com/center/gridlist.html#hs_a_board")return Nonedef process_response(self, request, response, spider):# Called with the response returned from the downloader.# Must either;# - return a Response object# - return a Request object# - or raise IgnoreRequestresponse_body = spider.driver.page_sourcereturn HtmlResponse(url=request.url, body=response_body, encoding='utf-8', request=request)

settings设置

# See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
SPIDER_MIDDLEWARES = {'stock_spider.middlewares.StockSpiderSpiderMiddleware': 543,
}# Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {'stock_spider.middlewares.StockSpiderDownloaderMiddleware': 543,
}

spider文件

    def parse(self, response):# 股票代码stock_code = response.css("table.table_wrapper-table tbody tr td:nth-child(2) a::text").extract()# 股票名称stock_name = response.css("table.table_wrapper-table tbody tr td:nth-child(3) a::text").extract()# 最新价last_price = response.css("table.table_wrapper-table tbody tr td:nth-child(5) span::text").extract()# 涨跌幅rise_fall_rate = response.css("table.table_wrapper-table tbody tr td:nth-child(6) span::text").extract()# 涨跌额rise_fall_price = response.css("table.table_wrapper-table tbody tr td:nth-child(7) span::text").extract()for i in range(len(stock_code)):item = StockSpiderItem()item["stock_code"] = stock_code[i]item["stock_name"] = stock_name[i]item["last_price"] = last_price[i]item["rise_fall_rate"] = rise_fall_rate[i]item["rise_fall_price"] = rise_fall_price[i]yield itemdef close(self, spider):spider.driver.quit()

pipelines持久化

    def process_item(self, item, spider):"""接收到提交过来的对象后，写入csv文件"""filename = f'stock_info.csv'with open(filename, 'a+', encoding='utf-8') as f:line = item["stock_code"] + "," + item["stock_name"] + "," + item["last_price"] + "," + \item["rise_fall_rate"] + "," + item["rise_fall_price"] + "\n"f.write(line)return item

readme文件

1.安装依赖包 
- python 3.0+
- pip install -r requirements.txt2.将最第二层stock_spider文件夹设置为根目录3.将firefox驱动程序包放到python环境的Scripts文件夹里4.必须要安装firefox浏览器才会调用到浏览器5.执行spider_main.py文件启动爬虫