参考：https://blog.csdn.net/s150503/article/details/72571680
CONCURRENT_REQUESTS 与 DOWNLOAD_DELAY
Scrapy 中 CONCURRENT_REQUESTS 与 DOWNLOAD_DELAY 的联系，先建立一个项目来找CONCURRENT_REQUESTS与DOWNLOAD_DELAY的联系
以豆瓣电影top250 为例
douban_spider.py
# -*- coding: utf-8 -*-import scrapy
import time
import re
from lxml import etree"""
scrapy 豆瓣登录响应结果乱码问题
https://www.jianshu.com/p/9974fc338242
"""class ExampleSpider(scrapy.Spider):name = 'douban'allowed_domains = ['example.com']# start_urls = ['https://movie.douban.com/top250?start={}&filter='.format(i) for i in range(0, 250, 25)]start_urls = ['https://movie.douban.com/top250?start={}&filter='.format(i) for i in range(10000)]custom_settings = {'DEFAULT_REQUEST_HEADERS': {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,''*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',"Accept-Encoding": "gzip, deflate","Accept-Language": "zh-CN,zh;q=0.9","Connection": "keep-alive","Host": "movie.douban.com","Upgrade-Insecure-Requests": "1","User-Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'' (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36',},'CONCURRENT_REQUESTS': 10,'DOWNLOAD_DELAY': 0.01,'CONCURRENT_REQUESTS_PER_IP': 0,'CONCURRENT_REQUESTS_PER_DOMAIN': 10000,'FEED_EXPORT_ENCODING': 'utf-8'}def parse(self, response):current_url = response.urlprint(current_url)time.sleep(3)returnoffset = re.findall(r'start=(\d+)', current_url)[0]page_num = int(offset) // 25html = etree.HTML(text=response.text)# 先定位到 li 标签，data 是一个包含25个li标签的list，就是包含25部电影信息的listdata = html.xpath('//ol[@class="grid_view"]/li')index = 0for d in data:data_title = d.xpath('div/div[2]/div[@class="hd"]/a/span[1]/text()')data_info = d.xpath('div/div[2]/div[@class="bd"]/p[1]/text()')data_quote = d.xpath('div/div[2]/div[@class="bd"]/p[2]/span/text()')data_score = d.xpath('div/div[2]/div[@class="bd"]/div/span[@class="rating_num"]/text()')data_num = d.xpath('div/div[2]/div[@class="bd"]/div/span[4]/text()')data_pic_url = d.xpath('div/div[1]/a/img/@src')print(f"No: {str(page_num * 25 + index + 1)} {data_title}")index += 1passif __name__ == '__main__':from scrapy import cmdlinecmdline.execute('scrapy crawl douban'.split())pass
验证 1：
'CONCURRENT_REQUESTS': 10,
'DOWNLOAD_DELAY': 0.01,CONCURRENT_REQUESTS 设置 为 10 时，理论上 可以并发 10个请求。但是 DOWNLOAD_DELAY 设置为 0.01 时，按 DOWNLOAD_DELAY 来算，可以并发 1 / 0.01 = 100 个请求，这两个取最小值 为 10，所以 并发 10个 请求。
几乎同一秒 并发 10 个左右的 请求
验证 2：
'CONCURRENT_REQUESTS': 10,
'DOWNLOAD_DELAY': 0.5,CONCURRENT_REQUESTS 设置 为 10 时，理论上 可以并发 10个请求。但是 DOWNLOAD_DELAY 设置为 0.5 时，按 DOWNLOAD_DELAY 来算，可以并发 1 / 0.5 = 2 个请求，这两个取最小值 为 2，所以 并发 2个 请求。
总结：
DOWNLOAD_DELAY 会影响 CONCURRENT_REQUESTS，不能使并发显现出来。
思考：
1. 当有 CONCURRENT_REQUESTS，没有 DOWNLOAD_DELAY 时，服务器会在同一时间收到大量的请求。
'CONCURRENT_REQUESTS': 10,
# 'DOWNLOAD_DELAY': 0.5,DOWNLOAD_DELAY 注释后，会使用默认值 0，
2. 当有 CONCURRENT_REQUESTS，有 DOWNLOAD_DELAY 时，服务器不会在同一时间收到大量的请求。
# 'CONCURRENT_REQUESTS': 0,
'DOWNLOAD_DELAY': 0.5,CONCURRENT_REQUESTS 注释后，会使用默认值 16，
