python爬虫单线程的多任务异步协程

在input()、sleep(2)、request.get()等时，都会导致线程阻塞，协程可以解决IO等操作时的阻塞现象，提高CPU利用效率。

1.单线程的多任务异步协程
main.py

"""=== coding: UTF8 ==="""
import asyncio
import timeasync def func1():print("hello python1")await asyncio.sleep(2)print("hello python1")async def func2():print("hello python2")await asyncio.sleep(3)print("hello python2")async def func3():print("hello python3")await asyncio.sleep(4)print("hello python3")async def main():# 准备异步协程对象列表tasks = [asyncio.create_task(func1()),asyncio.create_task(func2()),asyncio.create_task(func3())]await asyncio.wait(tasks)"""
========================================
主函数功能测试
========================================
"""
if __name__ == '__main__':t1 = time.time()# 一次性运行多个任务asyncio.run(main())t2 = time.time()print(t2 - t1)  # 打印耗时的时间

运行效果：

"C:\Program Files\Python38\python.exe" E:/PythonSourceCode/test/main.py
hello python1
hello python2
hello python3
hello python1
hello python2
hello python3
4.022439956665039Process finished with exit code 0

2.单线程的多任务异步协程在爬虫领域的模拟应用
只是模拟，可以作为模板，在实际爬虫时，需要重点解决

await asyncio.sleep(3)  # 模拟网络请求（网络耗时）

main.py

"""=== coding: UTF8 ==="""
import asyncio
import time# 在爬虫领域的应用
async def download(url):print("准备开始下载")await asyncio.sleep(3)  # 模拟网络请求（网络耗时）print("下载完成")async def main():urls = ["https://www.hao123.com/", "https://www.baidu.com/", "https://www.bilibili.com/"]# 准备异步协程对象列表tasks = []for url in urls:d = asyncio.create_task(download(url))tasks.append(d)await asyncio.wait(tasks)"""
========================================
主函数功能测试
========================================
"""
if __name__ == '__main__':t1 = time.time()# 一次性运行多个任务asyncio.run(main())t2 = time.time()print(t2 - t1)  # 打印耗时的时间