下面有三个常见的方法:
1、增加重试连接次数:
requests.DEFAULT_RETRIES = 5
2、关闭多余的链接:
默认的http connection是keep-alive的,在post请求中,header中有这样一个字段:Connection,我们将其置为’close’
monitor_header = {"Content-type":"application/json","Accept":"*/*","Connection":"close"
}
s = requests.session()
s.keep_alive = False
requests使用了urllib3库,默认的http connection是keep-alive的,requests设置False关闭。
3、请求时增加缓冲延时sleep
多线程进行post请求时,可以进行分批次的发请求,发完一次sleep一段时间:
time.sleep(60)
def get_machine_quality(mapping):threads = []list_machine_obj = []now_time = time.time()for value in mapping.values():for v in value:machine_obj = quality_data(v, 0, now_time)list_machine_obj.append(machine_obj)index = 0for value in mapping.values():for v in value:# 创建新线程thread_tmp = my_thread(v, threadfunc_get_machine_quality, list_machine_obj[index])# 添加线程到线程列表threads.append(thread_tmp)index += 1# 开启新线程length = len(threads)part = 4for i in range(0,part):start = int(i / part * length)end = int((i + 1) / part * length)for i in range(start, end):threads[i].start()for i in range(start, end):threads[i].join()time.sleep(60)print(str(start) + "to" + str(end))logging.critical(str(start) + "to" + str(end))# print ("退出主线程")return list_machine_obj