logger对象在创建时必须从父进程传递给子进程才能记录日志,有两种传递方式:
第一种:通过类定义一个日志变量,然后外部传入logger对象来记录日志。
# workers_a.py
class Worker:_logger = None@staticmethoddef set_logger(logger_):Worker._logger = logger_def work(self, x):self._logger.info("Square rooting {}", x)return x**0.5
第二种:通过将logger对象定义为全局变量来记录日志。
# workers_b.py
from loguru import loggerdef set_logger(logger_):global loggerlogger = logger_def work(x):logger.info("Square rooting {}", x)return x**0.5
在主程序中调用方式如下:
# main.py
from multiprocessing import Pool
from loguru import logger
import workers_a
import workers_bif __name__ == "__main__":logger.remove()logger.add("file.log", enqueue=True)worker = workers_a.Worker()with Pool(4, initializer=worker.set_logger, initargs=(logger, )) as pool:results = pool.map(worker.work, [1, 10, 100])with Pool(4, initializer=workers_b.set_logger, initargs=(logger, )) as pool:results = pool.map(workers_b.work, [1, 10, 100])logger.info("Done")
上面通过multiprocessing 中的Pool创建线程池可记录多线程执行日志。(测试正常执行)上面日志配置时需要把 enqueue=True 放开。
但是上面有个问题:linux系统下的python线程并发与windows环境下的python线程并发调用方式不相同。linux下会通过fork创建,windows是通过spawn拷贝创建。为了避免在windows系统下运行正常的代码在linux系统下出问题,需要借助multiprocessing 的context。代码如下:
import multiprocessing
from loguru import logger
import workers_aif __name__ == "__main__":context = multiprocessing.get_context("spawn")logger.remove()logger.add("file.log", enqueue=True, context=context)worker = workers_a.Worker()with context.Pool(4, initializer=worker.set_logger, initargs=(logger, )) as pool:results = pool.map(worker.work, [1, 10, 100])
更详细内容可参考下面地址:Compatibility with multiprocessing using enqueue argument.