异步日志系统设计demo

目录

  • 简单版本1
  • 优化版本1
  • 优化版本2

对于QPS要求很高或者对性能有一定要求的服务器程序，同步写日志会对服务的关键性逻辑的快速执行和及时响应带来一定的性能损失，因为写日志时等待磁盘IO完成工作也需要一定时间。为了减少这种损失，一般采用异步写日志。
本质上仍然是一个生产者与消费者模型，产生日志的线程是生产者，将日志写入文件的线程是消费者。
如果有多个消费者线程，可能存在写日志的时间顺序错位，所以一般将日志消费者线程数量设置为1.

简单版本1

下面给出一个简单的版本：

#include <iostream>
#include <thread>
#include <mutex>
#include <list>
#include <string>
#include <sstream>
#include <vector>
// 保护队列的mutex
std::mutex log_mutex;
std::list<std::string> cached_logs;
FILE* log_file = nullptr;bool init_log_file()
{// 以追加的模式写入文件，如果文件不存在，则创建log_file = fopen("my.log","a+");return log_file != nullptr;
}void uninit_log_file()
{if (log_file != nullptr)fclose(log_file);
}bool write_log_tofile(const std::string& line)
{if (log_file == nullptr)return false;if (fwrite((void *)line.c_str(), 1, line.length(), log_file) != line.length())return false;// 将日志flush到文件中fflush(log_file);return true;
}void log_producer()
{int index = 0;while (true) {++index;std::ostringstream os;os << "This is log, index :" << index << ", producer threadID:" << std::this_thread::get_id() << "\n";{std::lock_guard<std::mutex> lock(log_mutex);cached_logs.emplace_back(os.str());}// 生产出一个log之后，休眠100ms再生产std::chrono::milliseconds duration(100);std::this_thread::sleep_for(duration);}
}void log_consumer()
{std::string line;while (true) {{std::lock_guard<std::mutex> lock(log_mutex);if (!cached_logs.empty()) {line = cached_logs.front();cached_logs.pop_front();}}// 如果取出来的行为空，说明队列里面是空的，消费者休眠一会儿再去消费if (line.empty()) {std::chrono::milliseconds duration(1000);std::this_thread::sleep_for(duration);continue;}// 否则将line写入到log_file中write_log_tofile(line);line.clear();}
}int main(int argc, char* argv[])
{if (!init_log_file()) {std::cout << "init log file error." << std::endl;return -1;}std::thread log_producer1(log_producer);std::thread log_producer2(log_producer);std::thread log_producer3(log_producer);std::thread log_consumer1(log_consumer);std::thread log_consumer2(log_consumer);std::thread log_consumer3(log_consumer);log_producer1.join();log_producer2.join();log_producer3.join();log_consumer1.join();log_consumer2.join();log_consumer3.join();uninit_log_file();return 0;
}

效果：

This is log, index :1, producer threadID:139910877185792
This is log, index :1, producer threadID:139910868793088
This is log, index :1, producer threadID:139910860400384
This is log, index :2, producer threadID:139910877185792
This is log, index :2, producer threadID:139910860400384
This is log, index :3, producer threadID:139910877185792
This is log, index :3, producer threadID:139910860400384
This is log, index :2, producer threadID:139910868793088
This is log, index :3, producer threadID:139910868793088
This is log, index :4, producer threadID:139910877185792
This is log, index :4, producer threadID:139910860400384
This is log, index :4, producer threadID:139910868793088
This is log, index :5, producer threadID:139910877185792
This is log, index :5, producer threadID:139910860400384
This is log, index :5, producer threadID:139910868793088
This is log, index :6, producer threadID:139910860400384
This is log, index :6, producer threadID:139910868793088
This is log, index :7, producer threadID:139910877185792
This is log, index :7, producer threadID:139910860400384
This is log, index :7, producer threadID:139910868793088
This is log, index :8, producer threadID:139910877185792
This is log, index :8, producer threadID:139910860400384
This is log, index :8, producer threadID:139910868793088
This is log, index :9, producer threadID:139910877185792
This is log, index :9, producer threadID:139910860400384
This is log, index :9, producer threadID:139910868793088
This is log, index :6, producer threadID:139910877185792
This is log, index :10, producer threadID:139910860400384
This is log, index :10, producer threadID:139910868793088
This is log, index :10, producer threadID:139910877185792
This is log, index :11, producer threadID:139910860400384
This is log, index :11, producer threadID:139910868793088
This is log, index :12, producer threadID:139910860400384
This is log, index :12, producer threadID:139910868793088
This is log, index :11, producer threadID:139910877185792
This is log, index :12, producer threadID:139910877185792
This is log, index :13, producer threadID:139910860400384
This is log, index :13, producer threadID:139910868793088
This is log, index :13, producer threadID:139910877185792
This is log, index :14, producer threadID:139910868793088
This is log, index :14, producer threadID:139910877185792
This is log, index :15, producer threadID:139910868793088
This is log, index :15, producer threadID:139910877185792
This is log, index :14, producer threadID:139910860400384
This is log, index :15, producer threadID:139910860400384
This is log, index :16, producer threadID:139910877185792
This is log, index :16, producer threadID:139910860400384
This is log, index :17, producer threadID:139910877185792
This is log, index :17, producer threadID:139910860400384
This is log, index :17, producer threadID:139910868793088
This is log, index :16, producer threadID:139910868793088
This is log, index :18, producer threadID:139910877185792
This is log, index :19, producer threadID:139910860400384
This is log, index :19, producer threadID:139910877185792
This is log, index :19, producer threadID:139910868793088
This is log, index :20, producer threadID:139910860400384
This is log, index :18, producer threadID:139910860400384
This is log, index :20, producer threadID:139910877185792
This is log, index :18, producer threadID:139910868793088
This is log, index :20, producer threadID:139910868793088
This is log, index :21, producer threadID:139910868793088
This is log, index :21, producer threadID:139910877185792
This is log, index :21, producer threadID:139910860400384
This is log, index :22, producer threadID:139910860400384
This is log, index :22, producer threadID:139910877185792
This is log, index :22, producer threadID:139910868793088
This is log, index :23, producer threadID:139910860400384
This is log, index :23, producer threadID:139910877185792
This is log, index :23, producer threadID:139910868793088
This is log, index :24, producer threadID:139910860400384
This is log, index :24, producer threadID:139910868793088
This is log, index :24, producer threadID:139910877185792
This is log, index :25, producer threadID:139910860400384
This is log, index :25, producer threadID:139910868793088
This is log, index :25, producer threadID:139910877185792
This is log, index :26, producer threadID:139910868793088
This is log, index :26, producer threadID:139910860400384
This is log, index :26, producer threadID:139910877185792
This is log, index :27, producer threadID:139910868793088
This is log, index :27, producer threadID:139910860400384
This is log, index :27, producer threadID:139910877185792
This is log, index :28, producer threadID:139910868793088
This is log, index :28, producer threadID:139910877185792
This is log, index :28, producer threadID:139910860400384
This is log, index :29, producer threadID:139910877185792
This is log, index :29, producer threadID:139910868793088
This is log, index :29, producer threadID:139910860400384
This is log, index :30, producer threadID:139910877185792
This is log, index :30, producer threadID:139910868793088
This is log, index :30, producer threadID:139910860400384
This is log, index :31, producer threadID:139910868793088
This is log, index :31, producer threadID:139910877185792
This is log, index :31, producer threadID:139910860400384
This is log, index :32, producer threadID:139910877185792
This is log, index :32, producer threadID:139910868793088
This is log, index :32, producer threadID:139910860400384
This is log, index :33, producer threadID:139910860400384
This is log, index :33, producer threadID:139910868793088
This is log, index :33, producer threadID:139910877185792
This is log, index :34, producer threadID:139910860400384
This is log, index :34, producer threadID:139910877185792
This is log, index :34, producer threadID:139910868793088
This is log, index :35, producer threadID:139910860400384
This is log, index :35, producer threadID:139910868793088
This is log, index :35, producer threadID:139910877185792
This is log, index :36, producer threadID:139910877185792
This is log, index :36, producer threadID:139910868793088
This is log, index :37, producer threadID:139910860400384
This is log, index :37, producer threadID:139910877185792
This is log, index :37, producer threadID:139910868793088
This is log, index :38, producer threadID:139910860400384
This is log, index :38, producer threadID:139910877185792
This is log, index :38, producer threadID:139910868793088
This is log, index :39, producer threadID:139910860400384
This is log, index :39, producer threadID:139910877185792
This is log, index :39, producer threadID:139910868793088
This is log, index :40, producer threadID:139910860400384
This is log, index :40, producer threadID:139910877185792
This is log, index :40, producer threadID:139910868793088
This is log, index :36, producer threadID:139910860400384
This is log, index :41, producer threadID:139910860400384
This is log, index :41, producer threadID:139910877185792
This is log, index :42, producer threadID:139910860400384
This is log, index :42, producer threadID:139910877185792
This is log, index :42, producer threadID:139910868793088
This is log, index :43, producer threadID:139910860400384
This is log, index :43, producer threadID:139910868793088
This is log, index :43, producer threadID:139910877185792
This is log, index :44, producer threadID:139910860400384
This is log, index :44, producer threadID:139910868793088
This is log, index :44, producer threadID:139910877185792
This is log, index :45, producer threadID:139910860400384
This is log, index :45, producer threadID:139910868793088
This is log, index :45, producer threadID:139910877185792
This is log, index :46, producer threadID:139910860400384
This is log, index :46, producer threadID:139910868793088
This is log, index :46, producer threadID:139910877185792
This is log, index :47, producer threadID:139910860400384
This is log, index :47, producer threadID:139910868793088
This is log, index :47, producer threadID:139910877185792
This is log, index :48, producer threadID:139910860400384
This is log, index :48, producer threadID:139910868793088
This is log, index :48, producer threadID:139910877185792
This is log, index :49, producer threadID:139910860400384
This is log, index :49, producer threadID:139910877185792
This is log, index :49, producer threadID:139910868793088
This is log, index :50, producer threadID:139910877185792
This is log, index :50, producer threadID:139910860400384
This is log, index :50, producer threadID:139910868793088
This is log, index :41, producer threadID:139910868793088

优化版本1

上面的代码，在当前缓存队列没有日志记录的时候，消费日志线程会做无用功。
这里可以使用条件变量，如果当前队列中没有日志记录，就将日志消费者线程挂起；
当产生了新的日志后，signal条件变量，唤醒消费线程，将被日志从队列中取出，并写入文件。
下面是主要修改

#include <condition_variable>
std::condition_variable log_cv;
void log_producer()
{int index = 0;while (true) {++index;std::ostringstream os;os << "This is log, index :" << index << ", producer threadID:" << std::this_thread::get_id() << "\n";{std::lock_guard<std::mutex> lock(log_mutex);cached_logs.emplace_back(os.str());log_cv.notify_one();}// 生产出一个log之后，休眠100ms再生产std::chrono::milliseconds duration(100);std::this_thread::sleep_for(duration);}
}void log_consumer()
{std::string line;while (true) {{std::unique_lock<std::mutex> lock(log_mutex);while (cached_logs.empty()) {// 无限等待log_cv.wait(lock);}line = cached_logs.front();cached_logs.pop_front();}// 如果取出来的行为空，说明队列里面是空的，消费者休眠一会儿再去消费if (line.empty()) {std::chrono::milliseconds duration(1000);std::this_thread::sleep_for(duration);continue;}// 否则将line写入到log_file中write_log_tofile(line);line.clear();}
}

优化版本2

还可以使用信号量来设计异步日志系统。
信号量是带有资源计数的线程同步对象，每产生一条日志，就将信号量资源计数+1，日志消费线程默认等待这个信号量是否signal，如果signal，就唤醒一个日志消费线程，信号量计数自动-1。如果当前资源计数为0，则将消费者自动挂起。
C++现在还没有提供不同平台的信号量对象封装，这里以Linux系统为例：
明显能感觉运行相同时间，写入的日志更多了。。

#include <unistd.h>
#include <iostream>
#include <thread>
#include <mutex>
#include <list>
#include <string>
#include <sstream>
#include <semaphore.h>// 保护队列的mutex
pthread_mutex_t log_mutex = PTHREAD_MUTEX_INITIALIZER;
sem_t log_semphore;
std::list<std::string> cached_logs;
FILE* log_file = nullptr;bool init()
{pthread_mutex_init(&log_mutex, nullptr);// 初始信号量资源数量为0sem_init(&log_semphore, 0, 0);// 以追加的模式写入文件，如果文件不存在，则创建log_file = fopen("my.log","a++");return log_file != nullptr;
}void uninit()
{pthread_mutex_destroy(&log_mutex);sem_destroy(&log_semphore);if (log_file != nullptr)fclose(log_file);
}bool write_log_tofile(const std::string& line)
{if (log_file == nullptr)return false;// 对于比较长的日志应该分段写入，因为单次写入可能只写入部分内容// 这里逻辑从简if (fwrite((void *)line.c_str(), 1, line.length(), log_file) != line.length())return false;// 将日志flush到文件中fflush(log_file);return true;
}void* log_producer(void* arg)
{int index = 0;while (true) {++index;std::ostringstream os;os << "This is log, index :" << index << ", producer threadID:" << std::this_thread::get_id() << "\n";pthread_mutex_lock(&log_mutex);cached_logs.emplace_back(os.str());pthread_mutex_unlock(&log_mutex);sem_post(&log_semphore);usleep(1000);}
}void* log_consumer(void* arg)
{std::string line;while (true) {// 无限等待sem_wait(&log_semphore);pthread_mutex_lock(&log_mutex);if (!cached_logs.empty()) {line = cached_logs.front();cached_logs.pop_front();}pthread_mutex_unlock(&log_mutex);// 如果取出来的行为空，说明队列里面是空的，消费者休眠一会儿再去消费if (line.empty()) {sleep(1);continue;}// 否则将line写入到log_file中write_log_tofile(line);line.clear();}
}int main(int argc, char* argv[])
{if (!init()) {std::cout << "init log file error." << std::endl;return -1;}// 创建三个生产日志线程pthread_t producer_thread_id[3];for (size_t i = 0; i < sizeof(producer_thread_id) / sizeof(producer_thread_id[0]); ++i) {pthread_create(&producer_thread_id[i], NULL, log_producer, NULL);}// 创建三个消费日志线程pthread_t consumer_thread_id[3];for (size_t i = 0; i < sizeof(consumer_thread_id) / sizeof(consumer_thread_id[0]); ++i) {pthread_create(&consumer_thread_id[i], NULL, log_consumer, NULL);}// 等待生产者线程退出for (size_t i = 0; i < sizeof(producer_thread_id) / sizeof(producer_thread_id[0]); ++i) {pthread_join(producer_thread_id[i], NULL);}// 等待消费者线程退出for (size_t i = 0; i < sizeof(consumer_thread_id) / sizeof(consumer_thread_id[0]); ++i) {pthread_join(consumer_thread_id[i], NULL);}uninit();return 0;
}

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.mzph.cn/news/376788.shtml

如若内容造成侵权/违法违规/事实不符，请联系多彩编程网进行投诉反馈email:809451989@qq.com，一经查实，立即删除！

相关文章

js unix时间戳转换

js unix时间戳转换

一、unix时间戳转普通时间&#xff1a; var unixtime1358932051; var unixTimestamp new Date(unixtime* 1000); commonTime unixTimestamp.toLocaleString(); alert("普通时间为&#xff1a;"commonTime); 二、普通时间转unix时间戳 var str "2013-01-01 00…
阅读更多...
hdu 1025（最长非递减子序列的n*log(n)求法）

hdu 1025（最长非递减子序列的n*log(n)求法）

题目链接&#xff1a;http://acm.hdu.edu.cn/showproblem.php?pid1025 经典题。。。最长非递减序列的n*log(n)求法。。。orz... View Code 1 #include<iostream>2 const int N500007;3 using namespace std;4 int city[N];5 int dp[N];//dp[i]保存的是长度为i的最长不降…
阅读更多...
消息队列重要机制讲解以及MQ设计思路（kafka、rabbitmq、rocketmq）

消息队列重要机制讲解以及MQ设计思路（kafka、rabbitmq、rocketmq）

目录《Kafka篇》简述kafka的架构设计原理&#xff08;入口点&#xff09;消息队列有哪些作用&#xff08;简单&#xff09;消息队列的优缺点&#xff0c;使用场景&#xff08;基础&#xff09;消息队列如何保证消息可靠传输死信队列是什么&#xff1f;延时队列是什么&#xff1…
阅读更多...
js判断手机浏览器

js判断手机浏览器

最新浏览器识别合并。 demo&#xff1a;http://v.qq.com -> http://v.qq.com/h5　　　　http://v.qq.com/ -> http://v.qq.com/h5　　　　http://v.qq.com/h5 -> http://v.qq.com/h5 <script type"text/javascript"> (function(W){ …
阅读更多...
数据库归档模式

数据库归档模式

1、在sys身份下登陆oracle&#xff0c;执行命令archive log list; SQL> archive log list; Database log mode Archive Mode Automatic archival Enabled Archive destination USE_DB_RECOVERY_FILE_DEST Oldest online log sequence …
阅读更多...
转载|网络编程中阻塞式函数的底层逻辑

转载|网络编程中阻塞式函数的底层逻辑

逛知乎看到的&#xff0c;觉得写的挺透彻的&#xff0c;转载一下&#xff0c;原文链接&#xff1a;Unix网络编程里的阻塞是在操作系统的内核态创建一个线程来死循环吗&#xff1f; 原文以阻塞式的recv函数作为讲解&#xff0c;但是所有阻塞式的api底层逻辑基本相通。 下面是正文…
阅读更多...
把txt文件中的json字符串写到plist文件中

把txt文件中的json字符串写到plist文件中

- (void)json2Plist {NSString *filePath [self applicationDocumentsDirectoryFileName:"json"];NSMutableArray *tempArray [[NSMutableArray alloc] initWithContentsOfFile:filePath];//第一次添加数据时,数组为空if (tempArray.count 0) {tempArray [NSMuta…
阅读更多...
树的存储结构2 - 数据结构和算法42

树的存储结构2 - 数据结构和算法42

树的存储结构 让编程改变世界 Change the world by program 孩子表示法 我们这次换个角度来考虑&#xff0c;由于树中每个结点可能有多棵子树&#xff0c;可以考虑用多重链表来实现。 就像我们虽然有计划生育&#xff0c;但我们还是无法确保每个家庭只养育一个孩子的冲动&a…
阅读更多...
海量数据去重

海量数据去重

海量数据去重 一个文件中有40亿条数据&#xff0c;每条数据是一个32位的数字串&#xff0c;设计算法对其去重&#xff0c;相同的数字串仅保留一个&#xff0c;内存限制1G. 方法一&#xff1a;排序 对所有数字串进行排序&#xff0c;重复的数据传必然相邻&#xff0c;保留第一…
阅读更多...
Sharepoint 2013 发布功能（Publishing features）

Sharepoint 2013 发布功能（Publishing features）

一、默认情况下&#xff0c;在创建网站集时&#xff0c;只有选择的模板为‘ Publishing Portal&#xff08;发布门户&#xff09;’与‘ Enterprise Wiki&#xff08;企业 Wiki&#xff09;’时才默认启用发布功能&#xff0c;如下图所示&#xff1a; 二、发布功能包含两块&…
阅读更多...
【原】android启动时白屏或者黑屏的问题

【原】android启动时白屏或者黑屏的问题

解决应用启动时白屏或者黑屏的问题 由于Activity只能到onResume时&#xff0c;才能展示到前台&#xff0c;所以&#xff0c;如果为MAIN activity设置背景的话&#xff0c;无论onCreate-onResume速度多快&#xff0c;都会出现短暂的白屏或者黑屏 其实解决的办法很简单&#xff0…
阅读更多...
【草稿】windows + vscode 远程开发

【草稿】windows + vscode 远程开发

主要分为三个步骤&#xff1a; 1、开启openssh服务 2、通过ssh命令连接到远程服务器 3、通过vscode连接远程服务器进行开发调试 ssh概念 SSH是较可靠&#xff0c;专为远程登陆会话和其他网络服务提供安全性得协议&#xff0c;利用ssh协议可以有效防止远程管理过程中得信息…
阅读更多...
POJ3185(简单BFS，主要做测试使用)

POJ3185(简单BFS，主要做测试使用)

没事做水了一道POJ的简单BFS的题目 这道题的数据范围是20,所以状态总数就是&#xff08;1<<20&#xff09; 第一次提交使用STL的queue&#xff0c;并且是在队首判断是否达到终点&#xff0c;达到终点就退出&#xff0c;超时&#xff1a;&#xff08;其实这里我是很不明白…
阅读更多...
tomcat站点配置

tomcat站点配置

tomcat版本&#xff1a;tomcat5.5.91、打开tomcat\conf\server.xml&#xff0c;在里面找到<Engine name"Catalina" defaultHost"localhost">.....</Engine>2、在<Engine name"Catalina" defaultHost"localhost"><…
阅读更多...
新的视频会议模式：StarlineProject

新的视频会议模式：StarlineProject

目录效果展示部分用户参与度部分技术细节机械装置以及硬件配置。视频系统照明人脸跟踪压缩和传输图像渲染音频系统step1&#xff1a;捕获音频step2&#xff1a;音频去噪处理step3&#xff1a;压缩、传输、解压step4&#xff1a;渲染可以改进的点效果展示部分 〔映维网〕谷歌光场…
阅读更多...
HDU 3934

HDU 3934

/*这是用的有旋转卡壳的思想。 首先确定i&#xff0c;j&#xff0c;对k进行循环&#xff0c;知道找到第一个k使得cross(i,j,k)>cross(i,j,k1),如果ki进入下一次循环。 对j&#xff0c;k进行旋转&#xff0c;每次循环之前更新最大值&#xff0c;然后固定一个j&#xff0c;同样…
阅读更多...
[ios] UILocalNotification实现本地的闹钟提醒【转】

[ios] UILocalNotification实现本地的闹钟提醒【转】

http://www.cnblogs.com/jiangshiyong/archive/2012/06/06/2538204.html转载于:https://www.cnblogs.com/jinjiantong/archive/2013/04/01/2992624.html
阅读更多...
sql server根据表中数据生成insert语句

sql server根据表中数据生成insert语句

几个收藏的根据数据库生成Insert语句的存储过程[修正版]----根据表中数据生成insert语句的存储过程--建立存储过程&#xff0c;执行spGenInsertSQL 表名--感谢playyuer----感谢szyicol--CREATEproc[dbo].[spGenInsertSQL](tablenamevarchar(256))asbegindeclaresqlvarchar(8000…
阅读更多...
Javascript eval()函数 基础回顾

Javascript eval()函数 基础回顾

如果您想详细了解ev al和JSON请参考以下链接&#xff1a; eval &#xff1a;https://developer.mozilla.org/En/Core_JavaScript_1.5_Reference/Global_Functions/Eval JSON&#xff1a;http://www.json.org/ eval函数的工作原理 eval函数会评估一个给定的含有JavaScript代码的…
阅读更多...
杂感无题|

杂感无题|

今天中午和组里面的人吃饭&#xff0c;聊起了科兴跳楼的事情。这事其实前几天我华为的mentor就转给我了&#xff0c;当时也没太在意&#xff0c;在脉脉上看了看&#xff0c;也不知晓是谁&#xff0c;想着可能又是抑郁症吧。 饭后依旧绕着食堂散步&#xff0c;ly说那个人好像还是…
阅读更多...
最新文章

Copyright @ 2022~2023