Day05-filebeat常用的输出组件，logstash的输入输出组件及date，grok，geoip过滤插件实战案例
- 1、使用filebeat采集docker日志
- 2、filebeat的input类型之filestream实战案例
- 2.1 课堂练习案例
- 2.2 将数据写入到本地文件案例
- 2.3 写入数据到ES集群
- 2.4 将多个数据源写入到ES集群不同索引
- 3、logstash
- 3.1 基于rpm方式安装logstash
- 3.2 基于二进制方式安装logstash
- 3.3 编写第一个logstash配置文件
- 3.4 logstash搭配filebeat实战案例
- 3.5 logstash的过滤插件之geoip实战案例
- 3.6 logstash解析nginx原生日志并分析IP地址实战
- 3.7 logstash解析将实际写入时间更正案例
多行匹配案例补充:
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/10-log_multiline-to-console.yaml
filebeat.inputs:
- type: logpaths:- /tmp/oldboyedu-linux85/linux85.logmultiline:# 指定多行匹配的类型为counttype: count# 将3行数据合并为1行数据count_lines: 3output.console:pretty: true
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]#
1、使用filebeat采集docker日志
(1)安装docker
wget http://192.168.15.253/ElasticStack/day05-/softwares/oldboyedu-docker-ce-23_0_1.tar.gz
tar xf oldboyedu-docker-ce-23_0_1.tar.gz
yum -y localinstall oldboyedu-docker-ce-23_0_1/*.rpm
(2)配置docker的镜像加速
[root@elk103.oldboyedu.com ~]# cat /etc/docker/daemon.json
{"data-root": "/var/lib/docker","registry-mirrors": ["https://tuv7rqqq.mirror.aliyuncs.com","https://hub-mirror.c.1com/","https://docker.mirrors.ustc.edu.cn","https://reg-mirror.qiniu.com"]
}[root@elk103.oldboyedu.com ~]# systemctl enable --now docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
(3)下载nginx镜像
docker run -dp 88:80 --name mynginx --restart always nginx:1.22.1-alpine
docker run -dp 89:8080 --name mytomcat --restart always tomcat:jre8-alpine
(3)使用filebeat采集容器日志
# 通过docker
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/11-docker-to-console.yaml
filebeat.inputs:# 指定输入类型为docker类型
- type: docker# 指定容器的IDcontainers.ids: - '*'output.console:pretty: true
[root@elk103 filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/11-docker-to-console.yaml# 通过container
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/12-container-to-console.yaml
filebeat.inputs:
- type: containerpaths: - '/var/lib/docker/containers/*/*.log'# output.console:
# pretty: trueoutput.elasticsearch:hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"]
[root@elk103 filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/12-container-to-console.yaml
2、filebeat的input类型之filestream实战案例
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/13-filestream-to-console.yaml
filebeat.inputs:# 指定类型为filestream，在7.16版本中已经弃用log类型
- type: filestreamenabled: falsepaths:- /tmp/oldboyedu-linux85/linux85.log- type: filestreamenabled: falsepaths:- /tmp/oldboyedu-linux85/docker.json# 配置解析parsers:# 配置json格式解析- ndjson:# 将错误消息记录到error字段中add_error_key: true# 如果解析的json格式字段和filebeat内置的顶级字段冲突，则覆盖，默认是不覆盖的。overwrite_keys: true# 将message解析的字段放入一个自定义的字段下。若不指定该字段，则默认解析的键值对会在顶级字段.#target: oldboyedu-linux85- type: filestreamenabled: falsepaths:- /tmp/oldboyedu-linux85/linux85.log# 配置解析parsers:- multiline:type: countcount_lines: 3- type: filestreamenabled: truepaths:- /tmp/oldboyedu-linux85/demo.logparsers:- multiline:type: countcount_lines: 4- ndjson:add_error_key: trueoverwrite_keys: truetarget: oldboyedu-linux85-demooutput.console:pretty: true[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]#
2.1 课堂练习案例
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/14-ketanglianxi.yaml
filebeat.inputs:
- type: filestreamenabled: truepaths:- /tmp/oldboyedu-linux85/shopping.jsonparsers:- multiline:type: countcount_lines: 7- ndjson:add_error_key: trueoverwrite_keys: trueoutput.elasticsearch:hosts: ["http://10.0.0.101:9200","http://10.0.0.102:9200","http://10.0.0.103:9200"] [root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# ./filebeat -e -c config/14-ketanglianxi.yaml
2.2 将数据写入到本地文件案例
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/15-stdin-to-file.yaml
filebeat.inputs:
- type: stdin# 指定输出的类型为本地文件
output.file:# 指定文件存储的路径path: "/tmp/oldboyedu-linux85"# 指定文件的名称filename: stdin.log[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# ./filebeat -e -c config/15-stdin-to-file.yaml
2.3 写入数据到ES集群
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/16-log-to-es.yaml
filebeat.inputs:
- type: filestreamenabled: truepaths:- /tmp/oldboyedu-linux85/shopping.jsonparsers:- multiline:type: countcount_lines: 7- ndjson:add_error_key: trueoverwrite_keys: true# 将日志输出到ES集群
output.elasticsearch:# 指定ES集群地址hosts: - "http://10.0.0.101:9200"- "http://10.0.0.102:9200"- "http://10.0.0.103:9200"# 指定索引index: "oldboyedu-linux85-shopping-%{+yyyy.MM.dd}"
# 禁用索引声明管理周期，若不禁用则自动忽略自定义索引名称
setup.ilm.enabled: false
# 设置索引模板的名称
setup.template.name: "oldboyedu-linux85-shopping"
# 指定索引模板的匹配模式
setup.template.pattern: "oldboyedu-linux85-shopping-*"
# 是否覆盖原有的索引模板
setup.template.overwrite: true
# 设置索引模板
setup.template.settings:# 指定分片数量为8index.number_of_shards: 8# 指定副本数量为0index.number_of_replicas: 0
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]#
2.4 将多个数据源写入到ES集群不同索引
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/17-filestream-to-es.yaml
filebeat.inputs:
- type: filestreamenabled: truetags: "docker"paths:- /tmp/oldboyedu-linux85/docker.jsonparsers:- ndjson:add_error_key: true#overwrite_keys: true- type: filestreamenabled: truetags: "linux85"paths:- /tmp/oldboyedu-linux85/linux85.logparsers:- multiline:type: countcount_lines: 3- type: filestreamenabled: truetags: "demo"paths:- /tmp/oldboyedu-linux85/demo.logparsers:- multiline:type: countcount_lines: 4- ndjson:add_error_key: trueoverwrite_keys: truetarget: oldboyedu-linux85-demooutput.elasticsearch:hosts: - "http://10.0.0.101:9200"- "http://10.0.0.102:9200"- "http://10.0.0.103:9200"# index: "oldboyedu-linux85-shopping-%{+yyyy.MM.dd}"indices:- index: "oldboyedu-jiaoshi07-docker-%{+yyyy.MM.dd}"when.contains:tags: "docker"- index: "oldboyedu-jiaoshi07-linux85-%{+yyyy.MM.dd}"when.contains:tags: "linux85"- index: "oldboyedu-jiaoshi07-demo-%{+yyyy.MM.dd}"when.contains:tags: "demo"setup.ilm.enabled: false
setup.template.name: "oldboyedu-jiaoshi07"
setup.template.pattern: "oldboyedu-jiaoshi07-*"
setup.template.overwrite: true
setup.template.settings:index.number_of_shards: 3index.number_of_replicas: 0[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/17-filestream-to-es.yaml
3、logstash
3.1 基于rpm方式安装logstash
(1)下载软件包
wget http://192.168.15.253/ElasticStack/day05-/softwares/logstash-7.17.5-x86_64.rpm
(2)安装logstash
rpm -ivh logstash-7.17.5-x86_64.rpm
(3)验证logstash版本
ln -svf /usr/share/logstash/bin/logstash /usr/local/sbin
logstash -V
(4)基于命令行启动logstash实例
logstash -e "input { stdin { type => stdin } } output { stdout { codec => rubydebug } }"
(5)测试logstash
自行输入数据即可。
3.2 基于二进制方式安装logstash
(1)下载软件包
wget http://192.168.15.253/ElasticStack/day05-/softwares/logstash-7.17.5-linux-x86_64.tar.gz
(2)解压软件包
tar xf logstash-7.17.5-linux-x86_64.tar.gz -C /oldboyedu/softwares/
(3)验证logstash版本
ln -svf /oldboyedu/softwares/logstash-7.17.5/bin/logstash /usr/local/sbin/
logstash -V
(4)基于命令行启动logstash实例
logstash -e "input { stdin { type => stdin } } output { stdout {} }"
(5)测试logstash
自行输入数据即可。
3.3 编写第一个logstash配置文件
(1)编写配置文件
[root@elk101.oldboyedu.com ~]# cat config/01-stdin-to-stdout.conf
input { stdin { type => stdin }
} output { stdout {}
}
[root@elk101.oldboyedu.com ~]#
(2)启动logstash实例
[root@elk101.oldboyedu.com ~]# logstash -f /config/01-stdin-to-stdout.conf
3.4 logstash搭配filebeat实战案例
(1)编写logstash配置文件并启动
[root@elk101.oldboyedu.com ~]# cat /config/02-beats-to-stdout.conf
input { # 指定输入的类型是一个beatsbeats {# 指定监听的端口号port => 8888}
} output { # 将数据在标准输出显示stdout {} # 将数据写入ES集群elasticsearch {# 指定ES主机地址hosts => ["http://localhost:9200"]# 指定索引名称index => "oldboyedu-linux85-logstash"}
}[root@elk101.oldboyedu.com ~]# logstash -rf config/02-beats-to-stdout.conf
(2)启动filebeat实例并写入数据
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/18-nginx-to-logstash.yaml
filebeat.inputs:
- type: logpaths:- /var/log/nginx/access.log*
# 将数据输出到logstash中
output.logstash:# 指定logstash的主机和端口hosts: ["10.0.0.101:8888"][root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/18-nginx-to-logstash.yaml
3.5 logstash的过滤插件之geoip实战案例
(1)logstash配置文件
[root@elk101.oldboyedu.com ~]# cat config/03-beats-geoip-es.conf
input { # 指定输入的类型是一个beatsbeats {# 指定监听的端口号port => 8888}
} filter {# 根据IP地址分析客户端的经纬度，国家，城市信息等。geoip {source => "clientip"remove_field => [ "agent","log","input","host","ecs","tags" ]}}output { # 将数据在标准输出显示stdout {} # 将数据写入ES集群elasticsearch {# 指定ES主机地址hosts => ["http://localhost:9200"]# 指定索引名称index => "oldboyedu-linux85-logstash"}
}[root@elk101.oldboyedu.com ~]# logstash -rf config/03-beats-geoip-es.conf
(2)filebeat采集数据到logstash
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/18-nginx-to-logstash.yaml
filebeat.inputs:
- type: logpaths:- /var/log/nginx/access.log*json.keys_under_root: truejson.add_error_key: true# 将数据输出到logstash中
output.logstash:# 指定logstash的主机和端口hosts: ["10.0.0.101:8888"]
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]#
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/18-nginx-to-logstash.yaml
(3)日志格式
[root@elk103.oldboyedu.com oldboyedu-linux85]# cat /var/log/nginx/access.log
{"@timestamp":"2023-04-06T16:17:43+08:00","host":"10.0.0.103","clientip":"110.110.110.110","SendBytes":615,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.103","uri":"/index.html","domain":"10.0.0.103","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"curl/7.29.0","status":"200"}
{"@timestamp":"2023-04-06T18:18:18+08:00","host":"10.0.0.103","clientip":"101.231.54.100","SendBytes":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.103","uri":"/index.html","domain":"10.0.0.103","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"Mozilla/5.0 (iPad; CPU OS 13_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/87.0.4280.77 Mobile/15E148 Safari/604.1","status":"304"}
{"@timestamp":"2023-04-07T08:18:32+08:00","host":"10.0.0.103","clientip":"219.141.136.10","SendBytes":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.103","uri":"/index.html","domain":"10.0.0.103","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1","status":"304"}
{"@timestamp":"2023-04-07T10:18:52+08:00","host":"10.0.0.103","clientip":"221.118.208.184","SendBytes":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.103","uri":"/index.html","domain":"10.0.0.103","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1","status":"304"}
{"@timestamp":"2023-04-07T12:19:07+08:00","host":"10.0.0.103","clientip":"21.118.208.84","SendBytes":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.103","uri":"/index.html","domain":"10.0.0.103","xff":"-","referer":"-","tcp_xff":"-","http_user_agent":"Mozilla/5.0 (Linux; Android 10; SM-G981B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Mobile Safari/537.36","status":"404"}
[root@elk103.oldboyedu.com oldboyedu-linux85]#
3.6 logstash解析nginx原生日志并分析IP地址实战
(1)logstash配置文件编写
[root@elk101.oldboyedu.com ~]# cat config/04-beats-grok_geoip-es.conf
input { beats {port => 8888}
} filter {grok {match => { "message" => "%{HTTPD_COMBINEDLOG}" }remove_field => [ "agent","log","input","host","ecs","tags" ]}geoip {source => "clientip"}}output { # stdout {} elasticsearch {hosts => ["http://localhost:9200"]index => "oldboyedu-linux85-logstash-nginx"}
}
[root@elk101.oldboyedu.com ~]#
[root@elk101.oldboyedu.com ~]# logstash -rf config/04-beats-grok_geoip-es.conf
(2)filbeat采集日志
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/19-nginx-to-logstash.yaml
filebeat.inputs:
- type: logpaths:- /tmp/oldboyedu-linux85/access.log# 将数据输出到logstash中
output.logstash:# 指定logstash的主机和端口hosts: ["10.0.0.101:8888"]
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]#
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/19-nginx-to-logstash.yaml
3.7 logstash解析将实际写入时间更正案例
(1)logstash配置文件
[root@elk101.oldboyedu.com ~]# cat config/05-beats-grok_geoip_date-es.conf
input { beats {port => 8888}
} filter {grok {match => { "message" => "%{HTTPD_COMBINEDLOG}" }remove_field => [ "agent","log","input","host","ecs","tags" ]}geoip {source => "clientip"}date {# 匹配时间字符串字段并格式化# "22/Nov/2015:11:57:34 +0800"match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]# 匹配时区timezone => "Asia/Shanghai"# 将转后的日期替换为指定字段，若不指定，则默认值为"@timestamp"target => "oldboyedu-linux85-date"}}output { #stdout {} elasticsearch {hosts => ["http://localhost:9200"]index => "oldboyedu-linux85-logstash-nginx-date"}
}[root@elk101.oldboyedu.com ~]# logstash -rf config/05-beats-grok_geoip_date-es.conf
(2)filebeat配置文件
[root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# cat config/19-nginx-to-logstash.yaml
filebeat.inputs:
- type: logpaths:- /tmp/oldboyedu-linux85/access.log# 将数据输出到logstash中
output.logstash:# 指定logstash的主机和端口hosts: ["10.0.0.101:8888"][root@elk103.oldboyedu.com filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/19-nginx-to-logstash.yaml
周六作业:
(1)完成课堂的所有练习并整理思维导图;
(2)使用shell脚本编写一个nginx原生日志格式，要求生成100w条记录，并使用ElasticStack技术栈分析并能查询数据;
扩展作业:
config/05-beats-grok_geoip_date-es.conf
![[ruby on rails]部署时候产生ActiveRecord::PreparedStatementCacheExpired错误的原因及解决方法](https://i-blog.csdnimg.cn/direct/9e7759f7390747a5a939d6429d1c450a.png)
