ELK套件FileBeat部署

1.简介

FileBeat用于文件数据采集并输出到ElasticSearch或Logstash中。

ELK搭建过程参见： ELK搭建及Java程序接入

2.下载

下载FileBeats OSS 7.2.1版本
https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-oss-7.2.1-linux-x86_64.tar.gz

tar xzvf filebeat-oss-7.2.1-linux-x86_64.tar.gz
cd filebeat-7.2.1-*
#启用nginx模块
./filebeat modules enable nginx

3.直接输出到ElasticSearch

#设置初始环境
./filebeat setup -e

此时报错连接不上kibana
instance/beat.go:877 Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get http://localhost:5601/api/status: dial tcp [::1]:5601: connect: connection refused. Response: .
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get http://localhost:5601/api/status: dial tcp [::1]:5601: connect: connection refused. Response: .

修改连接kibana的配置
vi filebeat.yml
setup.kibana:
host: "192.168.81.145:5601"

重新设置初始环境

#运行filebeat
./filebeat -e

#附：静默运行
nohup ./filebeat -e > service.out&

此时
kibana中就有了filebeat-*索引，可以看到日志数据

多跑一些数据，打开kibana的Dashboard，选择filebeat给创建的[Filebeat Nginx] Overview ECS，可以看到如下页面，地图展示、柱状图饼图统计等均有。

4.输出到Logstash

vi filebeat.yml
output.logstash:
hosts: ["localhost:5044"]
把以下两个配置节注释掉
setup.kibana:
output.elasticsearch:

重启filebeat后，nginx访问日志会写入到logstash中，但此时logstash写入elasticsearch会报错：failed to parse field [host] of type [text] in document with id 'E0lsjW4BTdp_eLcgfhbu'
看elasticsearch日志发现此时host为一个json对象，需要变为字符串才行
修改配置，添加过滤器，把host.name赋值为host
vi config/logstash.conf
filter {
mutate {
rename => { "[host][name]" => "host" }
}
}

重启后即可

5.更改nginx日志路径

vi modules.d/nginx.yml
- module: nginx
access:
enabled: true
var.paths: ["/path/to/log/nginx/access.log*"]
error:
enabled: true
var.paths: ["/path/to/log/nginx/error.log*"]