ARM64环境部署EFK8.15.3收集K8S集群容器日志

环境规划

主机IP	系统	部署方式	ES版本	CPU架构	用户名	密码
192.168.1.225	Ubuntu 22.04.4 LTS	docker	elasticsearch:8.15.3	ARM64	elastic	llodyi4TMmZD

ES集群部署

创建持久化目录(所有节点)

mkdir -p  /data/es/{data,certs,logs,plugins}
mkdir -p /data/es/certs/{ca,es01}

服务器优化/配置(所有节点)

#提高进程及资源使用限制上线
vim /etc/security/limits.conf* soft nofile 65536
* hard nofile 65536
* soft nproc 32000
* hard nproc 32000
* hard memlock unlimited
* soft memlock unlimitedvim /etc/systemd/system.confDefaultLimitNOFILE=65536
DefaultLimitNPROC=32000
DefaultLimitMEMLOCK=infinity#重载配置文件
systemctl daemon-reload
systemctl daemon-reexec #修改虚拟内存最大映射数
vim /etc/sysctl.conf#添加参数
vm.max_map_count = 262144#重新加载/etc/sysctl.conf配置
sysctl -p

创建证书elastic-certificates.p12文件

[root@llody-dev ~]#docker run -itd --name es01 swr.cn-southwest-2.myhuaweicloud.com/llody/elasticsearch:8.15.3  /bin/bash
c5ace1a6f5a6d1fa623cbebf96a75b9c9b6d71e3832d9973fff809848e8ae4ac
[root@llody-dev ~]#docker exec -it es01 /bin/bash
elasticsearch@c5ace1a6f5a6:~$ ./bin/elasticsearch-certutil ca
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.The 'ca' mode generates a new 'certificate authority'
This will create a new X.509 certificate and private key that can be used
to sign certificate when running in 'cert' mode.Use the 'ca-dn' option if you wish to configure the 'distinguished name'
of the certificate authorityBy default the 'ca' mode produces a single PKCS#12 output file which holds:* The CA certificate* The CA's private keyIf you elect to generate PEM format certificates (the -pem option), then the output will
be a zip file containing individual files for the CA certificate and private keyPlease enter the desired output file [elastic-stack-ca.p12]: 
Enter password for elastic-stack-ca.p12 : 
elasticsearch@c5ace1a6f5a6:~$ ./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --dns es01 --ip 192.168.1.225  
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.The 'cert' mode generates X.509 certificate and private keys.* By default, this generates a single certificate and key for useon a single instance.* The '-multiple' option will prompt you to enter details for multipleinstances and will generate a certificate and key for each one* The '-in' option allows for the certificate generation to be automated by describingthe details of each instance in a YAML file* An instance is any piece of the Elastic Stack that requires an SSL certificate.Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beatsmay all require a certificate and private key.* The minimum required value for each instance is a name. This can simply be thehostname, which will be used as the Common Name of the certificate. A fulldistinguished name may also be used.* A filename value may be required for each instance. This is necessary when thename would result in an invalid file or directory name. The name provided hereis used as the directory name (within the zip) and the prefix for the key andcertificate files. The filename is required if you are prompted and the nameis not displayed in the prompt.* IP addresses and DNS names are optional. Multiple values can be specified as acomma separated string. If no IP addresses or DNS names are provided, you maydisable hostname verification in your SSL configuration.* All certificates generated by this tool will be signed by a certificate authority (CA)unless the --self-signed command line option is specified.The tool can automatically generate a new CA for you, or you can provide your own withthe --ca or --ca-cert command line options.By default the 'cert' mode produces a single PKCS#12 output file which holds:* The instance certificate* The private key for the instance certificate* The CA certificateIf you specify any of the following options:* -pem (PEM formatted output)* -multiple (generate multiple certificates)* -in (generate certificates from an input file)
then the output will be be a zip file containing individual certificate/key filesEnter password for CA (elastic-stack-ca.p12) : 
Please enter the desired output file [elastic-certificates.p12]: 
Enter password for elastic-certificates.p12 : Certificates written to /usr/share/elasticsearch/elastic-certificates.p12This file should be properly secured as it contains the private key for 
your instance.
This file is a self contained file and can be copied and used 'as is'
For each Elastic product that you wish to configure, you should copy
this '.p12' file to the relevant configuration directory
and then follow the SSL configuration instructions in the product guide.For client applications, you may only need to copy the CA certificate and
configure the client to trust this certificate.
elasticsearch@c5ace1a6f5a6:~$ ls
LICENSE.txt  NOTICE.txt  README.asciidoc  bin  config  data  elastic-certificates.p12  elastic-stack-ca.p12  jdk  lib  logs  modules  plugins
elasticsearch@c5ace1a6f5a6:~$ pwd
/usr/share/elasticsearch
elasticsearch@c5ace1a6f5a6:~$ exit
exit

创建证书时。必须指定节点固定IP，否则证书将不可用。

拷贝加密文件到conf下(分发到所有节点)

[root@llody-dev ~]#docker cp es01:/usr/share/elasticsearch/elastic-stack-ca.p12 /data/es/certs/ca/ca.p12
[root@llody-dev ~]#docker cp es01:/usr/share/elasticsearch/elastic-certificates.p12 /data/es/certs/es01/es01.p12
[root@llody-dev ~]#openssl pkcs12 -in /data/es/certs/ca/ca.p12 -clcerts -nokeys -out /data/es/certs/ca/ca.crt
Enter Import Password:
MAC verified OK
[root@llody-dev ~]#openssl pkcs12 -in /data/es/certs/ca/ca.p12 -nocerts -nodes -out /data/es/certs/ca/ca.key
Enter Import Password:
MAC verified OK
[root@llody-dev ~]#openssl pkcs12 -in /data/es/certs/es01/es01.p12 -clcerts -nokeys -out /data/es/certs/es01/es01.crt
Enter Import Password:
MAC verified OK
[root@llody-dev ~]#openssl pkcs12 -in /data/es/certs/es01/es01.p12 -nocerts -nodes -out /data/es/certs/es01/es01.key
Enter Import Password:
MAC verified OK

密码与elastic密码保持一致即可。

数据目录授权

[root@llody-dev ~]#chown -R 1000:1000 /data/es/*
[root@llody-dev ~]#chmod 600 /data/es/certs/ca/*
[root@llody-dev ~]#chmod 600 /data/es/certs/es01/*

创建网络命名空间

[root@llody-dev ~]#docker network create elk

启动ElasticSearch服务

es01.yaml

# 创建运维目录
mkdir /root/shell
# 创建es01.yaml文件
vim /root/shell/es01.yaml

version: '3'services:es01:image: swr.cn-southwest-2.myhuaweicloud.com/llody/elasticsearch:8.15.3container_name: es01restart: alwaysports:- "9200:9200"- "9300:9300"environment:- node.name=es01- cluster.name=es-cluster- cluster.initial_master_nodes=es01- bootstrap.memory_lock=true- "ES_JAVA_OPTS=-Xms512m -Xmx512m"- xpack.security.enabled=true- xpack.security.http.ssl.enabled=true- xpack.security.http.ssl.key=certs/es01/es01.key- xpack.security.http.ssl.certificate=certs/es01/es01.crt- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt- xpack.security.transport.ssl.enabled=true- xpack.security.transport.ssl.verification_mode=certificate- xpack.security.transport.ssl.key=certs/es01/es01.key- xpack.security.transport.ssl.certificate=certs/es01/es01.crt- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crtuser: "1000:1000"ulimits:memlock:soft: -1hard: -1volumes:- /data/es/data/:/usr/share/elasticsearch/data- /data/es/logs/:/usr/share/elasticsearch/logs- /data/es/plugins/:/usr/share/elasticsearch/plugins- /data/es/certs:/usr/share/elasticsearch/config/certsnetworks:- elknetworks:elk:external: truename: elk

# es01节点启动
docker-compose -p es -f shell/es01.yml up -d

如果是集群，一定要使用固定IP，并在创建证书时进行指定。

设置密码

[root@llody-dev ~]#docker exec -it es01 /bin/bash
elasticsearch@437bd4edc09f:~$  /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive -u https://192.168.1.225:9200
******************************************************************************
Note: The 'elasticsearch-setup-passwords' tool has been deprecated. This       command will be removed in a future release.
******************************************************************************Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]yEnter password for [elastic]: 
Reenter password for [elastic]: 
Enter password for [apm_system]: 
Reenter password for [apm_system]: 
Enter password for [kibana_system]: 
Reenter password for [kibana_system]: 
Enter password for [logstash_system]: 
Reenter password for [logstash_system]: 
Enter password for [beats_system]: 
Reenter password for [beats_system]: 
Enter password for [remote_monitoring_user]: 
Reenter password for [remote_monitoring_user]: 
Changed password for user [apm_system]
Changed password for user [kibana_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]
elasticsearch@437bd4edc09f:~$ exit
exit

查看状态

[root@llody-dev ~]#curl -k  -u elastic:llodyi4TMmZD https://192.168.1.225:9200/_cat/nodes?v
ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
172.18.0.2           32          56  10    0.22    0.42     0.43 cdfhilmrstw *      es01
[root@llody-dev ~]#curl --cacert /data/es/certs/ca/ca.crt -u elastic:llodyi4TMmZD -X GET "https://192.168.1.225:9200/_cluster/health?pretty"
{"cluster_name" : "es-cluster","status" : "green","timed_out" : false,"number_of_nodes" : 1,"number_of_data_nodes" : 1,"active_primary_shards" : 1,"active_shards" : 1,"relocating_shards" : 0,"initializing_shards" : 0,"unassigned_shards" : 0,"delayed_unassigned_shards" : 0,"number_of_pending_tasks" : 0,"number_of_in_flight_fetch" : 0,"task_max_waiting_in_queue_millis" : 0,"active_shards_percent_as_number" : 100.0
}

部署Kibana

配置Kibana与ES认证证书

创建证书目录

[root@llody-dev ~]#mkdir -p /data/kibana/certs

拷贝ES的ca证书

[root@llody-dev ~/shell]#cp /data/es/certs/ca/ca.crt /data/kibana/certs/ca.crt

配置权限

[root@llody-dev ~]#chown -R 1000:1000 /data/kibana
[root@llody-dev ~]#chmod 600 /data/kibana/certs/*

配置kibana.yml文件

server.host: "0.0.0.0"
server.port: 5601
elasticsearch.requestTimeout: 60000
monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.hosts: ["https://192.168.1.225:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "llodyi4TMmZD"
elasticsearch.ssl.certificateAuthorities: ["/usr/share/kibana/config/certs/ca.crt"]
i18n.locale: "zh-CN"

配置kibana.yaml

version: '3'services:kibana:image: swr.cn-southwest-2.myhuaweicloud.com/llody/kibana:8.15.3 container_name: kibanarestart: alwaysuser: "1000:1000"ports:- "5601:5601"volumes:- /data/kibana/certs:/usr/share/kibana/config/certs- /data/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.ymlnetworks:- elknetworks:elk:external: truename: elk

[root@llody-dev ~]#docker-compose -p kibana -f /root/shell/kibana.yaml up -d

查看容器状态

容器状态

部署filebeat

Docker

创建数据目录

[root@dev-2 ~]# mkdir -p /data/fluentd/{certs,config}
[root@dev-2 ~]# tree /data/filebeat/
/data/filebeat/
├── certs
│   └── ca.crt
└── config└── filebeat.yml2 directories, 2 files

上传ES的ca证书到指定目录下。

配置filebeat.yml

filebeat.inputs:
- type: containerenabled: truestream: allformat: dockerpaths:- '/var/lib/docker/containers/*/*.log'multiline:# 做多行日志聚合，此处是针对java的异常堆栈信息聚合（所有非年-月-日开头的日志会被聚合到最开始匹配到的那一行展示）type: patternpattern: '^\d+.\d+.\d+|^\d+.\w{3}.\d+'negate: truematch: afterprocessors:- add_docker_metadata:host: "unix:///var/run/docker.sock"# 去掉不需要的字段- drop_fields:ignore_missing: true # 字段不存在时不报错fields: - agent.type.keyword- agent.type- agent.name- agent.id.keyword- agent.id- host.name- host.name.keyword- script:lang: javascripttimeout: 100source: > function process(event) { var s = event.Get('container.image.name');if (!s) {return event;}var sp = s.lastIndexOf('/');sp = sp < 0 ? 0 : sp + 1;var ep = s.indexOf(':');if (ep < 0) {event.Put('service', s.substring(sp));}event.Put('service', s.substring(sp, ep));}- drop_event:when: or:- contains:service: 'filebeat'output.elasticsearch:hosts: ["https://192.168.1.225:9200"]username: "elastic"password: "llodyi4TMmZD"# 传输压缩级别compression_level: 4ssl.certificate_authorities: ["/usr/share/filebeat/certs/ca.crt"]indices:# service会作为上报的日志文件的名字参与进去- index: "host-log-%{[service]}-%{+yyyy.MM.dd}"when.has_fields:- service- index: "host-log-default-%{+yyyy.MM.dd}"setup.template.enabled: false
setup.ilm.enabled: false

配置filebeat.yaml启动文件

version: '3'services:filebeat:image: swr.cn-southwest-2.myhuaweicloud.com/llody/filebeat:8.15.3container_name: filebeatrestart: alwaysuser: rootenvironment:- ELASTICSEARCH_HOSTS=https://192.168.1.225:9200- ELASTICSEARCH_USERNAME=elastic- ELASTICSEARCH_PASSWORD=llodyi4TMmZD- ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=/usr/share/filebeat/certs/ca/ca.crtvolumes:- /var/lib/docker/containers:/var/lib/docker/containers- /var/run/docker.sock:/var/run/docker.sock- /etc/localtime:/etc/localtime:ro- /data/filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml- /data/filebeat/certs:/usr/share/filebeat/certsnetworks:- elknetworks:elk:external: truename: elk

部署

[root@dev-2 ~]# docker-compose -p filebeat -f /root/shell/filebeat.yaml up -d
WARN[0000] /root/shell/filebeat.yaml: `version` is obsolete 
[+] Running 1/1✔ Container filebeat  Started

K8S

创建filebeat-dm.yaml文件

apiVersion: v1
kind: ConfigMap
metadata:name: filebeat-confignamespace: kube-system
data:filebeat.yml: |filebeat.inputs:- type: containerenabled: truestream: allformat: dockerpaths:- /var/log/containers/*_dev_*.logprocessors:- add_kubernetes_metadata:in_cluster: truematchers:- logs_path:logs_path: "/var/log/containers/"output.elasticsearch:hosts: ["https://192.168.1.225:9200"]username: "elastic"password: "llodyi4TMmZD"compression_level: 4ssl.certificate_authorities: ["/usr/share/filebeat/certs/ca.crt"]indices:- index: "k8s-log-%{[kubernetes.namespace]}-%{[kubernetes.pod.name]}-%{+yyyy.MM.dd}"when.has_fields:- kubernetes.namespace- kubernetes.pod.name- index: "k8s-log-default-%{+yyyy.MM.dd}"setup.template.enabled: falsesetup.ilm.enabled: false
---
apiVersion: v1
kind: ConfigMap
metadata:name: filebeat-es-canamespace: kube-system
data:ca.crt: |Bag AttributesfriendlyName: calocalKeyID: 54 69 6D 65 20 31 37 33 31 34 38 34 39 36 33 35 39 39 subject=/CN=Elastic Certificate Tool Autogenerated CAissuer=/CN=Elastic Certificate Tool Autogenerated CA-----BEGIN CERTIFICATE-----MIIDSjCCAjKgAwIBAgIVAPrLZp0pnA1sbAZRSsG0l5E+O9d5MA0GCSqGSIb3DQEBCwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2VuZXJhdGVkIENBMB4XDTI0MTExMzA4MDIzMFoXDTI3MTExMzA4MDIzMFowNDEyMDAGA1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5lcmF0ZWQgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVDUcg4ABSs7M984pxvw1Fg329j6FOxBk+NtmBWkMEl9aou0qMVcXGCBIp1MxGnhVSuxhDPG1hS3hX5PlfNGJMgzvymW6oVf6+SiTqbSTl+psEbSeZyK2nzrTEOfyB+lEZ11+Owo0XwTVSVqKUj5B1KaRBbBOo5O99eo97Rv8HkbVVlCD1R4TCJqQXh29c8RZuMFQ8DacqvH6Gwj1YvntmRT/QudQja1vED85MiSFPY8LdOrDHqzEhTY3scqC5gPrc/C7XJSbE+cfkPk56OEEUBCSZBiY8DR4b+30SiyZ3Miu4xUU/HRdLn1CtpZ6c8xooJipddInPlloyy3K0F24fAgMBAAGjUzBRMB0GA1UdDgQWBBT3xLSl9WS6321QvOUbP1qnzC9j8TAfBgNVHSMEGDAWgBT3xLSl9WS6321QvOUbP1qnzC9j8TAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQAfFkd4ocTbZhtb456E9p+eR25HX1CqhOXOUfKTCy48qXCvq3uXmCxUJY/4+oeMFtv2DLYUi9cZ3BMzjGpXVGlKPhTGat+aELUgCTvf6YelniveAYktjcDmc6VPrIzIZ9PcH+6BhTykgO6cV0MRgo4R0K3u+QydhokzX3cpiAGHbgHPDM2NF2tNYVKsOK0x8XrRVHZa75ZLsClkePtqWmWKcQ5Qh8vt8dDv8GCHPi95AiitIEXNEBfStwNV+xFmQdfbjO5KAilXJdYnbdN/t2xQt9xJBHQ3t+tAWNdXoY6reXui/EftuD/YRKSJT0GMtC98HlpHP/gVv5kMXkKwMPqI-----END CERTIFICATE-----
---
apiVersion: apps/v1
kind: DaemonSet
metadata:name: filebeatnamespace: kube-systemlabels:k8s-app: filebeat
spec:selector:matchLabels:k8s-app: filebeattemplate:metadata:labels:k8s-app: filebeatspec:serviceAccountName: filebeatterminationGracePeriodSeconds: 30containers:- name: filebeatimage: swr.cn-southwest-2.myhuaweicloud.com/llody/filebeat:8.15.3args: ["--strict.perms=false",  # 忽略权限问题"-c", "/usr/share/filebeat/filebeat.yml","-e",]env:- name: NODE_NAMEvalueFrom:fieldRef:fieldPath: spec.nodeNamesecurityContext:runAsUser: 0resources:limits:memory: 200Mirequests:cpu: 100mmemory: 100MivolumeMounts:- name: configmountPath: /usr/share/filebeat/filebeat.ymlsubPath: filebeat.yml- name: varlibdockercontainersmountPath: /var/lib/docker/containersreadOnly: true- name: varlogmountPath: /var/logreadOnly: true- name: certsmountPath: /usr/share/filebeat/certsreadOnly: truevolumes:- name: configconfigMap:defaultMode: 0600name: filebeat-config- name: certsconfigMap:defaultMode: 0600name: filebeat-es-ca- name: varlibdockercontainershostPath:path: /var/lib/docker/containers- name: varloghostPath:path: /var/log---
apiVersion: v1
kind: ServiceAccount
metadata:name: filebeatnamespace: kube-system---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: filebeat
subjects:
- kind: ServiceAccountname: filebeatnamespace: kube-system
roleRef:kind: ClusterRolename: filebeatapiGroup: rbac.authorization.k8s.io---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: filebeatlabels:k8s-app: filebeat
rules:
- apiGroups: [""]resources:- namespaces- podsverbs:- get- watch- list

注意：

1、挂载了filebeat.yml文件和ES的ca证书文件。

2、只收集dev命名空间日志，如要收集所有日志修改paths：/var/log/containers/*.log

3、创建的K8S集群日志索引规则：k8s-log-命名空间-容器名称-时间

部署

[root@node1 ~]#kubectl apply -f es/filebeat-dm.yaml 
configmap/filebeat-config configured
configmap/filebeat-es-ca unchanged
daemonset.apps/filebeat unchanged
serviceaccount/filebeat unchanged
clusterrolebinding.rbac.authorization.k8s.io/filebeat unchanged
clusterrole.rbac.authorization.k8s.io/filebeat unchanged