说明：这只是本人在家里组网搭建的集群，所以内容排版可能比较随意了，没有耐心的同学直接跳过去看别的文章吧，我只是放在这里留作记录方便以后翻阅
ELK 的坑实在太多了，自己在物理机(多台)逐渐摸索的，安装最新版本的记录
为了好看可能我下面会出现多个这种的命令行
命令行 mkdir -p /iscsi/elk/kibana/config 命令行 mkdir -p /iscsi/elk/kibana/data 命令行 mkdir -p /iscsi/elk/kibana/logs # 可以使用 mkdir -p /iscsi/elk/kibana/{config,data,logs}代替在一个文件夹下创建多个目录```
ELK安装:版本: 8.9.0准备:更新apt: 命令行 apt-get update命令行 apt-get upgrade命令行 apt-get install unzipJDK安装: - 1查看JDK版本: 命令行 apt search openjdk- 2安装JDK: 命令行 apt install openjdk-21-jdk -y （找一个最高的来安装） 创建ELK目录: 命令行 mkdir -p /iscsi/elk 命令行 mkdir -p /iscsi/elk/elasticsearch命令行 mkdir -p /iscsi/elk/elasticsearch/data命令行 mkdir -p /iscsi/elk/elasticsearch/logs命令行 mkdir -p /iscsi/elk/elasticsearch/plugins命令行 mkdir -p /iscsi/elk/kibana命令行 mkdir -p /iscsi/elk/kibana/config命令行 mkdir -p /iscsi/elk/kibana/data命令行 mkdir -p /iscsi/elk/kibana/logs命令行 mkdir -p /iscsi/elk/logstash 命令行 mkdir -p /iscsi/elk/logstash/config命令行 mkdir -p /iscsi/elk/logstash/logs命令行 mkdir -p /iscsi/elk/logstash/pipeline文件夹提权:命令行 chmod 777 -R /iscsi/elk/*创建用户和组:1查看用户: 命令行 cat /etc/passwd2查看组: 命令行 cat /etc/group3修改原来uid为1000的用户: 命令行 usermod -u 1001 tonywoo4新建elasticsearch 用户: 命令行 useradd -u 1000 -g root elasticsearch ??useradd -u 1000 -g tonywoo elasticsearch6新建组: 命令行 groupadd elasticsearch5修改elasticsearch 用户密码: 命令行 passwd elasticsearch设置目录权限:1设置目录拥有者: 命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/2设置目录拥有组: 命令行 chgrp -R 0 /iscsi/elk/elasticsearch3备注: /iscsi/elk/elasticsearch 下的子目录的拥有者都要设置为 elasticsearch 这个用户内存设置:- 1查看用户内存权限: 命令行 sysctl -a|grep vm.max_map_count- 2设置用户内存权限: 命令行 vim /etc/sysctl.conf- 3禁止内存与硬盘交换: vm.swappiness=1- 4配置最大映射数量: vm.max_map_count=262144- 5使配置生效: 退出vim，命令行 sysctl -p修改打开文件数:- 1进入文件: 命令行 vim /etc/security/limits.conf- 2追加内容:`# elasticsearch是用户，也可以使用*代替所有用户）elasticsearch soft nofile 65536elasticsearch hard nofile 65536#内存锁定交换soft memlock unlimitedhard memlock unlimited`查看docker网络: 命令行 docker network list开始:新建docker-compose文件: 命令行 touch /iscsi/elk/docker-compose-elk.yml修改docker-compose文件如下: `version: '3.7'services:elasticsearch:image: elasticsearch:8.9.0container_name: elasticsearchhostname: elasticsearchrestart: "no"volumes:- /etc/localtime:/etc/localtime#- /iscsi/elk/elasticsearch/data:/usr/share/elasticsearch/data:rw#- /iscsi/elk/elasticsearch/config:/usr/share/elasticsearch/config:rw#- /iscsi/elk/elasticsearch/logs:/usr/share/elasticsearch/logs:rw#- /iscsi/elk/elasticsearch/plugins:/usr/share/elasticsearch/plugins:rwenvironment:- TZ="Asia/Shanghai"- ES_JAVA_OPTS=-Xms512m -Xmx512m- discovery.type=single-nodeports:- "9200:9200"- "9300:9300"networks:elastic:ipv4_address: 172.99.0.2aliases:- elasticsearchkibana:image: kibana:8.9.0container_name: kibanahostname: kibanarestart: "no"volumes:- /etc/localtime:/etc/localtime#- /iscsi/elk/kibana/data:/usr/share/kibana/data:rw#- /iscsi/elk/kibana/config:/usr/share/kibana/config:rw#- /iscsi/elk/kibana/logs:/usr/share/kibana/logs:rwports:- 5601:5601depends_on:- elasticsearchnetworks:elastic:ipv4_address: 172.99.0.3aliases:- kibanalogstash:image: logstash:8.9.0container_name: logstashhostname: logstashrestart: "no"volumes:- /etc/localtime:/etc/localtime#- /iscsi/elk/logstash/config:/usr/share/logstash/config:rw#- /iscsi/elk/logstash/logs:/usr/share/logstash/logs:rw#- /iscsi/elk/logstash/pipeline:/usr/share/logstash/pipeline:rwports:- 5044:5044- 9066:9066- 21068:21068- "5000:5000/tcp"- "5000:5000/udp" depends_on:- elasticsearchnetworks:elastic:ipv4_address: 172.99.0.4aliases:- logstash ##自定义网络networks:elastic:ipam:driver: defaultconfig:- subnet: 172.99.0.0/16`上面带有#号 的后面要解除首次启动: /iscsi/elk下执行 命令行 docker-compose -f docker-compose-elk.yml up -d复制容器内目录到宿主机:命令行 docker cp elasticsearch:/usr/share/elasticsearch/config /iscsi/elk/elasticsearch/命令行 docker cp kibana:/usr/share/kibana/config /iscsi/elk/kibana/命令行 docker cp logstash:/usr/share/logstash/config /iscsi/elk/logstash/命令行 docker cp logstash:/usr/share/logstash/pipeline /iscsi/elk/logstash/命令行 chmod 777 -R /iscsi/elk/kibana/*命令行 chmod 777 -R /iscsi/elk/logstash/*命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/config/命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/config/certs/修改elasticsearch的jvm文件:打开 /iscsi/elk/elasticsearch/config/jvm.options添加下面两项命令行 vim /iscsi/elk/elasticsearch/config/jvm.options`-Xms512m-Xmx512m`修改logstash的jvm文件:打开 /iscsi/elk/logstash/config/jvm.options添加下面两项命令行 vim /iscsi/elk/logstash/config/jvm.options 修改如下#-Xms1g 改为 -Xms512m#-Xmx1g 改为 -Xmx512m放开注释: 放开docker-compose-elk.yml文件内挂载数据卷的注释命令行 cd /iscsi/elk命令行 docker-compose -f docker-compose-elk.yml stop命令行 docker-compose -f docker-compose-elk.yml rm命令行 docker-compose -f docker-compose-elk.yml up -d配置SSL:设置目录拥有者:命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/config/*命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/config/certs/*进入elasticsearch容器:命令行 docker exec -it elasticsearch /bin/bash生成elastic-stack-ca.p12文件:命令行 ./bin/elasticsearch-certutil ca需要在 `Enter password for elastic-stack-ca.p12:` 哪里设置密码生成elastic-certificates.p12:命令行 ./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12`Enter password for CA(elastic-stack-ca.p12):`后输入 elastic-stack-ca.p12设置的密码复制文件到config文件夹:命令行 mv elastic-certificates.p12 config/certs/命令行 mv elastic-stack-ca.p12 ./config/ 备注 开放了docker-compose-elk.yml的注释复制文件到config宿主机文件夹会同时改变；elastic-stack-ca.p12文件后续也需要用到设置文件拥有权:退出容器命令行 chmod 777 /iscsi/elk/elasticsearch/*命令行 chmod 777 /iscsi/elk/elasticsearch/config/*命令行 chmod 777 /iscsi/elk/elasticsearch/config/certs/*命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/*设置elasticsearch.yml配置文件:命令行 vim /iscsi/elk/elasticsearch/config/elasticsearch.yml修改为如下`# Enable encryption and mutual authentication between cluster nodesxpack.security.transport.ssl:enabled: trueverification_mode: certificatekeystore.path: certs/elastic-certificates.p12truststore.path: certs/elastic-certificates.p12 `修改密码:如果certificate设置了密码，需要执行一下两个命令退回到容器根目录命令行 ./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password命令行 ./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password备注 这个密码就是 elastic-certificates.p12 文件设置的密码重启elasticsearch容器:docker restart elasticsearch配置elasticsearch和kibana开启https访问:进入elasticsearch容器:命令行 docker exec -it elasticsearch /bin/bash生成elasticsearch-ssl-http.zip:命令行 ./bin/elasticsearch-certutil http操作如下## Elasticsearch HTTP Certificate UtilityThe 'http' command guides you through the process of generating certificatesfor use on the HTTP (Rest) interface for Elasticsearch.This tool will ask you a number of questions in order to generate the rightset of files for your needs.## Do you wish to generate a Certificate Signing Request (CSR)?A CSR is used when you want your certificate to be created by an existingCertificate Authority (CA) that you do not control (that is, you don't haveaccess to the keys for that CA). If you are in a corporate environment with a central security team, then youmay have an existing Corporate CA that can generate your certificate for you.Infrastructure within your organisation may already be configured to trust thisCA, so it may be easier for clients to connect to Elasticsearch if you use aCSR and send that request to the team that controls your CA.If you choose not to generate a CSR, this tool will generate a new certificatefor you. That certificate will be signed by a CA under your control. This is aquick and easy way to secure your cluster with TLS, but you will need toconfigure all your clients to trust that custom CA.## 生成CSR 输入nGenerate a CSR? [y/N]n## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate?If you have an existing CA certificate and key, then you can use that CA tosign your new http certificate. This allows you to use the same CA acrossmultiple Elasticsearch clusters which can make it easier to configure clients,and may be easier for you to manage.If you do not have an existing CA, one will be generated for you.## 是否使用存在的ca 输入y（在基础配置时生成了）Use an existing CA? [y/N]y## What is the path to your CA?Please enter the full pathname to the Certificate Authority that you wish touse for signing your new http certificate. This can be in PKCS#12 (.p12), JKS(.jks) or PEM (.crt, .key, .pem) format.## 输入ca文件的地址CA Path: /usr/share/elasticsearch/config/elastic-stack-ca.p12Reading a PKCS12 keystore requires a password.It is possible for the keystore's password to be blank,in which case you can simply press <ENTER> at the prompt## 输入文件设置的密码Password for elastic-stack-ca.p12:## How long should your certificates be valid?Every certificate has an expiry date. When the expiry date is reached clientswill stop trusting your certificate and TLS connections will fail.Best practice suggests that you should either:(a) set this to a short duration (90 - 120 days) and have automatic processesto generate a new certificate before the old one expires, or(b) set it to a longer duration (3 - 5 years) and then perform a manual updatea few months before it expires.You may enter the validity period in years (e.g. 3Y), months (e.g. 18M), or days (e.g. 90D)## 设置过期时间For how long should your certificate be valid? [5y] 10y## Do you wish to generate one certificate per node?If you have multiple nodes in your cluster, then you may choose to generate aseparate certificate for each of these nodes. Each certificate will have itsown private key, and will be issued for a specific hostname or IP address.Alternatively, you may wish to generate a single certificate that is validacross all the hostnames or addresses in your cluster.If all of your nodes will be accessed through a single domain(e.g. node01.es.example.com, node02.es.example.com, etc) then you may find itsimpler to generate one certificate with a wildcard hostname (*.es.example.com)and use that across all of your nodes.However, if you do not have a common domain name, and you expect to addadditional nodes to your cluster in the future, then you should generate acertificate per node so that you can more easily generate new certificates whenyou provision new nodes.## 是否为每一个节点生成证书 输入nGenerate a certificate per node? [y/N]n## Which hostnames will be used to connect to your nodes?These hostnames will be added as "DNS" names in the "Subject Alternative Name"(SAN) field in your certificate.You should list every hostname and variant that people will use to connect toyour cluster over http.Do not list IP addresses here, you will be asked to enter them later.If you wish to use a wildcard certificate (for example *.es.example.com) youcan enter that here.## 节点的hostname,设置为elasticsearch，敲两次回车Enter all the hostnames that you need, one per line.When you are done, press <ENTER> once more to move on to the next step.elasticsearchYou entered the following hostnames.- elasticsearch## 配置是否正确Is this correct [Y/n]y## Which IP addresses will be used to connect to your nodes?If your clients will ever connect to your nodes by numeric IP address, then youcan list these as valid IP "Subject Alternative Name" (SAN) fields in yourcertificate.If you do not have fixed IP addresses, or not wish to support direct IP accessto your cluster then you can just press <ENTER> to skip this step.## 节点的ip(可以在宿主机通过命令docker inspect elasticsearch查看),设置为172.99.0.2，敲两次回车Enter all the IP addresses that you need, one per line.When you are done, press <ENTER> once more to move on to the next step.172.99.0.2You entered the following IP addresses.- 172.99.0.2## 配置是否正确Is this correct [Y/n]y## Other certificate optionsThe generated certificate will have the following additional configurationvalues. These values have been selected based on a combination of theinformation you have provided above and secure defaults. You should not need tochange these values unless you have specific requirements.Key Name: elasticsearchSubject DN: CN=elasticsearchKey Size: 2048## 是否更改任意项Do you wish to change any of these options? [y/N]n## What password do you want for your private key(s)?Your private key(s) will be stored in a PKCS#12 keystore file named "http.p12".This type of keystore is always password protected, but it is possible to use ablank password.If you wish to use a blank password, simply press <enter> at the prompt below.## 输入生成文件的密码（可不设置，设置需要在后面进行配置）Provide a password for the "http.p12" file: [<ENTER> for none]## 再次输入生成文件的密码Repeat password to confirm: ## Where should we save the generated files?A number of files will be generated including your private key(s),public certificate(s), and sample configuration options for Elastic Stack products.These files will be included in a single zip archive.## 生成压缩文件的地址和名称，直接敲回车即可What filename should be used for the output zip file? [/usr/share/elasticsearch/elasticsearch-ssl-http.zip] 移动elasticsearch-ssl-http.zip压缩包:命令行 mv elasticsearch-ssl-http.zip ./config/ 解压文件:退出容器命令行 unzip /iscsi/elk/elasticsearch/config/elasticsearch-ssl-http.zip解压后会在原目录下新增两个目录分别是 elasticsearch 和 kibana命令行 mv /iscsi/elk/elasticsearch/config/elasticsearch/http.p12 /iscsi/elk/elasticsearch/config/certs/复制elasticsearch-ca.pem到kibana的config文件夹内:命令行 cp /iscsi/elk/elasticsearch/config/kibana/elasticsearch-ca.pem /iscsi/elk/kibana/config/删除文件夹:命令行 rm -rf /iscsi/elk/elasticsearch/certs/kibana文件提权:命令行 chmod 777 /iscsi/elk/elasticsearch/config/certs/http.p12 设置http密码:命令行 docker exec -it elasticsearch /bin/bash命令行 ./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password退出容器重启elasticsearch容器: 命令行 docker restart elasticsearch设置elastic用户的密码:进入容器: 命令行 docker exec -it elasticsearch /bin/bash设置密码: ./bin/elasticsearch-reset-password -u elastic -i设置kibana_system密码:命令行 ./bin/elasticsearch-reset-password -u kibana_system -i生成kibana用https访问的公钥和私钥:命令行 ./bin/elasticsearch-certutil csr -name kibana-server备注 生成csr-bundle.zip文件夹复制csr-bundle.zip到kibana:退出容器命令行 docker cp elasticsearch:/usr/share/elasticsearch/csr-bundle.zip /iscsi/elk/kibana/解压csr-bundle.zip:命令行 cd /iscsi/elk/kibana/命令行 unzip /iscsi/elk/kibana/csr-bundle.zip备注 解压后会生成 kibana-server 文件夹移动文件到kibana的配置目录:命令行 mv /iscsi/elk/kibana/kibana-server/* /iscsi/elk/kibana/config/命令行 rm -rf /iscsi/elk/kibana/kibana-server生成kibana-server.crt文件:命令行 cd /iscsi/elk/kibana/config命令行 openssl x509 -req -days 3650 -in kibana-server.csr -signkey kibana-server.key -out kibana-server.crt文件提权:命令行 chmod 777 elasticsearch-ca.pem kibana-server.csr kibana-server.key kibana-server.crt修改kibana.yml文件:命令行 vim /iscsi/elk/kibana/config/kibana.yml最终文件为`## ** THIS IS AN AUTO-GENERATED FILE **## Default Kibana configuration for docker targetserver.host: "0.0.0.0"server.shutdownTimeout: "5s"elasticsearch.hosts: [ "https:/172.99.0.2:9200" ]monitoring.ui.container.elasticsearch.enabled: trueelasticsearch.ssl.certificateAuthorities: ["/usr/share/kibana/config/elasticsearch-ca.pem"]elasticsearch.username: "kibana_system"elasticsearch.password: "tonglian@126.com"server.ssl.certificate: "/usr/share/kibana/config/kibana-server.crt"server.ssl.key: "/usr/share/kibana/config/kibana-server.key"server.ssl.enabled: true# 设置中文访问i18n.locale: "zh-CN" `重启kiban:命令行 docker restart kibana配置logstash:设置logstash_system密码:命令行 docker exec -it elasticsearch /bin/bash命令行 ./bin/elasticsearch-reset-password -u logstash_system -i生成logstash.pem文件:退出容器命令行 openssl pkcs12 -in elasticsearch/config/certs/elastic-certificates.p12 -cacerts -nokeys -chain -out logstash.pem移动logstash.pem文件到logstash配置文件目录下:命令行 mv /iscsi/elk/elasticsearch/config/certs/logstash.pem /iscsi/elk/logstash/config/提权logstash.pem:命令行 chmod 777 /iscsi/elk/logstash/config/logstash.pem配置logstash.yml文件:命令行 vim /iscsi/elk/logstash/config/logstash.yml最终文件`http.host: "0.0.0.0"xpack.monitoring.elasticsearch.hosts: [ "https://172.99.0.2:9200" ]#你的ca.pem 的所在路径xpack.monitoring.elasticsearch.ssl.certificate_authority: "/usr/share/logstash/config/logstash.pem"xpack.monitoring.elasticsearch.ssl.verification_mode: certificate# 探嗅 es节点，设置为 falsexpack.monitoring.elasticsearch.sniffing: falsexpack.monitoring.elasticsearch.username: "logstash_system"xpack.monitoring.elasticsearch.password: "tonglian@126.com" `配置logstash.conf文件:命令行 vim /iscsi/elk/logstash/pipeline/logstash.conf最终文件`input {tcp {port => 21068codec => json_lines}}output {elasticsearch {hosts => ["https://172.99.0.2:9200"]index => "tonywoo-%{+YYYY.MM.dd}"user => "elastic"password => "tonglian@126.com"ssl_enabled => truessl_certificate_authorities => ["/usr/share/logstash/config/logstash.pem"]}} `重启logstash:docker restart logstash设置自动启动:`cat > /etc/systemd/system/docker-compose-elk.service << EOF[Unit]Description=Docker Compose Application ServiceRequires=docker.serviceAfter=docker.service[Service]Type=oneshotRemainAfterExit=yesWorkingDirectory=/iscsi/elk/ExecStart=/iscsi/elk/docker-compose -f docker-compose-elk.yml up -dExecStop=/iscsi/elk/docker-compose -f docker-compose-elk.yml up downTimeoutStartSec=0[Install]WantedBy=multi-user.target `回车命令行 ctrl+d
