前言
企业开发中，一个应用会有多个测试环境，于是会有一个专门的服务器做日志收集，那就需要保存日志和应用隔离，这里就牵涉到异步存写日志的问题，异步消息队列选取kafka，高性能，日志消息消费我们使用logstash。这里也可以都使用elk（土豪请绕过）。
Kafka的搭建
习惯使用docker-compose启动，脚本如下:
version: '2'
services:zookeeper:image: wurstmeister/zookeeper ## 镜像ports:- "2181:2181" ## 对外暴露的端口号kafka:image: wurstmeister/kafka ## 镜像volumes:- /etc/localtime:/etc/localtime ## 挂载位置（kafka镜像和宿主机器之间时间保持一直）ports:- "9092:9092"environment:KAFKA_ADVERTISED_HOST_NAME: ## 修改:宿主机IPKAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 ## kafka运行是基于zookeeper的kafka-manager:image: sheepkiller/kafka-manager ## 镜像：开源的web管理kafka集群的界面environment:ZK_HOSTS: ## 修改:宿主机IPports:- "9000:9000"
docker-compose up -d 后台方式启动
docker-compose stop 关闭
docker-compose ps 查看有哪些应用
docker-compose build 构建镜像
Logstash的搭建
同样也是用docker-compose启动，脚本如下：
docker-compose.yml->
version: "3"
services:logstash-1:image: logstash:7.0.0container_name: logstashvolumes:- ${LOGSTASH_CONFIG_DIR}/logstash.conf:/usr/share/logstash/pipeline/logstash.conf:rw- ${LOGSTASH_CONFIG_DIR}/logstash.yml:/usr/share/logstash/config/logstash.yml:rwnetwork_mode: "host"
logstash.config
input {kafka {bootstrap_servers => "IP:9092"topics => ["abklog_topic"]group_id => "abklog_topic"}
}output {file {file_mode => 0777dir_mode => 0777path => "/path/to/%{+yyyy-MM-dd-HH}/%{host}.log"}stdout {codec => rubydebug}
}
logstash.yml
# set now host ip to http.host
http.host: 192.168.56.121
# set the es-tribe-node host. let logstash monitor the es.
#xpack.monitoring.elasticsearch.hosts:
#- http://10.2.114.110:9204
# enable or disable the logstash monitoring the es.
#xpack.monitoring.enabled: true
Java日志配置
在你改好的springboot工程中整好，加入这个依赖。
<!-- https://mvnrepository.com/artifact/com.github.danielwegener/logback-kafka-appender --><dependency><groupId>com.github.danielwegener</groupId><artifactId>logback-kafka-appender</artifactId><version>0.2.0-RC2</version></dependency>
logback.xml配置
<?xml version="1.0" encoding="UTF-8"?>
<configuration><!-- springProfile用于指定当前激活的环境，如果spring.profile.active的值是哪个，就会激活对应节点下的配置 --><springProfile name="dev"><!-- configuration to be enabled when the "staging" profile is active --><springProperty scope="context" name="module" source="spring.application.name"defaultValue="undefinded"/><!-- 该节点会读取Environment中配置的值，在这里我们读取application.yml中的值 --><springProperty scope="context" name="bootstrapServers" source="spring.kafka.bootstrap-servers"defaultValue="IP:9092"/><appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"><!-- encoders are assigned the typech.qos.logback.classic.encoder.PatternLayoutEncoder by default --><encoder><pattern>%boldYellow(${module}) | %d | %highlight(%-5level)| %cyan(%logger{15}) - %msg %n</pattern></encoder></appender><!-- kafka的appender配置 --><appender name="kafka" class="com.github.danielwegener.logback.kafka.KafkaAppender"><encoder><pattern>${module} | %d | %-5level| %logger{15} - %msg</pattern></encoder><topic>abklog_topic</topic><keyingStrategy class="com.github.danielwegener.logback.kafka.keying.NoKeyKeyingStrategy"/><deliveryStrategy class="com.github.danielwegener.logback.kafka.delivery.AsynchronousDeliveryStrategy"/><!-- Optional parameter to use a fixed partition --><!-- <partition>0</partition> --><!-- Optional parameter to include log timestamps into the kafka message --><!-- <appendTimestamp>true</appendTimestamp> --><!-- each <producerConfig> translates to regular kafka-client config (format: key=value) --><!-- producer configs are documented here: https://kafka.apache.org/documentation.html#newproducerconfigs --><!-- bootstrap.servers is the only mandatory producerConfig --><producerConfig>bootstrap.servers=${bootstrapServers}</producerConfig><!-- 如果kafka不可用则输出到控制台 --><appender-ref ref="STDOUT"/></appender><!-- 指定项目中的logger --><!--<logger name="org.springframework.test" level="INFO" ><appender-ref ref="kafka" /></logger>--><logger name="com.fast.cloud.fastelk.controller" level="INFO" ><appender-ref ref="kafka" /></logger><root level="info"><appender-ref ref="STDOUT" /></root></springProfile>
</configuration>
记得将kafka的ip换成你真实的ip
