参考
- https://www.cnblogs.com/hualess/p/11540477.html
- https://blog.csdn.net/susu1083018911/article/details/124551632
- https://blog.csdn.net/Dyanxier/article/details/131302723
- https://blog.csdn.net/qq_44768464/article/details/120101990
- https://www.cnblogs.com/coderxz/p/13268417.html
- https://blog.csdn.net/Baldprogrammer/article/details/119429530
- https://www.cnblogs.com/youqc/p/15397900.html
- https://blog.csdn.net/Anumbrella/article/details/102691831
注意
- mvn 依赖 logstash-logback-encoder 7.4 版本测试出错,7.3 可以。。
- 所有服务启动后,Logstash 命令行无输出 json 应该是 Spring Boot 项目没有正确发送日志,试试清理 Spring Boot 项目的依赖,重新启动项目;或重启电脑。
- 比较新的架构是 ELKB。
- Elasticsearch 的安全开关默认打开,请求协议为 https,访问:
https://127.0.0.1:9200
需要输入账号密码。
环境
环境 | 版本 | 说明 |
---|---|---|
windows | 10 | |
vs code | 1.85.1 | |
Spring Boot Extension Pack | v0.2.1 | vscode插件 |
Extension Pack for Java | v0.25.15 | vscode插件 |
JDK | 11 | |
Springboot | 2.3.12.RELEASE | |
Apache Maven | 3.8.6 | |
logstash-logback-encoder | 7.3 | mvn驱动,测试7.4版本可能存在问题,导致无法发送日志给logstash |
Elasticsearch | 8.11.3 | 下载 |
kibana | 8.11.3 | 需要与ES一致 下载 |
Logstash | 8.11.3 | 需要与ES一致,另外cn语言网站没有exe安装包,需要去fr语言网站下载 下载 |
正文
准备
-
下载 Elasticsearch 、Kibana 、 Logstash。
-
Elasticsearch
elasticsearch-8.11.3\config\elasticsearch.yml
部分配置调整:# Enable security features # 6.8 和 7.1 后默认开启安全,强制ssl与账号密码 # 默认true xpack.security.enabled: false # 追加到文件尾部,设置插件访问es配置。 http.cors.enabled: true http.cors.allow-origin: "*"
-
Kibana
kibana-8.11.3\config\kibana.yml
配置追加:# 提供服务的IP server.host: "localhost" # 提供服务的端口,也就是 Kibana 网页面板访问的端口号。 server.port: 5601 # es服务的地址 elasticsearch.hosts: ["http://localhost:9200"] # 本地化 i18n.locale: "zh-CN"
-
Logstash
logstash-8.11.3\config\logstash.conf
新增配置文件:# 启动命令 ./bin/logstash.bat -f ./config/logstash.conf input{ tcp { mode => "server" host => "0.0.0.0" #我这里是本地 port => 4567 #开放这个端口进行采集 codec => json_lines # 编解码器 } } output{ elasticsearch { #es地址 hosts=>"127.0.0.1:9200" # 在es里产生的index的名称 index => "springboot-%{+YYYY.MM.dd}" } stdout{ codec => rubydebug } }
-
Spring Boot 项目配置
pom.xml
追加依赖<!-- https://mvnrepository.com/artifact/net.logstash.logback/logstash-logback-encoder --> <dependency> <groupId>net.logstash.logback</groupId> <artifactId>logstash-logback-encoder</artifactId> <version>7.3</version> </dependency>
src\main\resources\application.properties
追加配置# 引入 logstash 配置 logging.config=classpath:logstash-spring.xml logging.level.root=INFO
- 创建
src\main\resources\logstash-spring.xml
并写入:<?xml version="1.0" encoding="UTF-8"?> <configuration> <include resource="org/springframework/boot/logging/logback/base.xml" /> <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender"> <!-- Logstash 配置中开放的端口 input{ tcp { mode => "server" host => "0.0.0.0" #我这里是本地 port => 4567 #开放这个端口进行采集 codec => json_lines # 编解码器 有的版本需要独自安装 } } --> <destination>localhost:4567</destination> <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder" /> </appender> <!-- <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"> <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder"> <jsonGeneratorDecorator class="net.logstash.logback.decorate.FeatureJsonGeneratorDecorator"/> <providers> <pattern> <pattern> { "date": "%date{yyyy-MM-dd'T'HH:mm:ss.SSSZ}", "level": "%level", "msg": "%msg" } </pattern> </pattern> </providers> </encoder> </appender> --> <root level="INFO"> <appender-ref ref="LOGSTASH" /> <!-- <appender-ref ref="STDOUT" /> --> </root> </configuration>
启动
启动顺序为 1. Elasticsearch、2. Kibana、Logstash、3. Spring Boot 项目。
-
Elasticsearch 双击启动
elasticsearch-8.11.3\bin\elasticsearch.bat
,访问http://127.0.0.1:9200/
显示如下内容为启动成功。{ "name" : "DESKTOP-XXXXXXX", "cluster_name" : "elasticsearch", "cluster_uuid" : "XXXXNTUgXXXJTTNwXXpXXw", "version" : { "number" : "8.11.3", "build_flavor" : "default", "build_type" : "zip", "build_hash" : "64cf0xxx3b56b1fd444xxx454cb88aca7e7xxx9a", "build_date" : "2023-12-08T11:33:53.634979452Z", "build_snapshot" : false, "lucene_version" : "9.8.0", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" }
-
Kibana 双击启动
kibana-8.11.3\bin\kibana.bat
,访问http://127.0.0.1:5601
(图片:https://zhuanlan.zhihu.com/p/649902671)。首次启动提示 kibana no such index [.kibana],重启es和 kibana 试试。
-
Logstash 命令启动(指定配置文件):
./logstash-8.11.3/bin/logstash.bat -f ./logstash-8.11.3/config/logstash.conf
出现如下类似的控制台输出为启动成功。
[2023-12-29T10:40:56,656][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.85} [2023-12-29T10:40:56,802][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"} [2023-12-29T10:40:56,803][INFO ][logstash.inputs.tcp ][main][66431899e5xxe6919xxaba30xx636afdf4xx4c6f458a9245473bd02c7c1650c6] Starting tcp input listener {:address=>"0.0.0.0:4567", :ssl_enabled=>false} [2023-12-29T10:40:56,815][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
-
启动 Spring Boot 项目。
校验
- 启动 Spring Boot 项目后,观察 Logstash 启动命令行,显示如下类似信息即表示接收到了数据。
...
{
"@version" => "1",
"level_value" => 20000,
"logger_name" => "org.springframework.web.servlet.DispatcherServlet",
"message" => "Initializing Servlet 'dispatcherServlet'",
"thread_name" => "http-nio-8080-exec-3",
"@timestamp" => 2023-12-29T02:54:04.553Z,
"level" => "INFO"
}
{
"@version" => "1",
"level_value" => 30000,
"logger_name" => "com.xiaqiuchu.elk.controller.IndexController",
"message" => "warn log",
"thread_name" => "http-nio-8080-exec-3",
"@timestamp" => 2023-12-29T02:54:04.557Z,
"level" => "WARN"
}
{
"@version" => "1",
"level_value" => 20000,
"logger_name" => "org.springframework.web.servlet.DispatcherServlet",
"message" => "Completed initialization in 1 ms",
"thread_name" => "http-nio-8080-exec-3",
"@timestamp" => 2023-12-29T02:54:04.554Z,
"level" => "INFO"
}
{
"@version" => "1",
"level_value" => 20000,
"logger_name" => "org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/]",
"message" => "Initializing Spring DispatcherServlet 'dispatcherServlet'",
"thread_name" => "http-nio-8080-exec-3",
"@timestamp" => 2023-12-29T02:54:04.549Z,
"level" => "INFO"
}
...
数据可视化
- 访问 Kibana
http://127.0.0.1:5601
(ip与端口都在配置文件中配置的)。 - 选择 左侧菜单->Machine Learning
- 选择 可视化来自视图的数据。
- 选择 创建数据视图。
- 设置名称与索引模式,然后选择 保存数据视图到 Kibana 。(名称任意设置,索引模式是指匹配数据流,在 Logstash 配置文件中可以设置这个索引。如果索引模式匹配不到你的数据流,那么需要确认下是否正确将日志发送到 Logstash 没有。)
- 完毕(如果没数据请检查数据是否成功发送,另外可以调整数据统计时间区间)。
仪表板(自定义的数据大屏)
- Elasticsearch Logstash Windows Kibana ELKelasticsearch logstash windows kibana elasticsearch logstash kibana平台 elasticsearch logstash kibana stack elasticsearch logstash kibana系统 elasticsearch logstash serilog kibana elasticsearch filebeat windows kibana elasticsearch windows环境kibana logstash企业elk logstash kibana https beats elasticsearch docker kibana