ELK_8.2.0

安装及部署手册

安装环境

系统环境

操作系统：CentOS 7

软件环境

基础软件

ELK所需基础服务及版本信息，如下表所示：

序号	软件名称	软件版本	备注
1	jdk	jdk-11.0.18
2	elasticsearch	8.2.0
3	filebeat	8.2.0
4	logstash	8.2.0
5	kibana	8.2.0
6

ELK服务

优化系统

序号	服务名称	操作命令	备注
0	修改主机名	hostnamectl set-hostname es01（主机名称）	需要重启机器
1	关闭selinux	setenforce 0	vi /etc/sysconfig/selinux 里边 enforcing 改为disabled 重启动机器生效
2	关闭firewalld	systemctl stop firewalld systemctl disable firewalld

3	打开句柄数限制	vi /etc/sysctl.conf 添加下面配置： vm.max_map_count=655360 并执行命令： sysctl -p
4	配置域名解析	/etc/hosts 10.209.22.105 esserver1 10.209.22.106 esserver2

JDK安装配置

我们要将jdk-17.0.8_linux-aarch64_bin.tar.gz解压到linux系统中，用于es使用。（解压位置自定义为：/ ）。

使用命令 tar -zxvf /opt/jdk-17.0.8_linux-aarch64_bin.tar.gz -C / 将opt文件夹下的jdk-17.0.8解压到根目录 /home/es 下。

export JAVA_HOME=/home/es/jdk-17.0.6

PATH=$JAVA_HOME/bin:$PATH

CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export JAVA_HOME

export PATH

export CLASSPATH

ES生成证书

如下操作在其中一个node节点执行即可，生成完证书传到集群其他节点即可

bin/elasticsearch-certutil ca

bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

两条命令es8.x开启xpark认证步骤均一路回车即可，不需要给秘钥再添加密码。

证书创建完成之后，一定放到elasticsearch的config目录下，放到其他目录会报路径无权限的错误！

ES配置文件更改

elasticsearch.yml

# ---------------------------------- Cluster -----------------------------------

# 集群中所有节点必须保持一致

cluster.name: elk-application

# ------------------------------------ Node ------------------------------------

# 需要随节点更改

node.name: es01

node.attr.rack: r1

# ----------------------------------- Paths ------------------------------------

path.data: /home/es/elasticsearch-8.2.0/data

path.logs: /home/es/elasticsearch-8.2.0/logs

# ----------------------------------- Memory -----------------------------------

bootstrap.memory_lock: false

# ---------------------------------- Network -----------------------------------

# 默认情况下，Elasticsearch 仅仅绑定回环地址，比如127.0.0.1 和[::1]

# 需要随节点更改

network.host: 10.20.12.103

http.port: 9200

transport.port: 9300

# 开启安全防护

http.cors.enabled: true

http.cors.allow-origin: "*"

http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type

# 集群所有 master-eligible 节点

# 对应旧版中的 discovery.zen.ping.unicast.hosts

discovery.seed_hosts: ["es01:9300", "es02:9300", "es03:9300"]

# Elasticsearch 7.0新引入的配置项

# 集群第一次启动达到这个数量后就开始引导

cluster.initial_master_nodes: ["es01", "es02", "es03"]

xpack.security.enabled: true

xpack.license.self_generated.type: basic

xpack.security.transport.ssl.enabled: true

xpack.security.transport.ssl.verification_mode: certificate

xpack.security.transport.ssl.keystore.path: elastic-certificates.p12

xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

启动es

bin/elasticsearch -d

es生成管理密码

bin/elasticsearch-setup-passwords interactive

jvm.options

-Xms8g

-Xmx8g

Filebeat配置

#=========================== Filebeat inputs =============================

filebeat.inputs:

- type: log

enabled: true

paths:

- /home/rsyslog/logs/2*/*.log ##获取路径下所有包含secure开头的文件。

#----------------------------- Logstash output --------------------------------

output.logstash:

# The Logstash hosts

hosts: ["10.20.12.106:5044"] ##logstash服务地址和端口

#----------------------------- elasticsearch output --------------------------------

#output.elasticsearch:

# hosts: ["10.20.12.103:9200", "10.20.12.104:9200", "10.20.12.105:9200"]

# username: "elastic"

# password: "Elastic0309#"

# indices:

# - index: "test-index-%{+yyyy.MM.dd}"

# allow_older_versions: true #兼容旧es

#----------------------------- console output --------------------------------

#output.console:

# pretty: true

# enable: true

Filebeat启动

Nohup ./filebeat -e &

Syslog server配置

# Provides UDP syslog reception

$ModLoad imudp

$UDPServerRun 514

# Provides TCP syslog reception

$ModLoad imtcp

$InputTCPServerRun 514

$template Remote,"/home/rsyslog/logs/%$YEAR%-%$MONTH%-%$DAY%/%fromhost-ip%.log" # 设置远程日志存放路径和文件格式

:fromhost-ip, !isequal, "127.0.0.1" ?Remote # 如果是本机日志则不记录

& ~ #丢弃无匹配日志

编辑/etc/sysconfig/rsyslog加入“-m 0 -r”

SYSLOGD_OPTIONS=”-m 0 -r”

Syslog client配置

*.* @@10.20.12.106:514 #发送到syslog server服务日志

如何记录操作命令：需配置/etc/bashrc

export PROMPT_COMMAND='{ msg=$(history 1 | { read x y; echo $y; }); logger $(date "+%Y-%m-%d %H:%M:%S"):$user:$(hostname -I):$msg:$(who am i|cut -d$ -f2|cut -d$ -f1); }'

Logstash配置

# Sample Logstash configuration for creating a simple

# Beats -> Logstash -> Elasticsearch pipeline.

input {

beats {

port => 5044

}

# -------清洗数据grok--------

filter{

grok{

match => {"message" => "(?<logTime>(%{MONTH}%{SPACE}%{MONTHDAY}%{SPACE}(?<Time>%{TIME}))) (?<localname>.*?) (?<sysuser>.*?) (?<timestamp>(%{TIMESTAMP_ISO8601}))::%{IPV4:local_ip} :(?<shell>.*?):%{IPV4:login_ip}"}

}

#-----------输出到es--------

output {

elasticsearch {

hosts => ["10.20.12.103:9200", "10.20.12.104:9200", "10.20.12.105:9200"]

index => "sys-log-%{+YYYY.MM.dd}"

user => "elastic"

password => "Elastic0309#"

}

stdout {

codec => rubydebug

}

#output {

# stdout {

# codec => rubydebug

# }

Logstash启动

Nohup bin/logstash -f config/logstash-sample.conf &

Logstash清洗linux主机日志

日志样例：

"message": "Mar 14 18:30:06 es03 root: 2023-03-14 18:30:06::10.20.12.105 :more bashrc:10.2.11.85"

Grok规则：(?<logTime>(%{MONTH}%{SPACE}%{MONTHDAY}%{SPACE}(?<Time>%{TIME}))) (?<localname>.*?) (?<sysuser>.*?) (?<timestamp>(%{TIMESTAMP_ISO8601}))::%{IPV4:localip} :(?<shell>.*?):%{IPV4:clientip}