本文介绍通过elk + filebeat方式收集k8s日志，其中filebeat以sidecar方式部署。elfk最新版本：7.6.2
k8s日志收集方案
3种日志收集方案：
1. node上部署一个日志收集程序
Daemonset方式部署日志收集程序，对本节点 /var/log 和 /var/lib/docker/containers 两个目录下的日志进行采集
2. sidecar方式部署日志收集程序
每个运行应用程序的pod中附加一个日志收集的容器，使用 emptyDir 共享日志目录让日志容器收集日志
3. 应用程序直接推送日志
常见的如 graylog 工具，直接修改代码推送日志到es，然后在graylog上展示出来
1
2
3
4
5
6
7
8
9
10
11
3种收集方案的优缺点：
方案 优点 缺点
1. node上部署一个日志收集程序 每个node仅需部署一个日志收集程序，消耗资源少，对应用无侵入 应用程序日志需要写到标准输出和标准错误输出，不支持多行日志
2. pod中附加一个日志收集容器 低耦合 每个pod启动一个日志收集容器，增加资源消耗
3. 应用程序直接推送日志 无需额外收集工具 侵入应用，增加应用复杂度
下面测试第1种方案：每个node上部署一个日志收集程序，注意elfk版本保持一致。
SideCar方式收集k8s日志
主机说明：
系统 ip 角色 cpu 内存 hostname
CentOS 7.8 192.168.30.128 master、deploy >=2 >=2G master1
CentOS 7.8 192.168.30.129 master >=2 >=2G master2
CentOS 7.8 192.168.30.130 node >=2 >=2G node1
CentOS 7.8 192.168.30.131 node >=2 >=2G node2
CentOS 7.8 192.168.30.132 node >=2 >=2G node3
CentOS 7.8 192.168.30.133 test >=2 >=2G test
搭建k8s集群：
搭建过程省略，具体参考：Kubeadm方式搭建k8s集群 或 二进制方式搭建k8s集群
搭建完成后，查看集群：
kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready master 4d16h v1.14.0
master2 Ready master 4d16h v1.14.0
node1 Ready <none> 4d16h v1.14.0
node2 Ready <none> 4d16h v1.14.0
node3 Ready <none> 4d16h v1.14.0
1
2
3
4
5
6
7
8
这里为了方便，直接使用之前的k8s集群，注意删除之前实验的k8s资源对象。
docker-compose部署elk：
部署过程参考：docker-compose部署elfk
mkdir /software & cd /software
git clone https://github.com/Tobewont/elfk-docker.git
cd elfk-docker/
echo 'ELK_VERSION=7.6.2' > .env #替换版本
vim elasticsearch/Dockerfile
1
2
3
4
5
6
7
8
9
ARG ELK_VERSION=7.6.2
# https://github.com/elastic/elasticsearch-docker
FROM docker.elastic.co/elasticsearch/elasticsearch-oss:${ELK_VERSION}
# FROM elasticsearch:${ELK_VERSION}
# Add your elasticsearch plugins setup here
# Example: RUN elasticsearch-plugin install analysis-icu
1
2
3
4
5
6
7
vim elasticsearch/config/elasticsearch.yml
1
---
## Default Elasticsearch configuration from Elasticsearch base image.
## https://github.com/elastic/elasticsearch/blob/master/distribution/docker/src/docker/config/elasticsearch.yml
#
cluster.name: "docker-cluster"
network.host: "0.0.0.0"
## X-Pack settings
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-xpack.html
#
#xpack.license.self_generated.type: trial #trial为试用版，一个月期限，可更改为basic版本
#xpack.security.enabled: true
#xpack.monitoring.collection.enabled: true
#http.cors.enabled: true
#http.cors.allow-origin: "*"
#http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
vim kibana/Dockerfile
1
ARG ELK_VERSION=7.6.2
# https://github.com/elastic/kibana-docker
FROM docker.elastic.co/kibana/kibana-oss:${ELK_VERSION}
# FROM kibana:${ELK_VERSION}
# Add your kibana plugins setup here
# Example: RUN kibana-plugin install <name|url>
1
2
3
4
5
6
7
8
vim kibana/config/kibana.yml
1
---
## Default Kibana configuration from Kibana base image.
## https://github.com/elastic/kibana/blob/master/src/dev/build/tasks/os_packages/docker_generator/templates/kibana_yml.template.js
#
server.name: "kibana"
server.host: "0.0.0.0"
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
# xpack.monitoring.ui.container.elasticsearch.enabled: true
## X-Pack security credentials
#
# elasticsearch.username: elastic
# elasticsearch.password: changeme
1
2
3
4
5
6
7
8
9
10
11
12
13
vim logstash/Dockerfile
1
ARG ELK_VERSION=7.6.2
# https://github.com/elastic/logstash-docker
FROM docker.elastic.co/logstash/logstash-oss:${ELK_VERSION}
# FROM logstash:${ELK_VERSION}
# Add your logstash plugins setup here
# Example: RUN logstash-plugin install logstash-filter-json
RUN logstash-plugin install logstash-filter-multiline
1
2
3
4
5
6
7
8
9
10
vim logstash/config/logstash.yml
1
---
## Default Logstash configuration from Logstash base image.
## https://github.com/elastic/logstash/blob/master/docker/data/logstash/config/logstash-full.yml
#
http.host: "0.0.0.0"
#xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]
## X-Pack security credentials
#
#xpack.monitoring.enabled: true
#xpack.monitoring.elasticsearch.username: elastic
#xpack.monitoring.elasticsearch.password: changeme
#xpack.monitoring.collection.interval: 10s
1
2
3
4
5
6
7
8
9
10
11
12
13
vim logstash/pipeline/logstash.conf #如果filebeat配置文件中没有配置多行合并，可以在logstash中进行多行合并
1
input {
beats {
port => 5040
}
}
filter {
if [type] == "nginx_access" {
#multiline {
#pattern => "^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}"
#negate => true
#what => "previous"
#}
grok {
match => [ "message", "%{IPV4:remote_addr} - (%{USERNAME:user}|-) \[%{HTTPDATE:log_timestamp}\] \"%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:status} %{NUMBER:bytes} %{QS:referer} %{QS:agent} %{QS:xforward}" ]
}
}
if [type] == "nginx_error" {
#multiline {
#pattern => "^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}"
#negate => true
#what => "previous"
#}
grok {
match => [ "message", "%{IPV4:remote_addr} - (%{USERNAME:user}|-) \[%{HTTPDATE:log_timestamp}\] \"%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:status} %{NUMBER:bytes} %{QS:referer} %{QS:agent} %{QS:xforward}" ]
}
}
if [type] == "tomcat_catalina" {
#multiline {
#pattern => "^\d{1,2}-\S{3}-\d{4}\s\d{1,2}:\d{1,2}:\d{1,2}"
#negate => true
#what => "previous"
#}
grok {
match => [ "message", "(?<timestamp>%{MONTHDAY}-%{MONTH}-%{YEAR} %{TIME}) %{LOGLEVEL:severity} \[%{DATA:exception_info}\] %{GREEDYDATA:message}" ]
}
}
}
output {
if [type] == "nginx_access" {
elasticsearch {
hosts => ["elasticsearch:9200"]
#user => "elastic"
#password => "changeme"
index => "nginx-access"
}
}
if [type] == "nginx_error" {
elasticsearch {
hosts => ["elasticsearch:9200"]
#user => "elastic"
#password => "changeme"
index => "nginx-error"
}
}
if [type] == "tomcat_catalina" {
elasticsearch {
hosts => ["elasticsearch:9200"]
#user => "elastic"
#password => "changeme"
index => "tomcat-catalina"
}
}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
vim docker-compose.yml
1
version: '3.7'
services:
elasticsearch:
build:
context: elasticsearch/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- type: bind
source: ./elasticsearch/config/elasticsearch.yml
target: /usr/share/elasticsearch/config/elasticsearch.yml
read_only: true
- type: volume
source: elasticsearch
target: /usr/share/elasticsearch/data
ports:
- "9200:9200"
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xmx256m -Xms256m"
ELASTIC_PASSWORD: changeme
discovery.type: single-node
networks:
- elk
logstash:
depends_on:
- elasticsearch
build:
context: logstash/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- type: bind
source: ./logstash/config/logstash.yml
target: /usr/share/logstash/config/logstash.yml
read_only: true
- type: bind
source: ./logstash/pipeline
target: /usr/share/logstash/pipeline
read_only: true
ports:
- "5040:5040"
- "9600:9600"
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
networks:
- elk
kibana:
depends_on:
- elasticsearch
build:
context: kibana/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- type: bind
source: ./kibana/config/kibana.yml
target: /usr/share/kibana/config/kibana.yml
read_only: true
ports:
- "5601:5601"
networks:
- elk
volumes:
elasticsearch:
# driver: local
# driver_opts:
# type: none
# o: bind
# device: /home/elfk/elasticsearch/data
networks:
elk:
driver: bridge
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
为了方便，建议提前在test节点上拉取需要的镜像：
docker pull docker.elastic.co/elasticsearch/elasticsearch-oss:7.6.2
docker pull docker.elastic.co/kibana/kibana-oss:7.6.2
docker pull docker.elastic.co/logstash/logstash-oss:7.6.2
1
2
3
4
5
docker-compose up --build -d
docker-compose ps
Name Command State Ports
-------------------------------------------------------------------------------------------------------------------------------
elfk-docker_elasticsearch_1 /usr/local/bin/docker-entr ... Up 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp
elfk-docker_kibana_1 /usr/local/bin/dumb-init - ... Up 0.0.0.0:5601->5601/tcp
elfk-docker_logstash_1 /usr/local/bin/docker-entr ... Up 0.0.0.0:5040->5040/tcp, 5044/tcp, 0.0.0.0:9600->9600/tcp
1
2
3
4
5
6
7
8
9
访问kibana页面：ip:5601，
elk容器启动正常，kibana访问正常。接下来sidecar方式部署filebeat收集应用日志。
以sidecar方式部署filebeat收集nginx日志：
vim filebeat-nginx.yaml
1
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: default
labels:
app: nginx
spec:
selector:
app: nginx
ports:
- port: 80
nodePort: 30090
type: NodePort
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: default
labels:
app: filebeat
data:
filebeat.yml: |-
filebeat.config:
inputs:
path: ${path.config}/inputs.d/*.yml
reload.enabled: false
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
filebeat.inputs:
- type: log
paths:
- /logdata/access.log
tail_files: true
fields:
pod_name: '${pod_name}'
POD_IP: '${POD_IP}'
type: nginx_access
fields_under_root: true
multiline.pattern: '^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}'
multiline.negate: true
multiline.match: after
- type: log
paths:
- /logdata/error.log
tail_files: true
fields:
pod_name: '${pod_name}'
POD_IP: '${POD_IP}'
type: nginx_error
fields_under_root: true
multiline.pattern: '^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}'
multiline.negate: true
multiline.match: after
output.logstash:
hosts: ["192.168.30.133:5040"]
enabled: true
worker: 1
compression_level: 3
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: default
spec:
replicas: 1
minReadySeconds: 15
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat-oss:7.6.2
args: [
"-c", "/etc/filebeat/filebeat.yml",
"-e",
]
env:
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: pod_name
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 200m
memory: 200Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat/
- name: data
mountPath: /usr/share/filebeat/data
- name: logdata
mountPath: /logdata
- name: nginx
image: nginx:1.17.0
ports:
- containerPort: 80
volumeMounts:
- name: logdata
mountPath: /var/log/nginx
volumes:
- name: data
emptyDir: {}
- name: logdata
emptyDir: {}
- name: config
configMap:
name: filebeat-config
items:
- key: filebeat.yml
path: filebeat.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
kubectl apply -f filebeat-nginx.yaml
kubectl get pod |grep nginx
nginx-865f745bdd-mkdl6 2/2 Running 0 15s
kubectl describe pod nginx-865f745bdd-mkdl6
Normal Scheduled 36s default-scheduler Successfully assigned default/nginx-865f745bdd-mkdl6 to node2
Normal Pulled 34s kubelet, node2 Container image "docker.elastic.co/beats/filebeat-oss:7.6.2" already present on machine
Normal Created 34s kubelet, node2 Created container filebeat
Normal Started 34s kubelet, node2 Started container filebeat
Normal Pulled 34s kubelet, node2 Container image "nginx:1.17.0" already present on machine
Normal Created 34s kubelet, node2 Created container nginx
Normal Started 34s kubelet, node2 Started container nginx
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
访问nginx页面以产生日志：ip:30090，
kibana创建索引，查看nginx日志：
可以看到，已经有指定的index——nginx-access产生，创建同名索引查看日志，
nginx的访问日志通过logstash处理之后已产生对应的字段。访问不存在的url（如ip:30090/index.php）以产生错误的nginx日志，
到kibana页面查看是否有index产生，
可以看到，已经有指定的index——nginx-error产生，创建同名索引查看日志，
查看对应的nginx访问日志，
nginx日志收集完成。
以sidecar方式部署filebeat收集tomcat日志：
vim filebeat-tomcat.yaml
1
apiVersion: v1
kind: Service
metadata:
name: tomcat
namespace: default
labels:
app: tomcat
spec:
selector:
app: tomcat
ports:
- port: 8080
nodePort: 30100
type: NodePort
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config-tomcat
namespace: default
labels:
app: filebeat
data:
filebeat.yml: |-
filebeat.config:
inputs:
path: ${path.config}/inputs.d/*.yml
reload.enabled: false
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
filebeat.inputs:
- type: log
paths:
- /logdata/catalina*.log
tail_files: true
fields:
pod_name: '${pod_name}'
POD_IP: '${POD_IP}'
type: tomcat_catalina
fields_under_root: true
multiline.pattern: '^\d{1,2}-\S{3}-\d{4}\s\d{1,2}:\d{1,2}:\d{1,2}'
multiline.negate: true
multiline.match: after
output.logstash:
hosts: ["192.168.30.133:5040"]
enabled: true
worker: 1
compression_level: 3
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat
namespace: default
spec:
replicas: 1
minReadySeconds: 15
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
selector:
matchLabels:
app: tomcat
template:
metadata:
labels:
app: tomcat
spec:
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat-oss:7.6.2
args: [
"-c", "/etc/filebeat/filebeat.yml",
"-e",
]
env:
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: pod_name
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 200m
memory: 200Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat/
- name: data
mountPath: /usr/share/filebeat/data
- name: logdata
mountPath: /logdata
- name: tomcat
image: tomcat:8.0.51-alpine
ports:
- containerPort: 8080
volumeMounts:
- name: logdata
mountPath: /usr/local/tomcat/logs
volumes:
- name: data
emptyDir: {}
- name: logdata
emptyDir: {}
- name: config
configMap:
name: filebeat-config-tomcat
items:
- key: filebeat.yml
path: filebeat.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
kubectl apply -f filebeat-tomcat.yaml
kubectl get pod |grep tomcat
tomcat-5c7b6644f4-jwmxc 2/2 Running 0 10s
kubectl describe pod tomcat-5c7b6644f4-jwmxc
Normal Scheduled 35s default-scheduler Successfully assigned default/tomcat-5c7b6644f4-jwmxc to node1
Normal Pulled 34s kubelet, node1 Container image "docker.elastic.co/beats/filebeat-oss:7.6.2" already present on machine
Normal Created 34s kubelet, node1 Created container filebeat
Normal Started 34s kubelet, node1 Started container filebeat
Normal Pulled 34s kubelet, node1 Container image "tomcat:8.0.51-alpine" already present on machine
Normal Created 34s kubelet, node1 Created container tomcat
Normal Started 34s kubelet, node1 Started container tomcat
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
访问tomcat页面以产生日志：ip:30100，
kibana创建索引，查看tomcat日志：
可以看到，已经有指定的index——tomcat-catalina产生，创建同名索引查看日志，
tomcat的catalina日志通过logstash处理之后已产生对应的字段。tomcat日志收集完成。
总结：
通过sidecar方式部署filebeat收集k8s日志，比使用logagent方式部署filebeat收集k8s日志更加直接，可以指定具体路径并自定义index，在查看日志时也更加方便，缺点就是会增加资源消耗，不过资源消耗在可接受的范围内。
需要注意的是，filebeat对不同应用程序日志收集的ConfigMap名称尽量不要相同，避免冲突和误删除。
在前面，filebeat收集日志是直接输出到es中的，而且elk是部署在k8s集群中的。本文中filebeat收集日志是传输到logstash之后进行日志处理再输出到es中的，而且elk单独部署（这里为了方便，采用docker-compose部署，也可以传统方式部署）。
至此，elk + filebeat收集k8s日志完毕。个人感觉sidecar方式比logagent方式更为合适，在日志的过滤处理和查看方面更为简单有效。另外对于错误日志的告警，参考 ELK集群部署（三），此处不再赘述。
————————————————
版权声明：本文为CSDN博主「最爱喝酸奶」的原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接及本声明。
原文链接：https://blog.csdn.net/miss1181248983/article/details/106327696