etcd容量不足报错解决

发布时间 2023-08-04 08:23:50作者: 大顺儿

报错日志信息:“etcdserver:mvcc:database space exceded”

解决步骤:

需要用到etcdctl工具,一般二进制部署的k8s,都会带有这个工具,如果没有,可以去github下载指定版本的etcd二进制文件即可

1.查看节点状态获取db size

ETCDCTL_API=3 etcdctl \
--cacert=/etc/kubernetes/cert/ca.pem \
--cert=/etc/kubernetes/cert/etcd.pem \
--key=/etc/kubernetes/cert/etcd-key.pem \
--endpoints=https://172.31.243.179:2379 \
endpoint status --write-out="table"

在etcd启动的时候,如果没有配置--quota-backend-bytes的大小,默认只有2G,超过此大小将导致了apiserver无法写入etcd

pem认证文件路径根据实际环境做修改

2.获取旧版本号

ETCDCTL_API=3 etcdctl \
--cacert=/etc/kubernetes/cert/ca.pem \
--cert=/etc/kubernetes/cert/etcd.pem \
--key=/etc/kubernetes/cert/etcd-key.pem \
--endpoints=https://172.31.243.179:2379 \
endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9]*'

3.压缩旧版本

ETCDCTL_API=3 etcdctl \
--cacert=/etc/kubernetes/cert/ca.pem \
--cert=/etc/kubernetes/cert/etcd.pem \
--key=/etc/kubernetes/cert/etcd-key.pem \
--endpoints=https://172.31.243.179:2379 \
compact 15151255   #(由步骤2获得)

4.清理碎片

ETCDCTL_API=3 etcdctl \
--cacert=/etc/kubernetes/cert/ca.pem \
--cert=/etc/kubernetes/cert/etcd.pem \
--key=/etc/kubernetes/cert/etcd-key.pem \
--endpoints=https://172.31.243.179:2379 \
defrag

5.再次查看节点db size

ETCDCTL_API=3 etcdctl \
--cacert=/etc/kubernetes/cert/ca.pem \
--cert=/etc/kubernetes/cert/etcd.pem \
--key=/etc/kubernetes/cert/etcd-key.pem \
--endpoints=https://172.31.243.179:2379 \
endpoint status --write-out="table"

此时db size应变为1Mb

6.清除告警

ETCDCTL_API=3 etcdctl \ --cacert=/etc/kubernetes/cert/ca.pem \ --cert=/etc/kubernetes/cert/etcd.pem \ --key=/etc/kubernetes/cert/etcd-key.pem \ --endpoints=https://172.31.243.179:2379 \ alarm disarm

结束

参考博文:https://www.cnblogs.com/chen2ha/p/15914169.html