创建步骤：
1. 下载PKI证书管理工具
wget -O cfssl https://github.com/cloudflare/cfssl/releases/download/v1.6.3/cfssl_1.6.3_linux_amd64 && chmod +x cfssl && mv cfssl /usr/local/bin/ wget -O cfssljson https://github.com/cloudflare/cfssl/releases/download/v1.6.3/cfssljson_1.6.3_linux_amd64 && chmod +x cfssljson && mv cfssljson /usr/local/bin/
2. 下载etcd二进制安装包
cd /opt wget https://github.com/etcd-io/etcd/releases/download/v3.5.4/etcd-v3.5.4-linux-amd64.tar.gz tar -xvf etcd-v3.5.4-linux-amd64.tar.gz mv etcd-v3.5.4-linux-amd64 etcd-v3.5.4 rm -rf etcd-v3.5.4-linux-amd64.tar.gz
3. 配置etcd环境
编辑/etc/profile或者/root/.bash_profile，增加如下内容 ETCD_HOME=/opt/etcd-v3.5.4 PATH=$PATH:$ETCD_HOME export PATH source /etc/profile source /root/.bash_profile 创建etcd数据存储目录，这个可以自己定义，随意设定，但是一定要和etcd.conf配置文件里的一致，否则启动失败 mkdir /opt/etcd-v3.5.4/data
4. 创建CA根证书
创建ca-config.json文件
cfssl print-defaults config > ca-config.json
生成的文件如下，其中的www可以改成自己设定的名称，“expiry”默认是168h，需要改成87600h
特别提醒，生成的profile部分缺少“client auth”的内容，需要补充进去
{ "signing": { "default": { "expiry": "168h" # 修改有效期，最大是87600h }, "profiles": { "www": { # 可以自己更改名称
"expiry": "8760h", "usages": [ "signing", "key encipherment", "server auth"，
“client auth” # 手动补充进来
] }, "client": { "expiry": "8760h", "usages": [ "signing", "key encipherment", "client auth" ] } } } }
创建ca-csr.json文件，使用命令cfssl print-defaults csr > ca-csr.json
注意，生成的文件为模板，需要自己更改的内容在下面做了标注
[root@host106 ~]# cfssl print-defaults csr > ca-csr.json [root@host106 ~]# cat ca-csr.json { "CN": "example.net", # 改成自己定义的名称，比如kubernetes "hosts": [ # 根证书的这部分内容可以为空 "example.net", "www.example.net" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "US", # 申请单位国家简称，这里改成CN "ST": "CA", #申请单位所在省份，这里改成Beijing "L": "San Francisco"， # 申请单位所在城市， 这里改成Beijing
"O": "kubernetes" # 这一行是新增的，为申请单位组织名称
"OU": "system" # 这一行为新增，是申请单位部门
} ] }
接下来，使用cfssl命令结合cfssljson命令生成根证书,结果如下
[root@host106 ~]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca - 2023/05/26 15:30:44 [INFO] generating a new CA key and certificate from CSR 2023/05/26 15:30:44 [INFO] generate received request 2023/05/26 15:30:44 [INFO] received CSR 2023/05/26 15:30:44 [INFO] generating key: ecdsa-256 2023/05/26 15:30:44 [INFO] encoded CSR 2023/05/26 15:30:44 [INFO] signed certificate with serial number 48377921078874758075973025036457528910908411835
结果生成了三个文件具体如下：
[root@host106 ~]# ls ca* ca.csr ca-key.pem ca.pem
5. 为etcd制作证书，其实其他证书都基本雷同，不再赘述
首先为etcd生成csr文件，命令为cfssl print-defaults csr > etcd-csr.json ,生成的内容和上面的ca-csr.json一样，我把修改后的贴出来，以供参考
[root@host106 cert]# cat etcd-csr.json { "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.0.106", "192.168.0.105", "192.168.0.189", "192.168.0.177" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "CN", "ST": "Hubei", "L": "Wuhan", "O": "kubernetes", "OU": "system" } ] }
这个步骤中 hosts 列表中的内容非常关键，不在此范围内的节点访问此证书将会被拒绝。所以，请将 kubernetes集群中的节点IP加入进来，或对应的域名加入进来，hosts 列表中的值支持泛域名。
接下来为etcd生成证书，里面需要用到ca的根证书，ca的key文件，ca的config文件，profile选项就是ca-config.json文件里的profile下面第一行的www，以及etcd的csr文件，命令如下：
[root@host106 ~]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www etcd-csr.json |cfssljson -bare etcd 2023/05/26 15:41:31 [INFO] generate received request 2023/05/26 15:41:31 [INFO] received CSR 2023/05/26 15:41:31 [INFO] generating key: ecdsa-256 2023/05/26 15:41:31 [INFO] encoded CSR 2023/05/26 15:41:31 [INFO] signed certificate with serial number 376033176847126118100694928117307248378090625546
这个命令生成的文件如下：
[root@host106 ~]# ls etcd* etcd.csr etcd-key.pem etcd.pem
6. 配置etcd的配置文件
注意 ETCD_NAME 变量为节点的名称，不同的节点请给予不同的名称，
在 ETCD_INITIAL_CLUSTER节点中引用了所有节点的地址信息。两者要对应上。
其中涉及到IP地址要根据实际节点的IP地址进行调整，如上边配置中的 192.168.0.106 要替换成实际节点的IP地址
ETCD_CLIENT_CERT_AUTH可以设置为false
注意，其中的key文件所在路径要用自己的，名称也要和实际的相符
cat >/opt/etcd-v3.5.4/etcd.conf <<EOF #[Member] ETCD_NAME="etcd2" ETCD_DATA_DIR="/opt/etcd-v3.5.4/data" ETCD_LISTEN_PEER_URLS="https://192.168.0.106:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.106:2379,https://127.0.0.1:2379" ETCD_CERT_FILE=/opt/cert/etcd.pem ETCD_KEY_FILE=/opt/cert/etcd-key.pem ETCD_TRUSTED_CA_FILE=/opt/cert/ca.pem #ETCD_CLIENT_CERT_AUTH=true ETCD_PEER_CERT_FILE=/opt/cert/etcd.pem ETCD_PEER_KEY_FILE=/opt/cert/etcd-key.pem ETCD_PEER_TRUSTED_CA_FILE=/opt/cert/ca.pem #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.106:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.106:2379" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.0.105:2380,etcd2=https://192.168.0.106:2380,etcd3=https://192.168.0.189:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" EOF
7. 配置etcd的系统启动项
cat > /usr/lib/systemd/system/etcd.service <<EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target Documentation=https://github.com/coreos [Service] Type=notify WorkingDirectory=/opt/etcd-v3.5.4/data EnvironmentFile=/opt/etcd-v3.5.4/etcd.conf ExecStart=/opt/etcd-v3.5.4/etcd \\ --auto-compaction-mode=periodic \\ --auto-compaction-retention=1 \\ --max-request-bytes=33554432 \\ --quota-backend-bytes=6442450944 \\ --heartbeat-interval=250 \\ --election-timeout=2000 Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
设置允许etcd开机启动
systemctl enable etcd
systemctl start etcd
查看etcd启动状态
systemctl status etcd
[root@host106 etcd-v3.5.4]# systemctl status etcd ● etcd.service - Etcd Server Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled) Active: active (running) since 五 2023-05-26 14:53:14 CST; 59min ago Docs: https://github.com/coreos Main PID: 25104 (etcd) Tasks: 10 Memory: 22.0M CGroup: /system.slice/etcd.service └─25104 /opt/etcd-v3.5.4/etcd --auto-compaction-mode=periodic --auto-compaction-retention=1 --max-request-bytes=33554432 --quota-backend-bytes=6442450944 --heartbeat-inter... 5月 26 14:53:27 host106 etcd[25104]: {"level":"warn","ts":"2023-05-26T14:53:27.506+0800","caller":"rafthttp/peer_status.go:66","msg":"peer became inactive (message send to peer fail... 5月 26 14:53:28 host106 etcd[25104]: {"level":"warn","ts":"2023-05-26T14:53:28.175+0800","caller":"rafthttp/stream.go:223","msg":"lost TCP streaming connection with remo...37db5e0599"} 5月 26 14:53:28 host106 etcd[25104]: {"level":"info","ts":"2023-05-26T14:53:28.650+0800","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"ce9e5937db5e0599"} 5月 26 14:53:28 host106 etcd[25104]: {"level":"info","ts":"2023-05-26T14:53:28.650+0800","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection wi...37db5e0599"} 5月 26 14:53:28 host106 etcd[25104]: {"level":"info","ts":"2023-05-26T14:53:28.651+0800","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection wi...37db5e0599"} 5月 26 14:53:28 host106 etcd[25104]: {"level":"info","ts":"2023-05-26T14:53:28.656+0800","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"eb83c838a5...am Message"} 5月 26 14:53:28 host106 etcd[25104]: {"level":"info","ts":"2023-05-26T14:53:28.656+0800","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection wi...37db5e0599"} 5月 26 14:53:28 host106 etcd[25104]: {"level":"info","ts":"2023-05-26T14:53:28.658+0800","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"eb83c838a5... MsgApp v2"} 5月 26 14:53:28 host106 etcd[25104]: {"level":"warn","ts":"2023-05-26T14:53:28.658+0800","caller":"rafthttp/stream.go:265","msg":"closed TCP streaming connection with re...37db5e0599"} 5月 26 14:53:28 host106 etcd[25104]: {"level":"info","ts":"2023-05-26T14:53:28.658+0800","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection wi...37db5e0599"} Hint: Some lines were ellipsized, use -l to show in full.
8. etcd健康行检查
[root@host106 ~]# ETCDCTL_API=3 etcdctl --cacert=/opt/cert//etcd.pem --cert=/opt/cert//etcd.pem --key=/opt/cert//etcd-key.pem --endpoints="https://192.168.0.105:2379,https://192.168.0.106:2379,https://192.168.0.189:2379" endpoint status --write-out='table' +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://192.168.0.105:2379 | a1521095cffde44b | 3.5.4 | 20 kB | false | false | 5 | 16 | 16 | | | https://192.168.0.106:2379 | eb83c838a536671f | 3.5.4 | 20 kB | true | false | 5 | 16 | 16 | | | https://192.168.0.189:2379 | ce9e5937db5e0599 | 3.5.4 | 20 kB | false | false | 5 | 16 | 16 | | +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
证书问题：
"server-name":"","error":"tls: failed to verify client certificate: x509: certificate specifies an incompatible key usage"}
May 26 14:17:46 host105 etcd: {"level":"warn","ts":"2023-05-26T14:17:46.698+0800","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.0.106:51385","server-name":"","error":"tls: failed to verify client certificate: x509: certificate specifies an incompatible key usage"}
May 26 14:17:46 host105 etcd: {"level":"warn","ts":"2023-05-26T14:17:46.758+0800","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"192.168.0.189:41778","server-name":"","error":"tls: failed to verify client certificate: x509: certificate specifies an incompatible key usage"}
处理措施：
经过研究发现是因为配置文件ca-config.json 的profiles缺少了“client auth”，添加之后，再执行初始化解决了问题
1 [root@host106 cert]# cat ca-config.json 2 { 3 "signing": { 4 "default": { 5 "expiry": "168h" 6 }, 7 "profiles": { 8 "kubernetes": { 9 "expiry": "8760h", 10 "usages": [ 11 "signing", 12 "key encipherment", 13 "server auth", 14 "client auth" 15 ] 16 }, 17 "client": { 18 "expiry": "8760h", 19 "usages": [ 20 "signing", 21 "key encipherment", 22 "client auth" 23 ] 24 } 25 } 26 } 27 }