静态搭建etcd集群(二进制安装)-526互联

软件	版本
etcd	v3.5.9
cfssl	v1.6.4

一、下载文件

通过githup官网https://github.com

wget https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl-certinfo_1.6.4_linux_amd64
或
curl -O https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz
curl -O https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64
curl -O https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64
curl -O https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl-certinfo_1.6.4_linux_amd64
可使用加速地址(下载githup上的文件可以通过在文件链接前面加https://ghproxy.com/加速)如：wget https://ghproxy.com/https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz

二、准备证书(cfssl)

1.重命名cfssl命令并授权

命令

mv cfssl_1.6.4_linux_amd64 cfssl
mv cfssljson_1.6.4_linux_amd64 cfssljson
mv cfssl-certinfo_1.6.4_linux_amd64 cfssl-certinfo
chmod +x cfssl*
mv cfssl* /usr/local/bin/

#查看版本
cfssl version

2.etcd证书准备

#可以通过命令获取默认配置进行修改也可以直接使用下面的json文件
cfssl print-defaults config > ca-config.json
cfssl print-defaults csr > ca-csr.json

ca-csr.json

cat > ca-csr.json << EOF
{
    "CN": "etcd",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C":  "CN",
            "L":  "ChenDu",
            "O":  "kubernetes",
            "OU": "sys",
            "ST": "ChenDu"
        }
    ],
   "ca": {
    	"expiry": "87600h"  
    }
}
EOF

ca-config.json

cat > ca-config.json <<EOF
{
    "signing": {
        "default": {
            "expiry": "8760h"
        },
        "profiles": {
            "etcd": {
                "expiry": "8760h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
		    "client auth"
                ]
            }
        }
    }
}
EOF

etcd-csr.json

#hosts为三台etcd的地址，可以按自己的进行修改
cat > etcd-csr.json << EOF
{
    "CN": "etcd",
    "hosts": [
        "192.168.2.11",
        "192.168.2.12",
        "192.168.2.13",
        "127.0.0.1"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C":  "CN",
            "L":  "ChenDu",
            "O":  "etcds",
            "OU": "sys",
            "ST": "ChenDu"
        }
    ]
}
EOF

#### 3、证书制作 ##### 3.1制作CA证书，会生成以下三个文件 `cfssl gencert -initca ca-csr.json | cfssljson -bare ca`

3.2使用ca证书签发etcd证书，也会生成三个文件，这样就制作完成了

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd etcd-csr.json | cfssljson -bare etcd

3.3验证证书与密匙是否匹配（可选）

openssl x509 -noout -modulus -in etcd.cpem | openssl md5

openssl rsa -noout -modulus -in etcd-key.pem | openssl md5

三、安装部署etcd集群（三台操作相同）

1.分发etcd可执行文件

tar -zxvf etcd-v3.5.9-linux-amd64.tar.gz
cd etcd-v3.5.9-linux-amd64
chmod +x etcd*
mv etcd* /usr/local/bin
scp etcd* root@192.168.2.12:/usr/local/bin
scp etcd* root@192.168.2.13:/usr/local/bin

2.添加配置文件

mkdir -p /etc/etcd/ssl
mkdir -p /var/lib/etcd/default.etcd
#移动证书到/etc/etcd/ssl
mv ca*.pem /etc/etcd/ssl
mv etcd*.pem /etc/etcd/ssl

编写etcd配置文件/etc/etcd/etcd.conf

##三台服务器ETCD_NAME分别为etcd1,etcd2,etcd3；ETCD_INITIAL_CLUSTER都一样，其余代IP改为所在服务器的ip
cat > /etc/etcd/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.2.11:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.2.11:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.2.11:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.2.11:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.2.11:2380,etcd2=https://192.168.2.12:2380,etcd3=https://192.168.2.13:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

3.配置systemd服务

编写systemd服务/etc/systemd/system/etcd.service

cat > /etc/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/etc/etcd/etcd.conf
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now etcd.service
systemctl status etcd

4、验证集群状态4、验证集群状态

ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.2.11:2379,https://192.168.2.12:2379,https://192.168.2.13:2379 endpoint health

四、问题及参考

启动报错可以使用以下命令查看日志进行排查：
journalctl -u etcd
tail -f /var/log/message

问题（使用的cfssl自签证书）：etcd failed to verify client certificate: x509: certificate specifies an incompatible key usage
解决：检查创建证书过程ca-csr.json,ca-config.json,etcd-csr.json重新创建证书（我是过程中一些名字填错了）
问题（前两台正常启动，第三台启动出现这两个错误）：
first record does not look like a TLS handshake
request sent was ignored by remote peer due to cluster ID mismatch

解决：检查配置文件/etc/etcd/etcd.conf中ETCD_INITIAL_CLUSTER集群配置，我是发现正常那两天把第三台设置为了http访问，结果第三台配置的https导致出错，修改后重启前两台后重启第三台（不行可以尝试删除三台的数据文件/var/lib/etcd/default.etcd后重启etcd）

参考文档：
etcd中文文档：http://www.zhaowenyu.com/etcd-doc/ops/etcd-ha-install.html
cfssl签发etcd证书：https://www.cnblogs.com/yangzp/p/15692046.html
验证ca证书与私钥是否匹配：https://www.jianshu.com/p/866ed37ae060

集群check-etcd execution时候

集群证书etcd ssl

集群实例docker etcd