Kubernetes v1.28安装手册 - 基于Ubuntu22.04

2023年12月17日，目前k8s社区的kubernetes更新到了1.29.0版，但国内阿里云等镜像服务器，k8s软件还在1.28.2版本，k8s组件的镜像为1.28.4版。

基于此，本次使用Ubuntu22.04.3+k8s1.28.2版本，在虚拟机上搭建一个3master、3worker的k8s集群。

一、规划

1 节点规划

6个节点：3个master，3个worker，基础OS为Ubuntu22.04.3，版本代号jammy，内核5.15.0-91-generic。

6个节点的统一域名为beety.com，并为apiserver设定一个统一VIP。

角色	IP	主机名	主机长名	功能	配置
VIP	192.168.172.140	kubeapi	kubeapi.beety.com	apiserver的统一入口	N/A
master01	192.168.172.141	m01	m01.beety.com	kubelet、etcd、apiserver、controller-manager、scheduler	2vc4G40G
master02	192.168.172.142	m02	m02.beety.com	kubelet、etcd、apiserver、controller-manager、scheduler	2vc4G40G
master03	192.168.172.143	m03	m03.beety.com	kubelet、etcd、apiserver、controller-manager、scheduler	2vc4G40G
worker01	192.168.172.151	w01	w01.beety.com	kubelet、proxy、应用pod	4vc4G40G
worker02	192.168.172.152	w02	w02.beety.com	kubelet、proxy、应用pod	4vc4G40G
worker03	192.168.172.153	w03	w03.beety.com	kubelet、proxy、应用pod	4vc4G40G

2 软件版本规划

使用docker 24.0.7，containerd 1.6.26, kubernetes 1.28.2，flannel cniVersion 0.3.1

软件	版本	安装性质	获取渠道	贡献者
docker-ce	24.0.7	dub	阿里云	docker
docker-ce-cli	24.0.7	dub	阿里云	docker
containerd.io	1.6.26	dub	阿里云	docker
kubelet	1.28.2	dub	阿里云	kubernetes
kubeadm	1.28.2	dub	阿里云	kubernetes
kubectl	1.28.2	dub	阿里云	kubernetes
kube-apiserver	1.28.2	pod	阿里云	kubernetes
kube-controller-manager	1.28.2	pod	阿里云	kubernetes
kube-scheduler	1.28.2	pod	阿里云	kubernetes
kube-proxy	1.28.2	pod	阿里云	kubernetes
pause	3.9	pod	阿里云	kubernetes
coredns	1.10.1	pod	阿里云	coredns
etcd	3.5.9-0	pod	阿里云	coreos
flannel	cniVersion 0.3.1	pod	github	coreos

3 安装规划

（1）容器选择

每台上都安装docker-ce、docker-ce-cli、containerd.io，使用Containerd作为容器运行时，和kubelet交互。

Cgroup管理器，k8s默认是systemd，需要将Containerd的Cgroup管理器也修改为systemd（默认是cgroupfs）。

（2）CNI网络插件

使用谷歌嫡系，CoreOS提供的flannel作为网络插件。

（3）安装步骤

所有节点都安装docker-ce、docker-ce-cli、containerd.io软件包，启动docker.socket、docker.service、containerd.service服务。

所有节点都安装kubelet、kubeadm、kubectl软件包，都启动kubelet.service服务。

m01节点使用kubeadm创建集群，且m01节点作为master节点；集群创建后马上安装flannel建立私网。

集群创建后，将w01-w03节点加入集群作为worker节点。

运行一个比helloworld稍微复杂一些的demo，验证应用pod可以启动并提供服务。

m02、m03节点作为备用master节点最后再加入集群。

二、操作系统准备工作

准备工作需要在所有节点上进行。

1 ubuntu前置检查

（1）ubuntu默认的ufw防火前已关

systemctl status ufw
###正确回显
○ ufw.service - Uncomplicated firewall
     Loaded: loaded (/lib/systemd/system/ufw.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:ufw(8)

该服务状态为inactive(dead)，表明防火墙已经关闭。

（2）时钟同步有效

timedatectl status
###正确回显
Local time: Sun 2023-12-17 11:25:08 CST
Universal time: Sun 2023-12-17 03:25:08 UTC
RTC time: Sun 2023-12-17 03:25:08
Time zone: Asia/Shanghai (CST, +0800)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no

可以看到System clock synchronized为yes，时区也已经配置为东八区，使用的是本地CST时间

（3）APT主源完成配置

apt update
###正确回显
......
Hit:3 http://mirrors.tuna.tsinghua.edu.cn/ubuntu jammy InRelease           
Get:4 http://mirrors.tuna.tsinghua.edu.cn/ubuntu jammy-updates InRelease [119 kB]                                                         
Get:5 http://mirrors.tuna.tsinghua.edu.cn/ubuntu jammy-updates/main amd64 Packages [1263 kB]
Hit:6 http://mirrors.tuna.tsinghua.edu.cn/ubuntu jammy-backports InRelease
Get:7 http://mirrors.tuna.tsinghua.edu.cn/ubuntu jammy-security InRelease [110 kB]
Fetched 1501 kB in 17s (86.8 kB/s) 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
......

表明可以成功地从国内镜像源获取主源。

2 关闭swap

k8s在创建集群的时候，默认要求关闭swap以保障性能，这里通过swapoff和注释fstab的方式，关闭swap。

swapoff -a
vim /etc/fstab
###将swap一行做注释，并保存退出

3 配置docker和k8s的APT源

通过add-apt-repostory命令，添加阿里云的apt docker源、apt k8s源

sudo curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"

sudo curl -fsSL https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -
sudo add-apt-repository "deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main"

4 绑定hosts表

按照上面的节点规划，全部6个节点，都绑定同样的hosts表。注意，因为安装规划中，暂不启用m02、m03节点的master功能，因此暂时将VIP的虚拟主机名绑到m01节点上。

192.168.172.141 m01.beety.com m01 kubeapi.beety.com kubeapi
192.168.172.142 m02.beety.com m02
192.168.172.143 m03.beety.com m03
192.168.172.151 w01.beety.com w01
192.168.172.152 w02.beety.com w02
192.168.172.153 w03.beety.com w03

三、安装docker

安装docker需要在所有节点上进行。

1 安装docker软件

使用apt安装docker。

apt install docker-ce docker-ce-cli containerd.io

等待dub包下载并完成安装。

使用apt可以查看安装的docker三个软件及关联软件。

apt list --installed | grep -i -E 'docker|containerd'
###已安装：
containerd.io/jammy,now 1.6.26-1 amd64 [installed]
docker-buildx-plugin/jammy,now 0.11.2-1~ubuntu.22.04~jammy amd64 [installed,automatic]
docker-ce-cli/jammy,now 5:24.0.7-1~ubuntu.22.04~jammy amd64 [installed]
docker-ce-rootless-extras/jammy,now 5:24.0.7-1~ubuntu.22.04~jammy amd64 [installed,automatic]
docker-ce/jammy,now 5:24.0.7-1~ubuntu.22.04~jammy amd64 [installed]
docker-compose-plugin/jammy,now 2.21.0-1~ubuntu.22.04~jammy amd64 [installed,automatic]

2 启动docker相关服务

ubuntu上的dub安装后，如果有服务，会被自动设置为开机自启动，且装完就会拉起，这里给出验证。

systemctl list-unit-files | grep -E 'docker|containerd'
###已经设置为开机自启动：
containerd.service                                    enabled         enabled
docker.service                                        enabled         enabled
docker.socket                                         enabled         enabled

systemctl status containerd.service
systemctl status docker.service
systemctl status docker.socket
###三个服务都应是running状态

运行docker version，查看并验证docker的版本。

docker version
###回显：
Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:07:41 2023
 OS/Arch:           linux/amd64
 Context:           default
Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:07:41 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.26
  GitCommit:        3dd1e886e55dd695541fdcd67420c2888645a495
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

3 配置containerd

从k8s 1.24版本开始，负责kubelet与docker直接握手的docker-shim已经不再支持，转而强制使用容器运行时Container Runtime作为kubelet和容器的桥梁。

目前容器运行时有两种选择，一种是与docker有强关联的cri-dockerd，一种是Containerd，本安装手册选用Containerd，在前面安装了对应的containerd.io包。

为了让Containerd能够在后面与kubelet成功协同，这里需要做几点优化：

（1）Containerd安装完成后，其自带的配置文件/etc/containerd/config.toml中的内容，需要用打印出的containerd默认配置替换。

（2）Containerd的Cgroup设为systemd，以和k8s默认的Cgroup保持一致。

（3）pause镜像路径改为国内源registry.aliyuncs.com/google_containers/pause:3.9。

cp /etc/containerd/config.toml /etc/containerd/config.toml.ori
containerd config default > /etc/containerd/config.toml

vim /etc/containerd/config.toml
约125行，[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]段落
默认：
SystemdCgroup = false
改为：
SystemdCgroup = true

约61行，[plugins."io.containerd.grpc.v1.cri"]段落
默认：
sandbox_image = "registry.k8s.io/pause:3.6"
改为：
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"

配置后，重启containerd服务，并保证containerd状态正确

systemctl restart containerd.service
systemctl status containerd.service

四、安装Kubernetes

安装k8s需要在所有节点上进行。

1 安装kubernetes软件

apt install kubelet kubeadm kubectl
###如果需要指定安装1.28.2这个版本，则可以这样：
apt install kubelet=1.28.2-00 kubeadm=1.28.2-00 kubectl=1.28.2-00

2 确认kubelet服务状态

同docker一样，kubelet安装后，服务会自动配置为开机启动，且服务已经启动

systemctl status kubelet
###但此时该服务状态为
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since Sun 2023-12-17 23:57:32 CST; 3s ago
       Docs: https://kubernetes.io/docs/home/
    Process: 1453 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/F>
   Main PID: 1453 (code=exited, status=1/FAILURE)
        CPU: 45ms
Dec 17 23:57:32 m01 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Dec 17 23:57:32 m01 systemd[1]: kubelet.service: Failed with result 'exit-code'.

这里没有启动成功是正常的，因为kubelet服务成功启动的先决条件，需要kubelet的配置文件，所在目录/var/lib/kubelet还没有建立。

可以用下面命令看日志，追踪到该临时问题。

journalctl -xeu kubelet
###有这么一句
"command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/

kubelet的正常启动，要等到下一步，m01节点做kubeadm的初始化后，才会正常。

五、k8s master建立单机集群

建立单机集群，只在m01节点上操作。

1 拉取kubernetes镜像

可以通过下面的命令看到kubeadm默认配置的kubernetes镜像，是外网的镜像

kubeadm config images list
###回显
I1218 00:12:18.076424    2047 version.go:256] remote version is much newer: v1.29.0; falling back to: stable-1.28
registry.k8s.io/kube-apiserver:v1.28.4
registry.k8s.io/kube-controller-manager:v1.28.4
registry.k8s.io/kube-scheduler:v1.28.4
registry.k8s.io/kube-proxy:v1.28.4
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.9-0
registry.k8s.io/coredns/coredns:v1.10.1

可以看到，在kubeadm版本为1.28.2时，默认列出的外网官方镜像会提示最新已到1.29.0，并给出了1.28最新镜像版本为1.28.4。

使用阿里的kubernetes镜像源，再次尝试

kubeadm config images list --image-repository registry.aliyuncs.com/google_containers
###回显
I1218 00:17:37.569367    2261 version.go:256] remote version is much newer: v1.29.0; falling back to: stable-1.28
registry.aliyuncs.com/google_containers/kube-apiserver:v1.28.4
registry.aliyuncs.com/google_containers/kube-controller-manager:v1.28.4
registry.aliyuncs.com/google_containers/kube-scheduler:v1.28.4
registry.aliyuncs.com/google_containers/kube-proxy:v1.28.4
registry.aliyuncs.com/google_containers/pause:3.9
registry.aliyuncs.com/google_containers/etcd:3.5.9-0
registry.aliyuncs.com/google_containers/coredns:v1.10.1

可以看到阿里云给出的也是1.28最新镜像版本为1.28.4。

将kubernetes的控制面的几个镜像拉到本地，为了保证镜像和安装的k8s软件版本严格一致，这里的镜像拉取时，显性指定版本为1.28.2。

kubeadm config images pull --kubernetes-version v1.28.2 --image-repository registry.aliyuncs.com/google_containers
###耐心等待拉取完成
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.28.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.28.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.28.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.28.2
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.9
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.9-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.10.1

在指定版本后，kube的几个镜像都是1.28.2版本，和安装的k8s软件严格对应。这些镜像会占据/var下约910MB空间。

2 kubernetes初始化

这是整个配置k8s集群中，最重要的一步，这步将完成单机集群的组建，之后的其他worker节点和master节点，只需要join进该集群即可。

这里使用kubeadm init加手工添加参数的方式做初始化

kubeadm init \
  --control-plane-endpoint="kubeapi.beety.com" \
  --image-repository registry.aliyuncs.com/google_containers \
  --kubernetes-version v1.28.2 \
  --service-cidr=10.96.0.0/12 \
  --pod-network-cidr=10.244.0.0/16 \
  --ignore-preflight-errors=all
### –-apiserver-advertise-address # 集群通告地址，单master时为控制面使用的m01的服务器IP
### --control-plane-endpoint="kubeapi.magedu.com" #和上面的参数二选一，如果用这个，则配置kubeapi这个规划好的多master虚拟主机长名
### –-image-repository #由于默认拉取镜像地址k8s.gcr.io国内无法访问，这里指定阿里云镜像仓库地址
### –-kubernetes-version #K8s版本，与上面安装的k8s软件一致
### –-service-cidr #集群内部虚拟网络，Pod统一访问入口，可以不用更改，直接用上面的参数
### –-pod-network-cidr #Pod网络，与下面部署的CNI网络组件yaml中保持一致，可以不用更改，直接用上面的参数

初始化的内容，可以参考附件3。初始化后，kubeadm会提示后续的工作，首先是在m01节点设置配置文件并加进环境变量。

vim /root/.bashrc
###追加以下内容：
# 超级用户变量
export KUBECONFIG=/etc/kubernetes/admin.conf
# （可选）设置别名
alias k=kubectl
# （可选）设置kubectl命令补齐功能
source <(kubectl completion bash)
###追加后保存退出，并生效环境变量
source /root/.bashrc

kubeadm还提示后续需要建立kubelet集群网络，并将节点加入，下面开始转入网络配置、组网组群。

因此每次初始化完成后，一定要把回显内容记下来，里面后续加入worker和其他master时候，需要用到的token信息。

查看pod信息，这是已经有了master节点必须的k8s级pod了

kubectl get pod --all-namespaces
###回显
NAMESPACE     NAME                          READY   STATUS    RESTARTS   AGE
kube-system   coredns-66f779496c-4r9x4      0/1     Pending   0          9m10s
kube-system   coredns-66f779496c-9z6l6      0/1     Pending   0          9m10s
kube-system   etcd-m01                      1/1     Running   0          9m24s
kube-system   kube-apiserver-m01            1/1     Running   0          9m24s
kube-system   kube-controller-manager-m01   1/1     Running   0          9m24s
kube-system   kube-proxy-g485x              1/1     Running   0          9m10s
kube-system   kube-scheduler-m01            1/1     Running   0          9m24s

这里两个coredns的状态暂时是Pending，这是因为CNI网络插件还没有安装。

3 安装CNI网络插件

CNI网络插件有多种，flannel、calico、canal和kube-router等，这里使用flannel。

从github上下载kube-flannel.yml配置文件到服务器，这一步也可以用URL在github上直接下载后上传到服务器任意目录

wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
###如果github无法访问，也可以尝试这个站点：
wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

检查kube-flannel.yml中的net-conf.json段落，使之与kubernetes初始化时，指定的pod-network-cidr参数中的网段一致。

如果不一致则应作出对应的修改。

net-conf.json: |
  {
    "Network": "10.244.0.0/16",
    "Backend": {
      "Type": "vxlan"
  }
}

安装：

kubectl apply -f kube-flannel.yml
###回显：
namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
###回显完

4 查看集群网络状态

（1）系统级pod状态

再次查看k8s级pod的状态

kubectl get pod --all-namespaces
###回显
NAMESPACE      NAME                            READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-2wds4           1/1     Running   0          92s
kube-system    coredns-6d8c4cb4d-7qxln         1/1     Running   0          5d17h
kube-system    coredns-6d8c4cb4d-c8k2r         1/1     Running   0          5d17h
kube-system    etcd-node1                      1/1     Running   0          5d17h
kube-system    kube-apiserver-node1            1/1     Running   0          5d17h
kube-system    kube-controller-manager-node1   1/1     Running   0          5d17h
kube-system    kube-proxy-5psbz                1/1     Running   0          5d17h
kube-system    kube-scheduler-node1            1/1     Running   0          5d17h

flannel加入后，注意所有的系统级pod都已经是running状态，包括之前pending的两个coredns的pod。

（2）指定namespace，查看pod状态

kubectl get pod -n kube-system
###回显
NAME                          READY   STATUS    RESTARTS   AGE
coredns-66f779496c-4r9x4      1/1     Running   0          14m
coredns-66f779496c-9z6l6      1/1     Running   0          14m
etcd-m01                      1/1     Running   0          14m
kube-apiserver-m01            1/1     Running   0          14m
kube-controller-manager-m01   1/1     Running   0          14m
kube-proxy-g485x              1/1     Running   0          14m
kube-scheduler-m01            1/1     Running   0          14m

（3）查看主节点状态

kubectl get cs
###回显
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
controller-manager   Healthy   ok        
etcd-0               Healthy   ok        
scheduler            Healthy   ok

注意主节点应该不存在任何ERROR，组件状态都应为Healthy。

（4）查看节点列表

这时还只有主节点

kubectl get node
###回显
NAME   STATUS   ROLES           AGE   VERSION
m01    Ready    control-plane   16m   v1.28.2

六、其他节点加入集群

1 m01节点创建永久token

m01节点在初始化结束后，已经创建了临时token，但该临时token只有24小时有效期。因此这里需要重新在1节点创建永久有效的token。

m01节点上创建token：

kubeadm token create --print-join-command
###回显
kubeadm join kubeapi.beety.com:6443 --token xpgmbd.fes7zfcz66kpftp9 --discovery-token-ca-cert-hash sha256:80943437321a578c95a90bc2dae6267f4955b1d09fdf0b8c4b76e938993a780e

创建后，回显会非常友好地给出需要在其他节点上操作的命令。

2 worker节点加入

把w01-03节点加入kubernetes集群。原理为在worker节点上，使用kubeadm join命令，与m01节点的api server组件（6443端口）建立联系。

上面m01节点回显的命令，在w01-03节点执行，保证都可以成功加入集群。

kubeadm join kubeapi.beety.com:6443 --token xpgmbd.fes7zfcz66kpftp9 --discovery-token-ca-cert-hash sha256:80943437321a578c95a90bc2dae6267f4955b1d09fdf0b8c4b76e938993a780e
###某个节点的回显
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

查看节点列表

这时m01和w01-03，共计4个节点都已经加入

kubectl get node
###回显
NAME   STATUS     ROLES           AGE     VERSION
m01    Ready      control-plane   37m     v1.28.2
w01    Ready      <none>          4m33s   v1.28.2
w02    Ready      <none>          3m25s   v1.28.2
w03    NotReady   <none>          38s     v1.28.2
###回显完

最后加入的w03节点，暂时的状态为NotReady，需要耐心等待2-3分钟，之后就为Ready了。

一个正常的集群，master和worker的状态都应该为Ready，VERSION应该一致。

3 其他master节点加入

其他master节点加入，和worker节点的加入类似，只是需要在命令中增加--control-plane参数，表示该节点是作为master加入的。

kubeadm join kubeapi.beety.com:6443 --token xpgmbd.fes7zfcz66kpftp9 --discovery-token-ca-cert-hash sha256:80943437321a578c95a90bc2dae6267f4955b1d09fdf0b8c4b76e938993a780e ----control-plane

七、应用demo

（正文完）

附件和参考

附件1 kubernetes证书有效期

用kubeadm的certs子命令查看所有证书的有效期，其中有10年有效期的，多数为1年。

实际上该命令显示 /etc/kubernetes/pki 文件夹中的客户端证书以及 kubeadm(admin.conf、controller-manager.conf 和 scheduler.conf) 使用的 KUBECONFIG 文件中嵌入的客户端证书的到期时间/剩余时间。

因此，有频率地查看证书有效期，并定期更新证书，是维护k8s集群的一项必备工作。此外，每次升级master上的Control Plane时，k8s会自动更新证书。

kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Dec 16, 2024 16:42 UTC   364d            ca                      no      
apiserver                  Dec 16, 2024 16:42 UTC   364d            ca                      no      
apiserver-etcd-client      Dec 16, 2024 16:42 UTC   364d            etcd-ca                 no      
apiserver-kubelet-client   Dec 16, 2024 16:42 UTC   364d            ca                      no      
controller-manager.conf    Dec 16, 2024 16:42 UTC   364d            ca                      no      
etcd-healthcheck-client    Dec 16, 2024 16:42 UTC   364d            etcd-ca                 no      
etcd-peer                  Dec 16, 2024 16:42 UTC   364d            etcd-ca                 no      
etcd-server                Dec 16, 2024 16:42 UTC   364d            etcd-ca                 no      
front-proxy-client         Dec 16, 2024 16:42 UTC   364d            front-proxy-ca          no      
scheduler.conf             Dec 16, 2024 16:42 UTC   364d            ca                      no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Dec 14, 2033 16:42 UTC   9y              no      
etcd-ca                 Dec 14, 2033 16:42 UTC   9y              no      
front-proxy-ca          Dec 14, 2033 16:42 UTC   9y              no

手动更新全部证书：

kubeadm certs renew all

附件2 k8s软件apt安装正确的回显

root@w03:/etc/containerd# apt install kubelet kubeadm kubectl
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  conntrack cri-tools ebtables kubernetes-cni socat
The following NEW packages will be installed:
  conntrack cri-tools ebtables kubeadm kubectl kubelet kubernetes-cni socat
0 upgraded, 8 newly installed, 0 to remove and 43 not upgraded.
Need to get 87.1 MB of archives.
After this operation, 336 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:2 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 cri-tools amd64 1.26.0-00 [18.9 MB]
Get:1 http://mirrors.tuna.tsinghua.edu.cn/ubuntu jammy/main amd64 conntrack amd64 1:1.4.6-2build2 [33.5 kB]
Get:3 http://mirrors.tuna.tsinghua.edu.cn/ubuntu jammy/main amd64 ebtables amd64 2.0.11-4build2 [84.9 kB]
Get:4 http://mirrors.tuna.tsinghua.edu.cn/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB]
Get:5 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 1.2.0-00 [27.6 MB]                   
Get:6 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubelet amd64 1.28.2-00 [19.5 MB]                           
Get:7 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubectl amd64 1.28.2-00 [10.3 MB]                           
Get:8 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubeadm amd64 1.28.2-00 [10.3 MB]                           
Fetched 87.1 MB in 57s (1540 kB/s)                                                                                                       
Selecting previously unselected package conntrack.
(Reading database ... 74637 files and directories currently installed.)
Preparing to unpack .../0-conntrack_1%3a1.4.6-2build2_amd64.deb ...
Unpacking conntrack (1:1.4.6-2build2) ...
Selecting previously unselected package cri-tools.
Preparing to unpack .../1-cri-tools_1.26.0-00_amd64.deb ...
Unpacking cri-tools (1.26.0-00) ...
Selecting previously unselected package ebtables.
Preparing to unpack .../2-ebtables_2.0.11-4build2_amd64.deb ...
Unpacking ebtables (2.0.11-4build2) ...
Selecting previously unselected package kubernetes-cni.
Preparing to unpack .../3-kubernetes-cni_1.2.0-00_amd64.deb ...
Unpacking kubernetes-cni (1.2.0-00) ...
Selecting previously unselected package socat.
Preparing to unpack .../4-socat_1.7.4.1-3ubuntu4_amd64.deb ...
Unpacking socat (1.7.4.1-3ubuntu4) ...
Selecting previously unselected package kubelet.
Preparing to unpack .../5-kubelet_1.28.2-00_amd64.deb ...
Unpacking kubelet (1.28.2-00) ...
Selecting previously unselected package kubectl.
Preparing to unpack .../6-kubectl_1.28.2-00_amd64.deb ...
Unpacking kubectl (1.28.2-00) ...
Selecting previously unselected package kubeadm.
Preparing to unpack .../7-kubeadm_1.28.2-00_amd64.deb ...
Unpacking kubeadm (1.28.2-00) ...
Setting up conntrack (1:1.4.6-2build2) ...
Setting up kubectl (1.28.2-00) ...
Setting up ebtables (2.0.11-4build2) ...
Setting up socat (1.7.4.1-3ubuntu4) ...
Setting up cri-tools (1.26.0-00) ...
Setting up kubernetes-cni (1.2.0-00) ...
Setting up kubelet (1.28.2-00) ...
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /lib/systemd/system/kubelet.service.
Setting up kubeadm (1.28.2-00) ...
Processing triggers for man-db (2.10.2-1) ...
Scanning processes...                                                                                                                     
Scanning linux images...                                                                                                                 
Running kernel seems to be up-to-date.
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
No VM guests are running outdated hypervisor (qemu) binaries on this host.

附件3 kubeadm初始化的全部显示结果

这里给出使用kubeadm进行初始化时，成功后的所有回显，作为参考。

root@m01:~# kubeadm init \
>   --control-plane-endpoint="kubeapi.beety.com" \
>   --image-repository registry.aliyuncs.com/google_containers \
>   --kubernetes-version v1.28.2 \
>   --service-cidr=10.96.0.0/12 \
>   --pod-network-cidr=10.244.0.0/16 \
>   --ignore-preflight-errors=all
[init] Using Kubernetes version: v1.28.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubeapi.beety.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local m01] and IPs [10.96.0.1 192.168.172.141]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost m01] and IPs [192.168.172.141 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost m01] and IPs [192.168.172.141 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 4.003652 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node m01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node m01 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: avjbin.trcl3dwub19jjcau
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
  export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
  kubeadm join kubeapi.beety.com:6443 --token avjbin.trcl3dwub19jjcau \
        --discovery-token-ca-cert-hash sha256:80943437321a578c95a90bc2dae6267f4955b1d09fdf0b8c4b76e938993a780e \
        --control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join kubeapi.beety.com:6443 --token avjbin.trcl3dwub19jjcau \
        --discovery-token-ca-cert-hash sha256:80943437321a578c95a90bc2dae6267f4955b1d09fdf0b8c4b76e938993a780e

附件4 删除节点

（1）删除一个worker节点

这里以删除w03为例，删除后需要保证删得干干净净，删除后该节点还能再添加回来

###在master节点操作
kubectl drain w03 --delete-local-data --force --ignore-daemonsets
kubectl delete node w03
###在w03节点上操作，先reset，再删除CNI网络
kubeadm reset
rm -rf /etc/cni/net.d
ifconfig docker0 down
ip link delete docker0
ifconfig flannel.1 down
ip link delete flannel.1

附件5 临时笔记

按照之前版本的安装手册，在kubeadm init前，应在所有k8s节点上，添加网桥过滤和地址转发功能。

本次1.28版本的安装及初始化，并未执行该步骤。

cat > /etc/sysctl.d/kubernetes.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system