kubeadm安装k8s集群
一、机器准备(所有的master和node节点需要执行)
部署k8s集群的节点按照用途可以划分为如下2类角色：
- master：集群的master节点，集群的初始化节点，基础配置不低于2c 4g
- slave：集群的slave节点，可以多台，基础配置不低于1c 2g
主机名、节点ip、部署组件
k8s-master 10.0.0.10 etcd, kube-apiserver, kube-controller-manager, kubectl, kubeadm, kubelet, kube-proxy, flannel
k8s-node1 10.0.0.11 kubeadm, kubectl, kubelet, kube-proxy, flannel,docker
k8s-node2 10.0.0.12 kubeadm, kubectl, kubelet, kube-proxy, flannel,docker
hosts解析
cat >>/etc/hosts <<'EOF'
10.0.0.10 k8s-master-10
10.0.0.11 k8s-node-11
10.0.0.12 k8s-node-12
EOF
ping -c 2 k8s-master-10
ping -c 2 k8s-node-11
ping -c 2 k8s-node-12
调整系统配置
如果节点间无安全组限制（内网机器间可以任意访问），可以忽略，否则，至少保证如下端口可通：
k8s-master节点：TCP：6443，2379，2380，60080，60081 UDP协议端口全部打开
k8s-slave节点：UDP协议端口全部打开
设置iptables
systemctl stop firewalld NetworkManager
systemctl disable firewalld NetworkManager
sed -ri 's#(SELINUX=).*#\1disabled#' /etc/selinux/config
setenforce 0
systemctl disable firewalld && systemctl stop firewalld
getenforce 0
iptables -F
iptables -X
iptables -Z
iptables -P FORWARD ACCEPT
关闭swap
swapoff -a
# 防止开机自动挂载 swap 分区
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
配置阿里云源
curl -o /etc/yum.repos.d/Centos-7.repo http://mirrors.aliyun.com/repo/Centos-7.repo
curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum clean all && yum makecache fast
确保ntp、网络正确
yum install chrony -y
systemctl start chronyd
systemctl enable chronyd
date
hwclock -w
ping -c 2 baidu.com
修改内核参数
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1
vm.max_map_count=262144
EOF
modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf
二、通过kubeadm安装k8s
kubeadm 是 Kubernetes 主推的部署工具之一，将k8s的组件打包为了镜像，然后通过kubeadm进行集群初始化创建。
安装docker(所有的master和node节点需要执行)
# 可添加到脚本文件执行
# 配置阿里源
yum remove docker docker-common docker-selinux docker-engine -y
curl -o /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum makecache fast
yum list docker-ce --showduplicates
yum install docker-ce-19.03.15 docker-ce-cli-19.03.15 -y
# 配置docker加速器、以及crgoup驱动，改为k8s官方推荐的systemd，否则初始化时会有报错。
mkdir -p /etc/docker
cat > /etc/docker/daemon.json <<'EOF'
{
"registry-mirrors" : [
"https://iq1ib072.mirror.aliyuncs.com"],
"exec-opts":["native.cgroupdriver=systemd"]
}
EOF
# 启动
systemctl daemon-reload && systemctl start docker && systemctl enable docker
安装kubeadm工具(所有的master和node节点需要执行)
# 可添加到脚本文件执行
# 设置阿里云源
curl -o /etc/yum.repos.d/Centos-7.repo http://mirrors.aliyun.com/repo/Centos-7.repo
curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum clean all && yum makecache
#yum list kubeadm --showduplicates 列出阿里云k8s源，提供了哪些k8s版本让你用
#安装指定版本kubeadm-1.19.3，安装kubeadm的版本就是决定了拉取什么版本的k8s集群镜像
yum install kubelet-1.19.3 kubeadm-1.19.3 kubectl-1.19.3 ipvsadm
# 查看kubeadm 版本
kubeadm version
# kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:47:53Z", Go
# 设置kubelet开机启动
# 该工具用于建立起k8s集群，master，node之间的联系。
systemctl enable kubelet
查看当前占用端口
netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1019/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1247/master
tcp6 0 0 :::22 :::* LISTEN 1019/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1247/master
udp 0 0 127.0.0.1:323 0.0.0.0:* 1650/chronyd
udp6 0 0 ::1:323 :::* 1650/chronyd
初始化master节点(master)
kubeadm init \
--apiserver-advertise-address=10.0.0.10 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.19.3 \
--service-cidr=10.1.0.0/16 \
--pod-network-cidr=10.2.0.0/16 \
--service-dns-domain=cluster.local \
--ignore-preflight-errors=Swap \
--ignore-preflight-errors=NumCPU
# 参数解释
kubeadm init \
--apiserver-advertise-address=10.0.0.10 \ API-Server地址
--image-repository registry.aliyuncs.com/google_containers \ 镜像仓库地址
--kubernetes-version v1.19.3 \ k8s版本，需要与kubeadm的版本一致
--service-cidr=10.1.0.0/16 \ clusterIP
--pod-network-cidr=10.2.0.0/16 \ podIP网段，定义自己想要的网络段
--service-dns-domain=cluster.local \ 集群内dns后缀
--ignore-preflight-errors=Swap \ 忽略swap报错
--ignore-preflight-errors=NumCPU 忽略CPU报错
# 初始化过程会去阿里云下载镜像，最终的初始化结果务必保留好
W1018 17:52:21.939904 5332 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.3
[preflight] Running pre-flight checks
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master-10 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.1.0.1 10.0.0.10]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master-10 localhost] and IPs [10.0.0.10 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master-10 localhost] and IPs [10.0.0.10 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 21.507001 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.19" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master-10 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master-10 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: yew79j.ob41k8e2rp340pvu
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.0.0.10:6443 --token yew79j.ob41k8e2rp340pvu \
--discovery-token-ca-cert-hash sha256:fcaef0c918aa146ce08d4816dcc4f194bb6933edb7ee055a63bfd3b506d6634f
整个初始化过程基本是
- 生成证书，集群内组件通信走证书加密
- 创建k8s组件配置文件
- 控制平面pod生成，几大组件生成
-生成认证、授权规则
-生成插件coredns,kube-proxy
-以及根据最后的要求，创建配置文件、设置网络插件、Node加入集群即可。
查看生成的端口，组件进程信息
netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 16367/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 16681/kube-proxy
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 16095/etcd
tcp 0 0 10.0.0.10:2379 0.0.0.0:* LISTEN 16095/etcd
tcp 0 0 10.0.0.10:2380 0.0.0.0:* LISTEN 16095/etcd
tcp 0 0 127.0.0.1:2381 0.0.0.0:* LISTEN 16095/etcd
tcp 0 0 127.0.0.1:10257 0.0.0.0:* LISTEN 16035/kube-controll
tcp 0 0 127.0.0.1:10259 0.0.0.0:* LISTEN 16048/kube-schedule
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1019/sshd
tcp 0 0 127.0.0.1:40311 0.0.0.0:* LISTEN 16367/kubelet
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1247/master
tcp6 0 0 :::10250 :::* LISTEN 16367/kubelet
tcp6 0 0 :::6443 :::* LISTEN 16043/kube-apiserve
tcp6 0 0 :::10256 :::* LISTEN 16681/kube-proxy
tcp6 0 0 :::22 :::* LISTEN 1019/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1247/master
udp 0 0 127.0.0.1:323 0.0.0.0:* 1650/chronyd
udp6 0 0 ::1:323 :::* 1650/chronyd
[root@k8s-master-10 ~]#
根据要求设置master
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
查看k8s节点信息
[root@k8s-master-10 ~]#kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master-10 NotReady master 3h14m v1.19.3
⚠️注意：此时使用 kubectl get nodes查看节点应该处于notReady状态，因为还未配置网络插件
若执行初始化过程中出错，根据错误信息调整后，执行kubeadm reset后再次执行init操作即可
报错提示,可能出现警告信息:
1. NumCPU
2. IsDockerSystemdCheck
3. SystemVerification
部署Flannel网络插件(master)
# 下载flannel,网址如下：
https://github.com/coreos/flannel.git
上传到服务器指定位置解压
修改网络插件信息(master)
vim /root/flannel-master/Documentation/kube-flannel.yml
因为自定义了PODIP网段，所有这里也需要修改，你也可以选择初始化的的时候不改PODIP网段
第一处，指定flannel创建的局域网段
net-conf.json: |
{
"Network": "10.2.0.0/16", #这个ip和初始化master节点的pod-network-cidr参数设置一致
"Backend": {
"Type": "vxlan"
}
}
第二处，如果机器存在多网卡的话，指定外网网卡的名称，默认不指定的话会找第一块网卡
150 containers:
151 - name: kube-flannel
152 #image: flannelcni/flannel:v0.19.2 for ppc64le and mips64le (dockerhub limitations may apply)
153 image: docker.io/rancher/mirrored-flannelcni-flannel:v0.19.2
154 command:
155 - /opt/bin/flanneld
156 args:
157 - --ip-masq
158 - --kube-subnet-mgr
159 - --iface=ens33 #这个参数值为可以连外网的网卡名，通过ip a可以查看
应用yaml清单创建flannel(master)
# 过程是创建pod、创建容器
[root@k8s-master-10 ~/flannel-master/Documentation]#kubectl create -f kube-flannel.yml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
# 创建的这些资源都是可以查看的
# flannel这种网络插件，以Daemonset控制器去运行pod，确保在主机上只有一个。
# 检查关于flannel的容器信息
kubectl get pods -n kube-flannel
# 查看docker容器
docker ps|grep flannel
确保flannel创建正确(master)
kubectl get pods -n kube-system
Node节点加入集群（node）
kubeadm join 10.0.0.10:6443 --token yew79j.ob41k8e2rp340pvu \
--discovery-token-ca-cert-hash sha256:fcaef0c918aa146ce08d4816dcc4f194bb6933edb7ee055a63bfd3b506d6634f
如果忘记添加命令，可以通过如下命令生成：
kubeadm token create --print-join-command
查看k8s集群所有节点状态(master)
[root@k8s-master-10 ~/flannel-master/Documentation]#kubectl get node --show-labels=true -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME LABELS
k8s-master-10 Ready master 4h9m v1.19.3 10.0.0.10 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://19.3.15 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master-10,kubernetes.io/os=linux,node-role.kubernetes.io/master=
k8s-node-11 Ready <none> 5m10s v1.19.3 10.0.0.11 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://19.3.15 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node-11,kubernetes.io/os=linux
k8s-node-12 Ready <none> 5m6s v1.19.3 10.0.0.12 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://19.3.15 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node-12,kubernetes.io/os=linux
[root@k8s-master-10 ~/flannel-master/Documentation]#
配置k8S命令补全(master)
yum install bash-completion -y
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
三、验证集群是否可用
$ kubectl get nodes #观察集群节点是否全部Ready
创建测试nginx:1.17.9服务测试
kubectl run yuchao-nginx --image=nginx:1.17.9
kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
yuchao-nginx 0/1 ContainerCreating 0 17s <none> k8s-node-11 <none> <none>
[root@k8s-master-10 ~/flannel-master/Documentation]#kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
yuchao-nginx 1/1 Running 0 2m47s 10.2.1.2 k8s-node-11 <none> <none>
# 查看pod描述信息
kubectl describe pod yuchao-nginx
访问集群内nginx
在node节点上创建的nginx容器名字，命名规则默认是 k8s_容器名_pod名_命名空间_随机id
可以去修改容器内信息，然后再访问
[root@k8s-node-11 ~]#docker exec 772 bash -c 'echo "k8s部署完成了" > /usr/share/nginx/html/index.html'
[root@k8s-master-10 ~]#curl 10.2.1.2
k8s部署完成了
四、删除k8s
如果部署过程出错，可以选择快照还原、或者环境清理
# 在全部集群节点执行
kubeadm reset
ifconfig cni0 down && ip link delete cni0
ifconfig flannel.1 down && ip link delete flannel.1
rm -rf /run/flannel/subnet.env
rm -rf /var/lib/cni/
mv /etc/kubernetes/ /tmp
mv /var/lib/etcd /tmp
mv ~/.kube /tmp
iptables -F
iptables -t nat -F
ipvsadm -C
ip link del kube-ipvs0
ip link del dummy0