526互联

kubeadm搭建单master多node的k8s集群

发布时间 2023-06-21 14:56:18作者: 休耕

一、实验环境准备

镜像选择：CentOS-7-x86_64-DVD-2009.iso
配置：4核、6G内存、80G硬盘
兼容性：ESXI 7.0及更高版本

服务器信息：

k8s集群角色 ip 主机名 安装的组件
控制节点 10.104.26.192 hqs-master apiserver、controller-manager、scheduler、etcd、kube-proxy、docker、calico
工作节点 10.104.26.193 hqs-node1 kubelet、kube-proxy、docker、calico、coredns
工作节点 10.104.26.194 hqs-node2 kubelet、kube-proxy、docker、calico、coredns

1、kubeadm 和二进制安装 k8s 适用场景分析

kubeadm是官方提供的开源工具，是一个开源项目，用于快速搭建 kubernetes 集群。

kubeadmin init 和 kubeadm join 是 kubeadm 中最重要的两个命令，前者用于初始化集群，后者用于加入节点。

kubeadm 初始化 k8s 的过程中，会创建 kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kube-proxy 等组件，这些组件都是以 Pod 的形式运行在 k8s 集群中的，具备故障自恢复能力。

二进制安装 k8s，需要自己手动创建这些组件，而且二进制安装的 k8s 集群，不具备故障自恢复能力。

二进制安装的 k8s 集群，不具备故障自恢复能力，而且二进制安装的 k8s 集群，不支持 kubeadm 升级。所以，kubeadm 适合用于生产环境，二进制安装适合用于测试环境。

2、初始化部署环境

（1）修改机器IP为静态IP

# 控制节点
[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens192 
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NAME=ens192
UUID=1070fb40-0984-46df-9559-6d193e974c6c
DEVICE=ens192
ONBOOT=yes
IPADDR=10.104.26.192
NETMASK=255.255.255.0
GATEWAY=10.104.26.252
ZONE=public
PREFIX=24

# 工作节点1
[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens192
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NAME=ens192
UUID=1070fb40-0984-46df-9559-6d193e974c6c
DEVICE=ens192
ONBOOT=yes
IPADDR=10.104.26.193
NETMASK=255.255.255.0
GATEWAY=10.104.26.252
ZONE=public
PREFIX=24

# 工作节点2
[root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens192
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NAME=ens192
UUID=1070fb40-0984-46df-9559-6d193e974c6c
DEVICE=ens192
ONBOOT=yes
IPADDR=10.104.26.194
NETMASK=255.255.255.0
GATEWAY=10.104.26.252
ZONE=public
PREFIX=24

（2）修改主机名

# 控制节点
[root@localhost ~]# hostnamectl set-hostname hqs-master && bash
[root@hqs-master ~]# 

# 工作节点1
[root@localhost ~]# hostnamectl set-hostname hqs-node1 && bash
[root@hqs-node1 ~]# 

# 工作节点2
[root@localhost ~]# hostnamectl set-hostname hqs-node2 && bash
[root@hqs-node2 ~]#

（3）修改hosts文件

让各个节点都能够通过主机名访问到其他节点。修改每个机器的 /etc/hosts 文件，执行如下内容：

echo '10.104.26.192 hqs-master
10.104.26.193 hqs-node1
10.104.26.194 hqs-node2' >> /etc/hosts

（4）配置主机间免密登录

k8s 集群中的各个节点之间需要通过 ssh 进行通信，所以需要配置主机间免密登录。

控制节点执行如下命令：

# 生成密钥（一路回车）
[root@hqs-master ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:/GSg0F2vu/qUv1TqjaHDrsm4Fgj08wwmzjw2HbqBLB8 root@hqs-master
The keys randomart image is:
+---[RSA 2048]----+
|          .      |
|  .  . . . .     |
| . .. . o   .    |
|  o *. o . .     |
|.= * B. S +   .  |
|o.E o +  + o o   |
|.o *   . .= +    |
|  o   .o oo* +   |
|     .o.===.=..  |
+----[SHA256]-----+
# 将本地生成的密钥文件和私钥文件拷贝到其他节点
[root@hqs-master ~]# ssh-copy-id hqs-master
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hqs-master (10.104.26.192)' cant be established.
ECDSA key fingerprint is SHA256:dPx3U0PFkordJ6nnl7V//yfM4LBJdqvn0ElacmkkHmk.
ECDSA key fingerprint is MD5:aa:87:49:8f:d8:30:5f:0c:1e:40:a5:03:16:56:53:27.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hqs-masters password:      <<== 这里输入密码
Number of key(s) added: 1
Now try logging into the machine, with:   "ssh 'hqs-master'"
and check to make sure that only the key(s) you wanted were added.

[root@hqs-master ~]# ssh-copy-id hqs-node1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hqs-node1 (10.104.26.193)' can't be established.
ECDSA key fingerprint is SHA256:dPx3U0PFkordJ6nnl7V//yfM4LBJdqvn0ElacmkkHmk.
ECDSA key fingerprint is MD5:aa:87:49:8f:d8:30:5f:0c:1e:40:a5:03:16:56:53:27.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hqs-node1's password:      <<== 这里输入密码
Number of key(s) added: 1
Now try logging into the machine, with:   "ssh 'hqs-node1'"
and check to make sure that only the key(s) you wanted were added.

[root@hqs-master ~]# ssh-copy-id hqs-node2
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'hqs-node2 (10.104.26.194)' can't be established.
ECDSA key fingerprint is SHA256:dPx3U0PFkordJ6nnl7V//yfM4LBJdqvn0ElacmkkHmk.
ECDSA key fingerprint is MD5:aa:87:49:8f:d8:30:5f:0c:1e:40:a5:03:16:56:53:27.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@hqs-node2's password:      <<== 这里输入密码
Number of key(s) added: 1
Now try logging into the machine, with:   "ssh 'hqs-node2'"
and check to make sure that only the key(s) you wanted were added.

node1 和 node2 也执行如上命令，将密钥文件拷贝到其他节点。

# node1
[root@hqs-node1 ~]# ssh-keygen
[root@hqs-node1 ~]# ssh-copy-id hqs-master
[root@hqs-node1 ~]# ssh-copy-id hqs-node1
[root@hqs-node1 ~]# ssh-copy-id hqs-node2

# node2
[root@hqs-node2 ~]# ssh-keygen
[root@hqs-node2 ~]# ssh-copy-id hqs-master
[root@hqs-node2 ~]# ssh-copy-id hqs-node1
[root@hqs-node2 ~]# ssh-copy-id hqs-node2

（5）关闭交换分区swap

swap交换分区，在机器内存不够时会使用，但是swap分区的性能较低，k8s设计时为了提升性能，默认不允许使用交换分区。

kubeadm初始化的时候会检查是否开启了swap分区，如果开启了，会报错，所以需要关闭swap分区。如果不想关闭交换分区，可以使用--ignore-preflight-errors=Swap参数忽略检查。

# 临时关闭
[root@hqs-master ~]# swapoff -a
[root@hqs-node1 ~]# swapoff -a
[root@hqs-node2 ~]# swapoff -a

# 永久关闭————注释swap挂载
[root@hqs-master ~]# vim /etc/fstab
#/dev/mapper/centos-swap swap                    swap    defaults        0 0

# 这两台因为时克隆的机器，还需要删除UUID
[root@hqs-node1 ~]# vim /etc/fstab
#UUID=b64332af-5acd-4202-8dbe-8dc83c50bfae /boot                   xfs     defaults        0 0
#/dev/mapper/centos-swap swap                    swap    defaults        0 0
[root@hqs-node2 ~]# vim /etc/fstab
#UUID=b64332af-5acd-4202-8dbe-8dc83c50bfae /boot                   xfs     defaults        0 0
#/dev/mapper/centos-swap swap                    swap    defaults        0 0

（6）修改机器内核参数

k8s需要修改机器内核参数，使其支持路由转发和桥接，否则会报错。

# 1.master节点修改
# 临时修改，加载 br_netfilter 模块(桥接模块)
[root@hqs-master ~]# modprobe br_netfilter

# 永久修改
[root@hqs-master ~]# echo "modprobe br_netfilter" >> /etc/profile

# k8s.conf作用是修改内核参数，使其支持路由转发和桥接
[root@hqs-master ~]# cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

# 使配置生效
# sysctl 命令用于在运行时动态地修改内核的运行参数，可用于修改网络相关的参数
[root@hqs-master ~]# sysctl -p /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

# 查看内核参数
[root@hqs-master ~]# sysctl net.bridge.bridge-nf-call-ip6tables

# 2.node1节点修改
[root@hqs-node1 ~]# modprobe br_netfilter
[root@hqs-node1 ~]# echo "modprobe br_netfilter" >> /etc/profile
[root@hqs-node1 ~]# cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
[root@hqs-node1 ~]# sysctl -p /etc/sysctl.d/k8s.conf

# 3.node2节点修改
[root@hqs-node2 ~]# modprobe br_netfilter
[root@hqs-node2 ~]# echo "modprobe br_netfilter" >> /etc/profile
[root@hqs-node2 ~]# cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
[root@hqs-node2 ~]# sysctl -p /etc/sysctl.d/k8s.conf

修改上述内核参数主要是为了解决以下问题：

  • 问题1：执行sysctl -p /etc/sysctl.d/k8s.conf 出现报错：sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory。该问题是因为没有加载br_netfilter模块，执行modprobe br_netfilter加载该模块即可。
  • 问题2：安装docker后，执行 docker info 出现报错：WARNING: bridge-nf-call-iptables is disabled. WARNING: bridge-nf-call-ip6tables is disabled.。该问题是因为没有开启路由转发，执行sysctl net.bridge.bridge-nf-call-ip6tables=1和sysctl net.bridge.bridge-nf-call-iptables=1开启路由转发即可。
  • 问题3：kubeadm初始化k8s报错：ERROR FileContent--proc-sys-net-ipv4-ip_forward contents are not set to 1。该问题是因为没有开启路由转发，执行sysctl net.ipv4.ip_forward=1开启路由转发即可。

（7）关闭防火墙

关闭防火墙，或者开放k8s需要的端口。

[root@hqs-master ~]# systemctl stop firewalld && systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

[root@hqs-node1 ~]# systemctl stop firewalld && systemctl disable firewalld

[root@hqs-node2 ~]# systemctl stop firewalld && systemctl disable firewalld

（8）关闭selinux

关闭selinux，或者设置为permissive模式。

# 修改配置文件
[root@hqs-master ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
[root@hqs-node1 ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
[root@hqs-node2 ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

# 修改selinux配置后，需要重启机器，配置才能永久生效
[root@hqs-master ~]# reboot
[root@hqs-node1 ~]# reboot
[root@hqs-node2 ~]# reboot

# 查看selinux状态
[root@hqs-master ~]# getenforce
Disabled

（9）配置阿里云yum源

配置阿里云yum源，或者使用其他yum源。

# 删除原有yum源
[root@hqs-master ~]# rm -rf /etc/yum.repos.d/*.repo
[root@hqs-node1 ~]# rm -rf /etc/yum.repos.d/*.repo
[root@hqs-node2 ~]# rm -rf /etc/yum.repos.d/*.repo

# 下载阿里云yum源
[root@hqs-master ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
# 如没有wget命令，可上传CentOS-Base.repo文件到/etc/yum.repos.d/目录下
[root@hqs-master yum.repos.d]# scp CentOS-Base.repo root@hqs-node1:/etc/yum.repos.d/
CentOS-Base.repo                                          100% 2523   253.5KB/s   00:00    
[root@hqs-master yum.repos.d]# scp CentOS-Base.repo root@hqs-node2:/etc/yum.repos.d/
CentOS-Base.repo                                          100% 2523   253.5KB/s   00:00

# 清除缓存
[root@hqs-master ~]# yum clean all && yum makecache
[root@hqs-node1 ~]# yum clean all && yum makecache
[root@hqs-node2 ~]# yum clean all && yum makecache

# 安装lrzsz、scp、vim、wget、net-tools
[root@hqs-master ~]# yum install -y lrzsz scp vim wget net-tools yum-utils
[root@hqs-node1 ~]# yum install -y lrzsz scp vim wget net-tools yum-utils
[root@hqs-node2 ~]# yum install -y lrzsz scp vim wget net-tools yum-utils

# 配置国内docker的repo源
[root@hqs-master ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
Loaded plugins: fastestmirror
adding repo from: http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
grabbing file http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo to /etc/yum.repos.d/docker-ce.repo
repo saved to /etc/yum.repos.d/docker-ce.repo
[root@hqs-master ~]# ls /etc/yum.repos.d/
CentOS-Base.repo  docker-ce.repo

[root@hqs-node1 ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

[root@hqs-node2 ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo


# 配置epel源
# 下载epel源
[root@hqs-master ~]# wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
--2023-06-21 13:47:03--  http://mirrors.aliyun.com/repo/epel-7.repo
Resolving mirrors.aliyun.com (mirrors.aliyun.com)... 119.96.33.219, 182.40.59.176, 182.40.41.199, ...
Connecting to mirrors.aliyun.com (mirrors.aliyun.com)|119.96.33.219|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 664 [application/octet-stream]
Saving to: ‘/etc/yum.repos.d/epel.repo’

100%[=======================================================================================================>] 664         --.-K/s   in 0s      

2023-06-21 13:47:03 (147 MB/s) - ‘/etc/yum.repos.d/epel.repo’ saved [664/664]

[root@hqs-master ~]# scp /etc/yum.repos.d/epel.repo hqs-node1:/etc/yum.repos.d/
epel.repo                                                                         100%  664     1.0MB/s   00:00    
[root@hqs-master ~]# scp /etc/yum.repos.d/epel.repo hqs-node2:/etc/yum.repos.d/
epel.repo                                                                         100%  664     1.2MB/s   00:00

（10）配置安装k8s组件的yum源

配置k8s组件的yum源，或者使用其他yum源。

# 编写kubernetes.repo文件
[root@hqs-master ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
EOF
[root@hqs-master ~]# ls /etc/yum.repos.d/
CentOS-Base.repo  docker-ce.repo  epel.repo  kubernetes.repo

# 将kubernetes.repo文件分发到其他节点
[root@hqs-master ~]# scp /etc/yum.repos.d/kubernetes.repo hqs-node1:/etc/yum.repos.d/
kubernetes.repo                                                                    100%  129   145.5KB/s   00:00    
[root@hqs-master ~]# scp /etc/yum.repos.d/kubernetes.repo hqs-node2:/etc/yum.repos.d/
kubernetes.repo                                                                    100%  129   237.0KB/s   00:00

(11)配置时间同步

使用ntpdate同步时间。

# 安装ntpdate
[root@hqs-master ~]# yum install -y ntpdate
[root@hqs-node1 ~]# yum install -y ntpdate
[root@hqs-node2 ~]# yum install -y ntpdate

# 与ntp服务器同步时间
[root@hqs-master ~]# ntpdate cn.pool.ntp.org
[root@hqs-node1 ~]# ntpdate cn.pool.ntp.org
[root@hqs-node2 ~]# ntpdate cn.pool.ntp.org

# 将时间同步命令写入crontab
[root@hqs-master ~]# crontab -e
* */1 * * * /usr/sbin/ntpdate cn.pool.ntp.org
[root@hqs-node1 ~]# crontab -e
* */1 * * * /usr/sbin/ntpdate cn.pool.ntp.org
[root@hqs-node2 ~]# crontab -e
* */1 * * * /usr/sbin/ntpdate cn.pool.ntp.org

# 重启crontab服务
[root@hqs-master ~]# systemctl restart crond
[root@hqs-node1 ~]# systemctl restart crond
[root@hqs-node2 ~]# systemctl restart crond

（12）开启ipvs

开启ipvs，需要在所有节点上执行。

ipvs(IP Vertual Server)：实现了传输层的负载均衡，是一种高性能、可扩展的负载均衡方案。

  • Linux内核的一部分，承担着负载均衡的功能。
  • 通过ipvsadm工具来配置ipvs。
  • ipvs可基于TCP和UDP的服务请求转发到真实服务器上

ipvs和iptables的区别：

  • ipvs是在传输层工作，iptables是在网络层工作。
  • ipvs是基于内容的负载均衡，iptables是基于地址的负载均衡。
  • ipvs为大型集群提供了高性能的负载均衡，iptables为小型集群提供了负载均衡。
  • ipvs支持更多的负载均衡算法，iptables只支持轮询算法。
  • ipvs支持服务器健康检查、连接重试等功能，iptables不支持。
# 准备ipvs.modules文件
[root@hqs-master ~]# cat <<EOF > /etc/sysconfig/modules/ipvs.modules
#!/bin/bash
ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack"
for kernel_module in ${ipvs_modules}; do
 /sbin/modinfo -F filename ${kernel_module} > /dev/null 2>&1
 if [ 0 -eq 0 ]; then
 /sbin/modprobe ${kernel_module}
 fi
done
EOF

# 修改ipvs.modules文件权限并执行
[root@hqs-master ~]# chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs

# 将ipvs.modules文件分发到其他节点
[root@hqs-master ~]# scp /etc/sysconfig/modules/ipvs.modules hqs-node1:/etc/sysconfig/modules/
ipvs.modules                                                                      100%  164   188.5KB/s   00:00
[root@hqs-master ~]# scp /etc/sysconfig/modules/ipvs.modules hqs-node2:/etc/sysconfig/modules/
ipvs.modules                                                                      100%  164   188.5KB/s   00:00

# 修改node节点 ipvs.modules文件权限并执行
[root@hqs-node1 ~]# chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs
[root@hqs-node2 ~]# chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs

（13）安装基础软件包

[root@hqs-master ~]# yum install -y device-mapper-persistent-data lvm2 net-tools conntrack-tools wget nfs-utils telnet gcc gcc-c++ make cmake libxml2-devel openssl-devel curl-devel unzip sudo ntp libaio-devel ncurses-devel autoconf automake zlib-devel python-devel epel-release openssh-server socat ipvsadm conntrack ntpdate telnet

[root@hqs-node1 ~]# yum install -y device-mapper-persistent-data lvm2 net-tools conntrack-tools wget nfs-utils telnet gcc gcc-c++ make cmake libxml2-devel openssl-devel curl-devel unzip sudo ntp libaio-devel ncurses-devel autoconf automake zlib-devel python-devel epel-release openssh-server socat ipvsadm conntrack ntpdate telnet

[root@hqs-node2 ~]# yum install -y device-mapper-persistent-data lvm2 net-tools conntrack-tools wget nfs-utils telnet gcc gcc-c++ make cmake libxml2-devel openssl-devel curl-devel unzip sudo ntp libaio-devel ncurses-devel autoconf automake zlib-devel python-devel epel-release openssh-server socat ipvsadm conntrack ntpdate telnet

（14）安装iptables