BC-Linux for Euler部署K8S-1.27-526互联

1、介绍及说明
- 1.1 主机信息
- 1.2 部署组件及规划
2、基础优化
3、部署

1、介绍及说明

背景

大云企业操作系统BC-Linux for Euler 是以 openEuler 社区操作系统为基础，借助开源社区的开放优势，通过定制化手段研发的企业级 Linux 操作系统；目前主要在移动内部是用，是国产化改造的主力系统；本次部署使用BCLinux for Euler 21.10；本次使用kubeadm部署k8s-1.27.4双栈环境(ipv6为主)

1.1 主机信息

主机名	内核版本	IPV4地址	IPV6地址	角色
bclinux-11	4.19.90-2107	172.168.80.11	1:2:3:4::11	master
bclinux-12	4.19.90-2107	172.168.80.12	1:2:3:4::12	master
bclinux-13	4.19.90-2107	172.168.80.13	1:2:3:4::13	node

1.2 部署组件及规划

组件	组件说明	规划	规划推荐
nginx	单节点k8s时不需要部署此组件，集群时主要让api-server实现负载均衡能力	部署在俩台master节点	多master节点时，找任意俩台或三台都可以
keepalived	单节点k8s时不需要部署此组件，集群时主要给nginx提供高可用能力	同上	同上
docker	主要负责运行容器	所有节点都部署
cri-dockerd	k8s1.24版本后，不再维护调用docker的接口，所以需要通过cri-docker来调用docker	所有节点都部署
containerd	也是运行容器的组件，与docker二选一，任意一个组件都可以	所有节点都部署
kubectl	k8s命令行工具，管理k8s集群用的	master节点安装	master节点安装
kubeadm	k8s的一个工具箱，可以创建k8s集群及加入集群	所有节点都部署
kubelet	运行在所有节点，主要作用是管理节点上的容器和pod，并对pod和节点做监控	所有节点都部署
kube-proxy	运行在虽有节点，主要作用是维护service的通信及负载均衡机制	所有节点以容器方式运行
kube-scheduler	运行在master节点，是k8s的调度器，将pod分配到合适的节点运行	master节点以容器方式运行
kube-controller-manager	运行在master节点，是k8s的控制器，控制pod的副本数、镜像版本等	master节点以容器方式运行
kube-apiserver	运行在master节点，是k8s的消息接收，所有组件都经过api-server	master节点以容器方式运行
coredns	运行在任意节点，是k8s内部的DNS服务，负责解析内部的svc名称，还能代理外部的dns服务器	任意k8s节点以容器方式运行
etcd	可内置也可外置，是k8s的数据库，k8s的数据都存储到etcd中	可容器运行或自己单独部署	单节点推荐内置，集群推荐自己部署
CNI-calico	k8s中的网络服务，pod跨主机通信主要是CNI网络实现的	所有节点容器方式运行
metrics-server	k8s中查看pod和node的资源使用情况	任意k8s节点以容器方式运行

2、基础优化

2.1 开启ipv6

检查是否支持ipv6

[root@bclinux-11 ~]# cat /proc/net/if_inet6 # 查看是否有输出，有输出就支持ipv6
[root@bclinux-11 ~]# grep ipv6 /etc/sysctl.conf 
#开启ipv6
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0

配置ipv6

[root@bclinux-11 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens160 
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=no
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens160
DEVICE=ens160
ONBOOT=yes
IPADDR=172.168.80.11
PREFIX=24
GATEWAY=172.168.80.2
DNS1=114.114.114.114
IPV6_PRIVACY=no
IPV6ADDR=1:2:3:4::11/64
IPV6_DEFAULTGW=1:2:3:4::2
[root@bclinux-11 ~]# ip a show ens160 # 检查是否有ipv6地址
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:16:ba:aa brd ff:ff:ff:ff:ff:ff
    inet 172.168.80.11/24 brd 172.168.80.255 scope global noprefixroute ens160
       valid_lft forever preferred_lft forever
    inet6 1:2:3:4::11/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever

2.2 修改文件最大数

[root@bclinux-11 ~]# egrep -v '^#|^$' /etc/security/limits.conf
* soft nofile 655350
* hard nofile 655350

2.3 配置hosts解析


[root@bclinux-11 ~]# cat /etc/hosts 
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
1:2:3:4::11 bclinux-11
1:2:3:4::12 bclinux-12
1:2:3:4::13 bclinux-13
172.168.80.11 bclinux-11
172.168.80.12 bclinux-12
172.168.80.13 bclinux-13

2.4 内核参数优化

[root@bclinux-11 ~]# cat /etc/sysctl.d/k8s.conf 
# ipv6配置
net.ipv6.conf.all.forwarding = 1
net.bridge.bridge-nf-call-ip6tables = 1
# ipv4配置
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.all.forwarding = 1

2.5 selinux关闭

[root@bclinux-11 ~]# sed -i 's/enforcing/disabled/' /etc/selinux/config  && setenforce 0

2.6 防火墙

防火墙关闭
配置防火墙规则，将k8s对应的端口开放或者将节点主机的开放不设限制

2.7 ipvs支持

检查是否加载ipvs

[root@bclinux-11 ~]# lsmod | grep ip_vs # 有结果就加载了，没有结果就没有加载

加载ipvs

[root@bclinux-11 ~]# cat >/etc/sysconfig/modules/ipvs.modules <<-"EOF"
#!/bin/bash
ipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs"
for mod in $(ls $ipvs_mods_dir |grep -o "^[^.]*");do
    /sbin/modinfo -F filename $mod &>/dev/null
    if [ $? -eq 0 ];then
        /sbin/modprobe $mod
    fi
done
EOF
[root@bclinux-11 ~]# chown 755 /etc/sysconfig/modules/ipvs.modules && /etc/sysconfig/modules/ipvs.modules

检查

[root@bclinux-11 ~]# lsmod | grep ip_vs
ip_vs_wrr              16384  0
ip_vs_wlc              16384  0
ip_vs_sh               16384  0
ip_vs_sed              16384  0
ip_vs_rr               16384  0
ip_vs_pe_sip           16384  0
nf_conntrack_sip       32768  1 ip_vs_pe_sip
ip_vs_ovf              16384  0
ip_vs_nq               16384  0
ip_vs_lc               16384  0
ip_vs_lblcr            16384  0
ip_vs_lblc             16384  0
ip_vs_ftp              16384  0
ip_vs_fo               16384  0
ip_vs_dh               16384  0
ip_vs                 172032  28 ip_vs_wlc,ip_vs_rr,ip_vs_dh,ip_vs_lblcr,ip_vs_sh,ip_vs_ovf,ip_vs_fo,ip_vs_nq,ip_vs_lblc,ip_vs_pe_sip,ip_vs_wrr,ip_vs_lc,ip_vs_sed,ip_vs_ftp
nf_nat                 36864  3 nf_nat_ipv6,nf_nat_ipv4,ip_vs_ftp
nf_conntrack          163840  6 xt_conntrack,nf_nat,nf_nat_ipv6,nf_nat_ipv4,nf_conntrack_sip,ip_vs
nf_defrag_ipv6         20480  2 nf_conntrack,ip_vs
libcrc32c              16384  3 nf_conntrack,nf_nat,ip_vs

2.8 内核版本要求

已知问题内核版本：3.10.0-957.el7.x86_64
已知问题系统版本：centos7.6以前的版本
问题现象：部署完成后，各组件、创建pod无问题，创建完service网络后，在pod里ping不通service网络的名称，同命名空间或跨空间都不可以ping通
现用系统内核版本为4.19无需升级

2.9 yum源配置

[root@bclinux-11 ~]# ls /etc/yum.repos.d/
BCLinux.repo  centos8.repo  docker-ce.repo  kubernetes.repo
# BC-Linux源
[root@bclinux-11 ~]# cat /etc/yum.repos.d/BCLinux.repo 
[baseos]
name=BC-Linux-release - baseos
baseurl=http://mirrors.bclinux.org/bclinux/oe21.10/OS/$basearch/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-BCLinux-For-Euler
[everything]
name=BC-Linux-release - everything
baseurl=http://mirrors.bclinux.org/bclinux/oe21.10/everything/$basearch/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-BCLinux-For-Euler
[update]
name=BC-Linux-release - update
baseurl=http://mirrors.bclinux.org/bclinux/oe21.10/update/$basearch/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-BCLinux-For-Euler
[extras]
name=BC-Linux-release - extras
baseurl=http://mirrors.bclinux.org/bclinux/oe21.10/extras/$basearch/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-BCLinux-For-Euler
# centos8源，安装docker时需要
[root@bclinux-11 ~]# cat /etc/yum.repos.d/centos8.repo 
[BaseOS]
name=CentOS-8-stream - Base - repo.huaweicloud.com
baseurl=https://repo.huaweicloud.com/centos/8-stream/BaseOS/$basearch/os/
#mirrorlist=https://mirrorlist.centos.org/?release=8-stream&arch=$basearch&repo=BaseOS&infra=$infra
gpgcheck=1
gpgkey=https://repo.huaweicloud.com/centos/RPM-GPG-KEY-CentOS-Official 
#released updates 
[AppStream]
name=CentOS-8-stream - AppStream - repo.huaweicloud.com
baseurl=https://repo.huaweicloud.com/centos/8-stream/AppStream/$basearch/os/
#mirrorlist=https://mirrorlist.centos.org/?release=8-stream&arch=$basearch&repo=AppStream&infra=$infra
gpgcheck=1
gpgkey=https://repo.huaweicloud.com/centos/RPM-GPG-KEY-CentOS-Official
[PowerTools]
name=CentOS-8-stream - PowerTools - repo.huaweicloud.com
baseurl=https://repo.huaweicloud.com/centos/8-stream/PowerTools/$basearch/os/
#mirrorlist=https://mirrorlist.centos.org/?release=8-stream&arch=$basearch&repo=PowerTools&infra=$infra
gpgcheck=1
gpgkey=https://repo.huaweicloud.com/centos/RPM-GPG-KEY-CentOS-Official
#additional packages that may be useful
[extras]
name=CentOS-8-stream - Extras - repo.huaweicloud.com
baseurl=https://repo.huaweicloud.com/centos/8-stream/extras/$basearch/os/
#mirrorlist=https://mirrorlist.centos.org/?release=8-stream&arch=$basearch&repo=extras
gpgcheck=1
gpgkey=https://repo.huaweicloud.com/centos/RPM-GPG-KEY-CentOS-Official
# docker源
[root@bclinux-11 ~]# cat /etc/yum.repos.d/docker-ce.repo 
[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/8/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
[docker-ce-stable-debuginfo]
name=Docker CE Stable - Debuginfo $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/8/debug-$basearch/stable
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
[docker-ce-stable-source]
name=Docker CE Stable - Sources
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/8/source/stable
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
[docker-ce-test]
name=Docker CE Test - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/8/$basearch/test
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
[docker-ce-test-debuginfo]
name=Docker CE Test - Debuginfo $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/8/debug-$basearch/test
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
[docker-ce-test-source]
name=Docker CE Test - Sources
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/8/source/test
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
[docker-ce-nightly]
name=Docker CE Nightly - $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/8/$basearch/nightly
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
[docker-ce-nightly-debuginfo]
name=Docker CE Nightly - Debuginfo $basearch
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/8/debug-$basearch/nightly
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
[docker-ce-nightly-source]
name=Docker CE Nightly - Sources
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/8/source/nightly
enabled=0
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
# k8s源
[root@bclinux-11 ~]# cat /etc/yum.repos.d/kubernetes.repo 
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

3、部署

3.1 docker及cri-docker安装及配置 - 所有主机

k8s在1.24版本后，开始不支持接入docker，仅接入使用CRI 标准接口的容器编排器，目前比较主流程的替代品是containerd。containerd相关的命令参数与docker基本一样，唯一的区别就是无法构建镜像，所以还得使用docker构建镜像；但是docker维护了一个cri-docker的项目，主要为了满足k8s的CRI标准。

3.1.1 docker

# 安装docker
[root@bclinux-11 ~]# yum install docker-ce -y
# 配置docker
[root@bclinux-11 ~]# cat >/etc/docker/daemon.json<<-"EOF"
{
  "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"],
  "insecure-registries":["test.harbor.org:18080"],
  "data-root": "/home/docker_data",
  "log-driver":"json-file",
  "log-opts": {"max-size" :"50m","max-file":"1"},
  "exec-opts": ["native.cgroupdriver=systemd"]
} 
EOF
# 启动并设置开机自启
[root@bclinux-11 ~]# systemctl start docker && systemctl enable docker && systemctl status docker

3.1.2 cri-dockerd

下载地址：https://github.com/Mirantis/cri-dockerd/releases

# 安装
[root@bclinux-11 ~]# yum install -y https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.4/cri-dockerd-0.3.4-3.el8.x86_64.rpm
# 配置
[root@bclinux-11 ~]# egrep -v '^#|^$' /usr/lib/systemd/system/cri-docker.service
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket
[Service]
Type=notify
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --pod-infra-container-image registry.aliyuncs.com/google_containers/pause:3.9 --ipv6-dual-stack
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
# 启动并设置开机自启动
[root@bclinux-11 ~]# systemctl start cri-docker.service && systemctl enable cri-docker.service && systemctl status cri-docker.service

3.2 containerd部署 - 所有主机

如果部署了docker就不需要部署contained，俩者二选一；为什么有了cri-dockerd，还要用containerd
- containerd好处
1. 在调用时比docker少了俩层，比docker更快
2. 比docker更安全
- containerd缺点(不一定是缺点)
1. 不支持制作镜像，做镜像还得用docker或者podman
2. 操作命令相比较docker复杂一点
调用示意图

docker	containerd
K8s	K8s
Docker-shim
Docker
Containerd	Containerd
Containers	Containers

下载地址：https://github.com/containerd/containerd/releases

3.2.1 安装containerd

[root@bclinux ~]# tar xf cri-containerd-1.7.2-linux-amd64.tar.gz -C /
# 配置containerd
[root@bclinux ~]# mkdir /etc/containerd -p && containerd config default > /etc/containerd/config.toml
# 2条配置修改：
# 65行：sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.8" 【修改pause容器镜像为国内】
# 137行：SystemdCgroup = true 【让Runc使用system cgroup驱动，对容器进行资源划分，隔离。】
[root@bclinux ~]# vim /etc/containerd/config.toml # 添加私有harbor仓库
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."172.168.80.14:18080".auth]
          username = "admin"
          password = "Harbor12345"
      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."172.168.80.14:18080"]
          endpoint = ["http://172.168.80.14:18080"]
[root@bclinux ~]# systemctl start containerd.service && systemctl enable containerd.service && systemctl status containerd.service

3.3 nginx部署 - 2台master节点

nginx下载地址：http://nginx.org/en/download.html
nginx-rpm包下载地址：http://nginx.org/packages/centos/8/x86_64/RPMS/
本文使用二进制安装nginx，因rpm安装的nginx没有ipv6模块，所以使用二进制将ipv6模块编译进去

# 下载依赖包
[root@bclinux-11 ~]# yum -y install gcc automake autoconf libtool make gcc-c++ openssl openssl-devel zlib-devel pcre-devel zlib pcre libxslt-devel  perl-devel
# 编译nginx
[root@bclinux-11 ~]# tar xf nginx-1.24.0.tar.gz
[root@bclinux-11 nginx-1.24.0]# ./configure --prefix=/data/nginx --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie' --with-ipv6
..............................
  nginx path prefix: "/data/nginx"
  nginx binary file: "/data/nginx/sbin/nginx"
  nginx modules path: "/data/nginx/modules"
  nginx configuration prefix: "/data/nginx/conf"
  nginx configuration file: "/data/nginx/conf/nginx.conf"
  nginx pid file: "/data/nginx/logs/nginx.pid"
  nginx error log file: "/data/nginx/logs/error.log"
  nginx http access log file: "/data/nginx/logs/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"
  nginx http fastcgi temporary files: "fastcgi_temp"
  nginx http uwsgi temporary files: "uwsgi_temp"
  nginx http scgi temporary files: "scgi_temp"

./configure: warning: the "--with-ipv6" option is deprecated # 提示添加了ipv6模块
[root@bclinux-11 nginx-1.24.0]# make && make install
# 配置启动
[root@bclinux-11 nginx-1.24.0]# cat >/usr/lib/systemd/system/nginx.service<<-"EOF"
[Unit]
Description=nginx - high performance web server
Documentation=http://nginx.org/en/docs/
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/data/nginx/logs/nginx.pid
ExecStart=/data/nginx/sbin/nginx -c /data/nginx/conf/nginx.conf
ExecReload=/bin/sh -c "/bin/kill -s HUP $(/bin/cat /data/nginx/logs/nginx.pid)"
ExecStop=/bin/sh -c "/bin/kill -s TERM $(/bin/cat /data/nginx/logs/nginx.pid)"
[Install]
WantedBy=multi-user.target
EOF
# 配置代理
[root@bclinux-11 ~]# cat /data/nginx/conf/nginx.conf

user  root;
worker_processes  auto;

error_log  logs/error.log;
error_log  logs/error.log  notice;
error_log  logs/error.log  info;

pid        logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  logs/access.log  main;

    sendfile        on;
    keepalive_timeout  65;
}
stream {
    log_format proxy '$remote_addr [$time_local] '
               '$protocol $status $bytes_sent $bytes_received '
               '$session_time -> $upstream_addr '
               '$upstream_bytes_sent $upstream_bytes_received $upstream_connect_time';
    access_log logs/tcp-access.log proxy;
    upstream kube-apiservers {
        hash  $remote_addr consistent;
        server [1:2:3:4::11]:6443 weight=6 max_fails=1 fail_timeout=10s;
        server [1:2:3:4::12]:6443 weight=6 max_fails=1 fail_timeout=10s;
    }
    server {
        listen [::]:8443 ipv6only=on;
        proxy_connect_timeout 30s;
        proxy_timeout 60s;
        proxy_pass kube-apiservers;
    }
}
# 启动并设置开机自启
[root@bclinux-11 nginx-1.24.0]# systemctl start nginx && systemctl enable nginx && systemctl status nginx

nginx修改banner值

# 修改变量 
cat src/core/nginx.h
#define NGINX_VERSION      ""
#define NGINX_VER          "流氓兔/" NGINX_VERSION

#ifdef NGX_BUILD
#define NGINX_VER_BUILD    NGINX_VER " (" NGX_BUILD ")"
#else
#define NGINX_VER_BUILD    NGINX_VER
#endif

#define NGINX_VAR          ""
#define NGX_OLDPID_EXT     ".oldbin"

3.4 keepalived部署 - 2台master节点

[root@bclinux-11 ~]# yum install keepalived -y
[root@bclinux-11 keepalived]# tee >/etc/keepalived/keepalived.conf << "EOF"
! Configuration File for keepalived

global_defs {
   router_id k8s
}

vrrp_script check_ng {
    script "/etc/keepalived/script/check_ng.sh"
    interval 3
    weight -2
}

vrrp_instance VI_1 {
    state MASTER # 备节点修改成BACKUP
    interface ens160
    virtual_router_id 351
    priority 200 # 备节点修改成150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass ceb1b3ec013d66163d6ab
    }
    unicast_src_ip 172.168.80.11 # 修改成备节点的ip
    unicast_peer { 
      172.168.80.12 # 修改成主节点的ip
    }
    virtual_ipaddress {
        172.168.80.10
    }
    virtual_ipaddress_excluded {
        1:2:3:4::10
    }
    track_script {
        check_ng
    }
}
EOF
[root@bclinux-11 keepalived]# mkdir -p /etc/keepalived/script/ && tee >/etc/keepalived/script/check_ng.sh << "EOF" && chmod +x /etc/keepalived/script/check_ng.sh
#!/bin/bash
nginx_num=`ps -ef|grep [n]ginx|wc -l`
pid_file='/data/nginx/logs/nginx.pid'
if [ "${nginx_num}" != 0 -a -f /data/nginx/logs/nginx.pid ];then
  exit 0
else
  systemctl stop keepalived && exit 1
fi
EOF
[root@bclinux-11 keepalived]# systemctl start keepalived.service  && systemctl enable keepalived.service && systemctl status keepalived.service

3.5 etcd部署 - 三个节点

Etcd 是一个分布式键值存储系统，k8s使用Etcd进行数据存储，kubeadm搭建默认情况下只启动一个Etcd Pod，存在单点故障，生产环境强烈不建议，所以我们这里使用3台服务器组建集群，可容忍1台机器故障，当然，你也可以使用5台组建集群，可容忍2台机器故障。

为了节省机器，这里与K8s节点机器复用。也可以独立于k8s集群之外部署，只要apiserver能连接到就行。
如果是单节点k8s，可用内置etcd ，也可用外置etcd

3.5.1 准备cfssl证书生成工具

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

3.5.2 生成etcd证书

# 自签证书颁发工具机构（CA）
# 创建工作目录：
[root@bclinux-11 ~]# mkdir -p ~/etcd_tls
[root@bclinux-11 ~]# cd ~/etcd_tls
# 自签CA：
[root@bclinux-11 etcd_tls]# cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF
[root@bclinux-11 etcd_tls]# cat > ca-csr.json << EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF
# 生成证书：
[root@bclinux-11 etcd_tls]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2021/05/11 16:36:24 [INFO] generating a new CA key and certificate from CSR
2021/05/11 16:36:24 [INFO] generate received request
2021/05/11 16:36:24 [INFO] received CSR
2021/05/11 16:36:24 [INFO] generating key: rsa-2048
2021/05/11 16:36:24 [INFO] encoded CSR
2021/05/11 16:36:24 [INFO] signed certificate with serial number 359821359061850962149376009879415970209566630594
[root@bclinux-11 etcd_tls]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem
# 会生成ca.pem和ca-key.pem文件。
# 使用自签CA签发etcd https正式
# 创建证书申请文件：
[root@bclinux-11 etcd_tls]# cat > server-csr.json << EOF
{
    "CN": "etcd",
    "hosts": [
    "1:2:3:4::11",
    "1:2:3:4::12",
    "1:2:3:4::13"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF
# 注：上述文件hosts字段中IP为所有etcd节点的集群内部通信IP，要修改成自己的，一个都不能少！为了方便后期扩容可以多写几个预留的IP；如果要修改为ipv6调用，则把ip写成ipv6地址。
# 生成证书：
[root@bclinux-11 etcd_tls]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
2021/05/11 16:38:34 [INFO] generate received request
2021/05/11 16:38:34 [INFO] received CSR
2021/05/11 16:38:34 [INFO] generating key: rsa-2048
2021/05/11 16:38:35 [INFO] encoded CSR
2021/05/11 16:38:35 [INFO] signed certificate with serial number 686508384315413943399934921027716200577151171267
2021/05/11 16:38:35 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@bclinux-11 etcd_tls]# ls server*
server.csr  server-csr.json  server-key.pem  server.pem
# 会生成server.pem和server-key.pem文件。

3.5.3 部署etcd

下载地址：https://github.com/etcd-io/etcd/releases
以下在一台上操作，操作完成将节点1生成的所有文件拷贝到节点2和节点3；如配置ipv6则把下面配置得ipv4地址换成ipv6即可。

# 创建工作目录并解压二进制文件-1
[root@bclinux-11 ~]# mkdir /opt/etcd/{bin,cfg,ssl} -p
[root@bclinux-11 ~]# tar zxvf etcd-v3.5.6-linux-amd64.tar.gz
[root@bclinux-11 ~]# mv etcd-v3.5.6-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/
# 创建etcd配置文件-1
[root@bclinux-11 ~]# cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://[1:2:3:4::11]:2380"
ETCD_LISTEN_CLIENT_URLS="https://[1:2:3:4::11]:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://[1:2:3:4::11]:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://[1:2:3:4::11]:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://[1:2:3:4::11]:2380,etcd-2=https://[1:2:3:4::12]:2380,etcd-3=https://[1:2:3:4::13]:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
# ETCD_NAME：节点名称，集群中唯一
# ETCDDATADIR：数据目录
# ETCDLISTENPEER_URLS：集群通信监听地址
# ETCDLISTENCLIENT_URLS：客户端访问监听地址
# ETCDINITIALADVERTISEPEERURLS：集群通告地址
# ETCDADVERTISECLIENT_URLS：客户端通告地址
# ETCDINITIALCLUSTER：集群节点地址
# ETCDINITIALCLUSTER_TOKEN：集群Token
# ETCDINITIALCLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群
# systemd管理etcd-1
[root@bclinux-11 ~]# cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
# 拷贝刚才生成的证书-1
[root@bclinux-11 ~]# cp ~/etcd_tls/ca*pem ~/etcd_tls/server*pem /opt/etcd/ssl/

3.5.4 将节点1生成的所有文件拷贝到节点2和节点3

[root@bclinux-11 ~]# scp -r /opt/etcd/ root@[1:2:3:4::12]:/opt/
[root@bclinux-11 ~]# scp /usr/lib/systemd/system/etcd.service root@[1:2:3:4::12]:/usr/lib/systemd/system/
[root@bclinux-11 ~]# scp -r /opt/etcd/ root@[1:2:3:4::13]:/opt/
[root@bclinux-11 ~]# scp /usr/lib/systemd/system/etcd.service root@[1:2:3:4::13]:/usr/lib/systemd/system/
# 然后在节点2和节点3分别修改etcd.conf配置文件中的节点名称和当前服务器IP：
[root@bclinux-11 ~]# cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"                                            # 修改此处，节点2改为etcd-2，节点3改为etcd-3
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://[1:2:3:4::11]:2380"            # 修改此处为当前服务器IP
ETCD_LISTEN_CLIENT_URLS="https://[1:2:3:4::11]:2379"          # 修改此处为当前服务器IP

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://[1:2:3:4::11]:2380" # 修改此处为当前服务器IP
ETCD_ADVERTISE_CLIENT_URLS="https://[1:2:3:4::11]:2379"       # 修改此处为当前服务器IP
ETCD_INITIAL_CLUSTER="etcd-1=https://[1:2:3:4::11]:2380,etcd-2=https://[1:2:3:4::12]:2380,etcd-3=https://[1:2:3:4::13]:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
# 启动etcd
[root@bclinux-11 ~]# systemctl daemon-reload && systemctl start etcd && systemctl enable etcd && systemctl status etcd
# 注：etcd节点要一起启动
# ansible操作
ansible etcd -m service -a 'name=etcd state=started enabled=yes daemon-reload=yes'
# 注：etcd是在ansible里做的分组，这个组下面只有etcd这三台的主机ip

3.5.5 查看集群状态

[root@bclinux-11 ~]# ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://[1:2:3:4::11]:2379,https://[1:2:3:4::12]:2379,https://[1:2:3:4::13]:2379" endpoint health --write-out=table
+----------------------------+--------+-------------+-------+
|          ENDPOINT          | HEALTH |    TOOK     | ERROR |
+----------------------------+--------+-------------+-------+
| https://[1:2:3:4::11]:2379 |   true | 17.543566ms |       |
| https://[1:2:3:4::13]:2379 |   true | 18.424848ms |       |
| https://[1:2:3:4::12]:2379 |   true | 18.384393ms |       |
+----------------------------+--------+-------------+-------+
# 如果输出上面信息，就说明集群部署成功。
# 如果有问题第一步先看日志：/var/log/message 或 journalctl -u etcd

3.6 k8s安装 - 所有主机

[root@bclinux-11 ~]# yum install kubeadm-1.27.4 kubelet-1.27.4 kubectl-1.27.4 -y
[root@bclinux-11 ~]# systemctl enable kubelet.service

3.6.1 修改k8s证书默认1年到期

kubeadm部署的k8s除ca证书是10年，其他均是1年，如想修改默认，需要对kubeadm的源码进行修改并重新打包。
下载地址：https://github.com/kubernetes/kubernetes/releases
本次安装因为测试用，就不修改了，如想修改，参考下文

# 修改默认证书时间
[root@bclinux-11 ~]# vim kubernetes-1.20.10/staging/src/k8s.io/client-go/util/cert/cert.go
 65                 NotBefore:             now.UTC(),
 66                 NotAfter:              now.Add(duration365d * 100).UTC(),
 # 此处修改的ca正式到期时间，把10改为100即可
[root@bclinux-11 ~]# vim kubernetes-1.20.10/cmd/kubeadm/app/constants/constants.go
 49         CertificateValidity = time.Hour * 24 * 365 * 100
 # 此处是修改其他证书到期时间，增加*100

3.6.2 安装go环境

[root@bclinux-11 ~]# cat kubernetes-1.20.10/build/build-image/cross/VERSION
v1.15.15-legacy-1
# 安装go环境
[root@bclinux-11 ~]# yum install gcc make rsync jq -y
[root@bclinux-11 ~]# wget https://dl.google.com/go/go1.15.15.linux-amd64.tar.gz
[root@bclinux-11 ~]# tar zxvf go1.15.linux-amd64.tar.gz  -C /usr/local
[root@bclinux-11 ~]# tee >> /etc/profile <<-"EOF"
export GOROOT=/usr/local/go
export GOPATH=/usr/local/gopath
export PATH=$PATH:$GOROOT/bin
EOF
[root@bclinux-11 ~]# source /etc/profile
[root@bclinux-11 ~]# go version
go version go1.15.15 linux/amd64
# 编译kubeadm
[root@bclinux-11 ~]# cd kubernetes-1.20.10/
[root@bclinux-11 kubernetes-1.20.10]# make all WHAT=cmd/kubeadm GOFLAGS=-v
# 替换原有的kubeadm-所有主机
[root@bclinux-11 kubernetes-1.20.10]# ansible k8s-all -m copy -a "src=./_output/local/bin/linux/amd64/kubeadm dest=/usr/bin/kubeadm backup=yes"
# ./_output/local/bin/linux/amd64/kubeadm 是编译完输出目录

3.7 k8s部署

3.7.1 初始化 - 1台master节点

3.7.1.1 单节点初始化

列出初始化的yaml文件

---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration # 初始Master节点的私有配置 
bootstrapTokens:        # 可以指定bootstrapToken，默认24小过期自动删除
- token: "9a08jv.c0izixklcxtmnze7"
  description: "kubeadm bootstrap token"
  ttl: "24h"
certificateKey: "e6a2eb8581237ab72a4f494f30285ec12a9694d750b9785706a83bfcbbbd2204"      # 可以指定certificateKey,默认两小时过期自动删除
localAPIEndpoint:
  advertiseAddress: "1:2:3:4::11"    # master节点ip
nodeRegistration:
  criSocket: /var/run/cri-dockerd.sock
  name: master
  kubeletExtraArgs:
    node-ip: "1:2:3:4::11,172.168.80.11"    # master节点ip

---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration      # 所有Master节点的公共配置
imageRepository: registry.aliyuncs.com/google_containers
kubernetesVersion: v1.27.4
controlPlaneEndpoint: "[1:2:3:4::11]:6443"    # api-server地址,建议用域名
networking:
  podSubnet: 172:244::/64,172.244.0.0/16    # ipv4放在前面，那么kubectl get node时显示的是ipv4地址
  serviceSubnet: 172:96::/112,172.96.0.0/18    # ipv4放在前面，那么kubectl get service时显示的是ipv4地址
etcd:
  local:
    dataDir: "/home/etcd_data"
    extralArgs:
      listen-metrics-urls: http://[::]:2381

apiServer:
  certSANs: 
  - "1:2:3:4::11"
  - "172.168.80.11"
  - "bclinux-11"
  - "172.96.0.1"
  - "172:96::1"
  extraArgs:
    service-cluster-ip-range: 172:96::/112,172.96.0.0/18
    bind-address: "::"
    secure-port: "6443"
scheduler:
  extraArgs:
    bind-address: "::"
controllerManager:
  extraArgs:
    bind-address: "::"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failSwapOn: false
cgroupDriver: systemd
healthzBindAddress: "::"
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "172:244::/64,172.244.0.0/16"    # Pod的地址范围
mode: "ipvs"

3.7.1.2 集群初始化

[root@bclinux-11 ~]# cat kube-init.yaml 
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration # 初始Master节点的私有配置 
bootstrapTokens:        # 可以指定bootstrapToken，默认24小过期自动删除
- token: "9a08jv.c0izixklcxtmnze7"
  description: "kubeadm bootstrap token"
  ttl: "24h"
certificateKey: "e6a2eb8581237ab72a4f494f30285ec12a9694d750b9785706a83bfcbbbd2204"      # 可以指定certificateKey,默认两小时过期自动删除
localAPIEndpoint:
  advertiseAddress: "1:2:3:4::11"    # master节点ip
nodeRegistration:
  criSocket: /var/run/cri-dockerd.sock
  name: bclinux-11
  kubeletExtraArgs:
    node-ip: "1:2:3:4::11,172.168.80.11"    # master节点ip

---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration      # 所有Master节点的公共配置
imageRepository: registry.aliyuncs.com/google_containers
kubernetesVersion: v1.27.4
controlPlaneEndpoint: "[1:2:3:4::10]:8443"    # api-server地址,建议用域名
networking:
  podSubnet: 172:244::/64,172.244.0.0/16    # ipv4放在前面，那么kubectl get node时显示的是ipv4地址
  serviceSubnet: 172:96::/112,172.96.0.0/18    # ipv4放在前面，那么kubectl get service时显示的是ipv4地址
etcd:
  external:  # 使用外部etcd
    endpoints:
    - https://[1:2:3:4::11]:2379 # etcd集群3个节点
    - https://[1:2:3:4::12]:2379 # etcd集群3个节点
    - https://[1:2:3:4::13]:2379 # etcd集群3个节点
    caFile: /data/etcd/ssl/ca.pem # 连接etcd所需证书
    certFile: /data/etcd/ssl/server.pem
    keyFile: /data/etcd/ssl/server-key.pem

apiServer:
  certSANs: 
  - "1:2:3:4::11"
  - "1:2:3:4::12"
  - "172.168.80.11"
  - "172.168.80.12"
  - "bclinux-11"
  - "bclinux-12"
  - "172.96.0.1"
  - "172:96::1"
  extraArgs:
    service-cluster-ip-range: 172:96::/112,172.96.0.0/18
    bind-address: "::"
    secure-port: "6443"
scheduler:
  extraArgs:
    bind-address: "::"
controllerManager:
  extraArgs:
    bind-address: "::"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failSwapOn: false
cgroupDriver: systemd
healthzBindAddress: "::"
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "172:244::/64,172.244.0.0/16"    # Pod的地址范围
mode: "ipvs"
[root@bclinux-11 ~]# kubeadm init --config kube-init-1.27.yaml --upload-certs
........................................
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join [1:2:3:4::10]:8443 --token 9a08jv.c0izixklcxtmnze7 \
	--discovery-token-ca-cert-hash sha256:71818aa4d010d77aa3f0864c04415da65db936e68d598710296c2bf38104e4cf \
	--control-plane --certificate-key e6a2eb8581237ab72a4f494f30285ec12a9694d750b9785706a83bfcbbbd2204

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join [1:2:3:4::10]:8443 --token 9a08jv.c0izixklcxtmnze7 \
	--discovery-token-ca-cert-hash sha256:71818aa4d010d77aa3f0864c04415da65db936e68d598710296c2bf38104e4cf
[root@bclinux-11 ~]# mkdir -p $HOME/.kube
[root@bclinux-11 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@bclinux-11 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

3.7.2 添加master节点 - 剩余master节点

[root@bclinux-12 ~]# cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--node-ip 1:2:3:4::12,172.168.80.12 --fail-swap-on=false"
[root@bclinux-12 ~]# kubeadm join [1:2:3:4::10]:8443 --token 9a08jv.c0izixklcxtmnze7 --discovery-token-ca-cert-hash sha256:71818aa4d010d77aa3f0864c04415da65db936e68d598710296c2bf38104e4cf --control-plane --certificate-key e6a2eb8581237ab72a4f494f30285ec12a9694d750b9785706a83bfcbbbd2204 --cri-socket=/var/run/cri-dockerd.sock
..........................
To start administering your cluster from this node, you need to run the following as a regular user:

	mkdir -p $HOME/.kube
	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.
[root@bclinux-12 ~]# kubectl get pod -n kube-system|grep api
kube-apiserver-bclinux-11            1/1     Running            0             15h
kube-apiserver-bclinux-12            0/1     CrashLoopBackOff   5 (16s ago)   3m38s # 此pod有问题
# 修改这个节点apiserver的静态文件
[root@bclinux-12 ~]# grep '\-\-advertise-address' /etc/kubernetes/manifests/kube-apiserver.yaml 
    - --advertise-address=172.168.80.12 # 换成ipv6的ip
[root@bclinux-12 ~]# kubectl get pod -n kube-system|grep api
kube-apiserver-bclinux-11            1/1     Running   0          15h
kube-apiserver-bclinux-12            1/1     Running   0          2m42s

3.7.3 添加node - 所以node节点

[root@bclinux-13 ~]# cat /etc/sysconfig/kubelet 
KUBELET_EXTRA_ARGS="--node-ip 1:2:3:4::13,172.168.80.13 --fail-swap-on=false"
[root@bclinux-13 ~]# kubeadm join [1:2:3:4::10]:8443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:d63a4cb2d94ee1fb7f882f14cae74f5ded7b0f187dc778130b9f3005168ed8cb --cri-socket=/var/run/cri-dockerd.sock 
.................................................
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

3.7.4 部署CNI网络 - 任意一台master节点

calico文件下载地址：https://raw.githubusercontent.com/projectcalico/calico/v3.26.0/manifests/calico.yaml

# 修改calico配置并关闭IPIP使用BGP，在同网络环境下使用qperf命令分别测试calico的BGP、IPIP、VXLAN三种模式损耗大概为：bgp损耗10%、ipip损耗30%、vxlan损耗80%；推荐使用BGP模式，但是BGP模式对网络环境有要求(网络搞不明白，可自行查找资料)
[root@bclinux-11 ~]# vim calico-3.26.0.yaml
              "type": "calico-ipam",
              "assign_ipv4": "true", # 增加
              "assign_ipv6": "true"  # 增加
........................
            # Auto-detect the BGP IP address.
            - name: IP
              value: "autodetect"
            - name: IP6
              value: "autodetect"
            # 开启ipv6的nat出口
            - name: CALICO_IPV6POOL_NAT_OUTGOING
              value: "true"
            # 选择calico绑定的网卡，如果v4和v6在俩个网卡上，则使用 IP4_AUTODETECTION_METHOD 和 IP6_AUTODETECTION_METHOD 分别配置
            - name: IP_AUTODETECTION_METHOD
              value: "interface=ens160"
            # Enable IPIP
            - name: CALICO_IPV4POOL_IPIP
              value: "Never" # 关闭ipip
            - name: CALICO_IPV6POOL_IPIP
              value: "Never" # 关闭ipip
            # Enable or Disable VXLAN on the default IP pool.
            - name: CALICO_IPV4POOL_VXLAN
              value: "Never"
            # Enable or Disable VXLAN on the default IPv6 IP pool.
            - name: CALICO_IPV6POOL_VXLAN
              value: "Never"
            # Disable IPv6 on Kubernetes.
            - name: FELIX_IPV6SUPPORT
              value: "true" # 将false改为true
[root@bclinux-11 ~]# kubectl apply -f calico-3.26.0.yaml
[root@bclinux-11 ~]# kubectl get pod -n kube-system |grep calico
calico-kube-controllers-786b679988-vm8bb   1/1     Running   0               9m42s
calico-node-d79wc                          1/1     Running   0               7m10s
calico-node-hvlwh                          1/1     Running   0               9m42s
calico-node-xt8lb                          1/1     Running   0               9m42s

3.7.5 metrics-server安装 - 任意一台master节点

下载地址：https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
镜像地址由 k8s.gcr.io/metrics-server 替换为 registry.aliyuncs.com/google_containers
增加启动参数 --kubelet-insecure-tls # 忽略证书校验

[root@bclinux-11 ~]# kubectl apply -f components.yaml
[root@bclinux-11 ~]# kubectl get pod -n kube-system |grep metrics
metrics-server-6467f9696d-926st            1/1     Running   0              35s
[root@bclinux-11 ~]# kubectl top node
NAME         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
bclinux-11   636m         31%    782Mi           57%       
bclinux-12   685m         34%    811Mi           59%       
bclinux-13   109m         5%     703Mi           51%

3.7.6 验证集群 - 任意一台master节点

# 创建deployment类型的pod，保证每个节点运行一个
[root@bclinux-11 ~]# kubectl create deployment nginx --image nginx:1.20.0 --replicas 3
# 常见nodeport类型的svc网络
[root@bclinux-11 ~]# kubectl expose deployment nginx --target-port 80 --port 80 --type NodePort
# 查看pod地址和svc地址
[root@bclinux-11 ~]# kubectl get pod,svc -o wide
NAME                         READY   STATUS    RESTARTS   AGE   IP                             NODE         NOMINATED NODE   READINESS GATES
pod/nginx-7cf478bb58-7bmpm   1/1     Running   0          85s   172:244::390e:eaa8:1a33:8703   bclinux-12   <none>           <none>
pod/nginx-7cf478bb58-rb4fz   1/1     Running   0          85s   172:244::be3f:5e5d:bf32:cf80   bclinux-13   <none>           <none>
pod/nginx-7cf478bb58-xl4zr   1/1     Running   0          85s   172:244::1f75:c21e:b2f9:de81   bclinux-11   <none>           <none>

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE   SELECTOR
service/kubernetes   ClusterIP   172:96::1      <none>        443/TCP        16h   <none>
service/nginx        NodePort    172:96::11a8   <none>        80:30587/TCP   12s   app=nginx
# 分别curl容器地址、svc地址、节点地址看看是否都可以curl通
[root@bclinux-11 ~]# curl -I6 [172:244::390e:eaa8:1a33:8703] 2>&1|grep HTTP
HTTP/1.1 200 OK
[root@bclinux-11 ~]# curl -I6 [172:244::be3f:5e5d:bf32:cf80] 2>&1|grep HTTP
HTTP/1.1 200 OK
[root@bclinux-11 ~]# curl -I6 [172:244::1f75:c21e:b2f9:de81] 2>&1|grep HTTP
HTTP/1.1 200 OK
[root@bclinux-11 ~]# curl -I6 [172:96::11a8] 2>&1|grep HTTP
HTTP/1.1 200 OK
[root@bclinux-11 ~]# curl -I6 [1:2:3:4::13]:30587 2>&1|grep HTTP
HTTP/1.1 200 OK
[root@bclinux-11 ~]# curl -I6 [1:2:3:4::12]:30587 2>&1|grep HTTP
HTTP/1.1 200 OK
[root@bclinux-11 ~]# curl -I6 [1:2:3:4::11]:30587 2>&1|grep HTTP
HTTP/1.1 200 OK

bc-linux linux euler 1.27

factorial enormous project euler

网卡系统euler

euler explicit implicit taylor