| master1 | 10.38.0.50 |
| master2 | 10.38.0.58 |
| master3 | 10.38.0.166 |
| node1 | 10.38.0.77 |
| lb1 | 10.38.0.182 |
| lb2 | 10.38.0.18 |
| vip | 10.38.0.144 |
1.安装及配置nginx+keepalived
需要安装nginx(haproxy)+keepalived 为apiserver提供高可用master的vip。可以在master节点直接安装nginx+keepalive，但是由于80/443端口会被ingress占用，所以在本机安装只能提供apiserver的负载均衡，而使用另外的两台机器安装，就可以在nginx同时提供k8s集群所有node节点的80http和443https的负载均衡，用这个vip添加域名解析，不会有域名解析的单点故障。
由于测试服务器是华为云服务器，所以vip需要进行注册，否则不能解析，在弹性负载均衡-子网中申请虚拟IP地址，并对两台lb服务器进行绑定。
1)安装keepalived yum install -y conntrack-tools libseccomp libtool-ltdl yum -y install keepalived 2)配置keepalive 编辑keepalived配置文件 在lb1服务器: vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc script_user root enable_script_security } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id NGINX vrrp_skip_check_adv_addr vrrp_garp_interval 0 vrrp_gna_interval 0 } vrrp_script nginx_check { script "/etc/keepalived/nginx_health.sh" interval 2 weight -20 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 55 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.38.0.144 } track_script { nginx_check } } 在lb2服务器： ! Configuration File for keepalived global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc script_user root enable_script_security } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id NGINX vrrp_skip_check_adv_addr vrrp_garp_interval 0 vrrp_gna_interval 0 } vrrp_script nginx_check { script "/etc/keepalived/nginx_health.sh" interval 2 weight -20 } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 55 priority 90 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.38.0.144 } track_script { nginx_check } } 其中定义自定义资源监控脚本（vrrp_script）：nginx_check，通过调用（track_script）这个具体的脚本/etc/keepalived/nginx_health.sh来实现对nginx的监控,并根据监控的结果实现动态调整 在lb1及lb2两台服务器添加监控脚本,如果nginx进程数量为0那么重启nginx,过两秒后再次查询进程数量，如果进程数量仍为0则关闭keepalived. vim /etc/keepalived/nginx_health.sh #!/bin/bash counter=$(ps -C nginx --no-heading|wc -l) if [ "${counter}" = "0" ]; then systemctl restart nginx sleep 2 counter=$(ps -C nginx --no-heading|wc -l) if [ "${counter}" = "0" ]; then systemctl stop keepalived fi fi 3)安装nginx yum install -y pcre pcre-devel zlib zlib-devel openssl openssl-devel wget http://nginx.org/download/nginx-1.23.4.tar.gz tar -zxvf nginx-1.23.4.tar.gz cd nginx-1.23.4 ./configure --prefix=/data/nginx --with-http_stub_status_module --with-http_ssl_module --with-stream make && make install 4)配置nginx 在lb1,lb2两台服务器上 vim /data/nginx/conf/nginx.conf user root; worker_processes auto; error_log logs/error.log; pid logs/nginx.pid; #include /usr/share/nginx/modules/*.conf; events { worker_connections 1024; } stream { log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent'; access_log logs/k8s-access.log main; upstream k8s-http { server 10.38.0.50:80; server 10.38.0.58:80; server 10.38.0.166:80; server 10.38.0.77:80; } upstream k8s-https { server 10.38.0.50:443; server 10.38.0.58:443; server 10.38.0.166:443; server 10.38.0.77:443; } upstream k8s-apiserver { server 10.38.0.50:6443; server 10.38.0.58:6443; server 10.38.0.166:6443; } server { listen 80; proxy_connect_timeout 2s; proxy_timeout 5m; proxy_upload_rate 0; proxy_download_rate 0; proxy_buffer_size 4k; proxy_pass k8s-http; } server { listen 443; proxy_connect_timeout 2s; proxy_timeout 5m; proxy_upload_rate 0; proxy_download_rate 0; proxy_buffer_size 4k; proxy_pass k8s-https; } server { listen 26443; proxy_connect_timeout 2s; proxy_timeout 5m; proxy_upload_rate 0; proxy_download_rate 0; proxy_buffer_size 4k; proxy_pass k8s-apiserver; } } 添加nginx到systemd服务 vim /usr/lib/systemd/system/nginx.service [Unit] Description=nginx After=network.target [Service] Type=forking PIDFile=/data/nginx/logs/nginx.pid ExecStart=/data/nginx/sbin/nginx -c /data/nginx/conf/nginx.conf ExecReload=/data/nginx/sbin/nginx -s reload ExecStop=/data/nginx/sbin/nginx -s quit PrivateTmp=true [Install] WantedBy=multi-user.target 添加后可使用systemctl start nginx进行nginx的服务的启动