K8S Only IPV6的创建过程之二 完整版
整体过程
1. 阿里云新增一台虚拟机, 开通IPV6.与数据库,redis实现物理隔离. 并且进行环境基本处理,安装kubeadm等组件.
2. kubeadm 部署 K8S 1.28.4 并且使用ipv6 SingleStack 的calico网络组件.
3. 修改 nginx-ingress 1.9的部署文件, 修改ipv6支持.
4. 使用endpoint的方式将 mysql/redis 等服务通过service方式暴露到IPV6的K8S内部.
5. 修改myapp的配置文件, 将redis和mysql服务修改为服务名模式, 并且在宿主机使用mysql 的service name 重新注册.
6. docker build myapp 并且使用service的方式进行部署.
7. 使用ingress 进行服务暴露, 增加https证书处理.
8. 地址为 k8sipv6.jnxlh.online .
8.1 镜像信息
myapp v1.0 2bed9a6fcf63 7 hours ago 6.77GB
8.2 k8s的服务信息
[root@k8sipv6 ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
myapp-service NodePort 2001:db8:42:1::5040 <none> 80:32063/TCP 10h
kubernetes ClusterIP 2001:db8:42:1::1 <none> 443/TCP 10h
mysql-service NodePort 2001:db8:42:1::7b09 <none> 3306:30406/TCP 8h
redis-service NodePort 2001:db8:42:1::9a1e <none> 6378:30410/TCP 8h
8.3 ingress信息
[root@k8sipv6 ~]# kubectl get ing
NAME CLASS HOSTS ADDRESS PORTS AGE
myapp-ingress <none> k8sipv6.jnxlh.online 2408:400a:ed:d4ff:7a7b:f5dc::1 80, 443 7h54m
8.4 pod信息 IPV6
[root@k8sipv6 ~]# kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP
myapp-deployment-5487fb6775-96nhh 1/1 Running 0 6h54m 2001:db8:1234:5678:8:2:d1:790c
第一步机器处理
修改hosts
2408:400a:ed:d4ff:7a7b:f5dc::1 k8sipv6
第二步修改配置文件
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
net.ipv6.ip_forward = 1
EOF
sysctl -p
安装必备软件
docker
cri-docker
kubeadm 等
修改 cri-docker
vim /usr/lib/systemd/system/cri-docker.service
修改为
ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9
启动docker
systemctl enable --now docker
然后导入镜像
for i in `ls *.tar`; do docker load -i $i ;done
启动cri-docker
systemctl daemon-reload
systemctl restart cri-docker && systemctl enable cri-docker
初始化K8S
kubeadm init --node-name=k8sipv6 \
--image-repository=registry.aliyuncs.com/google_containers \
--cri-socket=unix:///var/run/cri-dockerd.sock \
--pod-network-cidr=2001:db8:42:0::/64 \
--service-cidr=2001:db8:42:1::/112 \
--apiserver-advertise-address=2408:400a:ed:d4ff:7a7b:f5dc::1 \
--apiserver-bind-port=6443
注意需要修改一下 节点的taint
这个适合单节点部署的测试环境
kubectl taint nodes --all node-role.kubernetes.io/control-plane-
安装calico
wget https://raw.githubusercontent.com/sgryphon/kubernetes-ipv6/main/calico-ipv6.yaml
直接可以用就可以了
kubectl apply -f calico-ipv6.yaml
nginx-ingress的处理
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.9.0/deploy/static/provider/baremetal/deploy.yaml
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20230407
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.9.0
注意 可以先拉去镜像, 然后在修改 deploy.yaml 里面的镜像信息 再执行 deploy
注意 要将里面的IPv4 修改为IPv6
ipFamilies:
- IPv6
ipFamilyPolicy: SingleStack
效果
kubectl get pods -A -owide
NAMESPACE NAME READY STATUS RESTARTS AGE IP
ingress-nginx ingress-nginx-admission-create-6ggqn 0/1 Completed 0 106s 2001:db8:1234:5678:8:2:d1:7903
ingress-nginx ingress-nginx-admission-patch-2nnfq 0/1 Completed 0 106s 2001:db8:1234:5678:8:2:d1:7904
ingress-nginx ingress-nginx-controller-7c76845d84-6t6f8 1/1 Running 0 106s 2408:400a:ed:d4ff:7a7b:f5dc:a12f
kube-system calico-kube-controllers-77455c6d7c-cg6gr 1/1 Running 0 5m1s 2001:db8:1234:5678:8:2:d1:7902
kube-system calico-node-dthhl 1/1 Running 0 5m1s 2408:400a:ed:d4ff:7a7b:f5dc:a12f
kube-system coredns-66f779496c-dzh5c 1/1 Running 0 5m14s 2001:db8:1234:5678:8:2:d1:7900
kube-system coredns-66f779496c-lrmn7 1/1 Running 0 5m14s 2001:db8:1234:5678:8:2:d1:7901
kube-system etcd-k8sipv6 1/1 Running 0 5m28s 2408:400a:ed:d4ff:7a7b:f5dc:a12f
kube-system kube-apiserver-k8sipv6 1/1 Running 0 5m29s 2408:400a:ed:d4ff:7a7b:f5dc:a12f
kube-system kube-controller-manager-k8sipv6 1/1 Running 0 5m29s 2408:400a:ed:d4ff:7a7b:f5dc:a12f
kube-system kube-proxy-v8ztl 1/1 Running 0 5m13s 2408:400a:ed:d4ff:7a7b:f5dc:a12f
kube-system kube-scheduler-k8sipv6 1/1 Running 0 5m28s 2408:400a:ed:d4ff:7a7b:f5dc:a12f
redis 与 mysql的处理
apiVersion: v1
kind: Service
metadata:
name: redis-service
spec:
ports:
- port: 6378
type: NodePort
---
apiVersion: v1
kind: Endpoints
metadata:
name: redis-service
subsets:
- addresses:
- ip: 2408:400a:ed:d4ff:7a7b:f5dc::2
ports:
- port: 6378
部署service和ingress
kubectl create secret tls k8sipv6_secret --key=k8sipv6.jnxlh.online.key --cert=k8sipv6.jnxlh.online.pem
cat > myapp.service <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: myapp
name: myapp-deployment
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- image: myapp:v1.0
ports:
- containerPort: 5200
name: myapp
---
apiVersion: v1
kind: Service
metadata:
labels:
app: myapp
name: myapp-service
namespace: default
spec:
sessionAffinity: ClientIP
ports:
- port: 80
name: myapp-service
protocol: TCP
targetPort: 5200
selector:
app: myapp
type: NodePort
EOF
部署 ingress
cat >ingress.yaml<<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
tls:
- hosts:
- k8sipv6.jnxlh.online
secretName: k8sipv6secret
rules:
- host: k8sipv6.jnxlh.online
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: myapp-service
port:
number: 80
EOF
kubectl apply -f ingress2.yaml
如果没有证书 可能会报错: 可以通过下面的命令来解决:
kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission
Warning: annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
Error from server (InternalError): error when creating "ingress2.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": no service port 443 found for service "ingress-nginx-controller-admission"