526互联

kubernetes之 Ingress及Ingress Controller资源

发布时间 2023-07-12 17:57:49作者: wang_wei123

第七部分 ingress及ingress controller配置
如何使用ingress服务，优先配置系统参数

a、编辑kubelet配置文件/etc/sysconfig/kubelet，设置其忽略Swap启用的状态错误，内容如下，
KUBELET_EXTRA_ARGS="--fail-swap-on=false"
KUBE_PROXY_MODE="ipvs"
b、开启模块
$ modprobe ip_vs
$ modprobe ip_vs_rr
$ modprobe ip_vs_wrr
$ modprobe ip_vs_sh
$ modprobe nf_conntrack_ipv4
$ lsmod |egrep "ip_vs|nf_conntrack_ipv4"

Secvice

模型：userpace、iptables、ipvs
ClusterIP、NodePort、
  NodePort：client-->NodeIP:NodePort-->ClusterIP:ServicePort-->PodIP:containerPort
  LoadBalancer
  ExtenelName:FQDN  CNAME
NO Cluster:Headless Service 
  ServiceName-->PodIP
每一个都是前一个类型的增强版。

三个核心资源：Pod，控制器，Service
Service标签选择器，通过标签来选择

先讲解Nginx ingress原理。
微服务，daemonset 污点标记，指定运行主机。专门负责集群外部调度使用。

Ingress网络走向，URL映射或做虚拟主机。

Ingress安装和部署。
4个核心部件：dns、hipster，doress，ingress-controller。
$ kubectl explain ing.spec.backend
参考：https://github.com/kubernetes/ingress-nginx,下载安装包。
[root@k8s-master ~]# mkdir ingress-nginx
[root@k8s-master ~]# cd ingress-nginx/
[root@k8s-master ingress-nginx]# for file in namespace.yaml configmap.yaml rbac.yaml with-rbac.yaml;do wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/$file;done
以上是比较旧的版本了，现在已经下载不了包了。
新版本有所变化：https://github.com/kubernetes/ingress-nginx/blob/main/deploy/static/provider/baremetal/deploy.yaml
https://kubernetes.github.io/ingress-nginx/deploy/#quick-start
复制和下载该文件，创建过程有如下报错
configmap/ingress-nginx-controller created
error: error validating "deploy.yaml": error validating data: [ValidationError(Service.spec): unknown field "ipFamilies" in io.k8s.api.core.v1.ServiceSpec, ValidationError(Service.spec): unknown field "ipFamilyPolicy" in io.k8s.api.core.v1.ServiceSpec, ValidationError(Service.spec.ports[0]): unknown field "appProtocol" in io.k8s.api.core.v1.ServicePort, ValidationError(Service.spec.ports[1]): unknown field "appProtocol" in io.k8s.api.core.v1.ServicePort]; if you choose to ignore these errors, turn validation off with --validate=false
回归旧版本的ingress配置：https://blog.csdn.net/weixin_41083358/article/details/114831714
[root@k8s-master ingress-nginx]# ls -l
总用量 20
-rw-r--r-- 1 root root 580 5月 23 2019 configmap.yaml
-rw-r--r-- 1 root root 166 5月 23 2019 namespace.yaml
-rw-r--r-- 1 root root 2866 5月 23 2019 rbac.yaml
-rw-r--r-- 1 root root 515 5月 23 2019 service-nodeport.yaml
-rw-r--r-- 1 root root 2365 5月 23 2019 with-rbac.yaml

[root@k8s-master ingress-nginx]# kubectl create -f namespace.yaml 
namespace/ingress-nginx created
[root@k8s-master ingress-nginx]# kubectl create -f configmap.yaml 
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
[root@k8s-master ingress-nginx]# kubectl create -f rbac.yaml 
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
[root@k8s-master ingress-nginx]# kubectl create -f with-rbac.yaml 
deployment.apps/nginx-ingress-controller created
[root@k8s-master ingress-nginx]# kubectl create -f service-nodeport.yaml 
The Service "ingress-nginx" is invalid: spec.ports[0].nodePort: Invalid value: 30080: provided port is already allocated

先执行namespace，等4个文件，最后一个是参考示例不用执行。

[root@k8s-master ~]# kubectl get pod -owide -n ingress-nginx                                           
NAME                                        READY   STATUS    RESTARTS   AGE     IP            NODE        NOMINATED NODE   READINESS GATES
nginx-ingress-controller-5694ccb578-lh5hm   1/1     Running   0          5m33s   10.244.2.33   k8s-node2   <none>           <none>
[root@k8s-master ~]# kubectl get deploy -n ingress-nginx           
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
nginx-ingress-controller   1/1     1            1           5m44s

创建Service ClusterIP示例，
[root@k8s-master ~]# kubectl explain ingress.spec
[root@k8s-master ~]# kubectl explain ingress.metadata.annotations
[root@k8s-master ingress-nginx]# cp service-nodeport.yaml service-nodeport-demo.yaml

[root@k8s-master ingress-nginx]# cat service-nodeport-demo.yaml 
apiVersion: v1
kind: Service
metadata:
  name: myapp-ing-s
  namespace: default
spec:
  ports:
    - name: http
      port: 80
      targetPort: 80
  selector:
    app: myapp
    release: canary
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-ing-d
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
      release: canary
  template:
    metadata:
      name: myapp
      labels:
        app: myapp
        release: canary
    spec:
      containers:
      - name: myapp
        image: ikubernetes/myapp:v2
        ports:
        - name: http
          containerPort: 80
View Code
[root@k8s-master ingress-nginx]# kubectl create -f service-nodeport-demo.yaml             
service/myapp-ing-s created
deployment.apps/myapp-ing-d created
[root@k8s-master ingress-nginx]# kubectl get svc myapp-ing-s
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
myapp-ing-s   ClusterIP   10.100.16.111   <none>        80/TCP    58s
[root@k8s-master ingress-nginx]# kubectl get deploy myapp-ing-d
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
myapp-ing-d   2/2     2            2           65s
[root@k8s-master ingress-nginx]# curl 10.100.16.111
Hello MyApp | Version: v2 | <a href="hostname.html">Pod Name</a>

创建Service NodePort访问示例，
[root@k8s-master ingress-nginx]# cat service-nodeport.yaml

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
      nodePort: 30080
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
      nodePort: 30443
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
View Code

[root@k8s-master ingress-nginx]# kubectl create -f service-nodeport.yaml
service/ingress-nginx created
[root@k8s-master ingress-nginx]# kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx NodePort 10.111.193.30 <none> 80:30080/TCP,443:30443/TCP 3m4s
[root@k8s-master ingress-nginx]# curl 10.111.193.30:80
...
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.15.10</center>
...
[root@k8s-master ingress-nginx]# curl k8s-master:30080
...
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.15.10</center>
...
以上访问出现clusterip和nodeip访问一致，说明nginx-ingress、nginx-ingress-controller部署成功。

编写发布后端应用，ingress 暴露服务。

[root@k8s-master ingress-nginx]# cat ingress_myapp.yaml               
apiVersion:  extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-app
  namespace: default
  annotations:
    app.kubernetes.io/ingress.class: "ingress-nginx"
spec:
  rules:
  - host: myapp.weiwei.com
    http:
      paths:
      - path:
        backend:
          serviceName: myapp-ing-s
          servicePort: 80
View Code
[root@k8s-master ingress-nginx]# kubectl create -f ingress_myapp.yaml        
ingress.extensions/ingress-app created
[root@k8s-master ~]# kubectl get ing ingress-app
NAME          HOSTS              ADDRESS   PORTS   AGE
ingress-app   myapp.weiwei.com             80      87s
[root@k8s-master ~]# kubectl get svc myapp-ing-s
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
myapp-ing-s   ClusterIP   10.100.16.111   <none>        80/TCP    47m
[root@k8s-master ~]# kubectl get svc myapp-ing-s -owide
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE   SELECTOR
myapp-ing-s   ClusterIP   10.100.16.111   <none>        80/TCP    47m   app=myapp,release=canary
[root@k8s-master ~]# kubectl get pods -l app=myapp -owide
NAME                           READY   STATUS    RESTARTS   AGE    IP             NODE        NOMINATED NODE   READINESS GATES
myapp-deploy-9699554f5-ccgs2   1/1     Running   0          156m   10.244.1.245   k8s-node1   <none>           <none>
myapp-deploy-9699554f5-k6w49   1/1     Running   0          156m   10.244.1.247   k8s-node1   <none>           <none>
myapp-deploy-9699554f5-xjzkg   1/1     Running   0          156m   10.244.1.244   k8s-node1   <none>           <none>
myapp-ing-6b59768b8-m7zhj      1/1     Running   0          51m    10.244.1.248   k8s-node1   <none>           <none>
myapp-ing-6b59768b8-rd548      1/1     Running   0          51m    10.244.2.34    k8s-node2   <none>           <none>
myapp-ing-d-6b59768b8-2bx4w    1/1     Running   0          49m    10.244.2.35    k8s-node2   <none>           <none>
myapp-ing-d-6b59768b8-mmssx    1/1     Running   0          49m    10.244.1.249   k8s-node1   <none>           <none>
[root@k8s-master ~]# kubectl describe ing ingress-app   
Name:             ingress-app

验证测试：
[root@k8s-node2 ~]# curl k8s-master:30080 -H "host: myapp.weiwei.com"
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
[root@k8s-node2 ~]# curl k8s-master:30080/hostname.html -H "host: myapp.weiwei.com"
myapp-ing-d-6b59768b8-2bx4w
可以在Windows上验证测试，设置hosts文件：C:\Windows\System32\drivers\etc\hosts
192.168.1.203 k8s-master myapp.weiwei.com
192.168.1.202 k8s-node1 myapp.weiwei.com
192.168.1.201 k8s-node2 myapp.weiwei.com
浏览器访问测试如下截图，

整个访问过程看下nginx配置：：
[root@k8s-master ~]# kubectl exec -it nginx-ingress-controller-5694ccb578-lh5hm -n ingress-nginx -- /bin/sh
$ ps -ef
UID PID PPID C STIME TTY TIME CMD
www-data 1 0 0 02:00 ? 00:00:00 /usr/bin/dumb-init -- /nginx-ingress-controller --configmap=ingress-ng
www-data 5 1 0 02:00 ? 00:00:42 /nginx-ingress-controller --configmap=ingress-nginx/nginx-configuratio
www-data 24 5 0 02:00 ? 00:00:01 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
www-data 241 24 0 02:59 ? 00:00:00 nginx: worker process
www-data 301 0 0 03:14 ? 00:00:00 /bin/sh
www-data 308 301 0 03:18 ? 00:00:00 ps -ef
$ more /etc/nginx/nginx.conf
$ exit

完整案例实战
解析多个主机名，或者虚拟路径
部署一个tomcat，生成环境尽量使用有调试工具的镜像，方便调试。
镜像地址：https://hub.docker.com/_/tomcat/?tab=tags&page=4，自己测试的话，选择镜像文件比较小的镜像。
Tomcat暴露服务器示意图，需要将tomcat8080、8009端口暴露出去。

[root@k8s-master ingress]# ls /root/ingress-nginx/ingress # 两个配置文件
ingress_tomcat.yaml tomcat-deploy.yaml

[root@k8s-master ingress]# cat tomcat-deploy.yaml  
apiVersion: v1
kind: Service
metadata:
  name: tomcat-svc
  namespace: default
spec:
  ports:
    - name: http
      port: 8080
      targetPort: 8080
    - name: ajp
      port: 8009
      targetPort: 8009
  selector:
    app: tomcat
    release: canary
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-deploy
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: tomcat
      release: canary
  template:
    metadata:
      name: tomcat
      labels:
        app: tomcat
        release: canary
    spec:
      containers:
      - name: tomcat
        image: tomcat:8.5.32-jre8-alpine
        ports:
        - name: http
          containerPort: 8080
        - name: ajp
          containerPort: 8009
[root@k8s-master ingress]# cat ingress_tomcat.yaml 
apiVersion:  extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-timcat
  namespace: default
  annotations:
    app.kubernetes.io/ingress.class: "ingress-tomcat"
spec:
  rules:
  - host: tomcat.weiwei.com
    http:
      paths:
      - path:
        backend:
          serviceName: tomcat-svc
          servicePort: 8080
ingress_tomcat.yaml&tomcat-deploy.yaml

项目需求：另个端口，只有8080http服务需要对外提供服务，因此Ing只需要将8080端口服务暴露对外。

[root@k8s-master ingress]# kubectl apply -f ingress_tomcat.yaml
[root@k8s-master ingress]# kubectl apply -f tomcat-deploy.yaml  
service/tomcat-svc created
deployment.apps/tomcat-deploy created
[root@k8s-master ingress]# kubectl get pods -l app=tomcat
NAME                             READY   STATUS              RESTARTS   AGE
tomcat-deploy-569fbc468d-prn82   1/1     Running             0          8m52s
tomcat-deploy-569fbc468d-vpjqc   0/1     ContainerCreating   0          8m52s
[root@k8s-master ingress]# kubectl get deploy tomcat-deploy 
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
tomcat-deploy   1/2     2            1           8m54s
[root@k8s-master ingress]# kubectl get ing -owide
NAME             HOSTS               ADDRESS   PORTS   AGE
ingress-app      myapp.weiwei.com              80      86m
ingress-timcat   tomcat.weiwei.com             80      14m
[root@k8s-master ingress]# kubectl get svc tomcat-svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
tomcat-svc   ClusterIP   10.111.214.235   <none>        8080/TCP,8009/TCP   4m41s
[root@k8s-master ingress]# kubectl get svc -n ingress-nginx 
NAME            TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx   NodePort   10.111.193.30   <none>        80:30080/TCP,443:30443/TCP   125m

[root@k8s-node2 ~]# curl k8s-master:30080 -H "host: tomcat.weiwei.com"
[root@k8s-node2 ~]# curl k8s-master:30080 -H "host: tomcat.weiwei.com" -I
HTTP/1.1 200

win浏览器设置了hosts文件访问http://tomcat.weiwei.com:30080/，可以正常访问tomcat。
自此，完成了http服务暴露出去，是明文的http服务。

配置https证书服务

https服务暴露，比较麻烦，需要配置证书和私钥。
作为自签证书，不做ca证书了，仅做测试而已。证书跟域名挂钩。
secret可以把证书明文隐藏起来了，但是这样是可以按照规律实现解密的，因此并非安全的。
制作自签证书

[root@k8s-master keys]# openssl genrsa -out tls.key 2048
[root@k8s-master keys]# openssl req -new -x509 -key tls.key -out tls.crt -subj "/C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=tomcat.weiwei.com"
[root@k8s-master keys]# kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.keyd
[root@k8s-master keys]# kubectl get secret 2 26s
[root@k8s-master keys]# kubectl describe secret tomcat-ingress-secret

将证书信息放到ing上面，
[root@k8s-master ~]# kubectl explain ing.spec.tls

[root@k8s-master ingress]# cat ingress_tomcat_tls.yaml
apiVersion:  extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-tomcat-tls
  namespace: default
  annotations:
    app.kubernetes.io/ingress.class: "ingress-tomcat"
spec:
  tls:
  - hosts:
    - tomcat.weiwei.com
    secretName: tomcat-ingress-secret
  rules:
  - host: tomcat.weiwei.com
    http:
      paths:
      - path:
        backend:
          serviceName: tomcat-svc
          servicePort: 8080

[root@k8s-master ingress]# kubectl create -f ingress_tomcat_tls.yaml
ingress.extensions/ingress-tomcat-tls created
[root@k8s-master ingress]# kubectl get pod -n ingress-nginx
NAME READY STATUS RESTARTS AGE
nginx-ingress-controller-5694ccb578-lh5hm 1/1 Running 0 6h36m
[root@k8s-master ingress]# kubectl exec -it nginx-ingress-controller-5694ccb578-lh5hm -n ingress-nginx -- /bin/sh  # 查看配置

配置如上，此时只能通过https形式，访问验证下，
[root@knode2 ~]# curl https://k8s-master:30443 -H "host: tomcat.weiwei.com" --insecure
[root@knode2 ~]# curl https://k8s-master:30443 -H "host: tomcat.weiwei.com" -k
浏览器访问，https://tomcat.weiwei.com:30443/

到此为止，ingress配置完毕。