526互联

kubernetes ingress金丝雀发布/灰度发布

发布时间 2023-03-22 21:11:49作者: 若-飞

1.金丝雀发布介绍

灰度发布（又名金丝雀发布）是指在黑与白之间，能够平滑过渡的一种发布方式。在其上可以进行A/B testing，即让一部分用户继续用产品特性A，一部分用户开始用产品特性B

蓝绿发布是部署一套新版本的环境，待充分测试验证以后再发布，这样比较消耗资源，所以可以先更新一部分pod或者一定比例的pod，新版本的pod运行一段时间后，如果没有报错，那么就可以逐步扩大新版本的pod的数量，并逐步完成更新。

所以可以先更新一部分pod或者一定比例的pod，新版本的pod运行一段时间后，如果没有报错，那么就可以逐步扩大新版本的pod的数量，并逐步完成更新

 

Ingress-Nginx 是一个K8S ingress工具，支持配置 Ingress Annotations 来实现不同场景下的灰度发布和测试。 Nginx Annotations 支持以下 4 种 Canary 规则：

nginx.ingress.kubernetes.io/canary-by-header：基于 Request Header 的流量切分，适用于灰度发布以及 A/B 测试。当 Request Header 设置为 always时，请求将会被一直发送到 Canary 版本；当 Request Header 设置为 never时，请求不会被发送到 Canary 入口；对于任何其他 Header 值，将忽略 Header，并通过优先级将请求与其他金丝雀规则进行优先级的比较。

nginx.ingress.kubernetes.io/canary-by-header-value：要匹配的 Request Header 的值，用于通知 Ingress 将请求路由到 Canary Ingress 中指定的服务。当 Request Header 设置为此值时，它将被路由到 Canary 入口。该规则允许用户自定义 Request Header 的值，必须与上一个 annotation (即：canary-by-header）一起使用。

nginx.ingress.kubernetes.io/canary-weight：基于服务权重的流量切分，适用于蓝绿部署，权重范围 0 - 100 按百分比将请求路由到 Canary Ingress 中指定的服务。权重为 0 意味着该金丝雀规则不会向 Canary 入口的服务发送任何请求。权重为 100 意味着所有请求都将被发送到 Canary 入口。

nginx.ingress.kubernetes.io/canary-by-cookie：基于 Cookie 的流量切分，适用于灰度发布与 A/B 测试。用于通知 Ingress 将请求路由到 Canary Ingress 中指定的服务的cookie。当 cookie 值设置为 always时，它将被路由到 Canary 入口；当 cookie 值设置为 never时，请求不会被发送到 Canary 入口；对于任何其他值，将忽略 cookie 并将请求与其他金丝雀规则进行优先级的比较。

注意：金丝雀规则按优先顺序进行如下排序：

 canary-by-header - > canary-by-cookie - > canary-weight

我们可以把以上的四个 annotation 规则可以总体划分为以下两类：

  • 基于权重的 Canary 规则：

  • 基于用户请求的 Canary 规则：

注意： Ingress-Nginx 实在0.21.0 版本 中，引入的Canary 功能，因此要确保ingress版本OK

本文演示通过nginx.ingress.kubernetes.io/canary-weight进行金丝雀测试

2.部署green的service

green_service.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-green
spec:
  replicas: 4
  selector:
    matchLabels:
      app: nginx-green
  template:
    metadata:
      labels:
        app: nginx-green
    spec:
      containers:
      - name: nginx-green
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
      volumes:
        - name: www
          configMap:
            name: nginx-green
---
apiVersion: v1
kind: ConfigMap
metadata:
    name: nginx-green
data:
  index.html: |
        green green green  -->  If you see  this message  --> The  route to green,bye ~~~;

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-green
  labels:
    app: nginx-green
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: nginx-green

部署：

kubectl apply -f green_service.yaml

 

查看pods：

qiteck@server:~/program/k8s/blue_green_split$ sudo kubectl get pod

nginx-green-78968cd575-bngtx   1/1     Running            0                79m

nginx-green-78968cd575-cv986   1/1     Running            0                79m

nginx-green-78968cd575-pkpcr   1/1     Running            0                79m

nginx-green-78968cd575-xz4dj   1/1     Running            0                79m

查看service:

qiteck@server:~/program/k8s/blue_green_split$ sudo kubectl get service 
nginx-green     ClusterIP   10.96.255.252   <none>        80/TCP                                                                                                                                              79m

 

3. 部署blue的service

 

blue_service.yaml 
 apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-blue
spec:
  replicas: 4
  selector:
    matchLabels:
      app: nginx-blue
  template:
    metadata:
      labels:
        app: nginx-blue
    spec:
      containers:
      - name: nginx-blue
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
      volumes:
        - name: www
          configMap:
            name: nginx-blue
---
apiVersion: v1
kind: ConfigMap
metadata:
    name: nginx-blue
data:
  index.html: |
        blue blue blue  --> If you see  this message  --> The  route to blue,bye ~~~;

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-blue
  labels:
    app: nginx-blue
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: nginx-blue

 

部署：

kubectl apply -f blue_service.yaml

查看pods：

qiteck@server:~/program/k8s/blue_green_split$ sudo kubectl get pods |grep nginx
nginx-blue-766b9f6df4-4ldzx    1/1     Running            0                  38m
nginx-blue-766b9f6df4-bmd85    1/1     Running            0                  38m
nginx-blue-766b9f6df4-h5bdd    1/1     Running            0                  38m
nginx-blue-766b9f6df4-qsn5w    1/1     Running            0                  38m
nginx-green-78968cd575-bngtx   1/1     Running            0                  97m
nginx-green-78968cd575-cv986   1/1     Running            0                  97m
nginx-green-78968cd575-pkpcr   1/1     Running            0                  97m
nginx-green-78968cd575-xz4dj   1/1     Running            0                  97m

查看service：

qiteck@server:~/program/k8s/blue_green_split$ sudo kubectl get service |grep nginx
nginx-blue      ClusterIP   10.96.95.252    <none>        80/TCP                                                                                                                                              39m
nginx-green     ClusterIP   10.96.255.252   <none>        80/TCP                                                                                                                                              98m

目前可以看到blue和green两套服务都是存在的

 4.部署green的ingress

green_ingress.yaml 
apiVersion:  networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-green
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: nginx.test.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-green
            port:
              number: 80

域名nginx.test.com映射到nginx_green服务：

部署：

sudo kubectl apply -f ingress.yaml

 

查看ingress：

qiteck@server:~/program/k8s/blue_green_split$ sudo kubectl get ingress
NAME    CLASS    HOSTS                 ADDRESS        PORTS   AGE
nginx   <none>   nginx.test.com   10.96.180.91   80      49s

 

5.部署blue的ingress

blue_ingress.yaml 
apiVersion:  networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-blue
  annotations:
    kubernetes.io/ingress.class: nginx     #指定ingress服务，无意义，可不写
    nginx.ingress.kubernetes.io/canary: "true"    #打开金丝雀开关
    nginx.ingress.kubernetes.io/canary-weight: "50"   #权重，按照50%的流量区分
spec:
  rules:
  - host: nginx.test.com    #和上面一样的域名
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-blue     #请求转到什么后端
            port:
              number: 80

这里设置了金丝雀开发和weight：

   nginx.ingress.kubernetes.io/canary: "true"    #打开金丝雀开关

     nginx.ingress.kubernetes.io/canary-weight: "50"   #权重，按照50%的流量区分

域名nginx.test.com映射到nginx_blue服务：

部署：

sudo kubectl apply -f ingress.yaml

 

查看ingress：

qiteck@server:~/program/k8s/blue_green_split$ sudo kubectl get ingress
NAME          CLASS    HOSTS                 ADDRESS        PORTS   AGE
nginx-blue    <none>   nginx.test.com   10.96.180.91   80      23m
nginx-green   <none>   nginx.test.com   10.96.180.91   80      37m

可以看到nginx.test.com同时影射到nginx-blue和nginx-green2服务了。

如果不设置canary的话，会提示域名和path已经存在 
 apiVersion:  networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-blue
  annotations:
    kubernetes.io/ingress.class: nginx     #指定ingress服务，无意义，可不写
spec:
  rules:
  - host: nginx.yemaodiyi.com    #和上面一样的域名
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-blue     #请求转到什么后端
            port:
              number: 80

一个path只能定位到一个service：

 

6.测试一下金丝雀权重

 

qiteck@server:~/program/k8s/blue_green_split$ for i in `seq 20`;do curl nginx.test.com;done
green green green  -->  If you see  this message  --> The  route to green,bye ~~~;
blue blue blue  --> If you see  this message  --> The  route to blue,bye ~~~;
blue blue blue  --> If you see  this message  --> The  route to blue,bye ~~~;
green green green  -->  If you see  this message  --> The  route to green,bye ~~~;
green green green  -->  If you see  this message  --> The  route to green,bye ~~~;
green green green  -->  If you see  this message  --> The  route to green,bye ~~~;
blue blue blue  --> If you see  this message  --> The  route to blue,bye ~~~;
blue blue blue  --> If you see  this message  --> The  route to blue,bye ~~~;
blue blue blue  --> If you see  this message  --> The  route to blue,bye ~~~;
green green green  -->  If you see  this message  --> The  route to green,bye ~~~;
green green green  -->  If you see  this message  --> The  route to green,bye ~~~;
blue blue blue  --> If you see  this message  --> The  route to blue,bye ~~~;
green green green  -->  If you see  this message  --> The  route to green,bye ~~~;
green green green  -->  If you see  this message  --> The  route to green,bye ~~~;
green green green  -->  If you see  this message  --> The  route to green,bye ~~~;
blue blue blue  --> If you see  this message  --> The  route to blue,bye ~~~;
green green green  -->  If you see  this message  --> The  route to green,bye ~~~;
green green green  -->  If you see  this message  --> The  route to green,bye ~~~;
green green green  -->  If you see  this message  --> The  route to green,bye ~~~;
blue blue blue  --> If you see  this message  --> The  route to blue,bye ~~~;

可以看到blue和green同时在提供服务了。

注意这里权重不是一个精确的百分比，使用过程当中，只是会看到一个近似分布。