问题描述

在Azure Kubernetes 服务中，创建一个Internal Load Balancer服务，使用以下yaml内容：

internallb.yaml

apiVersion: v1
kind: Service
metadata:
  name: ilb-myapp
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: myapp
apply yaml

kubectl apply -f internallb.yaml

查看service状态，一直保持pending

kubectl get service

## 输出结果：
NAME               TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
ilb-myapp          LoadBalancer   10.0.32.246   <pending>     80:30198/TCP   92m
kubernetes         ClusterIP      10.0.0.1      <none>        443/TCP        13h

查看service日志

"error_description":"AADSTS7000222: The provided client secret keys for app '********-****-****-****-************' are expired

kubectl describe service  ilb-myapp 

##输出结果：
Events:
  Type     Reason                    Age                  From                  Message
  ----     ------                    ----                 ----                  -------
  Warning  UpdateLoadBalancerFailed  45m (x26 over 82m)   service-controller    (combined from similar events): Error updating load balancer with new hosts map[aks-vmss000002:{} aks-vmss000003:{} aks-vmss000006:{} aks-vmss000007:{}]: shouldUpdateLoadBalancer: failed to list managed load balancers: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 401, RawError: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to http://localhost:7788/*****: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys for app '********-****-****-****-************' are expired. Visit the Azure portal to create new keys for your app: https://aka.ms/NewClientSecret, or consider using certificate credentials for added security: https://aka.ms/certCreds.\r\nTrace ID: fa5035d2-1a6e-4b5a-85d0-910862a12501\r\nCorrelation ID: a184c126-96cb-42a9-8c4b-92869210e295\r\nTimestamp: 2023-05-31 02:33:46Z","error_codes":[7000222],"timestamp":"2023-05-31 02:33:46Z","trace_id":"fa5035d2-1a6e-4b5a-85d0-910862a12501","correlation_id":"a184c126-96cb-42a9-8c4b-92869210e295","error_uri":"https://login.chinacloudapi.cn/error?code=7000222"} 
  Normal   EnsuringLoadBalancer      37s (x25 over 95m)   service-controller    Ensuring load balancer
  Warning  ListLoadBalancers         37s (x152 over 91m)  azure-cloud-provider  (combined from similar events): Retriable: false, RetryAfter: 0s, HTTPStatusCode: 401, RawError: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to http://localhost:7788/*****: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys for app '********-****-****-****-************' are expired. Visit the Azure portal to create new keys for your app: https://aka.ms/NewClientSecret, or consider using certificate credentials for added security: https://aka.ms/certCreds.\r\nTrace ID: 277e6d9c-d663-4415-bff9-ef6610dc4f01\r\nCorrelation ID: 383589a0-2dd7-4c8e-a1a3-23a48ae8bd6d\r\nTimestamp: 2023-05-31 03:19:01Z","error_codes":[7000222],"timestamp":"2023-05-31 03:19:01Z","trace_id":"277e6d9c-d663-4415-bff9-ef6610dc4f01","correlation_id":"383589a0-2dd7-4c8e-a1a3-23a48ae8bd6d","error_uri":"https://login.chinacloudapi.cn/error?code=7000222"}

问题解答

根据错误消息，已经得知是 app client secret 已经过期。那么要修复这个问题，就是更换新的 secret。操作步骤有两步：

第一步：根据错误消息中的Application ID，在Azure AD中查找到对应的注册应用

在注册应用中，重新生成新的Client Secret，复制保存新的Secret(用于第二步中)

第二步 : 使用 az aks update-credentials 命令，更新AKS集群中的密钥值(Secret Value)

  az aks update-credentials --resource-group <resource group name>  --name <AKS Cluster name>  --reset-service-principal --service-principal <App Application ID>    --client-secret  <第一步中的Secret Value>

命令参考文档：https://docs.azure.cn/zh-cn/aks/update-credentials#update-aks-cluster-with-service-principal-credentials

当命令执行完成后，在此检查 Service 状态，不在是Pending。而是分配了正确的IP地址