注册到K8S上的Nacos报错com.alibaba.nacos.api.exception.NacosException: Client not connected, current status:STARTING

发布时间 2023-09-13 10:10:42作者: 剑握在手

近期升级SpringCloudAlibaba版本,从2021.0.1.0升级到2021.0.5

问题现象

注册到K8S平台上2.1.2版本的Nacos报错,错误信息如下:

2023-09-13 09:29:53.712 INFO 19988 — [ main] c.a.n.p.a.s.c.ClientAuthPluginManager : ClientAuthPluginManager Load ClientAuthService com.alibaba.nacos.client.auth.impl.NacosClientAuthServiceImpl success.
2023-09-13 09:29:53.712 INFO 19988 — [ main] c.a.n.p.a.s.c.ClientAuthPluginManager : ClientAuthPluginManager Load ClientAuthService com.alibaba.nacos.client.auth.ram.RamClientAuthServiceImpl success.
2023-09-13 09:30:02.660 ERROR 19988 — [ main] c.a.c.n.registry.NacosServiceRegistry : nacos registry, oneboot register failed...NacosRegistration{nacosDiscoveryProperties=NacosDiscoveryProperties{serverAddr='10.xxx:xxx23', username='nacos', password='Ld5*d7!u^Y9%', endpoint='', namespace='', watchDelay=30000, logName='', service='oneboot', weight=1.0, clusterName='DEFAULT', group='DEFAULT_GROUP', namingLoadCacheAtStart='false', metadata=

Unknown macro: {IPv6=null, preserved.register.source=SPRING_CLOUD}
, registerEnabled=true, ip='10.xxxx', networkInterface='', port=8081, secure=false, accessKey='', secretKey='', heartBeatInterval=null, heartBeatTimeout=null, ipDeleteTimeout=null, instanceEnabled=true, ephemeral=true, failureToleranceEnabled=false}, ipDeleteTimeout=null, failFast=true}},

com.alibaba.nacos.api.exception.NacosException: Client not connected, current status:STARTING
at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:639) ~[nacos\-client\-2.2.0.jar:na]
at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:619) ~[nacos\-client\-2.2.0.jar:na]
at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:356) ~[nacos\-client\-2.2.0.jar:na]
at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.doRegisterService(NamingGrpcClientProxy.java:209) ~[nacos\-client\-2.2.0.jar:na]
at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.registerService(NamingGrpcClientProxy.java:123) ~[nacos\-client\-2.2.0.jar:na]
at com.alibaba.nacos.client.naming.remote.NamingClientProxyDelegate.registerService(NamingClientProxyDelegate.java:98) ~[nacos\-client\-2.2.0.jar:na]
at com.alibaba.nacos.client.naming.NacosNamingService.registerInstance(NacosNamingService.java:152) ~[nacos\-client\-2.2.0.jar:na]
at com.alibaba.cloud.nacos.registry.NacosServiceRegistry.register(NacosServiceRegistry.java:75) ~[spring\-cloud\-starter\-alibaba\-nacos\-discovery\-2021.0.5.0.jar:2021.0.5.0]
at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.register(AbstractAutoServiceRegistration.java:232) [spring\-cloud\-commons\-3.1.5.jar:3.1.5]
at com.alibaba.cloud.nacos.registry.NacosAutoServiceRegistration.register(NacosAutoServiceRegistration.java:78) [spring\-cloud\-starter\-alibaba\-nacos\-discovery\-2021.0.5.0.jar:2021.0.5.0]
at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.start(AbstractAutoServiceRegistration.java:133) [spring\-cloud\-commons\-3.1.5.jar:3.1.5]
at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.bind(AbstractAutoServiceRegistration.java:98) [spring\-cloud\-commons\-3.1.5.jar:3.1.5]
at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.onApplicationEvent(AbstractAutoServiceRegistration.java:86) [spring\-cloud\-commons\-3.1.5.jar:3.1.5]
at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.onApplicationEvent(AbstractAutoServiceRegistration.java:47) [spring\-cloud\-commons\-3.1.5.jar:3.1.5]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) [spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) [spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) [spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:421) [spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378) [spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.boot.web.servlet.context.WebServerStartStopLifecycle.start(WebServerStartStopLifecycle.java:46) [spring\-boot\-2.6.13.jar:2.6.13]
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) [spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) [spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) [spring\-context\-5.3.23.jar:5.3.23]
at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0\_144]
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) [spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) [spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935) [spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586) [spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring\-boot\-2.6.13.jar:2.6.13]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:745) ~[spring\-boot\-2.6.13.jar:2.6.13]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:420) ~[spring\-boot\-2.6.13.jar:2.6.13]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:307) ~[spring\-boot\-2.6.13.jar:2.6.13]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1317) ~[spring\-boot\-2.6.13.jar:2.6.13]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) ~[spring\-boot\-2.6.13.jar:2.6.13]
at com.yundingtech.Main.main(Main.java:14) ~[classes/:na]

2023-09-13 09:30:02.661 WARN 19988 — [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'; nested exception is java.lang.reflect.UndeclaredThrowableException
2023-09-13 09:30:03.497 INFO 19988 — [ main] o.apache.catalina.core.StandardService : Stopping service Tomcat
2023-09-13 09:30:03.519 INFO 19988 — [ main] ConditionEvaluationReportLoggingListener :

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2023-09-13 09:30:03.534 ERROR 19988 — [ main] o.s.boot.SpringApplication : Application run failed

org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'; nested exception is java.lang.reflect.UndeclaredThrowableException
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:181) ~[spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) ~[spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) ~[spring\-context\-5.3.23.jar:5.3.23]
at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0\_144]
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) ~[spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) ~[spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935) ~[spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586) ~[spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring\-boot\-2.6.13.jar:2.6.13]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:745) [spring\-boot\-2.6.13.jar:2.6.13]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:420) [spring\-boot\-2.6.13.jar:2.6.13]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:307) [spring\-boot\-2.6.13.jar:2.6.13]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1317) [spring\-boot\-2.6.13.jar:2.6.13]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) [spring\-boot\-2.6.13.jar:2.6.13]
at com.yundingtech.Main.main(Main.java:14) [classes/:na]
Caused by: java.lang.reflect.UndeclaredThrowableException: null
at org.springframework.util.ReflectionUtils.rethrowRuntimeException(ReflectionUtils.java:147) ~[spring\-core\-5.3.23.jar:5.3.23]
at com.alibaba.cloud.nacos.registry.NacosServiceRegistry.register(NacosServiceRegistry.java:83) ~[spring\-cloud\-starter\-alibaba\-nacos\-discovery\-2021.0.5.0.jar:2021.0.5.0]
at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.register(AbstractAutoServiceRegistration.java:232) ~[spring\-cloud\-commons\-3.1.5.jar:3.1.5]
at com.alibaba.cloud.nacos.registry.NacosAutoServiceRegistration.register(NacosAutoServiceRegistration.java:78) ~[spring\-cloud\-starter\-alibaba\-nacos\-discovery\-2021.0.5.0.jar:2021.0.5.0]
at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.start(AbstractAutoServiceRegistration.java:133) ~[spring\-cloud\-commons\-3.1.5.jar:3.1.5]
at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.bind(AbstractAutoServiceRegistration.java:98) ~[spring\-cloud\-commons\-3.1.5.jar:3.1.5]
at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.onApplicationEvent(AbstractAutoServiceRegistration.java:86) ~[spring\-cloud\-commons\-3.1.5.jar:3.1.5]
at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.onApplicationEvent(AbstractAutoServiceRegistration.java:47) ~[spring\-cloud\-commons\-3.1.5.jar:3.1.5]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) ~[spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) ~[spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) ~[spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:421) ~[spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378) ~[spring\-context\-5.3.23.jar:5.3.23]
at org.springframework.boot.web.servlet.context.WebServerStartStopLifecycle.start(WebServerStartStopLifecycle.java:46) ~[spring\-boot\-2.6.13.jar:2.6.13]
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) ~[spring\-context\-5.3.23.jar:5.3.23]
... 14 common frames omitted
Caused by: com.alibaba.nacos.api.exception.NacosException: Client not connected, current status:STARTING
at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:639) ~[nacos\-client\-2.2.0.jar:na]
at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:619) ~[nacos\-client\-2.2.0.jar:na]
at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:356) ~[nacos\-client\-2.2.0.jar:na]
at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.doRegisterService(NamingGrpcClientProxy.java:209) ~[nacos\-client\-2.2.0.jar:na]
at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.registerService(NamingGrpcClientProxy.java:123) ~[nacos\-client\-2.2.0.jar:na]
at com.alibaba.nacos.client.naming.remote.NamingClientProxyDelegate.registerService(NamingClientProxyDelegate.java:98) ~[nacos\-client\-2.2.0.jar:na]
at com.alibaba.nacos.client.naming.NacosNamingService.registerInstance(NacosNamingService.java:152) ~[nacos\-client\-2.2.0.jar:na]
at com.alibaba.cloud.nacos.registry.NacosServiceRegistry.register(NacosServiceRegistry.java:75) ~[spring\-cloud\-starter\-alibaba\-nacos\-discovery\-2021.0.5.0.jar:2021.0.5.0]
... 27 common frames omitted

2023-09-13 09:30:03.536 WARN 19988 — [ Thread-6] c.a.nacos.common.notify.NotifyCenter : NotifyCenter Start destroying Publisher
2023-09-13 09:30:03.536 WARN 19988 — [ Thread-6] c.a.nacos.common.notify.NotifyCenter : NotifyCenter Destruction of the end
2023-09-13 09:30:03.536 WARN 19988 — [ Thread-9] c.a.n.common.http.HttpClientBeanHolder : HttpClientBeanHolder Start destroying common HttpClient

Process finished with exit code 1

 

排查过程

1.本地部署2.1.2,注册到本地,成功。

2.降级SpringCloudAlibaba版本,到原来版本,注册到K8S,成功。

3.阅读报错信息,其大意为:建立rpc连接失败。

4.百度搜索相关报错信息关键字,结果均为配置错误导致,再次检查本地配置,无异常。

5.联系K8S同事分析Nacos服务端配置,无异常,重启Nacos后重试,问题仍未解决。

6.检查两版Nacos Client版本区别,发现客户端分别为1.X和2.X,调查两个版本的Nacos区别,发现2.X服务端比1.X多出3个端口,并且客户端对服务端的HTTP心跳改为RPC长连接。

得出结论,8848端口不提供RPC,通过别的端口提供RPC,该端口默认为9848,原因定位到,排查问题结束。

 

问题解决

1.尝试修改服务端配置,调整RPC端口配置,调查结果:无相关配置。

2.尝试修改客户端配置,调整其连接的RPC端口配置,调查结果:无相关配置。

3.思考其它方案若干

4.最终方案,通过其默认端口偏移量规律(+1000),调整云管映射端口。

规律:

端口
与主端口的偏移量
描述
8848 0 主端口,客户端、控制台及OpenAPI所使用的HTTP端口
9848 1000 客户端gRPC请求服务端端口,用于客户端向服务端发起连接和请求
9849 1001 服务端gRPC请求服务端端口,用于服务间同步等
7848 -1000 Jraft请求服务端端口,用于处理服务端间的Raft相关请求

K8S Service映射调整结果:

 

 

(问题解决,使用新版Nacos客户端可注册K8S上的Nacos成功。)