近期升级SpringCloudAlibaba版本,从2021.0.1.0升级到2021.0.5
问题现象
注册到K8S平台上2.1.2版本的Nacos报错,错误信息如下:
2023-09-13 09:29:53.712 INFO 19988 — [ main] c.a.n.p.a.s.c.ClientAuthPluginManager : ClientAuthPluginManager Load ClientAuthService com.alibaba.nacos.client.auth.impl.NacosClientAuthServiceImpl success. 2023-09-13 09:29:53.712 INFO 19988 — [ main] c.a.n.p.a.s.c.ClientAuthPluginManager : ClientAuthPluginManager Load ClientAuthService com.alibaba.nacos.client.auth.ram.RamClientAuthServiceImpl success. 2023-09-13 09:30:02.660 ERROR 19988 — [ main] c.a.c.n.registry.NacosServiceRegistry : nacos registry, oneboot register failed...NacosRegistration{nacosDiscoveryProperties=NacosDiscoveryProperties{serverAddr='10.xxx:xxx23', username='nacos', password='Ld5*d7!u^Y9%', endpoint='', namespace='', watchDelay=30000, logName='', service='oneboot', weight=1.0, clusterName='DEFAULT', group='DEFAULT_GROUP', namingLoadCacheAtStart='false', metadata= Unknown macro: {IPv6=null, preserved.register.source=SPRING_CLOUD} , registerEnabled=true, ip='10.xxxx', networkInterface='', port=8081, secure=false, accessKey='', secretKey='', heartBeatInterval=null, heartBeatTimeout=null, ipDeleteTimeout=null, instanceEnabled=true, ephemeral=true, failureToleranceEnabled=false}, ipDeleteTimeout=null, failFast=true}}, com.alibaba.nacos.api.exception.NacosException: Client not connected, current status:STARTING at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:639) ~[nacos\-client\-2.2.0.jar:na] at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:619) ~[nacos\-client\-2.2.0.jar:na] at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:356) ~[nacos\-client\-2.2.0.jar:na] at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.doRegisterService(NamingGrpcClientProxy.java:209) ~[nacos\-client\-2.2.0.jar:na] at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.registerService(NamingGrpcClientProxy.java:123) ~[nacos\-client\-2.2.0.jar:na] at com.alibaba.nacos.client.naming.remote.NamingClientProxyDelegate.registerService(NamingClientProxyDelegate.java:98) ~[nacos\-client\-2.2.0.jar:na] at com.alibaba.nacos.client.naming.NacosNamingService.registerInstance(NacosNamingService.java:152) ~[nacos\-client\-2.2.0.jar:na] at com.alibaba.cloud.nacos.registry.NacosServiceRegistry.register(NacosServiceRegistry.java:75) ~[spring\-cloud\-starter\-alibaba\-nacos\-discovery\-2021.0.5.0.jar:2021.0.5.0] at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.register(AbstractAutoServiceRegistration.java:232) [spring\-cloud\-commons\-3.1.5.jar:3.1.5] at com.alibaba.cloud.nacos.registry.NacosAutoServiceRegistration.register(NacosAutoServiceRegistration.java:78) [spring\-cloud\-starter\-alibaba\-nacos\-discovery\-2021.0.5.0.jar:2021.0.5.0] at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.start(AbstractAutoServiceRegistration.java:133) [spring\-cloud\-commons\-3.1.5.jar:3.1.5] at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.bind(AbstractAutoServiceRegistration.java:98) [spring\-cloud\-commons\-3.1.5.jar:3.1.5] at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.onApplicationEvent(AbstractAutoServiceRegistration.java:86) [spring\-cloud\-commons\-3.1.5.jar:3.1.5] at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.onApplicationEvent(AbstractAutoServiceRegistration.java:47) [spring\-cloud\-commons\-3.1.5.jar:3.1.5] at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) [spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) [spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) [spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:421) [spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378) [spring\-context\-5.3.23.jar:5.3.23] at org.springframework.boot.web.servlet.context.WebServerStartStopLifecycle.start(WebServerStartStopLifecycle.java:46) [spring\-boot\-2.6.13.jar:2.6.13] at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) [spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) [spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) [spring\-context\-5.3.23.jar:5.3.23] at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0\_144] at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) [spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) [spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935) [spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586) [spring\-context\-5.3.23.jar:5.3.23] at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring\-boot\-2.6.13.jar:2.6.13] at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:745) ~[spring\-boot\-2.6.13.jar:2.6.13] at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:420) ~[spring\-boot\-2.6.13.jar:2.6.13] at org.springframework.boot.SpringApplication.run(SpringApplication.java:307) ~[spring\-boot\-2.6.13.jar:2.6.13] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1317) ~[spring\-boot\-2.6.13.jar:2.6.13] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) ~[spring\-boot\-2.6.13.jar:2.6.13] at com.yundingtech.Main.main(Main.java:14) ~[classes/:na] 2023-09-13 09:30:02.661 WARN 19988 — [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'; nested exception is java.lang.reflect.UndeclaredThrowableException 2023-09-13 09:30:03.497 INFO 19988 — [ main] o.apache.catalina.core.StandardService : Stopping service Tomcat 2023-09-13 09:30:03.519 INFO 19988 — [ main] ConditionEvaluationReportLoggingListener : Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled. 2023-09-13 09:30:03.534 ERROR 19988 — [ main] o.s.boot.SpringApplication : Application run failed org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'; nested exception is java.lang.reflect.UndeclaredThrowableException at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:181) ~[spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) ~[spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) ~[spring\-context\-5.3.23.jar:5.3.23] at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0\_144] at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) ~[spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) ~[spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935) ~[spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586) ~[spring\-context\-5.3.23.jar:5.3.23] at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring\-boot\-2.6.13.jar:2.6.13] at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:745) [spring\-boot\-2.6.13.jar:2.6.13] at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:420) [spring\-boot\-2.6.13.jar:2.6.13] at org.springframework.boot.SpringApplication.run(SpringApplication.java:307) [spring\-boot\-2.6.13.jar:2.6.13] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1317) [spring\-boot\-2.6.13.jar:2.6.13] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) [spring\-boot\-2.6.13.jar:2.6.13] at com.yundingtech.Main.main(Main.java:14) [classes/:na] Caused by: java.lang.reflect.UndeclaredThrowableException: null at org.springframework.util.ReflectionUtils.rethrowRuntimeException(ReflectionUtils.java:147) ~[spring\-core\-5.3.23.jar:5.3.23] at com.alibaba.cloud.nacos.registry.NacosServiceRegistry.register(NacosServiceRegistry.java:83) ~[spring\-cloud\-starter\-alibaba\-nacos\-discovery\-2021.0.5.0.jar:2021.0.5.0] at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.register(AbstractAutoServiceRegistration.java:232) ~[spring\-cloud\-commons\-3.1.5.jar:3.1.5] at com.alibaba.cloud.nacos.registry.NacosAutoServiceRegistration.register(NacosAutoServiceRegistration.java:78) ~[spring\-cloud\-starter\-alibaba\-nacos\-discovery\-2021.0.5.0.jar:2021.0.5.0] at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.start(AbstractAutoServiceRegistration.java:133) ~[spring\-cloud\-commons\-3.1.5.jar:3.1.5] at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.bind(AbstractAutoServiceRegistration.java:98) ~[spring\-cloud\-commons\-3.1.5.jar:3.1.5] at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.onApplicationEvent(AbstractAutoServiceRegistration.java:86) ~[spring\-cloud\-commons\-3.1.5.jar:3.1.5] at org.springframework.cloud.client.serviceregistry.AbstractAutoServiceRegistration.onApplicationEvent(AbstractAutoServiceRegistration.java:47) ~[spring\-cloud\-commons\-3.1.5.jar:3.1.5] at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) ~[spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) ~[spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) ~[spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:421) ~[spring\-context\-5.3.23.jar:5.3.23] at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378) ~[spring\-context\-5.3.23.jar:5.3.23] at org.springframework.boot.web.servlet.context.WebServerStartStopLifecycle.start(WebServerStartStopLifecycle.java:46) ~[spring\-boot\-2.6.13.jar:2.6.13] at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) ~[spring\-context\-5.3.23.jar:5.3.23] ... 14 common frames omitted Caused by: com.alibaba.nacos.api.exception.NacosException: Client not connected, current status:STARTING at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:639) ~[nacos\-client\-2.2.0.jar:na] at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:619) ~[nacos\-client\-2.2.0.jar:na] at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:356) ~[nacos\-client\-2.2.0.jar:na] at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.doRegisterService(NamingGrpcClientProxy.java:209) ~[nacos\-client\-2.2.0.jar:na] at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.registerService(NamingGrpcClientProxy.java:123) ~[nacos\-client\-2.2.0.jar:na] at com.alibaba.nacos.client.naming.remote.NamingClientProxyDelegate.registerService(NamingClientProxyDelegate.java:98) ~[nacos\-client\-2.2.0.jar:na] at com.alibaba.nacos.client.naming.NacosNamingService.registerInstance(NacosNamingService.java:152) ~[nacos\-client\-2.2.0.jar:na] at com.alibaba.cloud.nacos.registry.NacosServiceRegistry.register(NacosServiceRegistry.java:75) ~[spring\-cloud\-starter\-alibaba\-nacos\-discovery\-2021.0.5.0.jar:2021.0.5.0] ... 27 common frames omitted 2023-09-13 09:30:03.536 WARN 19988 — [ Thread-6] c.a.nacos.common.notify.NotifyCenter : NotifyCenter Start destroying Publisher 2023-09-13 09:30:03.536 WARN 19988 — [ Thread-6] c.a.nacos.common.notify.NotifyCenter : NotifyCenter Destruction of the end 2023-09-13 09:30:03.536 WARN 19988 — [ Thread-9] c.a.n.common.http.HttpClientBeanHolder : HttpClientBeanHolder Start destroying common HttpClient Process finished with exit code 1
排查过程
1.本地部署2.1.2,注册到本地,成功。
2.降级SpringCloudAlibaba版本,到原来版本,注册到K8S,成功。
3.阅读报错信息,其大意为:建立rpc连接失败。
4.百度搜索相关报错信息关键字,结果均为配置错误导致,再次检查本地配置,无异常。
5.联系K8S同事分析Nacos服务端配置,无异常,重启Nacos后重试,问题仍未解决。
6.检查两版Nacos Client版本区别,发现客户端分别为1.X和2.X,调查两个版本的Nacos区别,发现2.X服务端比1.X多出3个端口,并且客户端对服务端的HTTP心跳改为RPC长连接。
得出结论,8848端口不提供RPC,通过别的端口提供RPC,该端口默认为9848,原因定位到,排查问题结束。
问题解决
1.尝试修改服务端配置,调整RPC端口配置,调查结果:无相关配置。
2.尝试修改客户端配置,调整其连接的RPC端口配置,调查结果:无相关配置。
3.思考其它方案若干
4.最终方案,通过其默认端口偏移量规律(+1000),调整云管映射端口。
规律:
端口
|
与主端口的偏移量
|
描述
|
---|---|---|
8848 | 0 | 主端口,客户端、控制台及OpenAPI所使用的HTTP端口 |
9848 | 1000 | 客户端gRPC请求服务端端口,用于客户端向服务端发起连接和请求 |
9849 | 1001 | 服务端gRPC请求服务端端口,用于服务间同步等 |
7848 | -1000 | Jraft请求服务端端口,用于处理服务端间的Raft相关请求 |
K8S Service映射调整结果:
(问题解决,使用新版Nacos客户端可注册K8S上的Nacos成功。)
- NacosException exception connected STARTING alibabanacosexception exception connected starting nacosexception connected starting instance connected starting current client connectionpool connection starting anaconda druid connection jdbcutils alibaba messagingexception quot exception connect sqlexception connection exception interrupt established connection exception办法 nacosexception starting