mq引发的线上事故【nested exception is org.springframework.amqp.AmqpIOException】-526互联

背景：

前一天晚上运维关闭了服务间公网访问，第二天早晨系统登录不进去了，重启进去也报错

原因：

mq连接不上，大量mq请求拖垮了系统；开发人员犯了两个错误(1. 内部通信用了公网地址访问， 2.mq发送请求没有做异步处理)

ERROR -Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.amqp.AmqpIOException: 
java.net.SocketTimeoutException: connect timed out] with root cause java.net.SocketTimeoutException: connect timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_191]
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_191]
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_191]
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_191]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_191]
        at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_191]
        at com.rabbitmq.client.impl.SocketFrameHandlerFactory.create(SocketFrameHandlerFactory.java:60) ~[amqp-client-5.10.0.jar!/:5.10.0]

 ERROR -Exception Processing ErrorPage[errorCode=0, location=/error] org.apache.catalina.connector.ClientAbortException: java.io.IOException: 断开的管道
        at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:309) ~[tomcat-embed-core-9.0.41.jar!/:9.0.41]
        at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:272) ~[tomcat-embed-core-9.0.41.jar!/:9.0.41]
        at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118) ~[tomcat-embed-core-9.0.41.jar!/:9.0.41]
        at org.springframework.session.web.http.OnCommittedResponseWrapper$SaveContextServletOutputStream.flush(OnCommittedResponseWrapper.java:499) ~[spring-session-core-2.4.1.jar!/:2.4.1]
        at java.io.FilterOutputStream.flush(FilterOutputStream.java:140) ~[?:1.8.0_191]
        at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1176) ~[jackson-core-2.11.3.jar!/:2.11.3]
        at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:1008) ~[jackson-databind-2.11.3.jar!/:2.11.3]
        at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:345) ~[spring-web-5.3.2.jar!/:5.3.2]
        at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:104) ~[spring-web-5.3.2.jar!/:5.3.2]
        at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:277)