526互联

服务器nf_conntrack(CT)表满导致虚拟机丢包

发布时间 2023-10-04 17:10:11作者: ishmaelwanglin

现象

虚拟机各种奇怪丢包(TCP的连接)
然后看到虚拟机所在CVK的dmesg里，有如下:
dmesg kern -l err,warn -T (/var/log/messages里也有)

提示:
nf_conntrack: nf_conntrack: table full, dropping packet

从日志看意思是:内核 netfilter 模块 conntrack 相关参数配置不合理，导致新连接被 drop 掉

原因

The iptables connection-tracking module uses a portion of the system memory to track connections in a table. The size of this table is set when the conntrack module is loaded, and is determined automatically based on the available system RAM. For example, a system with 256MB RAM will typically have a conntrack table of 8192 entries by default.
Expect 250 to 400 bytes of memory use per active connection.
The table full, dropping packet message means that the connection tracking table is full and a new entry for a new connection cannot be made because there is no more room. As a result, the packet fails connection tracking firewall check and is dropped by the firewall. The solution is to increase the number of connection tracking entries.

解决方法

  • 查看当前的ct表行数:
cat /proc/sys/net/netfilter/nf_conntrack_count
conntrack -L |wc -l
  • 查看当前的nf_conntrack参数
sysctl net.netfilter.nf_conntrack_max
cat /proc/sys/net/netfilter/nf_conntrack_max

修复方法分两步， 一是即时生效，二是修改配置文件

即时生效

1、 在物理服务器执行: sysctl -w net.netfilter.nf_conntrack_max=1024000
2、 调整CT hash size: echo 256000 > /sys/module/nf_conntrack/parameters/hashsize

关于ct表行数大小的限制和Hash表的size比例如下:

最好是2的N次方

On RHEL 6 and newer, the rule of thumb is hashsize = conntrack_max / 4

On RHEL 5, the rule of thumb is hashsize = conntrack_max / 8

配置文件


lsmod|grep nf_conntrack
or
cat /proc/modules |grep nf_conntrack

确认是否有该模块;
然后修改配置文件: 在/etc/sysctl.conf里添加
net.netfilter.nf_conntrack_max = 1024000
在/etc/modprobe.d/下新建一个nf_conntrack.conf(文件名关系不大)的配置文件 编辑文件/etc/modprobe.d/nf_conntrack.conf，添加如下内容并保存:
options nf_conntrack expect_hashsize=256000 hashsize=256000