oracle cloud一台小内存机器yum install/update OutofMemory的问题排查

发布时间 2023-10-02 01:31:56作者: zjhgx

一台Oracle Cloud Free Tier的机器,说是有1G内存,但free 一下只有680M,swap有1384M。yum install curl 就一直卡,过了一会就被Kill掉了。查看/var/message ,显示oom killer,yum被kill掉,内存不足。

在网上找了一些,都是些清理/var/lib/rpm/ 下面的__db.00x的文件,重新生成啥的。不是这个原因。后来又试过改在/etc/sysctl.confs加上vm_overcommit=2关掉oom,结果是fail to allocate 2xxxxxx bytes.

后来也看到说内存太小yum有可能OutofMemory,于是想不用yum,直接用rpm安装。因为我部署了一个python环境,需要安装mysqlclient, 这个必须用yum install  mysql-devel, 否则pip install mysqlclient会报错:

      raise Exception(
      Exception: Can not find valid pkg-config name.
      Specify MYSQLCLIENT_CFLAGS and MYSQLCLIENT_LDFLAGS env vars manually

但等我下载了mysql-devel的rpm包,用rpm -ivh 安装的时候又说要安装一些依赖,等安装依赖的时候又要依赖别的。

陷入了僵局。

后来又查,发现Centos 8的yum其实只是一个链接,实际用的包管理工具是dnf。后来在网上看到说dnf有个bug,在小内存机器上会需要大量的内存导致oom kill。https://discussion.fedoraproject.org/t/dnf-operations-use-large-amount-of-ram-and-may-fail-in-low-memory-environments/76389

The current design of the DNF package manager can cause it to use quite large amounts of memory (into the hundreds of megabytes) for most operations. This can mean that DNF operations fail (due to being killed by the kernel’s out-of-memory handler) in low-memory environments, especially systems or containers with 1GB or less of memory and no swap space.

Cause
This is due to parsing large amounts of repository metadata, and the exact amount of memory used varies over time and between Fedora releases as the metadata in their repositories changes.

Related Issues
Bugzilla report: #1907030 45

Workarounds
The alternative microdnf command can be used instead of dnf for most operations, and will use less memory. Of course, you somehow need to install it first. If possible, it’s a good idea to include microdnf in the initial deployment package set when deploying to low-memory environments. If it’s not installed, and you cannot install it with dnf, you may need to download the package and install it with rpm -ivh.

Another option is to create and use a swap space (a file, a partition, or a ZRAM device).

这里说可以用microdnf代替,但microdnf也是要用dnf下载。在github上下载了microdnf,想从source上build,但一些cmake工具也没装,也没build成功。后来又想到去升级下dnf,下载了一些依赖,但在删除旧dnf的时候发现还有好多依赖需要删除,那安装的时候也要安装这些依赖,疯了。

后来发现有人建议加swap文件增加虚拟内存。

If you would like to remain on the edition you have now, dnf should be able to complete when using a swap file:

fallocate -l 1G /swapfile
mkswap /swapfile
chmod 0600 /swapfile
swapon /swapfile

I would recommend disabling the swapfile after use to reduce writing to the SD card.

马上加了1G的swap,现在swap总共有2.3G。结果还是一样,感觉swap都没被用到,继续找,发现下面的解决方案。

sudo gedit /etc/sysctl.conf
在最后面添加如下语句
vm.min_free_kbytes=250000 
#更改swap配置,让系统RAM还有250000kbyte(可根据自己电脑的RAM自定义,系统的默认值很小,导致已经卡死了才会启动swap,我是8G内存所以设置250000)

于是照着改了,再把vm_overcommit=2删掉,reboot,这次yum install curl终于成功了!在后面运行yum就好像不卡了,好像就是第一次会卡。