526互联

服务器上mysqld，java的进程Out of Memory,被kernel kill 掉了

发布时间 2023-08-11 01:16:17作者: zjhgx

/var/log/messages 里面日志如下

Aug 10 19:47:16 VM-0-7-centos kernel: 8936 total pagecache pages
Aug 10 19:47:16 VM-0-7-centos kernel: 0 pages in swap cache
Aug 10 19:47:16 VM-0-7-centos kernel: Swap cache stats: add 0, delete 0, find 0/0
Aug 10 19:47:16 VM-0-7-centos kernel: Free swap  = 0kB
Aug 10 19:47:16 VM-0-7-centos kernel: Total swap = 0kB
Aug 10 19:47:16 VM-0-7-centos kernel: 2097016 pages RAM
Aug 10 19:47:16 VM-0-7-centos kernel: 0 pages HighMem/MovableOnly
Aug 10 19:47:16 VM-0-7-centos kernel: 94855 pages reserved
Aug 10 19:47:16 VM-0-7-centos kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Aug 10 19:47:16 VM-0-7-centos kernel: [13357]    27 13357  1244419   814184    2246        0             0 mysqld
Aug 10 19:47:16 VM-0-7-centos kernel: Out of memory: Kill process 13357 (mysqld) score 407 or sacrifice child
Aug 10 19:47:16 VM-0-7-centos kernel: Killed process 13357 (mysqld), UID 27, total-vm:4977676kB, anon-rss:3256736kB, file-rss:0kB, shmem-rss:0kB
Aug 10 19:47:16 VM-0-7-centos kernel: VM Periodic Tas invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Aug 10 19:47:16 VM-0-7-centos kernel: VM Periodic Tas cpuset=/ mems_allowed=0
Aug 10 19:47:16 VM-0-7-centos kernel: CPU: 0 PID: 18863 Comm: VM Periodic Tas Kdump: loaded Not tainted 3.10.0-1127.19.1.el7.x86_64 #1
Aug 10 19:47:16 VM-0-7-centos kernel: Hardware name: Smdbmds KVM, BIOS seabios-1.9.1-qemu-project.org 04/01/2014
Aug 10 19:47:16 VM-0-7-centos kernel: Call Trace:

Aug 10 19:47:16 VM-0-7-centos kernel: Out of memory: Kill process 13388 (mysqld) score 407 or sacrifice child
Aug 10 19:47:16 VM-0-7-centos kernel: Killed process 13388 (mysqld), UID 27, total-vm:4977676kB, anon-rss:3257420kB, file-rss:0kB, shmem-rss:0kB
Aug 10 19:47:16 VM-0-7-centos kernel: logback-3 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Aug 10 19:47:16 VM-0-7-centos kernel: logback-3 cpuset=/ mems_allowed=0
Aug 10 19:47:16 VM-0-7-centos kernel: CPU: 1 PID: 26485 Comm: logback-3 Kdump: loaded Not tainted 3.10.0-1127.19.1.el7.x86_64 #1
Aug 10 19:47:16 VM-0-7-centos kernel: Hardware name: Smdbmds KVM, BIOS seabios-1.9.1-qemu-project.org 04/01/2014

Aug 10 19:47:16 VM-0-7-centos kernel: [18851]  1000 18851  1442473   397376    1068        0             0 java
Aug 10 19:47:16 VM-0-7-centos kernel: Out of memory: Kill process 18851 (java) score 199 or sacrifice child
Aug 10 19:47:16 VM-0-7-centos kernel: Killed process 18851 (java), UID 1000, total-vm:5769892kB, anon-rss:1589504kB, file-rss:0kB, shmem-rss:0kB
Aug 10 19:47:16 VM-0-7-centos kernel: node /home/hjda invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Aug 10 19:47:16 VM-0-7-centos kernel: node /home/hjda cpuset=/ mems_allowed=0
Aug 10 19:47:16 VM-0-7-centos kernel: CPU: 3 PID: 8277 Comm: node /home/hjda Kdump: loaded Not tainted 3.10.0-1127.19.1.el7.x86_64 #1
Aug 10 19:47:16 VM-0-7-centos kernel: Hardware name: Smdbmds KVM, BIOS seabios-1.9.1-qemu-project.org 04/01/2014

Aug 10 19:47:16 VM-0-7-centos kernel: Out of memory: Kill process 18855 (VM Thread) score 201 or sacrifice child
Aug 10 19:47:16 VM-0-7-centos kernel: Killed process 18855 (VM Thread), UID 1000, total-vm:5769892kB, anon-rss:1612704kB, file-rss:0kB, shmem-rss:0kB
Aug 10 19:47:16 VM-0-7-centos kernel: node /home/hjda invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Aug 10 19:47:16 VM-0-7-centos kernel: node /home/hjda cpuset=/ mems_allowed=0
Aug 10 19:47:16 VM-0-7-centos kernel: CPU: 3 PID: 8277 Comm: node /home/hjda Kdum

Aug 10 19:47:16 VM-0-7-centos kernel: Out of memory: Kill process 18863 (VM Periodic Tas) score 201 or sacrifice child
Aug 10 19:47:16 VM-0-7-centos systemd: mysqld.service: main process exited, code=killed, status=9/KILL
Aug 10 19:47:16 VM-0-7-centos systemd: Unit mysqld.service entered failed state.
Aug 10 19:47:16 VM-0-7-centos systemd: mysqld.service failed.
Aug 10 19:47:16 VM-0-7-centos systemd: mysqld.service holdoff time over, scheduling restart.
Aug 10 19:47:16 VM-0-7-centos systemd: Stopped MySQL Server.
Aug 10 19:47:16 VM-0-7-centos systemd: Starting MySQL Server...
Aug 10 19:47:26 VM-0-7-centos systemd: Started MySQL Server.

 可以看到一开始是： VM Periodic Tas invoked oom-killer ，而VM Periodic Tag可能是JVM线程的东东。 Killed process 13357 (mysqld) 是指把分数最不理想的mysqld给kill了

后面则是 logback-3 invoked oom-killer ，killed process 13388 (mysqld)。

 

不过看监控在发生oom-killer时系统内存占用大概是6.8G左右，还有空闲。但还是发生了。而且看日志之前已经发生多次。

看网上的解释：The “OOM Killer” or “Out of Memory Killer” is a process that the Linux kernel employs when the system is critically low on memory.

 

看网上有人说建立swap分区会好一些，那我只好建了一个400M的swap分区文件。