top - 10:57:15 up 556 days, 19:08, 1 user, load average: 4.16, 4.18, 4.28 Tasks: 832 total, 2 running, 830 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.0 us, 0.9 sy, 0.0 ni, 98.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 26355433+total, 3788660 free, 48625552 used, 21114012+buff/cache KiB Swap: 4194300 total, 2725628 free, 1468672 used. 35909976 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 158755 oracle 20 0 180.2g 98.2g 98.2g S 0.0 39.1 289:09.67 oraclelionrdb (LOCAL=NO) 128933 oracle 20 0 180.3g 75.4g 75.4g S 0.0 30.0 86:20.59 oraclelionrdb (LOCAL=NO) 15235 oracle 20 0 180.2g 67.9g 67.9g S 0.0 27.0 297:42.39 oraclelionrdb (LOCAL=NO) 138638 oracle 20 0 180.3g 55.1g 55.1g S 0.0 21.9 576:47.05 ora_dbw0_lionrdb 138642 oracle 20 0 180.3g 54.8g 54.7g S 0.3 21.8 540:40.62 ora_dbw2_lionrdb 138640 oracle 20 0 180.3g 54.3g 54.3g S 0.0 21.6 514:38.55 ora_dbw1_lionrdb 138644 oracle 20 0 180.3g 53.7g 53.7g S 0.0 21.4 510:26.59 ora_dbw3_lionrdb 138654 oracle 20 0 180.3g 53.3g 53.2g S 0.0 21.2 471:21.38 ora_dbw8_lionrdb 138652 oracle 20 0 180.3g 50.7g 50.7g S 0.0 20.2 488:18.52 ora_dbw7_lionrdb 277386 oracle 20 0 180.2g 50.7g 50.7g S 0.0 20.2 31:14.25 oraclelionrdb (LOCAL=NO) 138658 oracle 20 0 180.3g 50.5g 50.4g S 0.0 20.1 508:04.70 ora_dbwa_lionrdb 138656 oracle 20 0 180.3g 50.4g 50.3g S 0.0 20.0 484:43.49 ora_dbw9_lionrdb 138660 oracle 20 0 180.3g 49.7g 49.7g S 0.0 19.8 475:52.51 ora_dbwb_lionrdb 138650 oracle 20 0 180.3g 49.0g 49.0g S 0.0 19.5 482:46.65 ora_dbw6_lionrdb 138646 oracle 20 0 180.3g 48.9g 48.9g S 0.3 19.5 501:02.68 ora_dbw4_lionrdb 138648 oracle 20 0 180.3g 48.6g 48.6g S 0.3 19.3 500:10.26 ora_dbw5_lionrdb 144512 oracle 20 0 180.2g 35.1g 35.1g S 0.0 14.0 9:48.15 oraclelionrdb (LOCAL=NO) 262425 oracle 20 0 180.2g 28.7g 28.7g S 0.0 11.4 1:41.59 oraclelionrdb (LOCAL=NO) 84154 oracle 20 0 180.3g 24.0g 24.0g S 0.0 9.6 6:43.15 oraclelionrdb (LOCAL=NO) 135744 oracle 20 0 180.2g 23.2g 23.2g S 0.0 9.2 1:31.53 oraclelionrdb (LOCAL=NO) 100314 oracle 20 0 180.2g 22.3g 22.3g S 0.0 8.9 2:08.14 oraclelionrdb (LOCAL=NO) 38139 oracle 20 0 180.2g 15.8g 15.8g S 0.0 6.3 0:19.81 oraclelionrdb (LOCAL=NO) 287159 oracle 20 0 180.2g 13.2g 13.2g S 0.0 5.3 0:28.17 oraclelionrdb (LOCAL=NO) 231971 oracle 20 0 182.0g 10.4g 9.0g S 0.0 4.1 2:30.58 oraclelionrdb (LOCAL=NO) 198652 oracle 20 0 180.2g 9.5g 9.5g S 0.3 3.8 146:08.87 ora_pr0j_lionrdb
查看top -c命令发现是
oraclelionrdb (LOCAL=NO) 占用内存很高，并且内存占用相加早已超过100%但是
KiB Mem : 26355433+total, 3788660 free, 48625552 used, 21114012+buff/cache
这里显示内存还有free ，就开始怀疑是共享内存占用，于是乎对占用比较高的内存进行排查
[oracle@cn1-prd-dba-oracle81 ~]$ pmap -x 158755 158755: oraclelionrdb (LOCAL=NO) Address Kbytes RSS Dirty Mode Mapping 0000000000400000 189188 17524 0 r-x-- oracle 000000000bec0000 8 8 4 r---- oracle 000000000bec2000 1988 204 40 rw--- oracle 000000000c0b3000 352 128 128 rw--- [ anon ] 000000000d5a6000 572 136 128 rw--- [ anon ] 0000000060000000 1048576 628 628 rw-s- [ shmid=0x3e ] 00000000a0000000 104857600 62720804 62720804 rw-s- [ shmid=0x3f ] 00000019a0000000 82837504 40276144 40276144 rw-s- [ shmid=0x8000 ] 0000002d60000000 2048 4 4 rw-s- [ shmid=0x8001 ] 00007fa088191000 256 256 256 rw--- zero 00007fa0881d1000 256 256 256 rw--- zero 00007fa088211000 192 192 192 rw--- zero 00007fa088241000 128 128 128 rw--- zero 00007fa088261000 128 128 128 rw--- zero 00007fa088281000 128 128 128 rw--- zero 00007fa08b6c1000 64 64 64 rw--- zero 00007fa08b6d1000 64 64 64 rw--- zero 00007fa08b6e1000 64 64 64 rw--- zero 00007fa08b6f1000 64 64 64 rw--- zero 00007fa08b701000 64 64 64 rw--- zero 00007fa08b711000 64 64 64 rw--- zero 00007fa08b721000 64 64 64 rw--- zero 00007fa08b731000 64 64 64 rw--- zero 00007fa08b741000 40 16 0 r-x-- libnque11.so 00007fa08b74b000 1020 0 0 ----- libnque11.so 00007fa08b84a000 4 0 0 rw--- libnque11.so
这里发现大部分内存占用是
0000000060000000 1048576 628 628 rw-s- [ shmid=0x3e ] 00000000a0000000 104857600 62720804 62720804 rw-s- [ shmid=0x3f ] 00000019a0000000 82837504 40276144 40276144 rw-s- [ shmid=0x8000 ]
并且定位到这里的占用高的shmid
然后通过
[oracle@cn1-prd-dba-oracle81 ~]$ ipcs -m ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x00000000 32768 oracle 640 84825604096 258 0x4d317030 32769 oracle 640 2097152 258 0x00000000 62 oracle 640 1073741824 258 0x00000000 63 oracle 640 107374182400 258
发现oracle共享内存shmid与占用高进程对应shmid吻合，并且对
SQL> select (84825604096+2097152+1073741824+107374182400)/1024/1024/1024 G from dual; G ---------- 180.001953
发现共享内存为180G
[oracle@cn1-prd-dba-oracle81 ~]$ free -g total used free shared buff/cache available Mem: 251 46 3 169 201 34 Swap: 3 1 2 [oracle@cn1-prd-dba-oracle81 ~]$
总大小为254 sga设置为180G之多
SQL> show parameter sga NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ lock_sga boolean FALSE pre_page_sga boolean FALSE sga_max_size big integer 180G sga_target big integer 180G
这是很不合理的，但是这台机器是一个单库而且是备库，暂时对外提供读服务，所以就有点不合理了，目前思路是对sga调小，
alter system set sga_max_size=160G scope=spfile;
等时候重启吧，备库单库只读库，也不会影响业务，内存也还够用，先这样吧