场景:
用springboot 开发的一个java程序部署到服务器上面,运行一段时间之后, 发现进程突然消失,没有错误日志打印,也没有内存溢出。查看服务器的状态还有部分内存。
原因:
一直以为是程序问题导致的内存不够,程序挂掉了,各种查找之后发现并没有内存泄漏的问题,后来通过网上搜索可能是linux自动把进程杀掉了,查看被系统杀掉进程日志的命令:
egrep -i -r 'killed process' /var/log(只有root权限才可以查看),
Binary file /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-000000000001ce8b-0005c260779456da.journal matches
Binary file /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-000000000000b262-00059e40d6bfec98.journal matches
Binary file /var/log/journal/20190711105006363114529432776998/system.journal matches
/var/log/messages:May 17 15:02:36 int-004 kernel: Killed process 24439 (java) total-vm:11058588kB, anon-rss:4500796kB, file-rss:0kB, shmem-rss:0kB
/var/log/messages:May 17 15:02:36 int-004 kernel: Killed process 27355 (hz.f7e1fb5e-8c4) total-vm:11058592kB, anon-rss:4512944kB, file-rss:800kB, shmem-rss:0kB
/var/log/messages:May 17 15:02:36 int-004 kernel: Killed process 27622 (nioEventLoopGro) total-vm:11058592kB, anon-rss:4513012kB, file-rss:624kB, shmem-rss:0kB
/var/log/messages:May 17 15:47:43 int-004 kernel: Killed process 28938 (java) total-vm:11077584kB, anon-rss:4511784kB, file-rss:632kB, shmem-rss:0kB
/var/log/messages:May 17 15:47:43 int-004 kernel: Killed process 28942 (java) total-vm:11077584kB, anon-rss:4534896kB, file-rss:1448kB, shmem-rss:0kB
/var/log/messages:May 17 15:50:08 int-004 kernel: Killed process 32128 (java) total-vm:10784608kB, anon-rss:4526060kB, file-rss:1032kB, shmem-rss:0kB
/var/log/messages:May 17 15:50:08 int-004 kernel: Killed process 32130 (java) total-vm:10784608kB, anon-rss:4545492kB, file-rss:1184kB, shmem-rss:0kB
/var/log/messages:May 17 16:25:58 int-004 kernel: Killed process 1793 (java) total-vm:11031212kB, anon-rss:4321012kB, file-rss:756kB, shmem-rss:0kB
/var/log/messages:May 17 16:25:58 int-004 kernel: Killed process 1802 (java) total-vm:11031212kB, anon-rss:4342368kB, file-rss:1200kB, shmem-rss:0kB
/var/log/messages:May 17 17:05:28 int-004 kernel: Killed process 4589 (java) total-vm:11109288kB, anon-rss:5218360kB, file-rss:112kB, shmem-rss:0kB
/var/log/messages:May 17 17:05:28 int-004 kernel: Killed process 4796 (Fresh-Jet-Job-S) total-vm:11109288kB, anon-rss:5243468kB, file-rss:828kB, shmem-rss:0kB
发现确实有被杀掉的进程,而且时间也对的上,到这知道原来是linux杀掉的进程。后来通过和我们运维了解,linux 会检测内存的变化,如果发现内存使用过大的进程,会被杀掉,保护系统正常运行。
解决:
我和我们运维的同学反馈之后,他们说可以通过加大虚拟内存解决问题(当linux的内存不够的时候,会使用虚拟内存swap),我们当时服务器的swap内存为512m ,后面加到了2G,后面进程被杀掉的情况发生的就很少了(我们服务器实际内存本身就是太小,很容易就溢出)