Hello,
I have sometimes a strange crash with xen. I can't reproduce it but it is very annoying because I have to reboot the system to recover from this state.
It seems that oom-killer is invoked while the memory resources on the dom0 are nevertheless available.
oom-killer is invoked 16 times in 5 minutes and kills essentials processes for xen.
I've seen few other reports but without solution :
- https://serverfault.com/questions/809944/oom-killer-on-xenserver-2832mb-memory-free-killing-xapi
- https://github.com/QubesOS/qubes-issues/issues/3079
Do you know what happens and how I can avoid it ?
Regards,
Guillaume
---
# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster
# uname -a
Linux dom0 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2 (2020-04-29) x86_64 GNU/Linux
# xl info
host : dom0
release : 4.19.0-9-amd64
version : #1 SMP Debian 4.19.118-2 (2020-04-29)
machine : x86_64
nr_cpus : 8
max_cpu_id : 55
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 2
cpu_mhz : 2194.858
hw_caps : bfebfbff:77fef3ff:2c100800:00000121:0000000f:d19ffffb:00000008:00000100
virt_caps : hvm hvm_directio
total_memory : 32419
free_memory : 634
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 11
xen_extra : .4-pre
xen_version : 4.11.4-pre
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset :
xen_commandline : placeholder dom0_mem=1536M,max:1536M smt=true no-real-mode edd=off
cc_compiler : gcc (Debian 8.3.0-6) 8.3.0
cc_compile_by : pkg-xen-devel
cc_compile_domain : lists.alioth.debian.org
cc_compile_date : Wed Jan 8 20:16:51 UTC 2020
build_id : b6822aa1d8f867753b92985e5cb0e806e520a08c
xend_config_format : 4
00:00:01 kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
05:05:01 22608 284004 214668 32,46 8316 102200 251068 2,96 58432 81660 80
05:15:01 22896 284808 213044 32,22 8684 103448 248984 2,94 58360 82900 76
05:25:01 66968 287588 211040 31,92 9036 61232 251820 2,97 60564 39072 20
05:35:01 63436 284952 213200 32,24 9480 61368 252000 2,97 62132 39248 72
05:45:02 64312 286516 210724 31,87 9980 63644 248060 2,93 60224 41472 52
05:55:02 56204 278760 218264 33,01 10488 63804 261020 3,08 66928 41508 52
06:05:02 70992 287216 209656 31,62 11676 59344 176992 2,09 54348 48044 52
06:15:02 79360 290312 221748 32,90 8444 53268 166460 1,96 56152 45792 44
06:25:01 60116 272472 237640 35,23 9900 54124 204768 2,41 64024 51272 92
06:35:01 11744 250120 258228 38,27 30896 58276 235632 2,78 91088 60764 8
06:45:01 39680 290064 219828 32,58 2336 95716 270092 3,18 60452 59588 32
06:55:01 19604 266392 241372 35,78 2596 91760 308968 3,64 67288 68540 16
07:05:01 10768 242652 262376 38,89 2904 76696 339884 4,00 68312 64324 20
07:15:01 10740 224596 277892 41,19 3076 58724 373104 4,40 65988 58920 4
07:25:01 25764 211088 287596 42,63 1432 33444 419220 4,94 35988 65928 24
07:35:02 13664 195904 300816 44,59 2088 29332 453316 5,34 51032 57736 44
07:45:01 11092 191848 302584 44,85 1124 28296 490896 5,78 59972 49644 20
07:55:01 12436 188440 304220 45,09 1312 22716 524504 6,18 57504 40980 0
08:05:01 14040 187564 302704 44,87 912 20112 562932 6,63 52432 38116 52
00:00:01 CPU %user %nice %system %iowait %steal %idle
05:05:01 all 0,10 0,00 0,52 0,01 0,95 98,42
05:15:01 all 0,10 0,00 0,53 0,00 1,15 98,22
05:25:01 all 0,14 0,00 0,65 0,02 1,58 97,60
05:35:01 all 0,10 0,00 0,54 0,01 1,00 98,35
05:45:02 all 0,11 0,00 0,56 0,01 1,21 98,11
05:55:02 all 0,11 0,00 0,55 0,01 1,09 98,24
06:05:02 all 0,14 0,00 0,62 0,01 1,09 98,14
06:15:02 all 0,09 0,00 0,36 0,01 0,87 98,67
06:25:01 all 0,05 0,00 0,12 0,01 0,71 99,12
06:35:01 all 0,02 0,01 0,12 0,01 0,60 99,24
06:45:01 all 0,12 0,03 0,21 0,01 0,21 99,42
06:55:01 all 0,01 0,00 0,08 0,01 0,19 99,72
07:05:01 all 0,01 0,00 0,07 0,00 0,21 99,70
07:15:01 all 0,01 0,00 0,08 0,00 0,19 99,72
07:25:01 all 0,04 0,00 0,13 0,01 0,28 99,55
07:35:02 all 0,01 0,00 0,08 0,00 0,21 99,69
07:45:01 all 0,01 0,00 0,09 0,00 0,17 99,73
07:55:01 all 0,01 0,00 0,08 0,00 0,18 99,73
08:05:01 all 0,01 0,00 0,09 0,01 0,17 99,73
00:00:01 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
05:05:01 1 469 0,55 0,32 0,18 0
05:15:01 10 466 0,29 0,41 0,30 0
05:25:01 3 470 0,21 0,28 0,31 0
05:35:01 3 470 0,53 0,35 0,30 0
05:45:02 2 469 0,24 0,27 0,27 0
05:55:02 4 476 0,27 0,32 0,28 0
06:05:02 5 464 0,08 0,16 0,20 0
06:15:02 3 470 1,17 0,93 0,58 0
06:25:01 6 527 1,06 1,05 0,84 0
06:35:01 2 582 1,12 1,19 1,04 0
06:45:01 1 640 1,11 1,17 1,09 0
06:55:01 1 707 1,11 1,16 1,10 0
07:05:01 2 764 1,08 1,12 1,09 0
07:15:01 5 818 1,11 1,11 1,09 0
07:25:01 3 885 1,09 1,09 1,08 0
07:35:02 9 943 1,10 1,07 1,07 0
07:45:01 1 1002 1,00 1,03 1,06 0
07:55:01 1 1061 1,17 1,06 1,03 0
08:05:01 1 1124 1,04 1,07 1,07 0
May 20 06:03:09 c00-xen kernel: [21662622.310610] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 c00-xen kernel: [21662622.356873] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 c00-xen kernel: [21662622.369082] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 c00-xen kernel: [21662622.380073] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 c00-xen kernel: [21662622.410332] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 c00-xen kernel: [21662622.422804] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 c00-xen kernel: [21662622.433052] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 c00-xen kernel: [21662622.446260] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:06:43 c00-xen kernel: [21662836.250666] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:06:43 c00-xen kernel: [21662836.317120] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:07:45 c00-xen kernel: [21662898.516879] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:07:45 c00-xen kernel: [21662898.561508] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:07:45 c00-xen kernel: [21662898.589353] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:07:46 c00-xen kernel: [21662898.610949] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:07:46 c00-xen kernel: [21662898.624386] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:07:46 c00-xen kernel: [21662898.648854] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 dom0 kernel: [21662622.310610] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 dom0 kernel: [21662622.310611] xenwatch cpuset=/ mems_allowed=0
May 20 06:03:09 dom0 kernel: [21662622.310616] CPU: 3 PID: 54 Comm: xenwatch Not tainted 4.19.0-6-amd64 0000001 Debian 4.19.67-2
May 20 06:03:09 dom0 kernel: [21662622.310617] Hardware name: Supermicro Super Server/X11SDV-4C-TLN2F, BIOS 1.1a 05/08/2019
May 20 06:03:09 dom0 kernel: [21662622.310617] Call Trace:
May 20 06:03:09 dom0 kernel: [21662622.310625] dump_stack+0x5c/0x80
May 20 06:03:09 dom0 kernel: [21662622.310630] dump_header+0x6b/0x283
May 20 06:03:09 dom0 kernel: [21662622.310633] oom_kill_process.cold.30+0xb/0x1cf
May 20 06:03:09 dom0 kernel: [21662622.310635] ? oom_badness+0xe9/0x140
May 20 06:03:09 dom0 kernel: [21662622.310636] out_of_memory+0x1a5/0x430
May 20 06:03:09 dom0 kernel: [21662622.310638] __alloc_pages_slowpath+0xbd8/0xcb0
May 20 06:03:09 dom0 kernel: [21662622.310640] __alloc_pages_nodemask+0x28b/0x2b0
May 20 06:03:09 dom0 kernel: [21662622.310643] kmalloc_order+0x14/0x30
May 20 06:03:09 dom0 kernel: [21662622.310644] kmalloc_order_trace+0x1d/0xa0
May 20 06:03:09 dom0 kernel: [21662622.310650] read_per_ring_refs+0x1c4/0x85e [xen_blkback]
May 20 06:03:09 dom0 kernel: [21662622.310652] ? xen_blkbk_unmap_and_respond_callback+0x160/0x160 [xen_blkback]
May 20 06:03:09 dom0 kernel: [21662622.310655] ? printk+0x58/0x6f
May 20 06:03:09 dom0 kernel: [21662622.310657] ? xen_blkbk_unmap_and_respond_callback+0x160/0x160 [xen_blkback]
May 20 06:03:09 dom0 kernel: [21662622.310659] frontend_changed.cold.18+0x80/0x11e [xen_blkback]
May 20 06:03:09 dom0 kernel: [21662622.310662] ? test_reply.isra.1+0x40/0x40
May 20 06:03:09 dom0 kernel: [21662622.310663] xenwatch_thread+0xcf/0x170
May 20 06:03:09 dom0 kernel: [21662622.310666] ? finish_wait+0x80/0x80
May 20 06:03:09 dom0 kernel: [21662622.310669] kthread+0x112/0x130
May 20 06:03:09 dom0 kernel: [21662622.310671] ? kthread_bind+0x30/0x30
May 20 06:03:09 dom0 kernel: [21662622.310674] ret_from_fork+0x35/0x40
May 20 06:03:09 dom0 kernel: [21662622.310676] Mem-Info:
May 20 06:03:09 dom0 kernel: [21662622.310680] active_anon:7819 inactive_anon:8715 isolated_anon:0
May 20 06:03:09 dom0 kernel: [21662622.310680] active_file:10879 inactive_file:7328 isolated_file:0
May 20 06:03:09 dom0 kernel: [21662622.310680] unevictable:0 dirty:18 writeback:0 unstable:0
May 20 06:03:09 dom0 kernel: [21662622.310680] slab_reclaimable:42838 slab_unreclaimable:35585
May 20 06:03:09 dom0 kernel: [21662622.310680] mapped:6081 shmem:1143 pagetables:738 bounce:0
May 20 06:03:09 dom0 kernel: [21662622.310680] free:10758 free_pcp:12 free_cma:0
May 20 06:03:09 dom0 kernel: [21662622.310682] Node 0 active_anon:31276kB inactive_anon:34860kB active_file:43516kB inactive_file:29312kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:24324kB dirty:72kB writeback:0kB shmem:4572kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
May 20 06:03:09 dom0 kernel: [21662622.310683] Node 0 DMA free:5216kB min:64kB low:80kB high:96kB active_anon:4kB inactive_anon:0kB active_file:4kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15448kB mlocked:0kB kernel_stack:16kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
May 20 06:03:09 dom0 kernel: [21662622.310686] lowmem_reserve[]: 0 846 846 846 846
May 20 06:03:09 dom0 kernel: [21662622.310688] Node 0 DMA32 free:37816kB min:3688kB low:4608kB high:5528kB active_anon:31276kB inactive_anon:34924kB active_file:43560kB inactive_file:29428kB unevictable:0kB writepending:72kB present:1008000kB managed:653980kB mlocked:0kB kernel_stack:7632kB pagetables:2952kB bounce:0kB free_pcp:48kB local_pcp:48kB free_cma:0kB
May 20 06:03:09 dom0 kernel: [21662622.310690] lowmem_reserve[]: 0 0 0 0 0
May 20 06:03:09 dom0 kernel: [21662622.310693] Node 0 DMA: 72*4kB (UME) 50*8kB (UMEH) 63*16kB (UMEH) 56*32kB (UMEH) 21*64kB (UMEH) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5216kB
May 20 06:03:09 dom0 kernel: [21662622.310700] Node 0 DMA32: 5442*4kB (UMEH) 1534*8kB (UMEH) 209*16kB (UEH) 12*32kB (H) 2*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37896kB
May 20 06:03:09 dom0 kernel: [21662622.310706] 21469 total pagecache pages
May 20 06:03:09 dom0 kernel: [21662622.310708] 2069 pages in swap cache
May 20 06:03:09 dom0 kernel: [21662622.310709] Swap cache stats: add 7057724, delete 7055780, find 47987948/48848494
May 20 06:03:09 dom0 kernel: [21662622.310710] Free swap = 7701752kB
May 20 06:03:09 dom0 kernel: [21662622.310710] Total swap = 7813112kB
May 20 06:03:09 dom0 kernel: [21662622.310710] 255999 pages RAM
May 20 06:03:09 dom0 kernel: [21662622.310711] 0 pages HighMem/MovableOnly
May 20 06:03:09 dom0 kernel: [21662622.310711] 88642 pages reserved
May 20 06:03:09 dom0 kernel: [21662622.310712] 0 pages hwpoisoned
May 20 06:03:09 dom0 kernel: [21662622.310712] Unreclaimable slab info:
May 20 06:03:09 dom0 kernel: [21662622.310713] Name Used Total
May 20 06:03:09 dom0 kernel: [21662622.312767] scsi_sense_cache 52KB 52KB
May 20 06:03:09 dom0 kernel: [21662622.312769] PINGv6 97KB 97KB
May 20 06:03:09 dom0 kernel: [21662622.312770] RAWv6 97KB 97KB
May 20 06:03:09 dom0 kernel: [21662622.312770] UDPv6 222KB 222KB
May 20 06:03:09 dom0 kernel: [21662622.312772] TCPv6 220KB 220KB
May 20 06:03:09 dom0 kernel: [21662622.312772] sgpool-128 32KB 32KB
May 20 06:03:09 dom0 kernel: [21662622.312774] mqueue_inode_cache 31KB 31KB
May 20 06:03:09 dom0 kernel: [21662622.312775] posix_timers_cache 63KB 63KB
May 20 06:03:09 dom0 kernel: [21662622.312775] UNIX 256KB 256KB
May 20 06:03:09 dom0 kernel: [21662622.312776] ip4-frags 7KB 7KB
May 20 06:03:09 dom0 kernel: [21662622.312777] secpath_cache 48KB 48KB
May 20 06:03:09 dom0 kernel: [21662622.312778] xfrm_dst_cache 367KB 367KB
May 20 06:03:09 dom0 kernel: [21662622.312779] PING 99KB 99KB
May 20 06:03:09 dom0 kernel: [21662622.312780] RAW 127KB 127KB
May 20 06:03:09 dom0 kernel: [21662622.312781] tw_sock_TCP 71KB 71KB
May 20 06:03:09 dom0 kernel: [21662622.312782] request_sock_TCP 61KB 61KB
May 20 06:03:09 dom0 kernel: [21662622.312783] TCP 510KB 510KB
May 20 06:03:09 dom0 kernel: [21662622.312784] eventpoll_pwq 31KB 31KB
May 20 06:03:09 dom0 kernel: [21662622.312784] inotify_inode_mark 35KB 35KB
May 20 06:03:09 dom0 kernel: [21662622.312785] request_queue 283KB 283KB
May 20 06:03:09 dom0 kernel: [21662622.312786] blkdev_ioc 47KB 47KB
May 20 06:03:09 dom0 kernel: [21662622.312788] biovec-max 1408KB 1544KB
May 20 06:03:09 dom0 kernel: [21662622.312789] biovec-128 32KB 32KB
May 20 06:03:09 dom0 kernel: [21662622.312790] biovec-64 288KB 308KB
May 20 06:03:09 dom0 kernel: [21662622.312791] dmaengine-unmap-256 30KB 30KB
May 20 06:03:09 dom0 kernel: [21662622.312792] dmaengine-unmap-128 31KB 31KB
May 20 06:03:09 dom0 kernel: [21662622.312792] dmaengine-unmap-16 35KB 35KB
May 20 06:03:09 dom0 kernel: [21662622.312795] dmaengine-unmap-2 403KB 480KB
May 20 06:03:09 dom0 kernel: [21662622.312797] skbuff_fclone_cache 256KB 256KB
May 20 06:03:09 dom0 kernel: [21662622.312797] skbuff_head_cache 424KB 424KB
May 20 06:03:09 dom0 kernel: [21662622.312798] file_lock_cache 58KB 58KB
May 20 06:03:09 dom0 kernel: [21662622.312799] shmem_inode_cache 585KB 585KB
May 20 06:03:09 dom0 kernel: [21662622.312800] taskstats 61KB 61KB
May 20 06:03:09 dom0 kernel: [21662622.312800] proc_dir_entry 878KB 878KB
May 20 06:03:09 dom0 kernel: [21662622.312801] pde_opener 31KB 31KB
May 20 06:03:09 dom0 kernel: [21662622.312802] sigqueue 39KB 39KB
May 20 06:03:09 dom0 kernel: [21662622.312815] kernfs_node_cache 7966KB 8060KB
May 20 06:03:09 dom0 kernel: [21662622.312820] mnt_cache 514KB 622KB
May 20 06:03:09 dom0 kernel: [21662622.312821] filp 474KB 504KB
May 20 06:03:09 dom0 kernel: [21662622.312822] names_cache 256KB 256KB
May 20 06:03:09 dom0 kernel: [21662622.312823] vm_area_struct 903KB 903KB
May 20 06:03:09 dom0 kernel: [21662622.312823] mm_struct 255KB 255KB
May 20 06:03:09 dom0 kernel: [21662622.312824] files_cache 126KB 126KB
May 20 06:03:09 dom0 kernel: [21662622.312825] signal_cache 892KB 978KB
May 20 06:03:09 dom0 kernel: [21662622.312826] sighand_cache 1109KB 1144KB
May 20 06:03:09 dom0 kernel: [21662622.312829] task_struct 2293KB 2409KB
May 20 06:03:09 dom0 kernel: [21662622.312830] cred_jar 224KB 224KB
May 20 06:03:09 dom0 kernel: [21662622.312830] anon_vma_chain 388KB 388KB
May 20 06:03:09 dom0 kernel: [21662622.312831] anon_vma 549KB 549KB
May 20 06:03:09 dom0 kernel: [21662622.312832] pid 152KB 152KB
May 20 06:03:09 dom0 kernel: [21662622.312832] Acpi-Operand 1212KB 1212KB
May 20 06:03:09 dom0 kernel: [21662622.312833] Acpi-Parse 31KB 31KB
May 20 06:03:09 dom0 kernel: [21662622.312833] Acpi-State 35KB 35KB
May 20 06:03:09 dom0 kernel: [21662622.312834] Acpi-Namespace 374KB 374KB
May 20 06:03:09 dom0 kernel: [21662622.312834] numa_policy 31KB 31KB
May 20 06:03:09 dom0 kernel: [21662622.312835] trace_event_file 130KB 130KB
May 20 06:03:09 dom0 kernel: [21662622.312836] ftrace_event_field 171KB 171KB
May 20 06:03:09 dom0 kernel: [21662622.312838] pool_workqueue 140KB 216KB
May 20 06:03:09 dom0 kernel: [21662622.312839] task_group 125KB 125KB
May 20 06:03:09 dom0 kernel: [21662622.312841] kmalloc-8192 7688KB 7776KB
May 20 06:03:09 dom0 kernel: [21662622.312845] kmalloc-4096 6480KB 6592KB
May 20 06:03:09 dom0 kernel: [21662622.312847] kmalloc-2048 5554KB 5600KB
May 20 06:03:09 dom0 kernel: [21662622.312851] kmalloc-1024 4396KB 4464KB
May 20 06:03:09 dom0 kernel: [21662622.312855] kmalloc-512 2113KB 2176KB
May 20 06:03:09 dom0 kernel: [21662622.312855] kmalloc-256 728KB 728KB
May 20 06:03:09 dom0 kernel: [21662622.312856] kmalloc-192 6044KB 6044KB
May 20 06:03:09 dom0 kernel: [21662622.312860] kmalloc-128 1319KB 1332KB
May 20 06:03:09 dom0 kernel: [21662622.312913] kmalloc-96 1003KB 2421KB
May 20 06:03:09 dom0 kernel: [21662622.312925] kmalloc-64 1839KB 1884KB
May 20 06:03:09 dom0 kernel: [21662622.312926] kmalloc-32 16088KB 16088KB
May 20 06:03:09 dom0 kernel: [21662622.312926] kmalloc-16 284KB 284KB
May 20 06:03:09 dom0 kernel: [21662622.312927] kmalloc-8 100KB 100KB
May 20 06:03:09 dom0 kernel: [21662622.312928] kmem_cache_node 1344KB 1344KB
May 20 06:03:09 dom0 kernel: [21662622.312928] kmem_cache 7983KB 7983KB
May 20 06:03:09 dom0 kernel: [21662622.312930] Tasks state (memory values in pages):
May 20 06:03:09 dom0 kernel: [21662622.312930] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
May 20 06:03:09 dom0 kernel: [21662622.312945] [ 449] 0 449 5312 200 86016 666 0 systemd-logind
[...]
May 20 06:03:09 dom0 kernel: [21662622.313081] Out of memory: Kill process 32670 (xl) score 3 or sacrifice child
May 20 06:03:09 dom0 kernel: [21662622.313978] Killed process 32670 (xl) total-vm:97036kB, anon-rss:25224kB, file-rss:2536kB, shmem-rss:0kB
May 20 06:03:09 dom0 kernel: [21662622.331673] oom_reaper: reaped process 32670 (xl), now anon-rss:0kB, file-rss:24kB, shmem-rss:0kB
I have sometimes a strange crash with xen. I can't reproduce it but it is very annoying because I have to reboot the system to recover from this state.
It seems that oom-killer is invoked while the memory resources on the dom0 are nevertheless available.
oom-killer is invoked 16 times in 5 minutes and kills essentials processes for xen.
I've seen few other reports but without solution :
- https://serverfault.com/questions/809944/oom-killer-on-xenserver-2832mb-memory-free-killing-xapi
- https://github.com/QubesOS/qubes-issues/issues/3079
Do you know what happens and how I can avoid it ?
Regards,
Guillaume
---
# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster
# uname -a
Linux dom0 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2 (2020-04-29) x86_64 GNU/Linux
# xl info
host : dom0
release : 4.19.0-9-amd64
version : #1 SMP Debian 4.19.118-2 (2020-04-29)
machine : x86_64
nr_cpus : 8
max_cpu_id : 55
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 2
cpu_mhz : 2194.858
hw_caps : bfebfbff:77fef3ff:2c100800:00000121:0000000f:d19ffffb:00000008:00000100
virt_caps : hvm hvm_directio
total_memory : 32419
free_memory : 634
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 11
xen_extra : .4-pre
xen_version : 4.11.4-pre
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset :
xen_commandline : placeholder dom0_mem=1536M,max:1536M smt=true no-real-mode edd=off
cc_compiler : gcc (Debian 8.3.0-6) 8.3.0
cc_compile_by : pkg-xen-devel
cc_compile_domain : lists.alioth.debian.org
cc_compile_date : Wed Jan 8 20:16:51 UTC 2020
build_id : b6822aa1d8f867753b92985e5cb0e806e520a08c
xend_config_format : 4
00:00:01 kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
05:05:01 22608 284004 214668 32,46 8316 102200 251068 2,96 58432 81660 80
05:15:01 22896 284808 213044 32,22 8684 103448 248984 2,94 58360 82900 76
05:25:01 66968 287588 211040 31,92 9036 61232 251820 2,97 60564 39072 20
05:35:01 63436 284952 213200 32,24 9480 61368 252000 2,97 62132 39248 72
05:45:02 64312 286516 210724 31,87 9980 63644 248060 2,93 60224 41472 52
05:55:02 56204 278760 218264 33,01 10488 63804 261020 3,08 66928 41508 52
06:05:02 70992 287216 209656 31,62 11676 59344 176992 2,09 54348 48044 52
06:15:02 79360 290312 221748 32,90 8444 53268 166460 1,96 56152 45792 44
06:25:01 60116 272472 237640 35,23 9900 54124 204768 2,41 64024 51272 92
06:35:01 11744 250120 258228 38,27 30896 58276 235632 2,78 91088 60764 8
06:45:01 39680 290064 219828 32,58 2336 95716 270092 3,18 60452 59588 32
06:55:01 19604 266392 241372 35,78 2596 91760 308968 3,64 67288 68540 16
07:05:01 10768 242652 262376 38,89 2904 76696 339884 4,00 68312 64324 20
07:15:01 10740 224596 277892 41,19 3076 58724 373104 4,40 65988 58920 4
07:25:01 25764 211088 287596 42,63 1432 33444 419220 4,94 35988 65928 24
07:35:02 13664 195904 300816 44,59 2088 29332 453316 5,34 51032 57736 44
07:45:01 11092 191848 302584 44,85 1124 28296 490896 5,78 59972 49644 20
07:55:01 12436 188440 304220 45,09 1312 22716 524504 6,18 57504 40980 0
08:05:01 14040 187564 302704 44,87 912 20112 562932 6,63 52432 38116 52
00:00:01 CPU %user %nice %system %iowait %steal %idle
05:05:01 all 0,10 0,00 0,52 0,01 0,95 98,42
05:15:01 all 0,10 0,00 0,53 0,00 1,15 98,22
05:25:01 all 0,14 0,00 0,65 0,02 1,58 97,60
05:35:01 all 0,10 0,00 0,54 0,01 1,00 98,35
05:45:02 all 0,11 0,00 0,56 0,01 1,21 98,11
05:55:02 all 0,11 0,00 0,55 0,01 1,09 98,24
06:05:02 all 0,14 0,00 0,62 0,01 1,09 98,14
06:15:02 all 0,09 0,00 0,36 0,01 0,87 98,67
06:25:01 all 0,05 0,00 0,12 0,01 0,71 99,12
06:35:01 all 0,02 0,01 0,12 0,01 0,60 99,24
06:45:01 all 0,12 0,03 0,21 0,01 0,21 99,42
06:55:01 all 0,01 0,00 0,08 0,01 0,19 99,72
07:05:01 all 0,01 0,00 0,07 0,00 0,21 99,70
07:15:01 all 0,01 0,00 0,08 0,00 0,19 99,72
07:25:01 all 0,04 0,00 0,13 0,01 0,28 99,55
07:35:02 all 0,01 0,00 0,08 0,00 0,21 99,69
07:45:01 all 0,01 0,00 0,09 0,00 0,17 99,73
07:55:01 all 0,01 0,00 0,08 0,00 0,18 99,73
08:05:01 all 0,01 0,00 0,09 0,01 0,17 99,73
00:00:01 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
05:05:01 1 469 0,55 0,32 0,18 0
05:15:01 10 466 0,29 0,41 0,30 0
05:25:01 3 470 0,21 0,28 0,31 0
05:35:01 3 470 0,53 0,35 0,30 0
05:45:02 2 469 0,24 0,27 0,27 0
05:55:02 4 476 0,27 0,32 0,28 0
06:05:02 5 464 0,08 0,16 0,20 0
06:15:02 3 470 1,17 0,93 0,58 0
06:25:01 6 527 1,06 1,05 0,84 0
06:35:01 2 582 1,12 1,19 1,04 0
06:45:01 1 640 1,11 1,17 1,09 0
06:55:01 1 707 1,11 1,16 1,10 0
07:05:01 2 764 1,08 1,12 1,09 0
07:15:01 5 818 1,11 1,11 1,09 0
07:25:01 3 885 1,09 1,09 1,08 0
07:35:02 9 943 1,10 1,07 1,07 0
07:45:01 1 1002 1,00 1,03 1,06 0
07:55:01 1 1061 1,17 1,06 1,03 0
08:05:01 1 1124 1,04 1,07 1,07 0
May 20 06:03:09 c00-xen kernel: [21662622.310610] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 c00-xen kernel: [21662622.356873] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 c00-xen kernel: [21662622.369082] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 c00-xen kernel: [21662622.380073] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 c00-xen kernel: [21662622.410332] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 c00-xen kernel: [21662622.422804] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 c00-xen kernel: [21662622.433052] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 c00-xen kernel: [21662622.446260] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:06:43 c00-xen kernel: [21662836.250666] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:06:43 c00-xen kernel: [21662836.317120] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:07:45 c00-xen kernel: [21662898.516879] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:07:45 c00-xen kernel: [21662898.561508] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:07:45 c00-xen kernel: [21662898.589353] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:07:46 c00-xen kernel: [21662898.610949] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:07:46 c00-xen kernel: [21662898.624386] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:07:46 c00-xen kernel: [21662898.648854] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 dom0 kernel: [21662622.310610] xenwatch invoked oom-killer: gfp_mask=0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null), order=3, oom_score_adj=0
May 20 06:03:09 dom0 kernel: [21662622.310611] xenwatch cpuset=/ mems_allowed=0
May 20 06:03:09 dom0 kernel: [21662622.310616] CPU: 3 PID: 54 Comm: xenwatch Not tainted 4.19.0-6-amd64 0000001 Debian 4.19.67-2
May 20 06:03:09 dom0 kernel: [21662622.310617] Hardware name: Supermicro Super Server/X11SDV-4C-TLN2F, BIOS 1.1a 05/08/2019
May 20 06:03:09 dom0 kernel: [21662622.310617] Call Trace:
May 20 06:03:09 dom0 kernel: [21662622.310625] dump_stack+0x5c/0x80
May 20 06:03:09 dom0 kernel: [21662622.310630] dump_header+0x6b/0x283
May 20 06:03:09 dom0 kernel: [21662622.310633] oom_kill_process.cold.30+0xb/0x1cf
May 20 06:03:09 dom0 kernel: [21662622.310635] ? oom_badness+0xe9/0x140
May 20 06:03:09 dom0 kernel: [21662622.310636] out_of_memory+0x1a5/0x430
May 20 06:03:09 dom0 kernel: [21662622.310638] __alloc_pages_slowpath+0xbd8/0xcb0
May 20 06:03:09 dom0 kernel: [21662622.310640] __alloc_pages_nodemask+0x28b/0x2b0
May 20 06:03:09 dom0 kernel: [21662622.310643] kmalloc_order+0x14/0x30
May 20 06:03:09 dom0 kernel: [21662622.310644] kmalloc_order_trace+0x1d/0xa0
May 20 06:03:09 dom0 kernel: [21662622.310650] read_per_ring_refs+0x1c4/0x85e [xen_blkback]
May 20 06:03:09 dom0 kernel: [21662622.310652] ? xen_blkbk_unmap_and_respond_callback+0x160/0x160 [xen_blkback]
May 20 06:03:09 dom0 kernel: [21662622.310655] ? printk+0x58/0x6f
May 20 06:03:09 dom0 kernel: [21662622.310657] ? xen_blkbk_unmap_and_respond_callback+0x160/0x160 [xen_blkback]
May 20 06:03:09 dom0 kernel: [21662622.310659] frontend_changed.cold.18+0x80/0x11e [xen_blkback]
May 20 06:03:09 dom0 kernel: [21662622.310662] ? test_reply.isra.1+0x40/0x40
May 20 06:03:09 dom0 kernel: [21662622.310663] xenwatch_thread+0xcf/0x170
May 20 06:03:09 dom0 kernel: [21662622.310666] ? finish_wait+0x80/0x80
May 20 06:03:09 dom0 kernel: [21662622.310669] kthread+0x112/0x130
May 20 06:03:09 dom0 kernel: [21662622.310671] ? kthread_bind+0x30/0x30
May 20 06:03:09 dom0 kernel: [21662622.310674] ret_from_fork+0x35/0x40
May 20 06:03:09 dom0 kernel: [21662622.310676] Mem-Info:
May 20 06:03:09 dom0 kernel: [21662622.310680] active_anon:7819 inactive_anon:8715 isolated_anon:0
May 20 06:03:09 dom0 kernel: [21662622.310680] active_file:10879 inactive_file:7328 isolated_file:0
May 20 06:03:09 dom0 kernel: [21662622.310680] unevictable:0 dirty:18 writeback:0 unstable:0
May 20 06:03:09 dom0 kernel: [21662622.310680] slab_reclaimable:42838 slab_unreclaimable:35585
May 20 06:03:09 dom0 kernel: [21662622.310680] mapped:6081 shmem:1143 pagetables:738 bounce:0
May 20 06:03:09 dom0 kernel: [21662622.310680] free:10758 free_pcp:12 free_cma:0
May 20 06:03:09 dom0 kernel: [21662622.310682] Node 0 active_anon:31276kB inactive_anon:34860kB active_file:43516kB inactive_file:29312kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:24324kB dirty:72kB writeback:0kB shmem:4572kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
May 20 06:03:09 dom0 kernel: [21662622.310683] Node 0 DMA free:5216kB min:64kB low:80kB high:96kB active_anon:4kB inactive_anon:0kB active_file:4kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15448kB mlocked:0kB kernel_stack:16kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
May 20 06:03:09 dom0 kernel: [21662622.310686] lowmem_reserve[]: 0 846 846 846 846
May 20 06:03:09 dom0 kernel: [21662622.310688] Node 0 DMA32 free:37816kB min:3688kB low:4608kB high:5528kB active_anon:31276kB inactive_anon:34924kB active_file:43560kB inactive_file:29428kB unevictable:0kB writepending:72kB present:1008000kB managed:653980kB mlocked:0kB kernel_stack:7632kB pagetables:2952kB bounce:0kB free_pcp:48kB local_pcp:48kB free_cma:0kB
May 20 06:03:09 dom0 kernel: [21662622.310690] lowmem_reserve[]: 0 0 0 0 0
May 20 06:03:09 dom0 kernel: [21662622.310693] Node 0 DMA: 72*4kB (UME) 50*8kB (UMEH) 63*16kB (UMEH) 56*32kB (UMEH) 21*64kB (UMEH) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5216kB
May 20 06:03:09 dom0 kernel: [21662622.310700] Node 0 DMA32: 5442*4kB (UMEH) 1534*8kB (UMEH) 209*16kB (UEH) 12*32kB (H) 2*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37896kB
May 20 06:03:09 dom0 kernel: [21662622.310706] 21469 total pagecache pages
May 20 06:03:09 dom0 kernel: [21662622.310708] 2069 pages in swap cache
May 20 06:03:09 dom0 kernel: [21662622.310709] Swap cache stats: add 7057724, delete 7055780, find 47987948/48848494
May 20 06:03:09 dom0 kernel: [21662622.310710] Free swap = 7701752kB
May 20 06:03:09 dom0 kernel: [21662622.310710] Total swap = 7813112kB
May 20 06:03:09 dom0 kernel: [21662622.310710] 255999 pages RAM
May 20 06:03:09 dom0 kernel: [21662622.310711] 0 pages HighMem/MovableOnly
May 20 06:03:09 dom0 kernel: [21662622.310711] 88642 pages reserved
May 20 06:03:09 dom0 kernel: [21662622.310712] 0 pages hwpoisoned
May 20 06:03:09 dom0 kernel: [21662622.310712] Unreclaimable slab info:
May 20 06:03:09 dom0 kernel: [21662622.310713] Name Used Total
May 20 06:03:09 dom0 kernel: [21662622.312767] scsi_sense_cache 52KB 52KB
May 20 06:03:09 dom0 kernel: [21662622.312769] PINGv6 97KB 97KB
May 20 06:03:09 dom0 kernel: [21662622.312770] RAWv6 97KB 97KB
May 20 06:03:09 dom0 kernel: [21662622.312770] UDPv6 222KB 222KB
May 20 06:03:09 dom0 kernel: [21662622.312772] TCPv6 220KB 220KB
May 20 06:03:09 dom0 kernel: [21662622.312772] sgpool-128 32KB 32KB
May 20 06:03:09 dom0 kernel: [21662622.312774] mqueue_inode_cache 31KB 31KB
May 20 06:03:09 dom0 kernel: [21662622.312775] posix_timers_cache 63KB 63KB
May 20 06:03:09 dom0 kernel: [21662622.312775] UNIX 256KB 256KB
May 20 06:03:09 dom0 kernel: [21662622.312776] ip4-frags 7KB 7KB
May 20 06:03:09 dom0 kernel: [21662622.312777] secpath_cache 48KB 48KB
May 20 06:03:09 dom0 kernel: [21662622.312778] xfrm_dst_cache 367KB 367KB
May 20 06:03:09 dom0 kernel: [21662622.312779] PING 99KB 99KB
May 20 06:03:09 dom0 kernel: [21662622.312780] RAW 127KB 127KB
May 20 06:03:09 dom0 kernel: [21662622.312781] tw_sock_TCP 71KB 71KB
May 20 06:03:09 dom0 kernel: [21662622.312782] request_sock_TCP 61KB 61KB
May 20 06:03:09 dom0 kernel: [21662622.312783] TCP 510KB 510KB
May 20 06:03:09 dom0 kernel: [21662622.312784] eventpoll_pwq 31KB 31KB
May 20 06:03:09 dom0 kernel: [21662622.312784] inotify_inode_mark 35KB 35KB
May 20 06:03:09 dom0 kernel: [21662622.312785] request_queue 283KB 283KB
May 20 06:03:09 dom0 kernel: [21662622.312786] blkdev_ioc 47KB 47KB
May 20 06:03:09 dom0 kernel: [21662622.312788] biovec-max 1408KB 1544KB
May 20 06:03:09 dom0 kernel: [21662622.312789] biovec-128 32KB 32KB
May 20 06:03:09 dom0 kernel: [21662622.312790] biovec-64 288KB 308KB
May 20 06:03:09 dom0 kernel: [21662622.312791] dmaengine-unmap-256 30KB 30KB
May 20 06:03:09 dom0 kernel: [21662622.312792] dmaengine-unmap-128 31KB 31KB
May 20 06:03:09 dom0 kernel: [21662622.312792] dmaengine-unmap-16 35KB 35KB
May 20 06:03:09 dom0 kernel: [21662622.312795] dmaengine-unmap-2 403KB 480KB
May 20 06:03:09 dom0 kernel: [21662622.312797] skbuff_fclone_cache 256KB 256KB
May 20 06:03:09 dom0 kernel: [21662622.312797] skbuff_head_cache 424KB 424KB
May 20 06:03:09 dom0 kernel: [21662622.312798] file_lock_cache 58KB 58KB
May 20 06:03:09 dom0 kernel: [21662622.312799] shmem_inode_cache 585KB 585KB
May 20 06:03:09 dom0 kernel: [21662622.312800] taskstats 61KB 61KB
May 20 06:03:09 dom0 kernel: [21662622.312800] proc_dir_entry 878KB 878KB
May 20 06:03:09 dom0 kernel: [21662622.312801] pde_opener 31KB 31KB
May 20 06:03:09 dom0 kernel: [21662622.312802] sigqueue 39KB 39KB
May 20 06:03:09 dom0 kernel: [21662622.312815] kernfs_node_cache 7966KB 8060KB
May 20 06:03:09 dom0 kernel: [21662622.312820] mnt_cache 514KB 622KB
May 20 06:03:09 dom0 kernel: [21662622.312821] filp 474KB 504KB
May 20 06:03:09 dom0 kernel: [21662622.312822] names_cache 256KB 256KB
May 20 06:03:09 dom0 kernel: [21662622.312823] vm_area_struct 903KB 903KB
May 20 06:03:09 dom0 kernel: [21662622.312823] mm_struct 255KB 255KB
May 20 06:03:09 dom0 kernel: [21662622.312824] files_cache 126KB 126KB
May 20 06:03:09 dom0 kernel: [21662622.312825] signal_cache 892KB 978KB
May 20 06:03:09 dom0 kernel: [21662622.312826] sighand_cache 1109KB 1144KB
May 20 06:03:09 dom0 kernel: [21662622.312829] task_struct 2293KB 2409KB
May 20 06:03:09 dom0 kernel: [21662622.312830] cred_jar 224KB 224KB
May 20 06:03:09 dom0 kernel: [21662622.312830] anon_vma_chain 388KB 388KB
May 20 06:03:09 dom0 kernel: [21662622.312831] anon_vma 549KB 549KB
May 20 06:03:09 dom0 kernel: [21662622.312832] pid 152KB 152KB
May 20 06:03:09 dom0 kernel: [21662622.312832] Acpi-Operand 1212KB 1212KB
May 20 06:03:09 dom0 kernel: [21662622.312833] Acpi-Parse 31KB 31KB
May 20 06:03:09 dom0 kernel: [21662622.312833] Acpi-State 35KB 35KB
May 20 06:03:09 dom0 kernel: [21662622.312834] Acpi-Namespace 374KB 374KB
May 20 06:03:09 dom0 kernel: [21662622.312834] numa_policy 31KB 31KB
May 20 06:03:09 dom0 kernel: [21662622.312835] trace_event_file 130KB 130KB
May 20 06:03:09 dom0 kernel: [21662622.312836] ftrace_event_field 171KB 171KB
May 20 06:03:09 dom0 kernel: [21662622.312838] pool_workqueue 140KB 216KB
May 20 06:03:09 dom0 kernel: [21662622.312839] task_group 125KB 125KB
May 20 06:03:09 dom0 kernel: [21662622.312841] kmalloc-8192 7688KB 7776KB
May 20 06:03:09 dom0 kernel: [21662622.312845] kmalloc-4096 6480KB 6592KB
May 20 06:03:09 dom0 kernel: [21662622.312847] kmalloc-2048 5554KB 5600KB
May 20 06:03:09 dom0 kernel: [21662622.312851] kmalloc-1024 4396KB 4464KB
May 20 06:03:09 dom0 kernel: [21662622.312855] kmalloc-512 2113KB 2176KB
May 20 06:03:09 dom0 kernel: [21662622.312855] kmalloc-256 728KB 728KB
May 20 06:03:09 dom0 kernel: [21662622.312856] kmalloc-192 6044KB 6044KB
May 20 06:03:09 dom0 kernel: [21662622.312860] kmalloc-128 1319KB 1332KB
May 20 06:03:09 dom0 kernel: [21662622.312913] kmalloc-96 1003KB 2421KB
May 20 06:03:09 dom0 kernel: [21662622.312925] kmalloc-64 1839KB 1884KB
May 20 06:03:09 dom0 kernel: [21662622.312926] kmalloc-32 16088KB 16088KB
May 20 06:03:09 dom0 kernel: [21662622.312926] kmalloc-16 284KB 284KB
May 20 06:03:09 dom0 kernel: [21662622.312927] kmalloc-8 100KB 100KB
May 20 06:03:09 dom0 kernel: [21662622.312928] kmem_cache_node 1344KB 1344KB
May 20 06:03:09 dom0 kernel: [21662622.312928] kmem_cache 7983KB 7983KB
May 20 06:03:09 dom0 kernel: [21662622.312930] Tasks state (memory values in pages):
May 20 06:03:09 dom0 kernel: [21662622.312930] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
May 20 06:03:09 dom0 kernel: [21662622.312945] [ 449] 0 449 5312 200 86016 666 0 systemd-logind
[...]
May 20 06:03:09 dom0 kernel: [21662622.313081] Out of memory: Kill process 32670 (xl) score 3 or sacrifice child
May 20 06:03:09 dom0 kernel: [21662622.313978] Killed process 32670 (xl) total-vm:97036kB, anon-rss:25224kB, file-rss:2536kB, shmem-rss:0kB
May 20 06:03:09 dom0 kernel: [21662622.331673] oom_reaper: reaped process 32670 (xl), now anon-rss:0kB, file-rss:24kB, shmem-rss:0kB