Dear list,
I have dozens of DomUs on several Debian Dom0s. I do backups of them by
taking LVM snapshots on Dom0 and attaching them to DomU. Occasionally
the block-attach fails with page allocation failure in DomU:
Sep 4 20:13:38 openvpn kernel: [696863.453080] xenwatch: page
allocation failure: order:5,
mode:0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null)
Sep 4 20:13:38 openvpn kernel: [696863.453093] xenwatch cpuset=/
mems_allowed=0
Sep 4 20:13:38 openvpn kernel: [696863.453099] CPU: 0 PID: 19 Comm:
xenwatch Not tainted 4.19.0-5-amd64 #1 Debian 4.19.37-5+deb10u2
Sep 4 20:13:38 openvpn kernel: [696863.453103] Call Trace:
Sep 4 20:13:38 openvpn kernel: [696863.453111] dump_stack+0x5c/0x80
Sep 4 20:13:38 openvpn kernel: [696863.453116]
warn_alloc.cold.117+0x6c/0xec
Sep 4 20:13:38 openvpn kernel: [696863.453120]
__alloc_pages_slowpath+0xc73/0xcb0
Sep 4 20:13:38 openvpn kernel: [696863.453127] ?
test_reply.isra.1+0x40/0x40
Sep 4 20:13:38 openvpn kernel: [696863.453131] ?
test_reply.isra.1+0x40/0x40
Sep 4 20:13:38 openvpn kernel: [696863.453134]
__alloc_pages_nodemask+0x28b/0x2b0
Sep 4 20:13:38 openvpn kernel: [696863.453139] kmalloc_order+0x14/0x30
Sep 4 20:13:38 openvpn kernel: [696863.453142]
kmalloc_order_trace+0x1d/0xa0
Sep 4 20:13:38 openvpn kernel: [696863.453144] ?
test_reply.isra.1+0x40/0x40
Sep 4 20:13:38 openvpn kernel: [696863.453149]
talk_to_blkback+0xc0/0xe00 [xen_blkfront]
Sep 4 20:13:38 openvpn kernel: [696863.453152] ? xenbus_gather+0x13b/0x160
Sep 4 20:13:38 openvpn kernel: [696863.453155] ?
test_reply.isra.1+0x40/0x40
Sep 4 20:13:38 openvpn kernel: [696863.453157]
blkback_changed+0x133/0xbf1 [xen_blkfront]
Sep 4 20:13:38 openvpn kernel: [696863.453160] ?
test_reply.isra.1+0x40/0x40
Sep 4 20:13:38 openvpn kernel: [696863.453162] ?
xenbus_read_driver_state+0x39/0x60
Sep 4 20:13:38 openvpn kernel: [696863.453165] ?
test_reply.isra.1+0x40/0x40
Sep 4 20:13:38 openvpn kernel: [696863.453168] xenwatch_thread+0xcf/0x170
Sep 4 20:13:38 openvpn kernel: [696863.453173] ? finish_wait+0x80/0x80
Sep 4 20:13:38 openvpn kernel: [696863.453177] kthread+0x112/0x130
Sep 4 20:13:38 openvpn kernel: [696863.453180] ? kthread_bind+0x30/0x30
Sep 4 20:13:38 openvpn kernel: [696863.453184] ret_from_fork+0x35/0x40
Sep 4 20:13:38 openvpn kernel: [696863.453193] Mem-Info:
Sep 4 20:13:38 openvpn kernel: [696863.453197] active_anon:48394
inactive_anon:1305 isolated_anon:0
Sep 4 20:13:38 openvpn kernel: [696863.453197] active_file:19592
inactive_file:27796 isolated_file:0
Sep 4 20:13:38 openvpn kernel: [696863.453197] unevictable:0 dirty:24
writeback:0 unstable:0
Sep 4 20:13:38 openvpn kernel: [696863.453197] slab_reclaimable:8236
slab_unreclaimable:6425
Sep 4 20:13:38 openvpn kernel: [696863.453197] mapped:6505 shmem:1369
pagetables:569 bounce:0
Sep 4 20:13:38 openvpn kernel: [696863.453197] free:4144 free_pcp:0
free_cma:0
Sep 4 20:13:38 openvpn kernel: [696863.453211] Node 0
active_anon:193576kB inactive_anon:5220kB active_file:78368kB
inactive_file:111184kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB mapped:26020kB dirty:96kB writeback:0kB shmem:5476kB
shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB
unstable:0kB all_unreclaimable? no
Sep 4 20:13:38 openvpn kernel: [696863.453219] Node 0 DMA free:8456kB
min:100kB low:124kB high:148kB active_anon:2592kB inactive_anon:0kB
active_file:1988kB inactive_file:1480kB unevictable:0kB writepending:8kB
present:15996kB managed:15912kB mlocked:0kB kernel_stack:16kB
pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 4 20:13:38 openvpn kernel: [696863.453230] lowmem_reserve[]: 0 377
377 377 377
Sep 4 20:13:38 openvpn kernel: [696863.453233] Node 0 DMA32 free:8120kB
min:2432kB low:3040kB high:3648kB active_anon:190984kB
inactive_anon:5220kB active_file:76380kB inactive_file:109704kB
unevictable:0kB writepending:88kB present:507904kB managed:477256kB
mlocked:0kB kernel_stack:2064kB pagetables:2276kB bounce:0kB
free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 4 20:13:38 openvpn kernel: [696863.453244] lowmem_reserve[]: 0 0 0 0 0
Sep 4 20:13:38 openvpn kernel: [696863.453248] Node 0 DMA: 506*4kB (UM)
346*8kB (UM) 147*16kB (UM) 35*32kB (UM) 3*64kB (UM) 0*128kB 0*256kB
0*512kB 0*1024kB 0*2048kB 0*4096kB = 8456kB
Sep 4 20:13:38 openvpn kernel: [696863.453258] Node 0 DMA32: 379*4kB
(UMEH) 446*8kB (UMEH) 96*16kB (UMEH) 29*32kB (UMEH) 9*64kB (UH) 0*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8124kB
Sep 4 20:13:38 openvpn kernel: [696863.453268] 48760 total pagecache pages
Sep 4 20:13:38 openvpn kernel: [696863.453272] 0 pages in swap cache
Sep 4 20:13:38 openvpn kernel: [696863.453274] Swap cache stats: add 0,
delete 0, find 0/0
Sep 4 20:13:38 openvpn kernel: [696863.453276] Free swap = 0kB
Sep 4 20:13:38 openvpn kernel: [696863.453277] Total swap = 0kB
Sep 4 20:13:38 openvpn kernel: [696863.453279] 130975 pages RAM
Sep 4 20:13:38 openvpn kernel: [696863.453282] 0 pages HighMem/MovableOnly
Sep 4 20:13:38 openvpn kernel: [696863.453283] 7683 pages reserved
Sep 4 20:13:38 openvpn kernel: [696863.453285] 0 pages hwpoisoned
Sep 4 20:13:38 openvpn kernel: [696863.453289] vbd vbd-268441857: 12
allocating ring_info structure
Sep 4 20:13:38 openvpn kernel: [696863.453347] vbd vbd-268441857:
failed to write error node for device/vbd/268441857 (12 allocating
ring_info structure)
Xen versions: 4.4.1 and 4.8.5-pre.
I can workaround the issue by re-attaching the snapshot, but how can I
fix it for good?
--
Karol J?drzejczyk
--
CONFIDENTIALITY NOTICE
This message and any attachment is intended
exclusively for the individual or entity to which it is addressed. This
communication may contain information that is proprietary, confidential,
legally privileged or otherwise exempt from disclosure. The security and
integrity of e-mails cannot be guaranteed. Any reply to this e-mail and any
attachment received may be subject to Icotera monitoring.
If you are not
the named addressee, you are not authorized to use, distribute, copy or
take any action in reliance on this message. The unauthorized use,
disclosure, or copying of this communication, or any attachment, is
strictly prohibited and may be unlawful. No waiver of confidentiality or
any applicable privilege is intended by any mistransmission. If you have
received this message in error, please notify the sender immediately by
replying to this e-mail and delete all copies of this message and any
attachments.
_______________________________________________
Xen-users mailing list
Xen-users@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-users
I have dozens of DomUs on several Debian Dom0s. I do backups of them by
taking LVM snapshots on Dom0 and attaching them to DomU. Occasionally
the block-attach fails with page allocation failure in DomU:
Sep 4 20:13:38 openvpn kernel: [696863.453080] xenwatch: page
allocation failure: order:5,
mode:0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null)
Sep 4 20:13:38 openvpn kernel: [696863.453093] xenwatch cpuset=/
mems_allowed=0
Sep 4 20:13:38 openvpn kernel: [696863.453099] CPU: 0 PID: 19 Comm:
xenwatch Not tainted 4.19.0-5-amd64 #1 Debian 4.19.37-5+deb10u2
Sep 4 20:13:38 openvpn kernel: [696863.453103] Call Trace:
Sep 4 20:13:38 openvpn kernel: [696863.453111] dump_stack+0x5c/0x80
Sep 4 20:13:38 openvpn kernel: [696863.453116]
warn_alloc.cold.117+0x6c/0xec
Sep 4 20:13:38 openvpn kernel: [696863.453120]
__alloc_pages_slowpath+0xc73/0xcb0
Sep 4 20:13:38 openvpn kernel: [696863.453127] ?
test_reply.isra.1+0x40/0x40
Sep 4 20:13:38 openvpn kernel: [696863.453131] ?
test_reply.isra.1+0x40/0x40
Sep 4 20:13:38 openvpn kernel: [696863.453134]
__alloc_pages_nodemask+0x28b/0x2b0
Sep 4 20:13:38 openvpn kernel: [696863.453139] kmalloc_order+0x14/0x30
Sep 4 20:13:38 openvpn kernel: [696863.453142]
kmalloc_order_trace+0x1d/0xa0
Sep 4 20:13:38 openvpn kernel: [696863.453144] ?
test_reply.isra.1+0x40/0x40
Sep 4 20:13:38 openvpn kernel: [696863.453149]
talk_to_blkback+0xc0/0xe00 [xen_blkfront]
Sep 4 20:13:38 openvpn kernel: [696863.453152] ? xenbus_gather+0x13b/0x160
Sep 4 20:13:38 openvpn kernel: [696863.453155] ?
test_reply.isra.1+0x40/0x40
Sep 4 20:13:38 openvpn kernel: [696863.453157]
blkback_changed+0x133/0xbf1 [xen_blkfront]
Sep 4 20:13:38 openvpn kernel: [696863.453160] ?
test_reply.isra.1+0x40/0x40
Sep 4 20:13:38 openvpn kernel: [696863.453162] ?
xenbus_read_driver_state+0x39/0x60
Sep 4 20:13:38 openvpn kernel: [696863.453165] ?
test_reply.isra.1+0x40/0x40
Sep 4 20:13:38 openvpn kernel: [696863.453168] xenwatch_thread+0xcf/0x170
Sep 4 20:13:38 openvpn kernel: [696863.453173] ? finish_wait+0x80/0x80
Sep 4 20:13:38 openvpn kernel: [696863.453177] kthread+0x112/0x130
Sep 4 20:13:38 openvpn kernel: [696863.453180] ? kthread_bind+0x30/0x30
Sep 4 20:13:38 openvpn kernel: [696863.453184] ret_from_fork+0x35/0x40
Sep 4 20:13:38 openvpn kernel: [696863.453193] Mem-Info:
Sep 4 20:13:38 openvpn kernel: [696863.453197] active_anon:48394
inactive_anon:1305 isolated_anon:0
Sep 4 20:13:38 openvpn kernel: [696863.453197] active_file:19592
inactive_file:27796 isolated_file:0
Sep 4 20:13:38 openvpn kernel: [696863.453197] unevictable:0 dirty:24
writeback:0 unstable:0
Sep 4 20:13:38 openvpn kernel: [696863.453197] slab_reclaimable:8236
slab_unreclaimable:6425
Sep 4 20:13:38 openvpn kernel: [696863.453197] mapped:6505 shmem:1369
pagetables:569 bounce:0
Sep 4 20:13:38 openvpn kernel: [696863.453197] free:4144 free_pcp:0
free_cma:0
Sep 4 20:13:38 openvpn kernel: [696863.453211] Node 0
active_anon:193576kB inactive_anon:5220kB active_file:78368kB
inactive_file:111184kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB mapped:26020kB dirty:96kB writeback:0kB shmem:5476kB
shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB
unstable:0kB all_unreclaimable? no
Sep 4 20:13:38 openvpn kernel: [696863.453219] Node 0 DMA free:8456kB
min:100kB low:124kB high:148kB active_anon:2592kB inactive_anon:0kB
active_file:1988kB inactive_file:1480kB unevictable:0kB writepending:8kB
present:15996kB managed:15912kB mlocked:0kB kernel_stack:16kB
pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 4 20:13:38 openvpn kernel: [696863.453230] lowmem_reserve[]: 0 377
377 377 377
Sep 4 20:13:38 openvpn kernel: [696863.453233] Node 0 DMA32 free:8120kB
min:2432kB low:3040kB high:3648kB active_anon:190984kB
inactive_anon:5220kB active_file:76380kB inactive_file:109704kB
unevictable:0kB writepending:88kB present:507904kB managed:477256kB
mlocked:0kB kernel_stack:2064kB pagetables:2276kB bounce:0kB
free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 4 20:13:38 openvpn kernel: [696863.453244] lowmem_reserve[]: 0 0 0 0 0
Sep 4 20:13:38 openvpn kernel: [696863.453248] Node 0 DMA: 506*4kB (UM)
346*8kB (UM) 147*16kB (UM) 35*32kB (UM) 3*64kB (UM) 0*128kB 0*256kB
0*512kB 0*1024kB 0*2048kB 0*4096kB = 8456kB
Sep 4 20:13:38 openvpn kernel: [696863.453258] Node 0 DMA32: 379*4kB
(UMEH) 446*8kB (UMEH) 96*16kB (UMEH) 29*32kB (UMEH) 9*64kB (UH) 0*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8124kB
Sep 4 20:13:38 openvpn kernel: [696863.453268] 48760 total pagecache pages
Sep 4 20:13:38 openvpn kernel: [696863.453272] 0 pages in swap cache
Sep 4 20:13:38 openvpn kernel: [696863.453274] Swap cache stats: add 0,
delete 0, find 0/0
Sep 4 20:13:38 openvpn kernel: [696863.453276] Free swap = 0kB
Sep 4 20:13:38 openvpn kernel: [696863.453277] Total swap = 0kB
Sep 4 20:13:38 openvpn kernel: [696863.453279] 130975 pages RAM
Sep 4 20:13:38 openvpn kernel: [696863.453282] 0 pages HighMem/MovableOnly
Sep 4 20:13:38 openvpn kernel: [696863.453283] 7683 pages reserved
Sep 4 20:13:38 openvpn kernel: [696863.453285] 0 pages hwpoisoned
Sep 4 20:13:38 openvpn kernel: [696863.453289] vbd vbd-268441857: 12
allocating ring_info structure
Sep 4 20:13:38 openvpn kernel: [696863.453347] vbd vbd-268441857:
failed to write error node for device/vbd/268441857 (12 allocating
ring_info structure)
Xen versions: 4.4.1 and 4.8.5-pre.
I can workaround the issue by re-attaching the snapshot, but how can I
fix it for good?
--
Karol J?drzejczyk
--
CONFIDENTIALITY NOTICE
This message and any attachment is intended
exclusively for the individual or entity to which it is addressed. This
communication may contain information that is proprietary, confidential,
legally privileged or otherwise exempt from disclosure. The security and
integrity of e-mails cannot be guaranteed. Any reply to this e-mail and any
attachment received may be subject to Icotera monitoring.
If you are not
the named addressee, you are not authorized to use, distribute, copy or
take any action in reliance on this message. The unauthorized use,
disclosure, or copying of this communication, or any attachment, is
strictly prohibited and may be unlawful. No waiver of confidentiality or
any applicable privilege is intended by any mistransmission. If you have
received this message in error, please notify the sender immediately by
replying to this e-mail and delete all copies of this message and any
attachments.
_______________________________________________
Xen-users mailing list
Xen-users@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-users