Hi all,
I've been working on getting a nested Xen setup working. Hardware is a
Dell PowerEdge R640 with 2x Intel Xeon Gold 6150 CPU, and a dual Intel
i350 NIC.
The Dom-0 is running Debian Bookworm (Linux 6.1.0-16-amd64) with Xen
version 4.17.3-pre (Debian 4.17.2+76-ge1f9cb16e2-1~deb12u1). This is
from package xen-system-amd64 version 4.17.2+76-ge1f9cb16e2-1~deb12u1.
I have a bunch of PV Dom-U's running that work perfectly fine with PV
networking. I also created a HVM Dom-U running Debian Bookworm (Linux
6.1.0-17-amd64) to run nested Xen. This nested Xen instance has a couple
of PV Dom-U's of its own.
When I install a fresh version of Debian Bookworm on a HVM guest it
nicely switches over to virtio, regardless of whether I select rtl8139
and e1000 as the driver. However, once I install Xen on this Dom-U and
reboot into Xen, it switches to either the 8139cp driver or e1000. Is
this expected behavior? Does the Xen kernel not support paravirtualized
network devices?
Anyway, I assumed this is the case so set up the PV Dom-U's on the
Nested xen instance. It does work but network performance deteriorates
over time and at some point during heavy network utilization (a backup
is created to a remote host), a transmit queue timeout occurs. I will
append the full log from syslog below.
From this point on, no further networking is possible via this network
adapter until I reboot the nested Xen instance.
I have now switched over to the e1000 network adapter to see if this
turns out to be more stable. I am wondering if this watchdog error is a
bug and should be reported somewhere, or if this is somewhat expected in
my setup?
The config of the Xen HVM host includes these settings:
---------- type = 'hvm' uuid = "<uid>" memory=131072 vcpus = 24 pae = 1
acpi = 1 viridian = 0 apic = 1 sdl = 0 usb = 1 name = "spotter" vif = [
'bridge=xenbr0,mac=<generated mac>,type=ioemu,model=rtl8139',
'bridge=xenbr1,mac=<generated mac>,type=ioemu,model=rtl8139' ] disk = [
'phy:/dev/zvol/main/spotter,xvda,w', 'phy:/dev/nvme1n1,sda,w',
'phy:/dev/nvme2n1,sdb,w', 'phy:/dev/nvme3n1,sdc,w', ] acpi = 1 serial =
'pty' vnc = 1 vnclisten = "127.0.0.1" vncdisplay = 0 hap=1 nestedhvm=1
----------
Any input is appreciated!
Kind regards,
Egbert
> 54 2024-01-30T01:03:56.499112+01:00 spotter kernel: [1222897.732550]
> ------------[ cut here ]------------ 53
> 2024-01-30T01:03:56.499135+01:00 spotter kernel: [1222897.732568]
> NETDEV WATCHDOG: eth0 (8139cp): transmit queue 0 timed out 52
> 2024-01-30T01:03:56.499141+01:00 spotter kernel: [1222897.732585]
> WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:525
> dev_watchdog+0x207/0x210 51 2024-01-30T01:03:56.499146+01:00 spotter
> kernel: [1222897.732596] Modules linked in: xt_recent xen_netfront
> xen_netback xen_blkback xen_gntdev xen_evtchn xenfs xen_p rivcmd
> rpcsec_gss_krb5 auth_rpcgss nft_chain_nat xt_MASQUERADE nf_nat nfsv4
> dns_resolver nfs lockd grace fscache netfs wireguard
> libchacha20poly1305 chacha_x86_64 po ly1305_x86_64 curve25519_x86_64
> libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel bridge stp
> llc ip6t_REJECT nf_reject_ipv6 xt_hl ip6_tables ip6t_rt ipt_REJE CT
> nf_reject_ipv4 xt_LOG nf_log_syslog nft_limit xt_limit xt_addrtype
> xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
> intel_rapl_msr intel_rapl_com mon nft_compat
> intel_uncore_frequency_common isst_if_common sunrpc nfit libnvdimm
> nf_tables nfnetlink binfmt_misc ghash_clmulni_intel sha512_ssse3
> sha512_generic sha 256_ssse3 sha1_ssse3 aesni_intel crypto_simd cryptd
> pcspkr button evdev serio_raw sg fuse loop efi_pstore configfs
> ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456
> async_raid6_recov async_memcpy async_pq async_xor 50
> 2024-01-30T01:03:56.499152+01:00 spotter kernel: [1222897.732858]
> async_tx xor raid6_pq libcrc32c crc32c_generic raid0 multipath linear
> virtio_pci virtio_pci_legacy _dev virtio_pci_modern_dev virtio_net
> net_failover failover virtio_blk virtio virtio_ring dm_mod raid1
> md_mod sd_mod t10_pi crc64_rocksoft crc64 crc_t10dif crct10dif
> _generic cirrus drm_shmem_helper 8139too ata_generic drm_kms_helper
> ata_piix uhci_hcd ehci_hcd libata usbcore drm sym53c8xx
> scsi_transport_spi crct10dif_pclmul crct1 0dif_common crc32_pclmul
> crc32c_intel psmouse scsi_mod 8139cp mii usb_common i2c_piix4
> scsi_common floppy 49 2024-01-30T01:03:56.499157+01:00 spotter kernel:
> [1222897.733007] CPU: 0 PID: 0 Comm: swapper/0 Not tainted
> 6.1.0-17-amd64 #1 Debian 6.1.69-1 48 2024-01-30T01:03:56.499162+01:00
> spotter kernel: [1222897.733013] Hardware name: Xen HVM domU, BIOS
> 4.17.3-pre 12/02/2023 47 2024-01-30T01:03:56.499168+01:00 spotter
> kernel: [1222897.733016] RIP: e030:dev_watchdog+0x207/0x210 46
> 2024-01-30T01:03:56.499173+01:00 spotter kernel: [1222897.733022]
> Code: 00 e9 40 ff ff ff 48 89 df c6 05 e6 a1 3e 01 01 e8 3e e3 fa ff
> 44 89 e9 48 89 de 48 c7 c7 50 b6 3f 82 48 89 c2 e8 79 f1 87 ff <0f> 0b
> e9 22 ff ff ff 66 90 0f 1f 44 00 00 55 53 48 89 fb 48 8b 6f 45
> 2024-01-30T01:03:56.499178+01:00 spotter kernel: [1222897.733026] RSP:
> e02b:ffffc90040003e88 EFLAGS: 00010282 44
> 2024-01-30T01:03:56.499182+01:00 spotter kernel: [1222897.733033] RAX:
> 0000000000000000 RBX: ffff888007122000 RCX: 0000000000000000 43
> 2024-01-30T01:03:56.499187+01:00 spotter kernel: [1222897.733036] RDX:
> ffff888004dc6a28 RSI: ffffc90040003d7c RDI: 0000000000000004 42
> 2024-01-30T01:03:56.499192+01:00 spotter kernel: [1222897.733040] RBP:
> ffff888007122488 R08: ffff888005000ae8 R09: ffffffff82ad4840 41
> 2024-01-30T01:03:56.499197+01:00 spotter kernel: [1222897.733043] R10:
> 0000000000000000 R11: 0000000000000000 R12: ffff8880071223dc 40
> 2024-01-30T01:03:56.499202+01:00 spotter kernel: [1222897.733047] R13:
> 0000000000000000 R14: ffffffff8181b480 R15: ffff888007122488 39
> 2024-01-30T01:03:56.499206+01:00 spotter kernel: [1222897.733058] FS:
> 0000000000000000(0000) GS:ffff88807d200000(0000)
> knlGS:0000000000000000 38 2024-01-30T01:03:56.499211+01:00 spotter
> kernel: [1222897.733062] CS: e030 DS: 0000 ES: 0000 CR0:
> 0000000080050033 37 2024-01-30T01:03:56.499216+01:00 spotter kernel:
> [1222897.733066] CR2: 00007fee61c0c048 CR3: 000000000ba60000 CR4:
> 0000000000050660 36 2024-01-30T01:03:56.499221+01:00 spotter kernel:
> [1222897.733074] Call Trace: 35 2024-01-30T01:03:56.499226+01:00
> spotter kernel: [1222897.733078] <IRQ> 34
> 2024-01-30T01:03:56.499231+01:00 spotter kernel: [1222897.733084] ?
> __warn+0x7d/0xc0 33 2024-01-30T01:03:56.499236+01:00 spotter kernel:
> [1222897.733090] ? dev_watchdog+0x207/0x210 32
> 2024-01-30T01:03:56.499259+01:00 spotter kernel: [1222897.733095] ?
> report_bug+0xe2/0x150 31 2024-01-30T01:03:56.499264+01:00 spotter
> kernel: [1222897.733102] ? xen_send_IPI_one+0x7f/0xe0 30
> 2024-01-30T01:03:56.499268+01:00 spotter kernel: [1222897.733108] ?
> handle_bug+0x41/0x70 29 2024-01-30T01:03:56.499273+01:00 spotter
> kernel: [1222897.733116] ? exc_invalid_op+0x13/0x60 28
> 2024-01-30T01:03:56.499278+01:00 spotter kernel: [1222897.733120] ?
> asm_exc_invalid_op+0x16/0x20 27 2024-01-30T01:03:56.499282+01:00
> spotter kernel: [1222897.733126] ? pfifo_fast_reset+0x140/0x140 26
> 2024-01-30T01:03:56.499287+01:00 spotter kernel: [1222897.733133] ?
> dev_watchdog+0x207/0x210 25 2024-01-30T01:03:56.499292+01:00 spotter
> kernel: [1222897.733137] ? dev_watchdog+0x207/0x210 24
> 2024-01-30T01:03:56.499297+01:00 spotter kernel: [1222897.733142] ?
> pfifo_fast_reset+0x140/0x140 23 2024-01-30T01:03:56.499302+01:00
> spotter kernel: [1222897.733146] call_timer_fn+0x24/0x130 22
> 2024-01-30T01:03:56.499307+01:00 spotter kernel: [1222897.733154]
> __run_timers+0x21c/0x2a0 21 2024-01-30T01:03:56.499318+01:00 spotter
> kernel: [1222897.733160] run_timer_softirq+0x19/0x30 20
> 2024-01-30T01:03:56.499323+01:00 spotter kernel: [1222897.733164]
> __do_softirq+0xc3/0x2ab 19 2024-01-30T01:03:56.499328+01:00 spotter
> kernel: [1222897.733169] ? xen_evtchn_do_upcall+0x77/0xb0 18
> 2024-01-30T01:03:56.499332+01:00 spotter kernel: [1222897.733174]
> __irq_exit_rcu+0xaa/0xe0 17 2024-01-30T01:03:56.499337+01:00 spotter
> kernel: [1222897.733180] xen_pv_evtchn_do_upcall+0x86/0xa0 16
> 2024-01-30T01:03:56.499342+01:00 spotter kernel: [1222897.733186]
> </IRQ> 15 2024-01-30T01:03:56.499346+01:00 spotter kernel:
> [1222897.733189] <TASK> 14 2024-01-30T01:03:56.499351+01:00 spotter
> kernel: [1222897.733192] exc_xen_hypervisor_callback+0x8/0x10 13
> 2024-01-30T01:03:56.499356+01:00 spotter kernel: [1222897.733196] RIP:
> e030:xen_hypercall_sched_op+0xa/0x20 12
> 2024-01-30T01:03:56.499361+01:00 spotter kernel: [1222897.733202]
> Code: 51 41 53 b8 1c 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc cc cc cc
> cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b
> 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 11
> 2024-01-30T01:03:56.499366+01:00 spotter kernel: [1222897.733205] RSP:
> e02b:ffffffff82a03db0 EFLAGS: 00000246 10
> 2024-01-30T01:03:56.499371+01:00 spotter kernel: [1222897.733211] RAX:
> 0000000000000000 RBX: ffffffff82a1aa40 RCX: ffffffff81a1f3aa 9
> 2024-01-30T01:03:56.499382+01:00 spotter kernel: [1222897.733215] RDX:
> 4000000000000000 RSI: 0000000000000000 RDI: 0000000000000001 8
> 2024-01-30T01:03:56.499386+01:00 spotter kernel: [1222897.733218] RBP:
> 0000000000000000 R08: 000000001d6b4193 R09: 0000000000000001 7
> 2024-01-30T01:03:56.499391+01:00 spotter kernel: [1222897.733221] R10:
> 0000000000007ff0 R11: 0000000000000246 R12: 0000000000000000 6
> 2024-01-30T01:03:56.499396+01:00 spotter kernel: [1222897.733225] R13:
> 0000000000000000 R14: ffffffff82a1a110 R15: 0000000000000000 5
> 2024-01-30T01:03:56.499401+01:00 spotter kernel: [1222897.733229] ?
> xen_hypercall_sched_op+0xa/0x20 4 2024-01-30T01:03:56.499405+01:00
> spotter kernel: [1222897.733235] ? xen_safe_halt+0xc/0x20 3
> 2024-01-30T01:03:56.499416+01:00 spotter kernel: [1222897.733241] ?
> default_idle+0xa/0x10 2 2024-01-30T01:03:56.499421+01:00 spotter
> kernel: [1222897.733246] ? default_idle_call+0x38/0xf0 1
> 2024-01-30T01:03:56.499426+01:00 spotter kernel: [1222897.733250] ?
> do_idle+0x21b/0x2a0 17742 2024-01-30T01:03:56.499430+01:00 spotter
> kernel: [1222897.733256] ? cpu_startup_entry+0x26/0x30 1
> 2024-01-30T01:03:56.499435+01:00 spotter kernel: [1222897.733261] ?
> rest_init+0xca/0xd0 2 2024-01-30T01:03:56.499440+01:00 spotter kernel:
> [1222897.733266] ? arch_call_rest_init+0xa/0x14 3
> 2024-01-30T01:03:56.499450+01:00 spotter kernel: [1222897.733272] ?
> start_kernel+0x6fe/0x727 4 2024-01-30T01:03:56.499455+01:00 spotter
> kernel: [1222897.733276] ? xen_start_kernel+0x5ee/0x5fe 5
> 2024-01-30T01:03:56.499460+01:00 spotter kernel: [1222897.733283] ?
> startup_xen+0x1f/0x1f 6 2024-01-30T01:03:56.499464+01:00 spotter
> kernel: [1222897.733291] </TASK> 7 2024-01-30T01:03:56.499469+01:00
> spotter kernel: [1222897.733294] ---[ end trace 0000000000000000 ]---
I've been working on getting a nested Xen setup working. Hardware is a
Dell PowerEdge R640 with 2x Intel Xeon Gold 6150 CPU, and a dual Intel
i350 NIC.
The Dom-0 is running Debian Bookworm (Linux 6.1.0-16-amd64) with Xen
version 4.17.3-pre (Debian 4.17.2+76-ge1f9cb16e2-1~deb12u1). This is
from package xen-system-amd64 version 4.17.2+76-ge1f9cb16e2-1~deb12u1.
I have a bunch of PV Dom-U's running that work perfectly fine with PV
networking. I also created a HVM Dom-U running Debian Bookworm (Linux
6.1.0-17-amd64) to run nested Xen. This nested Xen instance has a couple
of PV Dom-U's of its own.
When I install a fresh version of Debian Bookworm on a HVM guest it
nicely switches over to virtio, regardless of whether I select rtl8139
and e1000 as the driver. However, once I install Xen on this Dom-U and
reboot into Xen, it switches to either the 8139cp driver or e1000. Is
this expected behavior? Does the Xen kernel not support paravirtualized
network devices?
Anyway, I assumed this is the case so set up the PV Dom-U's on the
Nested xen instance. It does work but network performance deteriorates
over time and at some point during heavy network utilization (a backup
is created to a remote host), a transmit queue timeout occurs. I will
append the full log from syslog below.
From this point on, no further networking is possible via this network
adapter until I reboot the nested Xen instance.
I have now switched over to the e1000 network adapter to see if this
turns out to be more stable. I am wondering if this watchdog error is a
bug and should be reported somewhere, or if this is somewhat expected in
my setup?
The config of the Xen HVM host includes these settings:
---------- type = 'hvm' uuid = "<uid>" memory=131072 vcpus = 24 pae = 1
acpi = 1 viridian = 0 apic = 1 sdl = 0 usb = 1 name = "spotter" vif = [
'bridge=xenbr0,mac=<generated mac>,type=ioemu,model=rtl8139',
'bridge=xenbr1,mac=<generated mac>,type=ioemu,model=rtl8139' ] disk = [
'phy:/dev/zvol/main/spotter,xvda,w', 'phy:/dev/nvme1n1,sda,w',
'phy:/dev/nvme2n1,sdb,w', 'phy:/dev/nvme3n1,sdc,w', ] acpi = 1 serial =
'pty' vnc = 1 vnclisten = "127.0.0.1" vncdisplay = 0 hap=1 nestedhvm=1
----------
Any input is appreciated!
Kind regards,
Egbert
> 54 2024-01-30T01:03:56.499112+01:00 spotter kernel: [1222897.732550]
> ------------[ cut here ]------------ 53
> 2024-01-30T01:03:56.499135+01:00 spotter kernel: [1222897.732568]
> NETDEV WATCHDOG: eth0 (8139cp): transmit queue 0 timed out 52
> 2024-01-30T01:03:56.499141+01:00 spotter kernel: [1222897.732585]
> WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:525
> dev_watchdog+0x207/0x210 51 2024-01-30T01:03:56.499146+01:00 spotter
> kernel: [1222897.732596] Modules linked in: xt_recent xen_netfront
> xen_netback xen_blkback xen_gntdev xen_evtchn xenfs xen_p rivcmd
> rpcsec_gss_krb5 auth_rpcgss nft_chain_nat xt_MASQUERADE nf_nat nfsv4
> dns_resolver nfs lockd grace fscache netfs wireguard
> libchacha20poly1305 chacha_x86_64 po ly1305_x86_64 curve25519_x86_64
> libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel bridge stp
> llc ip6t_REJECT nf_reject_ipv6 xt_hl ip6_tables ip6t_rt ipt_REJE CT
> nf_reject_ipv4 xt_LOG nf_log_syslog nft_limit xt_limit xt_addrtype
> xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
> intel_rapl_msr intel_rapl_com mon nft_compat
> intel_uncore_frequency_common isst_if_common sunrpc nfit libnvdimm
> nf_tables nfnetlink binfmt_misc ghash_clmulni_intel sha512_ssse3
> sha512_generic sha 256_ssse3 sha1_ssse3 aesni_intel crypto_simd cryptd
> pcspkr button evdev serio_raw sg fuse loop efi_pstore configfs
> ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456
> async_raid6_recov async_memcpy async_pq async_xor 50
> 2024-01-30T01:03:56.499152+01:00 spotter kernel: [1222897.732858]
> async_tx xor raid6_pq libcrc32c crc32c_generic raid0 multipath linear
> virtio_pci virtio_pci_legacy _dev virtio_pci_modern_dev virtio_net
> net_failover failover virtio_blk virtio virtio_ring dm_mod raid1
> md_mod sd_mod t10_pi crc64_rocksoft crc64 crc_t10dif crct10dif
> _generic cirrus drm_shmem_helper 8139too ata_generic drm_kms_helper
> ata_piix uhci_hcd ehci_hcd libata usbcore drm sym53c8xx
> scsi_transport_spi crct10dif_pclmul crct1 0dif_common crc32_pclmul
> crc32c_intel psmouse scsi_mod 8139cp mii usb_common i2c_piix4
> scsi_common floppy 49 2024-01-30T01:03:56.499157+01:00 spotter kernel:
> [1222897.733007] CPU: 0 PID: 0 Comm: swapper/0 Not tainted
> 6.1.0-17-amd64 #1 Debian 6.1.69-1 48 2024-01-30T01:03:56.499162+01:00
> spotter kernel: [1222897.733013] Hardware name: Xen HVM domU, BIOS
> 4.17.3-pre 12/02/2023 47 2024-01-30T01:03:56.499168+01:00 spotter
> kernel: [1222897.733016] RIP: e030:dev_watchdog+0x207/0x210 46
> 2024-01-30T01:03:56.499173+01:00 spotter kernel: [1222897.733022]
> Code: 00 e9 40 ff ff ff 48 89 df c6 05 e6 a1 3e 01 01 e8 3e e3 fa ff
> 44 89 e9 48 89 de 48 c7 c7 50 b6 3f 82 48 89 c2 e8 79 f1 87 ff <0f> 0b
> e9 22 ff ff ff 66 90 0f 1f 44 00 00 55 53 48 89 fb 48 8b 6f 45
> 2024-01-30T01:03:56.499178+01:00 spotter kernel: [1222897.733026] RSP:
> e02b:ffffc90040003e88 EFLAGS: 00010282 44
> 2024-01-30T01:03:56.499182+01:00 spotter kernel: [1222897.733033] RAX:
> 0000000000000000 RBX: ffff888007122000 RCX: 0000000000000000 43
> 2024-01-30T01:03:56.499187+01:00 spotter kernel: [1222897.733036] RDX:
> ffff888004dc6a28 RSI: ffffc90040003d7c RDI: 0000000000000004 42
> 2024-01-30T01:03:56.499192+01:00 spotter kernel: [1222897.733040] RBP:
> ffff888007122488 R08: ffff888005000ae8 R09: ffffffff82ad4840 41
> 2024-01-30T01:03:56.499197+01:00 spotter kernel: [1222897.733043] R10:
> 0000000000000000 R11: 0000000000000000 R12: ffff8880071223dc 40
> 2024-01-30T01:03:56.499202+01:00 spotter kernel: [1222897.733047] R13:
> 0000000000000000 R14: ffffffff8181b480 R15: ffff888007122488 39
> 2024-01-30T01:03:56.499206+01:00 spotter kernel: [1222897.733058] FS:
> 0000000000000000(0000) GS:ffff88807d200000(0000)
> knlGS:0000000000000000 38 2024-01-30T01:03:56.499211+01:00 spotter
> kernel: [1222897.733062] CS: e030 DS: 0000 ES: 0000 CR0:
> 0000000080050033 37 2024-01-30T01:03:56.499216+01:00 spotter kernel:
> [1222897.733066] CR2: 00007fee61c0c048 CR3: 000000000ba60000 CR4:
> 0000000000050660 36 2024-01-30T01:03:56.499221+01:00 spotter kernel:
> [1222897.733074] Call Trace: 35 2024-01-30T01:03:56.499226+01:00
> spotter kernel: [1222897.733078] <IRQ> 34
> 2024-01-30T01:03:56.499231+01:00 spotter kernel: [1222897.733084] ?
> __warn+0x7d/0xc0 33 2024-01-30T01:03:56.499236+01:00 spotter kernel:
> [1222897.733090] ? dev_watchdog+0x207/0x210 32
> 2024-01-30T01:03:56.499259+01:00 spotter kernel: [1222897.733095] ?
> report_bug+0xe2/0x150 31 2024-01-30T01:03:56.499264+01:00 spotter
> kernel: [1222897.733102] ? xen_send_IPI_one+0x7f/0xe0 30
> 2024-01-30T01:03:56.499268+01:00 spotter kernel: [1222897.733108] ?
> handle_bug+0x41/0x70 29 2024-01-30T01:03:56.499273+01:00 spotter
> kernel: [1222897.733116] ? exc_invalid_op+0x13/0x60 28
> 2024-01-30T01:03:56.499278+01:00 spotter kernel: [1222897.733120] ?
> asm_exc_invalid_op+0x16/0x20 27 2024-01-30T01:03:56.499282+01:00
> spotter kernel: [1222897.733126] ? pfifo_fast_reset+0x140/0x140 26
> 2024-01-30T01:03:56.499287+01:00 spotter kernel: [1222897.733133] ?
> dev_watchdog+0x207/0x210 25 2024-01-30T01:03:56.499292+01:00 spotter
> kernel: [1222897.733137] ? dev_watchdog+0x207/0x210 24
> 2024-01-30T01:03:56.499297+01:00 spotter kernel: [1222897.733142] ?
> pfifo_fast_reset+0x140/0x140 23 2024-01-30T01:03:56.499302+01:00
> spotter kernel: [1222897.733146] call_timer_fn+0x24/0x130 22
> 2024-01-30T01:03:56.499307+01:00 spotter kernel: [1222897.733154]
> __run_timers+0x21c/0x2a0 21 2024-01-30T01:03:56.499318+01:00 spotter
> kernel: [1222897.733160] run_timer_softirq+0x19/0x30 20
> 2024-01-30T01:03:56.499323+01:00 spotter kernel: [1222897.733164]
> __do_softirq+0xc3/0x2ab 19 2024-01-30T01:03:56.499328+01:00 spotter
> kernel: [1222897.733169] ? xen_evtchn_do_upcall+0x77/0xb0 18
> 2024-01-30T01:03:56.499332+01:00 spotter kernel: [1222897.733174]
> __irq_exit_rcu+0xaa/0xe0 17 2024-01-30T01:03:56.499337+01:00 spotter
> kernel: [1222897.733180] xen_pv_evtchn_do_upcall+0x86/0xa0 16
> 2024-01-30T01:03:56.499342+01:00 spotter kernel: [1222897.733186]
> </IRQ> 15 2024-01-30T01:03:56.499346+01:00 spotter kernel:
> [1222897.733189] <TASK> 14 2024-01-30T01:03:56.499351+01:00 spotter
> kernel: [1222897.733192] exc_xen_hypervisor_callback+0x8/0x10 13
> 2024-01-30T01:03:56.499356+01:00 spotter kernel: [1222897.733196] RIP:
> e030:xen_hypercall_sched_op+0xa/0x20 12
> 2024-01-30T01:03:56.499361+01:00 spotter kernel: [1222897.733202]
> Code: 51 41 53 b8 1c 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc cc cc cc
> cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b
> 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 11
> 2024-01-30T01:03:56.499366+01:00 spotter kernel: [1222897.733205] RSP:
> e02b:ffffffff82a03db0 EFLAGS: 00000246 10
> 2024-01-30T01:03:56.499371+01:00 spotter kernel: [1222897.733211] RAX:
> 0000000000000000 RBX: ffffffff82a1aa40 RCX: ffffffff81a1f3aa 9
> 2024-01-30T01:03:56.499382+01:00 spotter kernel: [1222897.733215] RDX:
> 4000000000000000 RSI: 0000000000000000 RDI: 0000000000000001 8
> 2024-01-30T01:03:56.499386+01:00 spotter kernel: [1222897.733218] RBP:
> 0000000000000000 R08: 000000001d6b4193 R09: 0000000000000001 7
> 2024-01-30T01:03:56.499391+01:00 spotter kernel: [1222897.733221] R10:
> 0000000000007ff0 R11: 0000000000000246 R12: 0000000000000000 6
> 2024-01-30T01:03:56.499396+01:00 spotter kernel: [1222897.733225] R13:
> 0000000000000000 R14: ffffffff82a1a110 R15: 0000000000000000 5
> 2024-01-30T01:03:56.499401+01:00 spotter kernel: [1222897.733229] ?
> xen_hypercall_sched_op+0xa/0x20 4 2024-01-30T01:03:56.499405+01:00
> spotter kernel: [1222897.733235] ? xen_safe_halt+0xc/0x20 3
> 2024-01-30T01:03:56.499416+01:00 spotter kernel: [1222897.733241] ?
> default_idle+0xa/0x10 2 2024-01-30T01:03:56.499421+01:00 spotter
> kernel: [1222897.733246] ? default_idle_call+0x38/0xf0 1
> 2024-01-30T01:03:56.499426+01:00 spotter kernel: [1222897.733250] ?
> do_idle+0x21b/0x2a0 17742 2024-01-30T01:03:56.499430+01:00 spotter
> kernel: [1222897.733256] ? cpu_startup_entry+0x26/0x30 1
> 2024-01-30T01:03:56.499435+01:00 spotter kernel: [1222897.733261] ?
> rest_init+0xca/0xd0 2 2024-01-30T01:03:56.499440+01:00 spotter kernel:
> [1222897.733266] ? arch_call_rest_init+0xa/0x14 3
> 2024-01-30T01:03:56.499450+01:00 spotter kernel: [1222897.733272] ?
> start_kernel+0x6fe/0x727 4 2024-01-30T01:03:56.499455+01:00 spotter
> kernel: [1222897.733276] ? xen_start_kernel+0x5ee/0x5fe 5
> 2024-01-30T01:03:56.499460+01:00 spotter kernel: [1222897.733283] ?
> startup_xen+0x1f/0x1f 6 2024-01-30T01:03:56.499464+01:00 spotter
> kernel: [1222897.733291] </TASK> 7 2024-01-30T01:03:56.499469+01:00
> spotter kernel: [1222897.733294] ---[ end trace 0000000000000000 ]---