Hi Dario,
Thank you for your reply
This issue is made much worse with:
https://github.com/QubesOS/qubes-vmm-xen/commit/c28754bdb458281a22e9a9779213c941531b6dff#diff-d98b01176d360f55f58c25d2dfbfadc115718806181ef40d1838d2efa6b2bea1 Reverting `xen: credit2: limit the max number of CPUs in a
runqueue` results in stuttering even with dom0 pinned with 1vcpu, currently AMD Ryzen 4000 users are maintain a separate build without this change, I think this commit has something to do with the issues we're experiencing.
Without pinning dom0 1 vcpu this is what the lockups look like:
https://imgur.com/a/q7MQRez. Another weird artifact is the mouse (trackpad) will quickly jerk when being moved.
The other interesting thing is appVMs can only use 2 vcpus, allocating more results in that appVM exibiting stuttering/microlockups
So to get the device working, the following is required
1. Not revert `xen: credit2: limit the max number of CPUs in a
runqueue`
2. Pin 1 vcpu for donm0
3. Limit 2 vcpus per appVM
> So, just to be sure I am understanding the symptoms correctly: here you
> say that Credit (and RTDS) "don't boot correctly". In another mail, I
> think you said that Credit boots, but is unusable due to lag and
> lockups... Which is which?
credit2 is the only scheduler that I can get working, other schedulers don't boot at all
> Also, since this looks like it is SMT related, is Credit bootable
> and/or usable with SMT off? And with SMT on?
Qubes by default disables SMT, just after I sent my email yesterday I was actually able to boot with smt enabled as long as I had dom0 allocated 1 vcpu, without dom0 being allocated 1 vcpu the device won't boot at all.
> It'd be "wonderful" to see _how_ it does that, by seeing the
> stacktrace (preferrably of a debug build), if there is one. Or, if
> the system locks, e.g., knowing whether it is responsive at least
> to debug keys (and, if yest, how the output of the 'r' debug key
> looks like)
Because I'm compiling Xen myself, I should absolutely be able to enable debug/verbose logging, I'll try to capture more logging today
Here's what I can dig up currently
```
[chairman@dom0 ~]$ xl info
host : dom0
release : 5.8.18-200.fc32.x86_64
version : #1 SMP Mon Nov 2 19:49:11 UTC 2020
machine : x86_64
nr_cpus : 8
max_cpu_id : 15
nr_nodes : 1
cores_per_socket : 8
threads_per_core : 1
cpu_mhz : 1696.837
hw_caps : 178bf3ff:76d8320b:2e500800:244037ff:0000000f:219c91a9:00400004:00000500
virt_caps : pv hvm hvm_directio pv_directio hap
total_memory : 30439
free_memory : 6644
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 14
xen_extra : .0
xen_version : 4.14.0
xen_caps : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit2
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset :
xen_commandline : placeholder console=none dom0_mem=min:1024M dom0_mem=max:4096M dom0_max_vcpus=1 dom0_vcpus_pin ucode=scan smt=off gnttab_max_frames=2048 gnttab_max_maptrack_frames=4096 ept=exec-sp no-real-mode edd=off
cc_compiler : gcc (GCC) 10.2.1 20201125 (Red Hat 10.2.1-9)
cc_compile_by : user
cc_compile_domain : [unknown]
cc_compile_date : Wed Dec 16 00:00:00 UTC 2020
build_id : 9eb1d06c8bbc4686c4a8a6c9ee46d91e106df81d
xend_config_format : 4
[chairman@dom0 ~]$ xl vcpu-list
Name ID VCPU CPU State Time(s) Affinity (Hard / Soft)
Domain-0 0 0 0 r-- 367.3 0 / 0,2,4,6,8,10,12,14
sys-net 1 0 12 -b- 59.9 all / all
sys-net 1 1 10 -b- 61.0 all / all
sys-net-dm 2 0 10 -b- 19.4 all / all
sys-usb 3 0 8 -b- 31.0 all / all
sys-usb 3 1 14 -b- 36.9 all / all
sys-usb 3 2 10 -b- 34.5 all / all
sys-usb 3 3 12 -b- 32.8 all / all
sys-usb-dm 4 0 12 -b- 20.8 all / all
sys-firewall 5 0 10 -b- 1.4 all / all
sys-firewall 5 1 14 -b- 11.0 all / all
xxxxxxxxxxxxxx 6 0 12 -b- 9.2 all / all
xxxxxxxxxxxxxx 6 1 12 -b- 8.0 all / all
xxxxxxxxxxxxxxx 7 0 8 -b- 8.9 all / all
xxxxxxxxxxxxxxx 7 1 8 -b- 6.1 all / all
xxxxxxxxxxx 8 0 14 -b- 399.2 all / all
xxxxxxxxxxx 8 1 12 -b- 454.6 all / all
xxxxxxxxxxxxxxxx 9 0 14 -b- 33.7 all / all
xxxxxxxxxxxxxxxx 9 1 8 -b- 54.1 all / all
xxxxxxxxxxxxxxx 10 0 10 -b- 4.2 all / all
xxxxxxxxxxxxxxx 10 1 8 -b- 7.3 all / all
xxxxxxxxxxxx 11 0 2 -b- 39.7 all / all
xxxxxxxxxxxx 11 1 12 -b- 121.9 all / all
email 12 0 8 -b- 29.2 all / all
email 12 1 14 -b- 84.2 all / all
[2020-12-24 08:34:49] Logfile Opened
[2020-12-24 08:34:49] (XEN) Built-in command line: ept=exec-sp
[2020-12-24 08:34:49] (XEN) parameter no-real-mode unknown!
[2020-12-24 08:34:49] (XEN) parameter edd unknown!
[2020-12-24 08:34:49] Xen 4.14.0
[2020-12-24 08:34:49] (XEN) Xen version 4.14.0 (user@[unknown]) (gcc (GCC) 10.2.1 20201125 (Red Hat 10.2.1-9)) debug=n Wed Dec 16 00:00:00 UTC 2020
[2020-12-24 08:34:49] (XEN) Latest ChangeSet:
[2020-12-24 08:34:49] (XEN) Bootloader: GRUB 2.04
[2020-12-24 08:34:49] (XEN) Command line: placeholder console=none dom0_mem=min:1024M dom0_mem=max:4096M dom0_max_vcpus=1 dom0_vcpus_pin ucode=scan smt=off gnttab_max_frames=2048 gnttab_max_maptrack_frames=4096 ept=exec-sp no-real-mode edd=off
[2020-12-24 08:34:49] (XEN) Xen image load base address: 0xb8e00000
[2020-12-24 08:34:49] (XEN) Video information:
[2020-12-24 08:34:49] (XEN) VGA is graphics mode 1920x1080, 32 bpp
[2020-12-24 08:34:49] (XEN) Disc information:
[2020-12-24 08:34:49] (XEN) Found 0 MBR signatures
[2020-12-24 08:34:49] (XEN) Found 2 EDD information structures
[2020-12-24 08:34:49] (XEN) EFI RAM map:
[2020-12-24 08:34:49] (XEN) [0000000000000000, 000000000009efff] (usable)
[2020-12-24 08:34:49] (XEN) [000000000009f000, 000000000009ffff] (reserved)
[2020-12-24 08:34:49] (XEN) [00000000000e0000, 00000000000fffff] (reserved)
[2020-12-24 08:34:49] (XEN) [0000000000100000, 0000000009bfffff] (usable)
[2020-12-24 08:34:49] (XEN) [0000000009c00000, 0000000009d00fff] (reserved)
[2020-12-24 08:34:49] (XEN) [0000000009d01000, 0000000009efffff] (usable)
[2020-12-24 08:34:49] (XEN) [0000000009f00000, 0000000009f0ffff] (ACPI NVS)
[2020-12-24 08:34:49] (XEN) [0000000009f10000, 00000000bd9ddfff] (usable)
[2020-12-24 08:34:49] (XEN) [00000000bd9de000, 00000000ca37dfff] (reserved)
[2020-12-24 08:34:49] (XEN) [00000000ca37e000, 00000000cc37dfff] (ACPI NVS)
[2020-12-24 08:34:49] (XEN) [00000000cc37e000, 00000000cc3fdfff] (ACPI data)
[2020-12-24 08:34:49] (XEN) [00000000cc3fe000, 00000000cdffffff] (usable)
[2020-12-24 08:34:49] (XEN) [00000000ce000000, 00000000cfffffff] (reserved)
[2020-12-24 08:34:49] (XEN) [00000000f8000000, 00000000fbffffff] (reserved)
[2020-12-24 08:34:49] (XEN) [00000000fde00000, 00000000fdefffff] (reserved)
[2020-12-24 08:34:49] (XEN) [00000000fed80000, 00000000fed80fff] (reserved)
[2020-12-24 08:34:49] (XEN) [0000000100000000, 00000007af33ffff] (usable)
[2020-12-24 08:34:49] (XEN) [00000007af340000, 000000082fffffff] (reserved)
[2020-12-24 08:34:49] (XEN) ACPI: RSDP CC3FD014, 0024 (r2 LENOVO)
[2020-12-24 08:34:49] (XEN) ACPI: XSDT CC3FB188, 0104 (r1 LENOVO TP-R1C 1290 PTEC 2)
[2020-12-24 08:34:49] (XEN) ACPI: FACP BE499000, 0114 (r6 LENOVO TP-R1C 1290 PTEC 2)
[2020-12-24 08:34:49] (XEN) ACPI: DSDT BE484000, F08E (r1 LENOVO TP-R1C 1290 INTL 20180313)
[2020-12-24 08:34:49] (XEN) ACPI: FACS CC218000, 0040
[2020-12-24 08:34:49] (XEN) ACPI: SSDT BF751000, 00A2 (r1 LENOVO PID0Ssdt 1 INTL 20180313)
[2020-12-24 08:34:49] (XEN) ACPI: SSDT BF750000, 0CCC (r1 LENOVO UsbCTabl 1 INTL 20180313)
[2020-12-24 08:34:49] (XEN) ACPI: SSDT BF743000, 7216 (r2 LENOVO TP-R1C 2 MSFT 4000000)
[2020-12-24 08:34:49] (XEN) ACPI: IVRS BF742000, 01A4 (r2 LENOVO TP-R1C 1290 PTEC 2)
[2020-12-24 08:34:49] (XEN) ACPI: SSDT BF704000, 0266 (r1 LENOVO STD3 1 INTL 20180313)
[2020-12-24 08:34:49] (XEN) ACPI: SSDT BF6F0000, 0632 (r2 LENOVO Tpm2Tabl 1000 INTL 20180313)
[2020-12-24 08:34:49] (XEN) ACPI: TPM2 BF6EF000, 0034 (r3 LENOVO TP-R1C 1290 PTEC 2)
[2020-12-24 08:34:49] (XEN) ACPI: SSDT BF6EE000, 0924 (r1 LENOVO WmiTable 1 INTL 20180313)
[2020-12-24 08:34:49] (XEN) ACPI: MSDM BF6B5000, 0055 (r3 LENOVO TP-R1C 1290 PTEC 2)
[2020-12-24 08:34:49] (XEN) ACPI: BATB BF6A0000, 004A (r2 LENOVO TP-R1C 1290 PTEC 2)
[2020-12-24 08:34:49] (XEN) ACPI: HPET BE498000, 0038 (r1 LENOVO TP-R1C 1290 PTEC 2)
[2020-12-24 08:34:49] (XEN) ACPI: APIC BE497000, 0138 (r2 LENOVO TP-R1C 1290 PTEC 2)
[2020-12-24 08:34:49] (XEN) ACPI: MCFG BE496000, 003C (r1 LENOVO TP-R1C 1290 PTEC 2)
[2020-12-24 08:34:49] (XEN) ACPI: SBST BE495000, 0030 (r1 LENOVO TP-R1C 1290 PTEC 2)
[2020-12-24 08:34:49] (XEN) ACPI: WSMT BE494000, 0028 (r1 LENOVO TP-R1C 1290 PTEC 2)
[2020-12-24 08:34:49] (XEN) ACPI: VFCT BE476000, D484 (r1 LENOVO TP-R1C 1290 PTEC 2)
[2020-12-24 08:34:49] (XEN) ACPI: SSDT BE472000, 39F4 (r1 LENOVO TP-R1C 1 AMD 1)
[2020-12-24 08:34:49] (XEN) ACPI: CRAT BE471000, 0F00 (r1 LENOVO TP-R1C 1290 PTEC 2)
[2020-12-24 08:34:49] (XEN) ACPI: CDIT BE470000, 0029 (r1 LENOVO TP-R1C 1290 PTEC 2)
[2020-12-24 08:34:49] (XEN) ACPI: FPDT BF6C7000, 0034 (r1 LENOVO TP-R1C 1290 PTEC 2)
[2020-12-24 08:34:49] (XEN) ACPI: SSDT BE46E000, 13CF (r1 LENOVO TP-R1C 1 INTL 20180313)
[2020-12-24 08:34:49] (XEN) ACPI: SSDT BE46C000, 1576 (r1 LENOVO TP-R1C 1 INTL 20180313)
[2020-12-24 08:34:49] (XEN) ACPI: SSDT BE468000, 353C (r1 LENOVO TP-R1C 1 INTL 20180313)
[2020-12-24 08:34:49] (XEN) ACPI: BGRT BE467000, 0038 (r1 LENOVO TP-R1C 1290 PTEC 2)
[2020-12-24 08:34:49] (XEN) ACPI: UEFI CC217000, 013E (r1 LENOVO TP-R1C 1290 PTEC 2)
[2020-12-24 08:34:49] (XEN) ACPI: SSDT BF74F000, 0090 (r1 LENOVO TP-R1C 1 INTL 20180313)
[2020-12-24 08:34:49] (XEN) ACPI: SSDT BF74E000, 09AD (r1 LENOVO TP-R1C 1 INTL 20180313)
[2020-12-24 08:34:49] (XEN) System RAM: 30439MB (31170232kB)
[2020-12-24 08:34:49] (XEN) Domain heap initialised
[2020-12-24 08:34:49] (XEN) ACPI: 32/64X FACS address mismatch in FADT - cc218000/0000000000000000, using 32
[2020-12-24 08:34:49] (XEN) IOAPIC[0]: apic_id 32, version 33, address 0xfec00000, GSI 0-23
[2020-12-24 08:34:49] (XEN) IOAPIC[1]: apic_id 33, version 33, address 0xfec01000, GSI 24-55
[2020-12-24 08:34:49] (XEN) Enabling APIC mode: Phys. Using 2 I/O APICs
[2020-12-24 08:34:49] (XEN) CPU0: 1400..1700 MHz
[2020-12-24 08:34:49] (XEN) xstate: size: 0x380 and states: 0x207
[2020-12-24 08:34:49] (XEN) Speculative mitigation facilities:
[2020-12-24 08:34:49] (XEN) Hardware features: IBPB
[2020-12-24 08:34:49] (XEN) Compiled-in support: INDIRECT_THUNK
[2020-12-24 08:34:49] (XEN) Xen settings: BTI-Thunk LFENCE, SPEC_CTRL: No, Other: IBPB BRANCH_HARDEN
[2020-12-24 08:34:49] (XEN) Support for HVM VMs: RSB
[2020-12-24 08:34:49] (XEN) Support for PV VMs: RSB
[2020-12-24 08:34:49] (XEN) XPTI (64-bit PV only): Dom0 disabled, DomU disabled (without PCID)
[2020-12-24 08:34:49] (XEN) PV L1TF shadowing: Dom0 disabled, DomU disabled
[2020-12-24 08:34:49] (XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
[2020-12-24 08:34:49] (XEN) Initializing Credit2 scheduler
[2020-12-24 08:34:49] (XEN) Platform timer is 14.318MHz HPET
[2020-12-24 08:34:49] (XEN) Detected 1696.837 MHz processor.
[2020-12-24 08:34:49] (XEN) Unknown cachability for MFNs 0xe0-0xff
[2020-12-24 08:34:49] (XEN) AMD-Vi: IOMMU Extended Features:
[2020-12-24 08:34:49] (XEN) - Peripheral Page Service Request
[2020-12-24 08:34:49] (XEN) - x2APIC
[2020-12-24 08:34:49] (XEN) - NX bit
[2020-12-24 08:34:49] (XEN) - Invalidate All Command
[2020-12-24 08:34:49] (XEN) - Guest APIC
[2020-12-24 08:34:49] (XEN) - Performance Counters
[2020-12-24 08:34:49] (XEN) - Host Address Translation Size: 0x2
[2020-12-24 08:34:49] (XEN) - Guest Address Translation Size: 0
[2020-12-24 08:34:49] (XEN) - Guest CR3 Root Table Level: 0x1
[2020-12-24 08:34:49] (XEN) - Maximum PASID: 0xf
[2020-12-24 08:34:49] (XEN) - SMI Filter Register: 0x1
[2020-12-24 08:34:49] (XEN) - SMI Filter Register Count: 0x1
[2020-12-24 08:34:49] (XEN) - Guest Virtual APIC Modes: 0x1
[2020-12-24 08:34:49] (XEN) - Dual PPR Log: 0x2
[2020-12-24 08:34:49] (XEN) - Dual Event Log: 0x2
[2020-12-24 08:34:49] (XEN) - User / Supervisor Page Protection
[2020-12-24 08:34:49] (XEN) - Device Table Segmentation: 0x3
[2020-12-24 08:34:49] (XEN) - PPR Log Overflow Early Warning
[2020-12-24 08:34:49] (XEN) - PPR Automatic Response
[2020-12-24 08:34:49] (XEN) - Memory Access Routing and Control: 0
[2020-12-24 08:34:49] (XEN) - Block StopMark Message
[2020-12-24 08:34:49] (XEN) - Performance Optimization
[2020-12-24 08:34:49] (XEN) - MSI Capability MMIO Access
[2020-12-24 08:34:49] (XEN) - Guest I/O Protection
[2020-12-24 08:34:49] (XEN) - Enhanced PPR Handling
[2020-12-24 08:34:49] (XEN) - Attribute Forward
[2020-12-24 08:34:49] (XEN) - Invalidate IOTLB Type
[2020-12-24 08:34:49] (XEN) - VM Table Size: 0
[2020-12-24 08:34:49] (XEN) - Guest Access Bit Update Disable
[2020-12-24 08:34:49] (XEN) AMD-Vi: IOMMU 0 Enabled.
[2020-12-24 08:34:49] (XEN) I/O virtualisation enabled
[2020-12-24 08:34:49] (XEN) - Dom0 mode: Relaxed
[2020-12-24 08:34:49] (XEN) Interrupt remapping enabled
[2020-12-24 08:34:49] (XEN) ENABLING IO-APIC IRQs
[2020-12-24 08:34:49] (XEN) -> Using new ACK method
[2020-12-24 08:34:49] (XEN) Allocated console ring of 32 KiB.
[2020-12-24 08:34:49] (XEN) HVM: ASIDs enabled.
[2020-12-24 08:34:49] (XEN) SVM: Supported advanced features:
[2020-12-24 08:34:49] (XEN) - Nested Page Tables (NPT)
[2020-12-24 08:34:49] (XEN) - Last Branch Record (LBR) Virtualisation
[2020-12-24 08:34:49] (XEN) - Next-RIP Saved on #VMEXIT
[2020-12-24 08:34:49] (XEN) - VMCB Clean Bits
[2020-12-24 08:34:49] (XEN) - DecodeAssists
[2020-12-24 08:34:49] (XEN) - Virtual VMLOAD/VMSAVE
[2020-12-24 08:34:49] (XEN) - Virtual GIF
[2020-12-24 08:34:49] (XEN) - Pause-Intercept Filter
[2020-12-24 08:34:49] (XEN) - Pause-Intercept Filter Threshold
[2020-12-24 08:34:49] (XEN) - TSC Rate MSR
[2020-12-24 08:34:49] (XEN) HVM: SVM enabled
[2020-12-24 08:34:49] (XEN) HVM: Hardware Assisted Paging (HAP) detected
[2020-12-24 08:34:49] (XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
[2020-12-24 08:34:49] (XEN) CPU 1 still not dead...
[2020-12-24 08:34:49] (XEN) CPU 1 still not dead...
[2020-12-24 08:34:49] (XEN) Brought up 8 CPUs
[2020-12-24 08:34:49] (XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
[2020-12-24 08:34:49] (XEN) xenoprof: Initialization failed. AMD processor family 23 is not supported
[2020-12-24 08:34:49] (XEN) TSC warp detected, disabling TSC_RELIABLE
[2020-12-24 08:34:49] (XEN) Dom0 has maximum 264 PIRQs
[2020-12-24 08:34:49] (XEN) Xen kernel: 64-bit, lsb, compat32
[2020-12-24 08:34:49] (XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x3e00000
[2020-12-24 08:34:49] (XEN) PHYSICAL MEMORY ARRANGEMENT:
[2020-12-24 08:34:49] (XEN) Dom0 alloc.: 000000078c000000->0000000790000000 (1022908 pages to be allocated)
[2020-12-24 08:34:49] (XEN) Init. ramdisk: 00000007acdbc000->00000007af1ff5cd
[2020-12-24 08:34:49] (XEN) VIRTUAL MEMORY ARRANGEMENT:
[2020-12-24 08:34:49] (XEN) Loaded kernel: ffffffff81000000->ffffffff83e00000
[2020-12-24 08:34:49] (XEN) Init. ramdisk: 0000000000000000->0000000000000000
[2020-12-24 08:34:49] (XEN) Phys-Mach map: 0000008000000000->0000008000800000
[2020-12-24 08:34:49] (XEN) Start info: ffffffff83e00000->ffffffff83e004b8
[2020-12-24 08:34:49] (XEN) Xenstore ring: 0000000000000000->0000000000000000
[2020-12-24 08:34:49] (XEN) Console ring: 0000000000000000->0000000000000000
[2020-12-24 08:34:49] (XEN) Page tables: ffffffff83e01000->ffffffff83e24000
[2020-12-24 08:34:49] (XEN) Boot stack: ffffffff83e24000->ffffffff83e25000
[2020-12-24 08:34:49] (XEN) TOTAL: ffffffff80000000->ffffffff84000000
[2020-12-24 08:34:49] (XEN) ENTRY ADDRESS: ffffffff83128180
[2020-12-24 08:34:49] (XEN) Dom0 has maximum 1 VCPUs
[2020-12-24 08:34:49] (XEN) Initial low memory virq threshold set at 0x4000 pages.
[2020-12-24 08:34:49] (XEN) Scrubbing Free RAM in background
[2020-12-24 08:34:49] (XEN) Std. Loglevel: Errors and warnings
[2020-12-24 08:34:49] (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
[2020-12-24 08:34:49] (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
[2020-12-24 08:34:49] (XEN) Freed 536kB init memory
```
> In absence of that, I only have more questions. :-/ E.g., how are you
> enabling and disabling SMT, via the command line parameter, or via
> BIOS?
Currently via Xen's CMDLINE, the Lenovo X13 doesn't have an option to disable SMT :(
> Also, can you perhaps try either upstream 4.14 Xen (from sources, I
> mean) or the packages for a distro different than QubesOS (perhaps
> installing such distro, temporarily, in an external HD or whatever).
I can give this a shot, I'll try and dump out as much logging as I can, then I'll try build from the master branch of xen
> I can try Credit as well, later, but if this is something CPU arch/gen
> related, it seems to be a Ryzen rather than a Zen 2 thing...
Absolutely, I agree, this seems to be Ryzen 3000/4000 related, another Qubes user running a Ryzen 9 3900X CPU appears to be having the same issue:
https://imgur.com/a/EYOMmRe