Mailing List Archive

http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1698





------- Comment #2 from at@wavecon.de 2010-12-20 08:30 -------
Another R710, being an exact clone of this configuration crashes as well.

Some older machines (~ 2 months), again being R710 but having a slight
different configuration (< 2TB disk thus using DOS partition tables, slightly
different CPU, RAID controller and no SAS disks) don't have this problem. They
run Debian stable (thus and old style dom0+hypervisor with Xen 3.2). The output
may therefore not being that useful, however:


(XEN) Xen version 3.2-1 (Debian 3.2.1-2) (waldi@debian.org) (gcc version 4.3.1
(Debian 4.3.1-2) ) Sat Jun 28 09:32:18 UTC 2008
(XEN) Command line:
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: none; EDID transfer time: 0 seconds
(XEN) EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN) Found 1 MBR signatures
(XEN) Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 00000000000a0000 (usable)
(XEN) 0000000000100000 - 000000007f679000 (usable)
(XEN) 000000007f679000 - 000000007f68f000 (reserved)
(XEN) 000000007f68f000 - 000000007f6ce000 (ACPI data)
(XEN) 000000007f6ce000 - 0000000080000000 (reserved)
(XEN) 00000000e0000000 - 00000000f0000000 (reserved)
(XEN) 00000000fe000000 - 0000000100000000 (reserved)
(XEN) 0000000100000000 - 0000000c80000000 (usable)
(XEN) System RAM: 49142MB (50321508kB)
(XEN) Xen heap: 13MB (13472kB)
(XEN) Domain heap initialised: DMA width 32 bits
(XEN) Processor #16 7:10 APIC version 21
(XEN) Processor #0 7:10 APIC version 21
(XEN) Processor #18 7:10 APIC version 21
(XEN) Processor #2 7:10 APIC version 21
(XEN) Processor #20 7:10 APIC version 21
(XEN) Processor #4 7:10 APIC version 21
(XEN) Processor #22 7:10 APIC version 21
(XEN) Processor #6 7:10 APIC version 21
(XEN) Processor #17 7:10 APIC version 21
(XEN) Processor #1 7:10 APIC version 21
(XEN) Processor #19 7:10 APIC version 21
(XEN) Processor #3 7:10 APIC version 21
(XEN) Processor #21 7:10 APIC version 21
(XEN) Processor #5 7:10 APIC version 21
(XEN) Processor #23 7:10 APIC version 21
(XEN) Processor #7 7:10 APIC version 21
(XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 1, version 32, address 0xfec80000, GSI 32-55
(XEN) Enabling APIC mode: Phys. Using 2 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2394.040 MHz processor.
(XEN) , L1 D cache: 32K
(XEN) HVM: VMX enabled
(XEN) CPU0: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping 05
(XEN) Booting processor 1/0 eip 8c000
(XEN) , L1 D cache: 32K
(XEN) CPU1: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping 05
(XEN) Booting processor 2/18 eip 8c000
(XEN) , L1 D cache: 32K
(XEN) CPU2: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping 05
(XEN) Booting processor 3/2 eip 8c000
(XEN) , L1 D cache: 32K
(XEN) CPU3: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping 05
(XEN) Booting processor 4/20 eip 8c000
(XEN) , L1 D cache: 32K
(XEN) CPU4: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping 05
(XEN) Booting processor 5/4 eip 8c000
(XEN) , L1 D cache: 32K
(XEN) CPU5: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping 05
(XEN) Booting processor 6/22 eip 8c000
(XEN) , L1 D cache: 32K
(XEN) CPU6: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping 05
(XEN) Booting processor 7/6 eip 8c000
(XEN) , L1 D cache: 32K
(XEN) CPU7: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping 05
(XEN) Booting processor 8/17 eip 8c000
(XEN) , L1 D cache: 32K
(XEN) CPU8: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping 05
(XEN) Booting processor 9/1 eip 8c000
(XEN) , L1 D cache: 32K
(XEN) CPU9: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping 05
(XEN) Booting processor 10/19 eip 8c000
(XEN) , L1 D cache: 32K
(XEN) CPU10: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping 05
(XEN) Booting processor 11/3 eip 8c000
(XEN) , L1 D cache: 32K
(XEN) CPU11: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping 05
(XEN) Booting processor 12/21 eip 8c000
(XEN) , L1 D cache: 32K
(XEN) CPU12: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping 05
(XEN) Booting processor 13/5 eip 8c000
(XEN) , L1 D cache: 32K
(XEN) CPU13: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping 05
(XEN) Booting processor 14/23 eip 8c000
(XEN) , L1 D cache: 32K
(XEN) CPU14: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping 05
(XEN) Booting processor 15/7 eip 8c000
(XEN) , L1 D cache: 32K
(XEN) CPU15: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz stepping 05
(XEN) Total of 16 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using new ACK method
(XEN) Platform timer overflows in 14998 jiffies.
(XEN) Platform timer is 14.318MHz HPET
(XEN) Brought up 16 CPUs
(XEN) xenoprof: Initialization failed. Intel processor model 26 for P6 class
family is not supported
(XEN) AMD IOMMU: Disabled
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, lsb, paddr 0x200000 -> 0x631918
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 0000000c48000000->0000000c50000000 (12344123 pages to be
allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff80200000->ffffffff80631918
(XEN) Init. ramdisk: ffffffff80632000->ffffffff81b30800
(XEN) Phys-Mach map: ffffffff81b31000->ffffffff8799e9d8
(XEN) Start info: ffffffff8799f000->ffffffff8799f4a4
(XEN) Page tables: ffffffff879a0000->ffffffff879e1000
(XEN) Boot stack: ffffffff879e1000->ffffffff879e2000
(XEN) TOTAL: ffffffff80000000->ffffffff87c00000
(XEN) ENTRY ADDRESS: ffffffff80200000
(XEN) Dom0 has maximum 16 VCPUs
(XEN) Initrd len 0x14fe800, start at 0xffffffff80632000
(XEN) Scrubbing Free RAM: .done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to
Xen)
(XEN) Freed 104kB init memory.
(XEN) mm.c:665:d0 Error getting mfn 100 (pfn 3ed39) from L1 entry
8000000000100225 for dom32753
(XEN) mm.c:665:d0 Error getting mfn 100 (pfn 3ed39) from L1 entry
8000000000100225 for dom32753


and

host : xen25.c1.wavecon.info
release : 2.6.26-2-xen-amd64
version : #1 SMP Thu Sep 16 16:32:15 UTC 2010
machine : x86_64
nr_cpus : 16
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 2
cpu_mhz : 2394
hw_caps :
bfebfbff:28100800:00000000:00000140:009ce3bd:00000000:00000001
total_memory : 49142
free_memory : 4
node_to_cpu : node0:0-15
xen_major : 3
xen_minor : 2
xen_extra : -1
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
cc_compiler : gcc version 4.3.1 (Debian 4.3.1-2)
cc_compile_by : waldi
cc_compile_domain : debian.org
cc_compile_date : Sat Jun 28 09:32:18 UTC 2008
xend_config_format : 4


Debian stable failed on the crashing machines as well, although I cannot
remember exactly what the problem was. If you do care I can reinstall on of
those machines with Lenny/Xen 3.

Let me know if I can further assist you. I may also provide you or someone
being interested to dig the bug command line access to the machine if you want.


--
Configure bugmail: http://bugzilla.xensource.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

_______________________________________________
Xen-bugs mailing list
Xen-bugs@lists.xensource.com
http://lists.xensource.com/xen-bugs

http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1698





------- Comment #3 from konrad.wilk@oracle.com 2010-12-20 09:57 -------
(In reply to comment #2)
> Another R710, being an exact clone of this configuration crashes as well.
>
> Some older machines (~ 2 months), again being R710 but having a slight
> different configuration (< 2TB disk thus using DOS partition tables, slightly
> different CPU, RAID controller and no SAS disks) don't have this problem. They
> run Debian stable (thus and old style dom0+hypervisor with Xen 3.2). The output
> may

The issue here seems from the fact that the NUMA configuration steers the
hypervisor (not the Linux kernel) in allocating memory differently. I am not
sure what the allocation code in the hypervisor does but it looks that the
node 0 ends up having no DMA32 memory.

The older machines might not have the SRAT table, or the older hypervisor =
either which causes the hypervisor to have more DMA32 type memory available.

So what we need to try to see is if you can limit the amount of memory that
Dom0 gets so there is enough space in the hypervisor to swap with DMA32 type
memory. Try dom0_mem=max:4GB for example on your Hypervisor line.

The other thing to try is to disable the Hypervisor from parsing the NUMA
information (the SRAT table). I am not sure what the command line argument for
that is (I would think 'numa=off')

So try either one or both and see what the 'xm info' has for node_to_dma32_mem
value.


--
Configure bugmail: http://bugzilla.xensource.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

_______________________________________________
Xen-bugs mailing list
Xen-bugs@lists.xensource.com
http://lists.xensource.com/xen-bugs

http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1698





------- Comment #4 from at@wavecon.de 2010-12-20 10:33 -------
Passing dom0_mem=max:4GB to the hypervisor works around the issue. Some output
excerpts will follow (I hope I didn't strip off relevant parts):


(XEN) Xen version 4.0.1 (Debian 4.0.1-1) (waldi@debian.org) (gcc version 4.4.5
20100824 (prerelease) (Debian 4.4.4-11) ) Fri Sep 3 15:38:12 UTC 2010
(XEN) Bootloader: GRUB 1.98+20100804-10
(XEN) Command line: console=com2 dom0_mem=max:4GB
(XEN) Video information:
...
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x16b6000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 0000000c5c000000->0000000c60000000 (1032192 pages to be
allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff81000000->ffffffff816b6000
(XEN) Init. ramdisk: ffffffff816b6000->ffffffff83064200
(XEN) Phys-Mach map: ffffffff83065000->ffffffff83865000
(XEN) Start info: ffffffff83865000->ffffffff838654b4
(XEN) Page tables: ffffffff83866000->ffffffff83887000
(XEN) Boot stack: ffffffff83887000->ffffffff83888000
(XEN) TOTAL: ffffffff80000000->ffffffff83c00000
(XEN) ENTRY ADDRESS: ffffffff81506200
(XEN) Dom0 has maximum 16 VCPUs
(XEN) Scrubbing Free RAM:
...................................................................................................................................
(XEN) Xen trace buffers: disabled
...
[ 0.000000] Linux version 2.6.32-5-xen-amd64 (Debian 2.6.32-29)
(ben@decadent.org.uk) (gcc version 4.3.5 (Debian 4.3.5-4) ) #1 SMP Fri Dec 10
17:41:50 UT0
[ 0.000000] Command line: placeholder
root=UUID=4346c332-863c-4482-b9a1-f3d45dffc1e7 ro console=ttyS1 console=hvc0
earlyprintk=xen initcall_debug debug l0
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls
[ 0.000000] xen_release_chunk: looking at area pfn 80000-e0000: 393216 pages
freed
[ 0.000000] xen_release_chunk: looking at area pfn f0000-fe000: 57344 pages
freed
[ 0.000000] released 450560 pages of unused memory
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable)
[ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved)
[ 0.000000] Xen: 0000000000100000 - 000000007f679000 (usable)
[ 0.000000] Xen: 000000007f679000 - 000000007f68f000 (reserved)
[ 0.000000] Xen: 000000007f68f000 - 000000007f6ce000 (ACPI data)
[ 0.000000] Xen: 000000007f6ce000 - 0000000080000000 (reserved)
[ 0.000000] Xen: 00000000e0000000 - 00000000f0000000 (reserved)
[ 0.000000] Xen: 00000000fe000000 - 0000000100000000 (reserved)
[ 0.000000] bootconsole [xenboot0] enabled
[ 0.000000] DMI 2.6 present.
[ 0.000000] last_pfn = 0x7f679 max_arch_pfn = 0x400000000
[ 0.000000] x86 PAT enabled: cpu 0, old 0x50100070406, new 0x7010600070106
[ 0.000000] initial memory mapped : 0 - 20000000
[ 0.000000] init_memory_mapping: 0000000000000000-000000007f679000
[ 0.000000] 0000000000 - 007f679000 page 4k
[ 0.000000] kernel direct mapping tables up to 7f679000 @ 100000-4ff000
[ 0.000000] RAMDISK: 016b6000 - 03064200
[ 0.000000] ACPI: RSDP 00000000000f0a90 00024 (v02 DELL )
[ 0.000000] ACPI: XSDT 00000000000f0b94 0009C (v01 DELL PE_SC3 00000001
DELL 00000001)
[ 0.000000] ACPI: FACP 000000007f6b3f9c 000F4 (v03 DELL PE_SC3 00000001
DELL 00000001)
[ 0.000000] ACPI: DSDT 000000007f68f000 03D5A (v01 DELL PE_SC3 00000001
INTL 20050624)
[ 0.000000] ACPI: FACS 000000007f6b6000 00040
[ 0.000000] ACPI: APIC 000000007f6b3478 0015E (v01 DELL PE_SC3 00000001
DELL 00000001)
[ 0.000000] ACPI: SPCR 000000007f6b35d8 00050 (v01 DELL PE_SC3 00000001
DELL 00000001)
[ 0.000000] ACPI: HPET 000000007f6b362c 00038 (v01 DELL PE_SC3 00000001
DELL 00000001)
[ 0.000000] ACPI: XMAR 000000007f6b3668 001C0 (v01 DELL PE_SC3 00000001
DELL 00000001)
[ 0.000000] ACPI: MCFG 000000007f6b38c4 0003C (v01 DELL PE_SC3 00000001
DELL 00000001)
[ 0.000000] ACPI: WD__ 000000007f6b3904 00134 (v01 DELL PE_SC3 00000001
DELL 00000001)
[ 0.000000] ACPI: SLIC 000000007f6b3a3c 00024 (v01 DELL PE_SC3 00000001
DELL 00000001)
[ 0.000000] ACPI: ERST 000000007f692edc 00270 (v01 DELL PE_SC3 00000001
DELL 00000001)
[ 0.000000] ACPI: HEST 000000007f69314c 003A8 (v01 DELL PE_SC3 00000001
DELL 00000001)
[ 0.000000] ACPI: BERT 000000007f692d5c 00030 (v01 DELL PE_SC3 00000001
DELL 00000001)
[ 0.000000] ACPI: EINJ 000000007f692d8c 00150 (v01 DELL PE_SC3 00000001
DELL 00000001)
[ 0.000000] ACPI: SRAT 000000007f6b3bc0 00370 (v01 DELL PE_SC3 00000001
DELL 00000001)
[ 0.000000] ACPI: TCPA 000000007f6b3f34 00064 (v02 DELL PE_SC3 00000001
DELL 00000001)
[ 0.000000] ACPI: SSDT 000000007f6b7000 043F4 (v01 INTEL PPM RCM 80000001
INTL 20061109)
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] No NUMA configuration found
[ 0.000000] Faking a node at 0000000000000000-000000007f679000
[ 0.000000] Bootmem setup node 0 0000000000000000-000000007f679000
[ 0.000000] NODE_DATA [0000000000008000 - 000000000000ffff]
[ 0.000000] bootmap [0000000000010000 - 000000000001fecf] pages 10
[ 0.000000] (8 early reservations) ==> bootmem [0000000000 - 007f679000]
[ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000
- 0000001000]
[ 0.000000] #1 [0003866000 - 0003887000] XEN PAGETABLES ==> [0003866000
- 0003887000]
[ 0.000000] #2 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000
- 0000008000]
[ 0.000000] #3 [0001000000 - 0001695994] TEXT DATA BSS ==> [0001000000
- 0001695994]
[ 0.000000] #4 [00016b6000 - 0003064200] RAMDISK ==> [00016b6000
- 0003064200]
[ 0.000000] #5 [0003065000 - 0003866000] XEN START INFO ==> [0003065000
- 0003866000]
[ 0.000000] #6 [0001696000 - 00016964fd] BRK ==> [0001696000
- 00016964fd]
[ 0.000000] #7 [0000100000 - 00004db000] PGTABLE ==> [0000100000
- 00004db000]
[ 0.000000] found SMP MP-table at [ffff8800000fe710] fe710
[ 0.000000] Zone PFN ranges:
[ 0.000000] DMA 0x00000000 -> 0x00001000
[ 0.000000] DMA32 0x00001000 -> 0x00100000
[ 0.000000] Normal 0x00100000 -> 0x00100000
[ 0.000000] Movable zone start PFN for each node
[ 0.000000] early_node_map[2] active PFN ranges
[ 0.000000] 0: 0x00000000 -> 0x000000a0
[ 0.000000] 0: 0x00000100 -> 0x0007f679
[ 0.000000] On node 0 totalpages: 521753
[ 0.000000] DMA zone: 56 pages used for memmap
[ 0.000000] DMA zone: 990 pages reserved
[ 0.000000] DMA zone: 2954 pages, LIFO batch:0
[ 0.000000] DMA32 zone: 7079 pages used for memmap
[ 0.000000] DMA32 zone: 510674 pages, LIFO batch:31
[ 0.000000] ACPI: PM-Timer IO Port: 0x808
..



[root@xen27:/]# xm info
host : xen27.c1.wavecon.info
release : 2.6.32-5-xen-amd64
version : #1 SMP Fri Dec 10 17:41:50 UTC 2010
machine : x86_64
nr_cpus : 16
nr_nodes : 2
cores_per_socket : 4
threads_per_core : 2
cpu_mhz : 2394
hw_caps :
bfebfbff:28100800:00000000:00001b40:009ce3bd:00000000:00000001:00000000
virt_caps : hvm hvm_directio
total_memory : 49142
free_memory : 46190
node_to_cpu : node0:0,2,4,6,8,10,12,14
node1:1,3,5,7,9,11,13,15
node_to_memory : node0:21765
node1:24425
node_to_dma32_mem : node0:0
node1:1969
max_node_id : 1
xen_major : 4
xen_minor : 0
xen_extra : .1
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
xen_commandline : console=com2 dom0_mem=max:4GB
cc_compiler : gcc version 4.4.5 20100824 (prerelease) (Debian
4.4.4-11)
cc_compile_by : waldi
cc_compile_domain : debian.org
cc_compile_date : Fri Sep 3 15:38:12 UTC 2010
xend_config_format : 4



numa=off didn't do the trick. I can't tell whether the command line switch is
unknown or the problem yet exists. The output however, is very similar to the
first dmesg and the kernel keeps crashing with the same message. The only
difference is, that passing numa=off doesn't crash the the kernel immediately
but after a bit where no output is shown on the console in the meantime (I
guess Xen probes SRT tables on its own. Please note I have no idea what I'm
talking about here, since I'm not familiar with such low level tasks, although
I do know what NUMA is for).


--
Configure bugmail: http://bugzilla.xensource.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

_______________________________________________
Xen-bugs mailing list
Xen-bugs@lists.xensource.com
http://lists.xensource.com/xen-bugs