Mailing List Archive

On Wed, 2011-12-14 at 08:05 +0000, Liwei wrote:
> Hello list,
> Using the 3.1.5 kernel and 4.1.3-rc1-pre xen (Latest ChangeSet:
> Tue Dec 06 10:53:12 2011 +0000 23199:d9f8316a8291), these crashes
> occur after about a day or so of running a HVM:
> =================8<=================
> (XEN) realmode.c:115:d9 Failed to emulate insn.
> (XEN) realmode.c:165:d9 Real-mode emulation failed @ e40a:00005f6c: d9
> 33 01 fc 30 34

I think this is FNSTENV/FSTENV which doesn't seem to be emulated by
Xen :-( xen/arch/x86/x86_emulate/x86_emulate.c has:
/* case 6: fstenv - TODO */

Ian.

> (XEN) domain_crash called from realmode.c:166
> (XEN) Domain 9 (vcpu#0) crashed on cpu#6:
> (XEN) ----[ Xen-4.1.3-rc1-pre x86_64 debug=n Not tainted ]----
> (XEN) CPU: 6
> (XEN) RIP: e40a:[<0000000000005f6c>]
> (XEN) RFLAGS: 0000000000010002 CONTEXT: hvm guest
> (XEN) rax: 000000000000e7e5 rbx: 0000000000000000 rcx: 00000000000d0000
> (XEN) rdx: 00000000000001f7 rsi: 0000000000000000 rdi: 0000000000000dba
> (XEN) rbp: 0000000000000182 rsp: 0000000000000db0 r8: 0000000000000000
> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
> (XEN) r15: 0000000000000000 cr0: 0000000000000010 cr4: 0000000000000000
> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
> (XEN) ds: 9fc0 es: 9e00 fs: 0000 gs: 0000 ss: 9e00 cs: e40a
> =================8<=================
> (XEN) realmode.c:115:d5 Failed to emulate insn.
> (XEN) realmode.c:165:d5 Real-mode emulation failed @ c5d5:000242bc: d9
> 33 01 fc 30 34
> (XEN) domain_crash called from realmode.c:166
> (XEN) Domain 5 (vcpu#0) crashed on cpu#0:
> (XEN) ----[ Xen-4.1.3-rc1-pre x86_64 debug=n Not tainted ]----
> (XEN) CPU: 0
> (XEN) RIP: c5d5:[<00000000000242bc>]
> (XEN) RFLAGS: 0000000000010013 CONTEXT: hvm guest
> (XEN) rax: 000000000000001b rbx: 0000000000000702 rcx: 0000000000010ba7
> (XEN) rdx: 0000000000008fd5 rsi: 0000000000001333 rdi: 000000000000900e
> (XEN) rbp: 000000000000703e rsp: 000000000000c5cf r8: 0000000000000000
> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
> (XEN) r15: 0000000000000000 cr0: 0000000000000010 cr4: 0000000000000000
> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: c5d5
> (XEN) [VT-D]iommu.c:856: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]iommu.c:831: DMAR:[DMA Write] Request device [0d:00.1]
> fault addr d8964f44000, iommu reg = ffff82c3fff57000
> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
> (XEN) print_vtd_entries: iommu = ffff83045ce565c0 bdf = d:0.1 gmfn = d8964f44
> (XEN) root_entry = ffff8304519d1000
> (XEN) root_entry[d] = 4127af001
> (XEN) context = ffff8304127af000
> (XEN) context[1] = 2_414aca001
> (XEN) l4 = ffff830414aca000
> (XEN) l4_index = 1b
> (XEN) l4[1b] = 0
> (XEN) l4[1b] not present
> =================8<=================
>
> If I try to restart the crashed VM, the same crash always occurs
> again during POST (the last thing I see on the VNC console is the BIOS
> boot, before the connection terminates). A dom0 reboot is necessary to
> get things to work again.
> 0d:00.1 is part of an Intel 82575GB quad port network adapter. The
> adapter contains three ports, 0d:00.0, 0d:00.1, 0e:00.0, 0e:00.1. Each
> of them is assigned to a VM. So far the crashes have occurred to VMs
> using 0d:00.1, 0e:00.0 and 0e:00.1. (Pretty sure a crash involving
> 0d:00.0 will come soon).
> Is this a potential bug, or (more likely?) problematic hardware?
>
> The relevant PCI subtree looks like this:
> =================8<=================
> \-[0000:00]-+-00.0
> +-03.0-[01]--+-00.0
> | \-00.1
> +-05.0-[02-0e]----00.0-[03-0e]--+-00.0-[04]----00.0
>
> +-02.0-[05-0a]----00.0-[06-0a]--+-02.0-[07]----00.0
> |
> +-03.0-[08]----00.0
> |
> +-04.0-[09]----00.0
> |
> \-05.0-[0a]----00.0
>
> \-03.0-[0b-0e]----00.0-[0c-0e]--+-02.0-[0d]--+-00.0
>
> | \-00.1
>
> \-04.0-[0e]--+-00.0
>
> \-00.1
> =================8<=================
>
> Thanks!
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 14/12/2011 09:36, "Ian Campbell" <Ian.Campbell@citrix.com> wrote:

> On Wed, 2011-12-14 at 08:05 +0000, Liwei wrote:
>> Hello list,
>> Using the 3.1.5 kernel and 4.1.3-rc1-pre xen (Latest ChangeSet:
>> Tue Dec 06 10:53:12 2011 +0000 23199:d9f8316a8291), these crashes
>> occur after about a day or so of running a HVM:
>> =================8<=================
>> (XEN) realmode.c:115:d9 Failed to emulate insn.
>> (XEN) realmode.c:165:d9 Real-mode emulation failed @ e40a:00005f6c: d9
>> 33 01 fc 30 34
>
> I think this is FNSTENV/FSTENV which doesn't seem to be emulated by
> Xen :-( xen/arch/x86/x86_emulate/x86_emulate.c has:
> /* case 6: fstenv - TODO */

I wonder why a long-running HVM guest would be in real mode executing FP
code. That seems a bit fishy.

-- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> On 14.12.11 at 09:05, Liwei <xieliwei@gmail.com> wrote:
> Using the 3.1.5 kernel and 4.1.3-rc1-pre xen (Latest ChangeSet:

The kernel version is the HVM guest one or the host one, because ...

> Tue Dec 06 10:53:12 2011 +0000 23199:d9f8316a8291), these crashes
> occur after about a day or so of running a HVM:
> =================8<=================
> (XEN) realmode.c:115:d9 Failed to emulate insn.
> (XEN) realmode.c:165:d9 Real-mode emulation failed @ e40a:00005f6c: d9
> 33 01 fc 30 34

... if this is Linux I'm having a rather hard time seeing why it would
enter realmode at normal run time (and then even use floating point
instructions there). This would be expected at boot time and at most
during guest-internal suspend/resume, but I'm sure you would have
mentioned if the guest was in the process of doing the latter (and
you excluded the former).

I'm rather suspecting the VM to have gone astray earlier.

> (XEN) domain_crash called from realmode.c:166
> (XEN) Domain 9 (vcpu#0) crashed on cpu#6:
> (XEN) ----[ Xen-4.1.3-rc1-pre x86_64 debug=n Not tainted ]----
> (XEN) CPU: 6
> (XEN) RIP: e40a:[<0000000000005f6c>]
> (XEN) RFLAGS: 0000000000010002 CONTEXT: hvm guest
> (XEN) rax: 000000000000e7e5 rbx: 0000000000000000 rcx: 00000000000d0000
> (XEN) rdx: 00000000000001f7 rsi: 0000000000000000 rdi: 0000000000000dba
> (XEN) rbp: 0000000000000182 rsp: 0000000000000db0 r8: 0000000000000000
> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
> (XEN) r15: 0000000000000000 cr0: 0000000000000010 cr4: 0000000000000000
> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
> (XEN) ds: 9fc0 es: 9e00 fs: 0000 gs: 0000 ss: 9e00 cs: e40a
> =================8<=================
> (XEN) realmode.c:115:d5 Failed to emulate insn.
> (XEN) realmode.c:165:d5 Real-mode emulation failed @ c5d5:000242bc: d9
> 33 01 fc 30 34
> (XEN) domain_crash called from realmode.c:166
> (XEN) Domain 5 (vcpu#0) crashed on cpu#0:
> (XEN) ----[ Xen-4.1.3-rc1-pre x86_64 debug=n Not tainted ]----
> (XEN) CPU: 0
> (XEN) RIP: c5d5:[<00000000000242bc>]
> (XEN) RFLAGS: 0000000000010013 CONTEXT: hvm guest
> (XEN) rax: 000000000000001b rbx: 0000000000000702 rcx: 0000000000010ba7
> (XEN) rdx: 0000000000008fd5 rsi: 0000000000001333 rdi: 000000000000900e
> (XEN) rbp: 000000000000703e rsp: 000000000000c5cf r8: 0000000000000000
> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
> (XEN) r15: 0000000000000000 cr0: 0000000000000010 cr4: 0000000000000000
> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: c5d5

Further, you present two instances of a VM death, ...

> (XEN) [VT-D]iommu.c:856: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]iommu.c:831: DMAR:[DMA Write] Request device [0d:00.1]
> fault addr d8964f44000, iommu reg = ffff82c3fff57000
> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
> (XEN) print_vtd_entries: iommu = ffff83045ce565c0 bdf = d:0.1 gmfn =
> d8964f44
> (XEN) root_entry = ffff8304519d1000
> (XEN) root_entry[d] = 4127af001
> (XEN) context = ffff8304127af000
> (XEN) context[1] = 2_414aca001
> (XEN) l4 = ffff830414aca000
> (XEN) l4_index = 1b
> (XEN) l4[1b] = 0
> (XEN) l4[1b] not present

... but only one instance of an IOMMU fault. Hence it's not even clear
they are connected. It also can't be the result of an immediate guest
restart, as the second VM that's dying has a lower numbered domain
ID than the first one.

Further, the IOMMU fault is not about the lack of write access, but the
requested mapping simply is not present at all. Which is because the gmfn
looks rather bogus - I doubt you're running a guest with nearly 14Tb
of memory (Xen doesn't even support that much) or with an extremely
spares physical address space (you would have to have arranged this
yourself, no VMs get created this way afaik).

Please provide consistent and complete information.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 2011-12-14 at 09:41 +0000, Keir Fraser wrote:
> On 14/12/2011 09:36, "Ian Campbell" <Ian.Campbell@citrix.com> wrote:
>
> > On Wed, 2011-12-14 at 08:05 +0000, Liwei wrote:
> >> Hello list,
> >> Using the 3.1.5 kernel and 4.1.3-rc1-pre xen (Latest ChangeSet:
> >> Tue Dec 06 10:53:12 2011 +0000 23199:d9f8316a8291), these crashes
> >> occur after about a day or so of running a HVM:
> >> =================8<=================
> >> (XEN) realmode.c:115:d9 Failed to emulate insn.
> >> (XEN) realmode.c:165:d9 Real-mode emulation failed @ e40a:00005f6c: d9
> >> 33 01 fc 30 34
> >
> > I think this is FNSTENV/FSTENV which doesn't seem to be emulated by
> > Xen :-( xen/arch/x86/x86_emulate/x86_emulate.c has:
> > /* case 6: fstenv - TODO */
>
> I wonder why a long-running HVM guest would be in real mode executing FP
> code. That seems a bit fishy.

I assumed it was some driver calling into a BIOS ROM or something
horrific like that.

RFLAGS was 0x0000000000010002 (RF and the MB1 bit), no VM86 etc in
there.

CS is e40a -- Is that ring 2? surely not... I don't see anything in
Linux which uses that segment (it's not an easy grep though).

So, yes, fishy.

Ian.

>
> -- Keir
>
>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 09:57 +0000 on 14 Dec (1323856663), Jan Beulich wrote:
> > (XEN) realmode.c:115:d9 Failed to emulate insn.
> > (XEN) realmode.c:165:d9 Real-mode emulation failed @ e40a:00005f6c: d9
> > 33 01 fc 30 34
>
> ... if this is Linux I'm having a rather hard time seeing why it would
> enter realmode at normal run time (and then even use floating point
> instructions there). This would be expected at boot time and at most
> during guest-internal suspend/resume, but I'm sure you would have
> mentioned if the guest was in the process of doing the latter (and
> you excluded the former).
>
> I'm rather suspecting the VM to have gone astray earlier.
>
> > (XEN) domain_crash called from realmode.c:166
> > (XEN) Domain 9 (vcpu#0) crashed on cpu#6:
> > (XEN) ----[ Xen-4.1.3-rc1-pre x86_64 debug=n Not tainted ]----
> > (XEN) CPU: 6
> > (XEN) RIP: e40a:[<0000000000005f6c>]
> > (XEN) RFLAGS: 0000000000010002 CONTEXT: hvm guest
> > (XEN) rax: 000000000000e7e5 rbx: 0000000000000000 rcx: 00000000000d0000
> > (XEN) rdx: 00000000000001f7 rsi: 0000000000000000 rdi: 0000000000000dba
> > (XEN) rbp: 0000000000000182 rsp: 0000000000000db0 r8: 0000000000000000
> > (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
> > (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
> > (XEN) r15: 0000000000000000 cr0: 0000000000000010 cr4: 0000000000000000
> > (XEN) cr3: 0000000000000000 cr2: 0000000000000000
> > (XEN) ds: 9fc0 es: 9e00 fs: 0000 gs: 0000 ss: 9e00 cs: e40a

I'd say the guest has recently rebooted - even if it was going back
down to real mode for some reason, I doubt that cr2 and cr3 would be all
zeros.

Tim.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hello all,
My apologies for the unclear email. Here are the events in
chronological order:

======First instance of VM Crash======
Guest is openSuSE 11.4 on 2.6.37.6-0.5-desktop
No symptoms in guest logs, logging just stopped.
Looking at the serial logs, it appears that the guest just vapourised
(unless the "First re-creation of guest" below is actually the first
crash, though Jan and Tim mentioned that that crash couldn't have
occurred during run-time).
This VM has one part of the Intel 82575GB (0e:00.1) pass-throughed.

++++First re-creation of guest++++
xl create /etc/xen/vm-test4.cfg

----Xen log----
(XEN) HVM7: ata0-0: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
(XEN) HVM7: ata0 master: QEMU HARDDISK ATA-7 Hard-Disk ( 160 GBytes)
(XEN) realmode.c:115:d7 Failed to emulate insn.
(XEN) realmode.c:165:d7 Real-mode emulation failed @ a8c0:00005a8e: ff
fa 00 15 99 2c
(XEN) domain_crash called from realmode.c:166
(XEN) Domain 7 (vcpu#0) crashed on cpu#3:
(XEN) ----[ Xen-4.1.3-rc1-pre x86_64 debug=n Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: a8c0:[<0000000000005a8e>]
(XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest
(XEN) rax: 0000000000000120 rbx: 0000000000000000 rcx: 00000000000dffff
(XEN) rdx: 00000000000001f7 rsi: 000000000000ffff rdi: 000000000000fffe
(XEN) rbp: 0000000000000db5 rsp: 000000000000eff8 r8: 0000000000000000
(XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
(XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
(XEN) r15: 0000000000000000 cr0: 0000000000000010 cr4: 0000000000000000
(XEN) cr3: 0000000000000000 cr2: 0000000000000000
(XEN) ds: 9fc0 es: 9e00 fs: 0000 gs: 0000 ss: 9e00 cs: a8c0

++++Second re-creation of guest++++
xl create /etc/xen/vm-test4.cfg

(XEN) HVM8: ata0-0: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
(XEN) HVM8: ata0 master: QEMU HARDDISK ATA-7 Hard-Disk ( 160 GBytes)
(XEN) realmode.c:115:d8 Failed to emulate insn.
(XEN) realmode.c:165:d8 Real-mode emulation failed @ a8c0:0000fcb5: 72
5c 1b 80 c4 82
(XEN) domain_crash called from realmode.c:166
(XEN) Domain 8 (vcpu#0) crashed on cpu#5:
(XEN) ----[ Xen-4.1.3-rc1-pre x86_64 debug=n Not tainted ]----
(XEN) CPU: 5
(XEN) RIP: a8c0:[<000000000000fcb5>]
(XEN) RFLAGS: 0000000000010002 CONTEXT: hvm guest
(XEN) rax: 0000000000000000 rbx: 0000000000000000 rcx: 00000000000d0000
(XEN) rdx: 00000000000001f7 rsi: 0000000000000000 rdi: 0000000000000dba
(XEN) rbp: 0000000000000db8 rsp: 0000000000000d38 r8: 0000000000000000
(XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
(XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
(XEN) r15: 0000000000000000 cr0: 0000000000000010 cr4: 0000000000000000
(XEN) cr3: 0000000000000000 cr2: 0000000000000000
(XEN) ds: 9fc0 es: 9e00 fs: 0000 gs: 0000 ss: 9e00 cs: a8c0

[.Did the same thing a few more times with similar results and gave up,
rebooted dom0 and all was well]


[About 20 hours later]
======Second instance of VM Crash======
Guest is pfSense 2.1 on FreeBSD 8.1-RELEASE-p6
Unfortunately, I didn't get the chance to extract the guest logs as
they're purged on boot.
Similarly, looking at the serial logs, it appears that the guest just
vapourised (unless the "First re-creation of guest" below is actually
the first crash, though Jan and Tim mentioned that that crash couldn't
have occurred during run-time).
This VM has two parts of the Intel 82575GB (0d:00.0 and 0d:00.1) pass-throughed.

++++First re-creation of guest++++
xl create /etc/xen/vm-pfsense.cfg

(XEN) HVM5: Booting from Hard Disk...
(XEN) HVM5: Booting from 0000:7c00
(XEN) realmode.c:115:d5 Failed to emulate insn.
(XEN) realmode.c:165:d5 Real-mode emulation failed @ c5d5:000242bc: d9
33 01 fc 30 34
(XEN) domain_crash called from realmode.c:166
(XEN) Domain 5 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-4.1.3-rc1-pre x86_64 debug=n Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: c5d5:[<00000000000242bc>]
(XEN) RFLAGS: 0000000000010013 CONTEXT: hvm guest
(XEN) rax: 000000000000001b rbx: 0000000000000702 rcx: 0000000000010ba7
(XEN) rdx: 0000000000008fd5 rsi: 0000000000001333 rdi: 000000000000900e
(XEN) rbp: 000000000000703e rsp: 000000000000c5cf r8: 0000000000000000
(XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
(XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
(XEN) r15: 0000000000000000 cr0: 0000000000000010 cr4: 0000000000000000
(XEN) cr3: 0000000000000000 cr2: 0000000000000000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: c5d5
(XEN) [VT-D]iommu.c:856: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:831: DMAR:[DMA Write] Request device [0d:00.1]
fault addr d8964f44000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83045ce565c0 bdf = d:0.1 gmfn = d8964f44
(XEN) root_entry = ffff8304519d1000
(XEN) root_entry[d] = 4127af001
(XEN) context = ffff8304127af000
(XEN) context[1] = 2_414aca001
(XEN) l4 = ffff830414aca000
(XEN) l4_index = 1b
(XEN) l4[1b] = 0
(XEN) l4[1b] not present

++++Second re-creation of guest++++
xl create /etc/xen/vm-pfsense.cfg

(XEN) HVM6: Options: apmbios pcibios eltorito PMM
(XEN) HVM6:
(XEN) realmode.c:115:d6 Failed to emulate insn.
(XEN) realmode.c:165:d6 Real-mode emulation failed @ a8c0:00005a83: 72
5c 1b 80 c4 82
(XEN) domain_crash called from realmode.c:166
(XEN) Domain 6 (vcpu#0) crashed on cpu#6:
(XEN) ----[ Xen-4.1.3-rc1-pre x86_64 debug=n Not tainted ]----
(XEN) CPU: 6
(XEN) RIP: a8c0:[<0000000000005a83>]
(XEN) RFLAGS: 0000000000010086 CONTEXT: hvm guest
(XEN) rax: 0000000000000000 rbx: 0000000000000db4 rcx: 00000000000f0027
(XEN) rdx: 0000000000000004 rsi: 0000000000000006 rdi: 0000000000000db2
(XEN) rbp: 0000000000000dc8 rsp: 000000000000df33 r8: 0000000000000000
(XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
(XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
(XEN) r15: 0000000000000000 cr0: 0000000000000010 cr4: 0000000000000000
(XEN) cr3: 0000000000000000 cr2: 0000000000000000
(XEN) ds: 9fc0 es: c000 fs: 0000 gs: 0000 ss: 9e00 cs: a8c0
(XEN) [VT-D]iommu.c:856: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:831: DMAR:[DMA Write] Request device [0d:00.1]
fault addr 1a16ee8000000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 04h] Access beyond MGAW
(XEN) print_vtd_entries: iommu = ffff83045ce565c0 bdf = d:0.1 gmfn = 1a16ee8000
(XEN) root_entry = ffff8304519d1000
(XEN) root_entry[d] = 4127af001
(XEN) context = ffff8304127af000
(XEN) context[1] = 102_2159b1001
(XEN) l4 = ffff8302159b1000
(XEN) l4_index = 142
(XEN) l4[142] = 0
(XEN) l4[142] not present

[.Did the same thing a few more times with similar results and gave up,
rebooted dom0 and all was well]

Typical log of a booting openSuSE guest:
(XEN) HVM2: HVM Loader
(XEN) HVM2: Detected Xen v4.1.3-rc1-pre
(XEN) HVM2: CPU speed is 2930 MHz
(XEN) HVM2: Xenbus rings @0xfeffc000, event channel 9
(XEN) irq.c:264: Dom2 PCI link 0 changed 0 -> 5
(XEN) HVM2: PCI-ISA link 0 routed to IRQ5
(XEN) irq.c:264: Dom2 PCI link 1 changed 0 -> 10
(XEN) HVM2: PCI-ISA link 1 routed to IRQ10
(XEN) irq.c:264: Dom2 PCI link 2 changed 0 -> 11
(XEN) HVM2: PCI-ISA link 2 routed to IRQ11
(XEN) irq.c:264: Dom2 PCI link 3 changed 0 -> 5
(XEN) HVM2: PCI-ISA link 3 routed to IRQ5
(XEN) HVM2: pci dev 01:2 INTD->IRQ5
(XEN) HVM2: pci dev 01:3 INTA->IRQ10
(XEN) HVM2: pci dev 03:0 INTA->IRQ5
(XEN) HVM2: pci dev 04:0 INTA->IRQ5
(XEN) HVM2: pci dev 05:0 INTB->IRQ11
(XEN) HVM2: pci dev 02:0 bar 10 size 02000000: f0000008
(XEN) HVM2: pci dev 03:0 bar 14 size 01000000: f2000008
(XEN) HVM2: pci dev 05:0 bar 14 size 00200000: f3000000
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3000 mfn=fb400 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3200 mfn=fb620 nr_mfns=20
(XEN) HVM2: pci dev 05:0 bar 10 size 00020000: f3200000
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3220 mfn=fb644 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3220 mfn=fb644 nr_mfns=1
(XEN) HVM2: pci dev 05:0 bar 1c size 00004000: f3220000
(XEN) HVM2: pci dev 02:0 bar 14 size 00001000: f3224000
(XEN) HVM2: pci dev 04:0 bar 10 size 00000400: 0000c001
(XEN) HVM2: pci dev 03:0 bar 10 size 00000100: 0000c401
(XEN) HVM2: pci dev 04:0 bar 14 size 00000100: 0000c501
(XEN) HVM2: pci dev 01:2 bar 20 size 00000020: 0000c601
(XEN) HVM2: pci dev 05:0 bar 18 size 00000020: 0000c621
(XEN) domctl.c:1038:d0 ioport_map:add f_gport=c620 f_mport=a880 np=20
(XEN) HVM2: pci dev 01:1 bar 20 size 00000010: 0000c641
(XEN) HVM2: Multiprocessor initialisation:
(XEN) HVM2: - CPU0 ... 36-bit phys ... fixed MTRRs ... var MTRRs
[2/8] ... done.
(XEN) HVM2: - CPU1 ... 36-bit phys ... fixed MTRRs ... var MTRRs
[2/8] ... done.
(XEN) HVM2: - CPU2 ... 36-bit phys ... fixed MTRRs ... var MTRRs
[2/8] ... done.
(XEN) HVM2: - CPU3 ... 36-bit phys ... fixed MTRRs ... var MTRRs
[2/8] ... done.
(XEN) HVM2: - CPU4 ... 36-bit phys ... fixed MTRRs ... var MTRRs
[2/8] ... done.
(XEN) HVM2: - CPU5 ... 36-bit phys ... fixed MTRRs ... var MTRRs
[2/8] ... done.
(XEN) HVM2: - CPU6 ... 36-bit phys ... fixed MTRRs ... var MTRRs
[2/8] ... done.
(XEN) HVM2: - CPU7 ... 36-bit phys ... fixed MTRRs ... var MTRRs
[2/8] ... done.
(XEN) HVM2: Writing SMBIOS tables ...
(XEN) HVM2: Loading ROMBIOS ...
(XEN) HVM2: 12604 bytes of ROMBIOS high-memory extensions:
(XEN) HVM2: Relocating to 0xfc000000-0xfc00313c ... done
(XEN) HVM2: Creating MP tables ...
(XEN) HVM2: Loading Cirrus VGABIOS ...
(XEN) HVM2: Loading ACPI ...
(XEN) HVM2: - Lo data: 000ea020-000ea04f
(XEN) HVM2: - Hi data: fc003400-fc01355f
(XEN) HVM2: vm86 TSS at fc013800
(XEN) HVM2: BIOS map:
(XEN) HVM2: c0000-c8fff: VGA BIOS
(XEN) HVM2: eb000-eb26b: SMBIOS tables
(XEN) HVM2: f0000-fffff: Main BIOS
(XEN) HVM2: E820 table:
(XEN) HVM2: [00]: 00000000:00000000 - 00000000:0009e000: RAM
(XEN) HVM2: [01]: 00000000:0009e000 - 00000000:0009fc00: RESERVED
(XEN) HVM2: [02]: 00000000:0009fc00 - 00000000:000a0000: RESERVED
(XEN) HVM2: HOLE: 00000000:000a0000 - 00000000:000e0000
(XEN) HVM2: [03]: 00000000:000e0000 - 00000000:00100000: RESERVED
(XEN) HVM2: [04]: 00000000:00100000 - 00000000:7f800000: RAM
(XEN) HVM2: HOLE: 00000000:7f800000 - 00000000:fc000000
(XEN) HVM2: [05]: 00000000:fc000000 - 00000001:00000000: RESERVED
(XEN) HVM2: Invoking ROMBIOS ...
(XEN) HVM2: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $
(XEN) stdvga.c:147:d2 entering stdvga and caching modes
(XEN) HVM2: VGABios $Id: vgabios.c,v 1.67 2008/01/27 09:44:12 vruppert Exp $
(XEN) HVM2: Bochs BIOS - build: 06/23/99
(XEN) HVM2: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $
(XEN) HVM2: Options: apmbios pcibios eltorito PMM
(XEN) HVM2:
(XEN) HVM2: ata0-0: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
(XEN) HVM2: ata0 master: QEMU HARDDISK ATA-7 Hard-Disk ( 160 GBytes)
(XEN) HVM2: IDE time out
(XEN) HVM2:
(XEN) HVM2:
(XEN) HVM2:
(XEN) HVM2: Press F12 for boot menu.
(XEN) HVM2:
(XEN) HVM2: Booting from Hard Disk...
(XEN) HVM2: Booting from 0000:7c00
(XEN) HVM2: int13_harddisk: function 41, unmapped device for ELDL=81
(XEN) HVM2: int13_harddisk: function 08, unmapped device for ELDL=81
(XEN) HVM2: *** int 15h function AX=00c0, BX=0000 not yet supported!
(XEN) stdvga.c:151:d2 leaving stdvga
(XEN) stdvga.c:147:d2 entering stdvga and caching modes
(XEN) HVM2: *** int 15h function AX=ec00, BX=0002 not yet supported!
(XEN) HVM2: KBD: unsupported int 16h function 03
(XEN) HVM2: *** int 15h function AX=e980, BX=0000 not yet supported!
(XEN) HVM2: int13_harddisk: function 41, unmapped device for ELDL=81
(XEN) HVM2: int13_harddisk: function 02, unmapped device for ELDL=81
(XEN) HVM2: int13_harddisk: function 41, unmapped device for ELDL=82
(XEN) HVM2: int13_harddisk: function 02, unmapped device for ELDL=82
(XEN) HVM2: int13_harddisk: function 41, unmapped device for ELDL=83
(XEN) HVM2: int13_harddisk: function 02, unmapped device for ELDL=83
(XEN) HVM2: int13_harddisk: function 41, unmapped device for ELDL=84
(XEN) HVM2: int13_harddisk: function 02, unmapped device for ELDL=84
(XEN) HVM2: int13_harddisk: function 41, unmapped device for ELDL=85
(XEN) HVM2: int13_harddisk: function 02, unmapped device for ELDL=85
(XEN) HVM2: int13_harddisk: function 41, unmapped device for ELDL=86
(XEN) HVM2: int13_harddisk: function 02, unmapped device for ELDL=86
(XEN) HVM2: int13_harddisk: function 41, unmapped device for ELDL=87
(XEN) HVM2: int13_harddisk: function 02, unmapped device for ELDL=87
(XEN) HVM2: int13_harddisk: function 41, ELDL out of range 88
(XEN) HVM2: int13_harddisk: function 02, ELDL out of range 88
(XEN) HVM2: int13_harddisk: function 41, ELDL out of range 89
(XEN) HVM2: int13_harddisk: function 02, ELDL out of range 89
(XEN) HVM2: int13_harddisk: function 41, ELDL out of range 8a
(XEN) HVM2: int13_harddisk: function 02, ELDL out of range 8a
(XEN) HVM2: int13_harddisk: function 41, ELDL out of range 8b
(XEN) HVM2: int13_harddisk: function 02, ELDL out of range 8b
(XEN) HVM2: int13_harddisk: function 41, ELDL out of range 8c
(XEN) HVM2: int13_harddisk: function 02, ELDL out of range 8c
(XEN) HVM2: int13_harddisk: function 41, ELDL out of range 8d
(XEN) HVM2: int13_harddisk: function 02, ELDL out of range 8d
(XEN) HVM2: int13_harddisk: function 41, ELDL out of range 8e
(XEN) HVM2: int13_harddisk: function 02, ELDL out of range 8e
(XEN) HVM2: int13_harddisk: function 41, ELDL out of range 8f
(XEN) HVM2: int13_harddisk: function 02, ELDL out of range 8f
(XEN) stdvga.c:151:d2 leaving stdvga
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3200 mfn=fb620 nr_mfns=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3000 mfn=fb400 nr_mfns=200
(XEN) domctl.c:1062:d0 ioport_map:remove f_gport=c620 f_mport=a880 np=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3220 mfn=fb644 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3220 mfn=fb644 nr_mfns=4
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3200 mfn=fb620 nr_mfns=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3000 mfn=fb400 nr_mfns=200
(XEN) domctl.c:1038:d0 ioport_map:add f_gport=c620 f_mport=a880 np=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3220 mfn=fb644 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3220 mfn=fb644 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3200 mfn=fb620 nr_mfns=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3000 mfn=fb400 nr_mfns=200
(XEN) domctl.c:1062:d0 ioport_map:remove f_gport=c620 f_mport=a880 np=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3220 mfn=fb644 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3220 mfn=fb644 nr_mfns=4
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3200 mfn=fb620 nr_mfns=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3000 mfn=fb400 nr_mfns=200
(XEN) domctl.c:1038:d0 ioport_map:add f_gport=c620 f_mport=a880 np=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3220 mfn=fb644 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3220 mfn=fb644 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3200 mfn=fb620 nr_mfns=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3000 mfn=fb400 nr_mfns=200
(XEN) domctl.c:1062:d0 ioport_map:remove f_gport=c620 f_mport=a880 np=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3220 mfn=fb644 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3220 mfn=fb644 nr_mfns=4
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3200 mfn=fb620 nr_mfns=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3000 mfn=fb400 nr_mfns=200
(XEN) domctl.c:1038:d0 ioport_map:add f_gport=c620 f_mport=a880 np=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3220 mfn=fb644 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3220 mfn=fb644 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3200 mfn=fb620 nr_mfns=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3000 mfn=fb400 nr_mfns=200
(XEN) domctl.c:1062:d0 ioport_map:remove f_gport=c620 f_mport=a880 np=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3220 mfn=fb644 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3220 mfn=fb644 nr_mfns=4
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3200 mfn=fb620 nr_mfns=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3000 mfn=fb400 nr_mfns=200
(XEN) domctl.c:1038:d0 ioport_map:add f_gport=c620 f_mport=a880 np=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3220 mfn=fb644 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3220 mfn=fb644 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3200 mfn=fb620 nr_mfns=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3000 mfn=fb400 nr_mfns=200
(XEN) domctl.c:1062:d0 ioport_map:remove f_gport=c620 f_mport=a880 np=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3220 mfn=fb644 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3220 mfn=fb644 nr_mfns=4
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3200 mfn=fb620 nr_mfns=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3000 mfn=fb400 nr_mfns=200
(XEN) domctl.c:1038:d0 ioport_map:add f_gport=c620 f_mport=a880 np=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3220 mfn=fb644 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3220 mfn=fb644 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3200 mfn=fb620 nr_mfns=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3000 mfn=fb400 nr_mfns=200
(XEN) domctl.c:1062:d0 ioport_map:remove f_gport=c620 f_mport=a880 np=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3220 mfn=fb644 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3220 mfn=fb644 nr_mfns=4
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3200 mfn=fb620 nr_mfns=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3000 mfn=fb400 nr_mfns=200
(XEN) domctl.c:1038:d0 ioport_map:add f_gport=c620 f_mport=a880 np=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3220 mfn=fb644 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3220 mfn=fb644 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3200 mfn=fb620 nr_mfns=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3000 mfn=fb400 nr_mfns=200
(XEN) domctl.c:1062:d0 ioport_map:remove f_gport=c620 f_mport=a880 np=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3220 mfn=fb644 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3220 mfn=fb644 nr_mfns=4
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3200 mfn=fb620 nr_mfns=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3000 mfn=fb400 nr_mfns=200
(XEN) domctl.c:1038:d0 ioport_map:add f_gport=c620 f_mport=a880 np=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3220 mfn=fb644 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3220 mfn=fb644 nr_mfns=1
(XEN) irq.c:264: Dom2 PCI link 0 changed 5 -> 0
(XEN) irq.c:264: Dom2 PCI link 1 changed 10 -> 0
(XEN) irq.c:264: Dom2 PCI link 2 changed 11 -> 0
(XEN) irq.c:264: Dom2 PCI link 3 changed 5 -> 0
(XEN) irq.c:330: Dom2 callback via changed to PCI INTx Dev 0x03 IntA
(XEN) msi.c:867: MSI is already in use on device 0e:00.1
(XEN) msi.c:867: MSI is already in use on device 0e:00.1
(XEN) msi.c:867: MSI is already in use on device 0e:00.1
(XEN) msi.c:867: MSI is already in use on device 0e:00.1
(XEN) msi.c:867: MSI is already in use on device 0e:00.1
(XEN) msi.c:867: MSI is already in use on device 0e:00.1
(XEN) msi.c:867: MSI is already in use on device 0e:00.1
(XEN) msi.c:867: MSI is already in use on device 0e:00.1
(XEN) msi.c:867: MSI is already in use on device 0e:00.1
(XEN) physdev.c:170: dom3: 16:-1 already mapped to 16
(XEN) physdev.c:170: dom3: 19:-1 already mapped to 20

Typical log of a booting pfSense guest:
(XEN) HVM1: HVM Loader
(XEN) HVM1: Detected Xen v4.1.3-rc1-pre
(XEN) HVM1: CPU speed is 2930 MHz
(XEN) HVM1: Xenbus rings @0xfeffc000, event channel 9
(XEN) irq.c:264: Dom1 PCI link 0 changed 0 -> 5
(XEN) HVM1: PCI-ISA link 0 routed to IRQ5
(XEN) irq.c:264: Dom1 PCI link 1 changed 0 -> 10
(XEN) HVM1: PCI-ISA link 1 routed to IRQ10
(XEN) irq.c:264: Dom1 PCI link 2 changed 0 -> 11
(XEN) HVM1: PCI-ISA link 2 routed to IRQ11
(XEN) irq.c:264: Dom1 PCI link 3 changed 0 -> 5
(XEN) HVM1: PCI-ISA link 3 routed to IRQ5
(XEN) HVM1: pci dev 01:3 INTA->IRQ10
(XEN) HVM1: pci dev 03:0 INTA->IRQ5
(XEN) HVM1: pci dev 04:0 INTA->IRQ5
(XEN) HVM1: pci dev 05:0 INTB->IRQ11
(XEN) HVM1: pci dev 02:0 bar 10 size 02000000: f0000008
(XEN) HVM1: pci dev 03:0 bar 14 size 01000000: f2000008
(XEN) HVM1: pci dev 04:0 bar 14 size 00200000: f3000000
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3000 mfn=faa00 nr_mfns=200
(XEN) HVM1: pci dev 05:0 bar 14 size 00200000: f3200000
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3200 mfn=fac00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3400 mfn=fa900 nr_mfns=20
(XEN) HVM1: pci dev 04:0 bar 10 size 00020000: f3400000
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3420 mfn=fa920 nr_mfns=20
(XEN) HVM1: pci dev 05:0 bar 10 size 00020000: f3420000
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3440 mfn=fa940 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3440 mfn=fa940 nr_mfns=1
(XEN) HVM1: pci dev 04:0 bar 1c size 00004000: f3440000
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3444 mfn=fa944 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3444 mfn=fa944 nr_mfns=1
(XEN) HVM1: pci dev 05:0 bar 1c size 00004000: f3444000
(XEN) HVM1: pci dev 02:0 bar 14 size 00001000: f3448000
(XEN) HVM1: pci dev 03:0 bar 10 size 00000100: 0000c001
(XEN) HVM1: pci dev 04:0 bar 18 size 00000020: 0000c101
(XEN) domctl.c:1038:d0 ioport_map:add f_gport=c100 f_mport=9c00 np=20
(XEN) HVM1: pci dev 05:0 bar 18 size 00000020: 0000c121
(XEN) domctl.c:1038:d0 ioport_map:add f_gport=c120 f_mport=9880 np=20
(XEN) HVM1: pci dev 01:1 bar 20 size 00000010: 0000c141
(XEN) HVM1: Multiprocessor initialisation:
(XEN) HVM1: - CPU0 ... 36-bit phys ... fixed MTRRs ... var MTRRs
[2/8] ... done.
(XEN) HVM1: - CPU1 ... 36-bit phys ... fixed MTRRs ... var MTRRs
[2/8] ... done.
(XEN) HVM1: - CPU2 ... 36-bit phys ... fixed MTRRs ... var MTRRs
[2/8] ... done.
(XEN) HVM1: - CPU3 ... 36-bit phys ... fixed MTRRs ... var MTRRs
[2/8] ... done.
(XEN) HVM1: - CPU4 ... 36-bit phys ... fixed MTRRs ... var MTRRs
[2/8] ... done.
(XEN) HVM1: - CPU5 ... 36-bit phys ... fixed MTRRs ... var MTRRs
[2/8] ... done.
(XEN) HVM1: - CPU6 ... 36-bit phys ... fixed MTRRs ... var MTRRs
[2/8] ... done.
(XEN) HVM1: - CPU7 ... 36-bit phys ... fixed MTRRs ... var MTRRs
[2/8] ... done.
(XEN) HVM1: Writing SMBIOS tables ...
(XEN) HVM1: Loading ROMBIOS ...
(XEN) HVM1: 12604 bytes of ROMBIOS high-memory extensions:
(XEN) HVM1: Relocating to 0xfc000000-0xfc00313c ... done
(XEN) HVM1: Creating MP tables ...
(XEN) HVM1: Loading Cirrus VGABIOS ...
(XEN) HVM1: Loading ACPI ...
(XEN) HVM1: - Lo data: 000ea020-000ea04f
(XEN) HVM1: - Hi data: fc003400-fc01355f
(XEN) HVM1: vm86 TSS at fc013800
(XEN) HVM1: BIOS map:
(XEN) HVM1: c0000-c8fff: VGA BIOS
(XEN) HVM1: eb000-eb26b: SMBIOS tables
(XEN) HVM1: f0000-fffff: Main BIOS
(XEN) HVM1: E820 table:
(XEN) HVM1: [00]: 00000000:00000000 - 00000000:0009e000: RAM
(XEN) HVM1: [01]: 00000000:0009e000 - 00000000:0009fc00: RESERVED
(XEN) HVM1: [02]: 00000000:0009fc00 - 00000000:000a0000: RESERVED
(XEN) HVM1: HOLE: 00000000:000a0000 - 00000000:000e0000
(XEN) HVM1: [03]: 00000000:000e0000 - 00000000:00100000: RESERVED
(XEN) HVM1: [04]: 00000000:00100000 - 00000000:3f800000: RAM
(XEN) HVM1: HOLE: 00000000:3f800000 - 00000000:fc000000
(XEN) HVM1: [05]: 00000000:fc000000 - 00000001:00000000: RESERVED
(XEN) HVM1: Invoking ROMBIOS ...
(XEN) HVM1: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $
(XEN) stdvga.c:147:d1 entering stdvga and caching modes
(XEN) HVM1: VGABios $Id: vgabios.c,v 1.67 2008/01/27 09:44:12 vruppert Exp $
(XEN) HVM1: Bochs BIOS - build: 06/23/99
(XEN) HVM1: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $
(XEN) HVM1: Options: apmbios pcibios eltorito PMM
(XEN) HVM1:
(XEN) HVM1: ata0 master: QEMU DVD-ROM ATAPI-4 CD-Rom/DVD-Rom
(XEN) HVM1: ata0-1: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
(XEN) HVM1: ata0 slave: QEMU HARDDISK ATA-7 Hard-Disk (16384 MBytes)
(XEN) HVM1:
(XEN) HVM1:
(XEN) HVM1:
(XEN) HVM1: Press F12 for boot menu.
(XEN) HVM1:
(XEN) HVM1: Booting from Hard Disk...
(XEN) HVM1: Booting from 0000:7c00
(XEN) irq.c:264: Dom1 PCI link 0 changed 5 -> 0
(XEN) irq.c:264: Dom1 PCI link 1 changed 10 -> 0
(XEN) irq.c:264: Dom1 PCI link 2 changed 11 -> 0
(XEN) irq.c:264: Dom1 PCI link 3 changed 5 -> 0
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3400 mfn=fa900 nr_mfns=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3000 mfn=faa00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3440 mfn=fa940 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3440 mfn=fa940 nr_mfns=4
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3400 mfn=fa900 nr_mfns=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3000 mfn=faa00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3440 mfn=fa940 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3440 mfn=fa940 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3400 mfn=fa900 nr_mfns=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3000 mfn=faa00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3440 mfn=fa940 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3440 mfn=fa940 nr_mfns=4
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3400 mfn=fa900 nr_mfns=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3000 mfn=faa00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3440 mfn=fa940 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3440 mfn=fa940 nr_mfns=1
(XEN) domctl.c:1062:d0 ioport_map:remove f_gport=c100 f_mport=9c00 np=20
(XEN) domctl.c:1038:d0 ioport_map:add f_gport=c100 f_mport=9c00 np=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3400 mfn=fa900 nr_mfns=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3000 mfn=faa00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3440 mfn=fa940 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3440 mfn=fa940 nr_mfns=4
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3400 mfn=fa900 nr_mfns=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3000 mfn=faa00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3440 mfn=fa940 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3440 mfn=fa940 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3400 mfn=fa900 nr_mfns=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3000 mfn=faa00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3440 mfn=fa940 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3440 mfn=fa940 nr_mfns=4
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3400 mfn=fa900 nr_mfns=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3000 mfn=faa00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3440 mfn=fa940 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3440 mfn=fa940 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3400 mfn=fa900 nr_mfns=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3000 mfn=faa00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3440 mfn=fa940 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3440 mfn=fa940 nr_mfns=4
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3400 mfn=fa900 nr_mfns=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3000 mfn=faa00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3440 mfn=fa940 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3440 mfn=fa940 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3420 mfn=fa920 nr_mfns=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3200 mfn=fac00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3444 mfn=fa944 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3444 mfn=fa944 nr_mfns=4
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3420 mfn=fa920 nr_mfns=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3200 mfn=fac00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3444 mfn=fa944 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3444 mfn=fa944 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3420 mfn=fa920 nr_mfns=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3200 mfn=fac00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3444 mfn=fa944 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3444 mfn=fa944 nr_mfns=4
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3420 mfn=fa920 nr_mfns=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3200 mfn=fac00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3444 mfn=fa944 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3444 mfn=fa944 nr_mfns=1
(XEN) domctl.c:1062:d0 ioport_map:remove f_gport=c120 f_mport=9880 np=20
(XEN) domctl.c:1038:d0 ioport_map:add f_gport=c120 f_mport=9880 np=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3420 mfn=fa920 nr_mfns=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3200 mfn=fac00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3444 mfn=fa944 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3444 mfn=fa944 nr_mfns=4
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3420 mfn=fa920 nr_mfns=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3200 mfn=fac00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3444 mfn=fa944 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3444 mfn=fa944 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3420 mfn=fa920 nr_mfns=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3200 mfn=fac00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3444 mfn=fa944 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3444 mfn=fa944 nr_mfns=4
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3420 mfn=fa920 nr_mfns=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3200 mfn=fac00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3444 mfn=fa944 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3444 mfn=fa944 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3420 mfn=fa920 nr_mfns=20
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3200 mfn=fac00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3444 mfn=fa944 nr_mfns=1
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3444 mfn=fa944 nr_mfns=4
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3420 mfn=fa920 nr_mfns=20
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3200 mfn=fac00 nr_mfns=200
(XEN) domctl.c:982:d0 memory_map:add: gfn=f3444 mfn=fa944 nr_mfns=4
(XEN) domctl.c:992:d0 memory_map:remove: gfn=f3444 mfn=fa944 nr_mfns=1
(XEN) msi.c:867: MSI is already in use on device 0d:00.0
(XEN) msi.c:867: MSI is already in use on device 0d:00.0
(XEN) msi.c:867: MSI is already in use on device 0d:00.0
(XEN) msi.c:867: MSI is already in use on device 0d:00.0
(XEN) msi.c:867: MSI is already in use on device 0d:00.0
(XEN) msi.c:867: MSI is already in use on device 0d:00.1
(XEN) msi.c:867: MSI is already in use on device 0d:00.1
(XEN) msi.c:867: MSI is already in use on device 0d:00.1
(XEN) msi.c:867: MSI is already in use on device 0d:00.1
(XEN) msi.c:867: MSI is already in use on device 0d:00.1

Thanks!

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> On 14.12.11 at 15:31, Liwei <xieliwei@gmail.com> wrote:
> ======First instance of VM Crash======
> Guest is openSuSE 11.4 on 2.6.37.6-0.5-desktop
> No symptoms in guest logs, logging just stopped.
> Looking at the serial logs, it appears that the guest just vapourised
> (unless the "First re-creation of guest" below is actually the first
> crash, though Jan and Tim mentioned that that crash couldn't have
> occurred during run-time).
> This VM has one part of the Intel 82575GB (0e:00.1) pass-throughed.
>
> ++++First re-creation of guest++++
> xl create /etc/xen/vm-test4.cfg
>
> ----Xen log----
> (XEN) HVM7: ata0-0: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
> (XEN) HVM7: ata0 master: QEMU HARDDISK ATA-7 Hard-Disk ( 160 GBytes)
> (XEN) realmode.c:115:d7 Failed to emulate insn.
> (XEN) realmode.c:165:d7 Real-mode emulation failed @ a8c0:00005a8e: ff
> fa 00 15 99 2c
> (XEN) domain_crash called from realmode.c:166
>
> ++++Second re-creation of guest++++
> xl create /etc/xen/vm-test4.cfg
>
> (XEN) HVM8: ata0-0: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
> (XEN) HVM8: ata0 master: QEMU HARDDISK ATA-7 Hard-Disk ( 160 GBytes)
> (XEN) realmode.c:115:d8 Failed to emulate insn.
> (XEN) realmode.c:165:d8 Real-mode emulation failed @ a8c0:0000fcb5: 72
> 5c 1b 80 c4 82
> (XEN) domain_crash called from realmode.c:166
>
> [.Did the same thing a few more times with similar results and gave up,
> rebooted dom0 and all was well]

So this suggests that the virtual firmware has a problem coming back
up with the passed through device still active (and the IOMMU may in
fact be shielding the rest of the system from bad effects of this).

This by itself is a problem, and the sudden death of the guest is
another (particularly if indeed there's no trace of that event in any
logs). Debugging this may be difficult for anyone who can't
reproduce it, so we'll have to rely on you providing more insight.
Enable more debugging/verbosity in the hypervisor and try again is
probably all I can suggest at this point.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 14 December 2011 22:52, Jan Beulich <JBeulich@suse.com> wrote:
>
> So this suggests that the virtual firmware has a problem coming back
> up with the passed through device still active (and the IOMMU may in
> fact be shielding the rest of the system from bad effects of this).
>
> This by itself is a problem, and the sudden death of the guest is
> another (particularly if indeed there's no trace of that event in any
> logs). Debugging this may be difficult for anyone who can't
> reproduce it, so we'll have to rely on you providing more insight.
> Enable more debugging/verbosity in the hypervisor and try again is
> probably all I can suggest at this point.

Currently I have earlyprintk=xen, loglvl=all and guest_loglvl=all.
Other than adding loglevel=10 and debug initcall_debug, what other
options and/or procedures do you suggest?

>
> Jan
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> On 14.12.11 at 15:59, Liwei <xieliwei@gmail.com> wrote:
> On 14 December 2011 22:52, Jan Beulich <JBeulich@suse.com> wrote:
>>
>> So this suggests that the virtual firmware has a problem coming back
>> up with the passed through device still active (and the IOMMU may in
>> fact be shielding the rest of the system from bad effects of this).
>>
>> This by itself is a problem, and the sudden death of the guest is
>> another (particularly if indeed there's no trace of that event in any
>> logs). Debugging this may be difficult for anyone who can't
>> reproduce it, so we'll have to rely on you providing more insight.
>> Enable more debugging/verbosity in the hypervisor and try again is
>> probably all I can suggest at this point.
>
> Currently I have earlyprintk=xen, loglvl=all and guest_loglvl=all.

"earlyprintk=xen" is completely irrelevant here. For the others, as
I see them commonly misplaced, I'll take the risk of embarrassing you
by asking whether they are on the Xen command line (and not the
kernel one).

> Other than adding loglevel=10 and debug initcall_debug, what other
> options and/or procedures do you suggest?

"hvm_debug=" comes to mind (grep for the various DBG_LEVEL_*
defines and uses to see which one might provide greatest benefit,
and be prepared to go through a lot of data afterwards, though
presumably just the tail is going to be interesting if you make the
guest not immediately restart after death).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 14 December 2011 23:16, Jan Beulich <JBeulich@suse.com> wrote:
>
> "earlyprintk=xen" is completely irrelevant here. For the others, as
> I see them commonly misplaced, I'll take the risk of embarrassing you
> by asking whether they are on the Xen command line (and not the
> kernel one).
>
>
> "hvm_debug=" comes to mind (grep for the various DBG_LEVEL_*
> defines and uses to see which one might provide greatest benefit,
> and be prepared to go through a lot of data afterwards, though
> presumably just the tail is going to be interesting if you make the
> guest not immediately restart after death).
>
> Jan
>

Would this be okay? I enabled all levels except for the VLAPIC related ones.

GRUB_CMDLINE_LINUX_DEFAULT="quiet console=tty0 console=hvc0
earlyprintk=xen nomodeset xen-pciback.permissive
xen-pciback.hide=[snip] pci=resource_alignment=[snip] loglevel=10
debug initcall_debug hvm_debug=3647"
GRUB_CMDLINE_XEN="dom0_mem=1024M loglvl=all guest_loglvl=all
console_to_ring com1=115200,8n1,0xdc00,0 console=com1"

I'll report back as soon as the crash occurs again.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel