Mailing List Archive

On 10.09.2020 05:57, Marek Marczykowski-Górecki wrote:
> After updating from Xen 4.13 to Xen 4.14 I have troubles starting any
> HVM: just after hvmloader saying "Invoking SeaBIOS" I get "(XEN) MMIO
> emulation failed (1): d29v0 32bit @ 0008:fffeedf d -> "
>
> I come to a situation where seemingly the same domU started via xl
> works, while when started via libvirt it crashes. This seems to be
> related to xl setting b_info.{acpi,apic}, while libvirt setting
> b_info.u.hvm.{acpi,apic}. Modifying libvirt to use the former fixes the
> issue.
>
> So, handling the old option got broken sometime between 4.13 and 4.14.

The respective changes in libxl date back to 2016/2017, and the involved
code looks identical between 4.13 and 4.14. Perhaps there's some 4.13-
only logic in libvirt, which really means to be >= 4.13? In any event
libxl_types.idl clearly says the two u.hvm fields are deprecated;
nevertheless there are some uses left for u.hvm.acpi (the most
suspicious one in xl_sxp.c) and exactly one in a comment for u.hvm.apic.
Sadly this marking of being deprecated doesn't get reflected in the
_libxl_types.h which gets installed for public consumption.

Jan

On Thu, Sep 10, 2020 at 05:57:23AM +0200, Marek Marczykowski-Górecki wrote:
> Hi,
>
> After updating from Xen 4.13 to Xen 4.14 I have troubles starting any
> HVM: just after hvmloader saying "Invoking SeaBIOS" I get "(XEN) MMIO
> emulation failed (1): d29v0 32bit @ 0008:fffeedf d -> "
>
> I come to a situation where seemingly the same domU started via xl
> works, while when started via libvirt it crashes. This seems to be
> related to xl setting b_info.{acpi,apic}, while libvirt setting
> b_info.u.hvm.{acpi,apic}. Modifying libvirt to use the former fixes the
> issue.

Could you print the values of the involved fields at the end of
libxl__domain_build_info_setdefault in both cases?

I'm not able to spot what changed between 4.13 and 4.14 that could
alter the behavior, but knowing the values at that point might make
it easier.

> So, handling the old option got broken sometime between 4.13 and 4.14.
> Or perhaps it is some other related side effect.

Libvirt master tests seem to have been failing for some time (60 days)
in the build phase [0], maybe you are using a newer version of libvirt
than what osstest is currently testing? (Xen flights are currently
stuck at libvirt commit 2c846fa6bcc11929c9fb857a22430fb9945654ad).

Thanks, Roger.

[0] http://logs.test-lab.xenproject.org/osstest/logs/152881/build-amd64-libvirt/6.ts-libvirt-build.log

On Thu, Sep 10, 2020 at 10:51:48AM +0200, Roger Pau Monné wrote:
> On Thu, Sep 10, 2020 at 05:57:23AM +0200, Marek Marczykowski-Górecki wrote:
> > Hi,
> >
> > After updating from Xen 4.13 to Xen 4.14 I have troubles starting any
> > HVM: just after hvmloader saying "Invoking SeaBIOS" I get "(XEN) MMIO
> > emulation failed (1): d29v0 32bit @ 0008:fffeedf d -> "
> >
> > I come to a situation where seemingly the same domU started via xl
> > works, while when started via libvirt it crashes. This seems to be
> > related to xl setting b_info.{acpi,apic}, while libvirt setting
> > b_info.u.hvm.{acpi,apic}. Modifying libvirt to use the former fixes the
> > issue.
>
> Could you print the values of the involved fields at the end of
> libxl__domain_build_info_setdefault in both cases?
>
> I'm not able to spot what changed between 4.13 and 4.14 that could
> alter the behavior, but knowing the values at that point might make
> it easier.

Sure, will do.
It may be also something else: maybe it acpi/apic settings were broken
before, but did not results in a domU crash this way.
FWIW when looking into /var/lib/xen/*-libxl-json I clearly see
difference between b_info.{acpi,apic} and b_info.u.hvm.{acpi,apic}.

> > So, handling the old option got broken sometime between 4.13 and 4.14.
> > Or perhaps it is some other related side effect.
>
> Libvirt master tests seem to have been failing for some time (60 days)
> in the build phase [0], maybe you are using a newer version of libvirt
> than what osstest is currently testing? (Xen flights are currently
> stuck at libvirt commit 2c846fa6bcc11929c9fb857a22430fb9945654ad).

I use libvirt 6.6.0, so even older one, and in fact updated it from
6.4.0 together with Xen 4.14. But I don't see relevant change between
those two versions.

BTW The libvirt build may require more attention, they've changed build
system... Now it is built with meson/ninja (haven't figured it out yet).

> Thanks, Roger.
>
> [0] http://logs.test-lab.xenproject.org/osstest/logs/152881/build-amd64-libvirt/6.ts-libvirt-build.log

--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

Adding toolstack maintainers.

On Thu, Sep 10, 2020 at 12:29:21PM +0200, Marek Marczykowski-Górecki wrote:
> On Thu, Sep 10, 2020 at 10:51:48AM +0200, Roger Pau Monné wrote:
> > On Thu, Sep 10, 2020 at 05:57:23AM +0200, Marek Marczykowski-Górecki wrote:
> > > Hi,
> > >
> > > After updating from Xen 4.13 to Xen 4.14 I have troubles starting any
> > > HVM: just after hvmloader saying "Invoking SeaBIOS" I get "(XEN) MMIO
> > > emulation failed (1): d29v0 32bit @ 0008:fffeedf d -> "
> > >
> > > I come to a situation where seemingly the same domU started via xl
> > > works, while when started via libvirt it crashes. This seems to be
> > > related to xl setting b_info.{acpi,apic}, while libvirt setting
> > > b_info.u.hvm.{acpi,apic}. Modifying libvirt to use the former fixes the
> > > issue.
> >
> > Could you print the values of the involved fields at the end of
> > libxl__domain_build_info_setdefault in both cases?
> >
> > I'm not able to spot what changed between 4.13 and 4.14 that could
> > alter the behavior, but knowing the values at that point might make
> > it easier.
>
> Sure, will do.
> It may be also something else: maybe it acpi/apic settings were broken
> before, but did not results in a domU crash this way.
> FWIW when looking into /var/lib/xen/*-libxl-json I clearly see
> difference between b_info.{acpi,apic} and b_info.u.hvm.{acpi,apic}.

I think libxl__domain_build_info_setdefault should check whether
b_info.u.hvm.{acpi,apic} is set and copy those into b_info.{acpi,apic}
if those are not set?

Toolstack people is more likely to have an opinion here, or to help
debug the issue.

> > > So, handling the old option got broken sometime between 4.13 and 4.14.
> > > Or perhaps it is some other related side effect.
> >
> > Libvirt master tests seem to have been failing for some time (60 days)
> > in the build phase [0], maybe you are using a newer version of libvirt
> > than what osstest is currently testing? (Xen flights are currently
> > stuck at libvirt commit 2c846fa6bcc11929c9fb857a22430fb9945654ad).
>
> I use libvirt 6.6.0, so even older one, and in fact updated it from
> 6.4.0 together with Xen 4.14. But I don't see relevant change between
> those two versions.
>
> BTW The libvirt build may require more attention, they've changed build
> system... Now it is built with meson/ninja (haven't figured it out yet).

If you can figure out and send a patch to fix oostest that would be
great!

Thanks, Roger.

On Thu, Sep 10, 2020 at 12:41:04PM +0200, Roger Pau Monné wrote:
> Adding toolstack maintainers.
>
> On Thu, Sep 10, 2020 at 12:29:21PM +0200, Marek Marczykowski-Górecki wrote:
> > On Thu, Sep 10, 2020 at 10:51:48AM +0200, Roger Pau Monné wrote:
> > > On Thu, Sep 10, 2020 at 05:57:23AM +0200, Marek Marczykowski-Górecki wrote:
> > > > Hi,
> > > >
> > > > After updating from Xen 4.13 to Xen 4.14 I have troubles starting any
> > > > HVM: just after hvmloader saying "Invoking SeaBIOS" I get "(XEN) MMIO
> > > > emulation failed (1): d29v0 32bit @ 0008:fffeedf d -> "
> > > >
> > > > I come to a situation where seemingly the same domU started via xl
> > > > works, while when started via libvirt it crashes. This seems to be
> > > > related to xl setting b_info.{acpi,apic}, while libvirt setting
> > > > b_info.u.hvm.{acpi,apic}. Modifying libvirt to use the former fixes the
> > > > issue.
> > >
> > > Could you print the values of the involved fields at the end of
> > > libxl__domain_build_info_setdefault in both cases?
> > >
> > > I'm not able to spot what changed between 4.13 and 4.14 that could
> > > alter the behavior, but knowing the values at that point might make
> > > it easier.
> >
> > Sure, will do.
> > It may be also something else: maybe it acpi/apic settings were broken
> > before, but did not results in a domU crash this way.
> > FWIW when looking into /var/lib/xen/*-libxl-json I clearly see
> > difference between b_info.{acpi,apic} and b_info.u.hvm.{acpi,apic}.
>
> I think libxl__domain_build_info_setdefault should check whether
> b_info.u.hvm.{acpi,apic} is set and copy those into b_info.{acpi,apic}
> if those are not set?

Looking at libxl__domain_build_info_setdefault this is not the case.
Instead there is libxl__acpi_defbool_val which looks at both.
Oh, and there is no similar thing for apic -> b_info.u.hvm.apic is
ignored!

> Toolstack people is more likely to have an opinion here, or to help
> debug the issue.

--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

On Thu, Sep 10, 2020 at 12:58:57PM +0200, Marek Marczykowski-Górecki wrote:
> On Thu, Sep 10, 2020 at 12:41:04PM +0200, Roger Pau Monné wrote:
> > Adding toolstack maintainers.
> >
> > On Thu, Sep 10, 2020 at 12:29:21PM +0200, Marek Marczykowski-Górecki wrote:
> > > On Thu, Sep 10, 2020 at 10:51:48AM +0200, Roger Pau Monné wrote:
> > > > On Thu, Sep 10, 2020 at 05:57:23AM +0200, Marek Marczykowski-Górecki wrote:
> > > > > Hi,
> > > > >
> > > > > After updating from Xen 4.13 to Xen 4.14 I have troubles starting any
> > > > > HVM: just after hvmloader saying "Invoking SeaBIOS" I get "(XEN) MMIO
> > > > > emulation failed (1): d29v0 32bit @ 0008:fffeedf d -> "
> > > > >
> > > > > I come to a situation where seemingly the same domU started via xl
> > > > > works, while when started via libvirt it crashes. This seems to be
> > > > > related to xl setting b_info.{acpi,apic}, while libvirt setting
> > > > > b_info.u.hvm.{acpi,apic}. Modifying libvirt to use the former fixes the
> > > > > issue.
> > > >
> > > > Could you print the values of the involved fields at the end of
> > > > libxl__domain_build_info_setdefault in both cases?
> > > >
> > > > I'm not able to spot what changed between 4.13 and 4.14 that could
> > > > alter the behavior, but knowing the values at that point might make
> > > > it easier.
> > >
> > > Sure, will do.
> > > It may be also something else: maybe it acpi/apic settings were broken
> > > before, but did not results in a domU crash this way.
> > > FWIW when looking into /var/lib/xen/*-libxl-json I clearly see
> > > difference between b_info.{acpi,apic} and b_info.u.hvm.{acpi,apic}.
> >
> > I think libxl__domain_build_info_setdefault should check whether
> > b_info.u.hvm.{acpi,apic} is set and copy those into b_info.{acpi,apic}
> > if those are not set?
>
> Looking at libxl__domain_build_info_setdefault this is not the case.
> Instead there is libxl__acpi_defbool_val which looks at both.
> Oh, and there is no similar thing for apic -> b_info.u.hvm.apic is
> ignored!
>
> > Toolstack people is more likely to have an opinion here, or to help
> > debug the issue.

Ok, The crash reported initially was caused by a different thing: using
seabios.bin instead of seabios-256k.bin (should that really cause the
crash? shouldn't 128k seabios build work too?). But in any case, I think
the b_info.u.hvm.{acpi,apic} is also not in a good shape.

--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

On Sun, Sep 13, 2020 at 01:12:39PM +0200, Marek Marczykowski-Górecki wrote:
> On Thu, Sep 10, 2020 at 12:58:57PM +0200, Marek Marczykowski-Górecki wrote:
> > On Thu, Sep 10, 2020 at 12:41:04PM +0200, Roger Pau Monné wrote:
> > > Adding toolstack maintainers.
> > >
> > > On Thu, Sep 10, 2020 at 12:29:21PM +0200, Marek Marczykowski-Górecki wrote:
> > > > On Thu, Sep 10, 2020 at 10:51:48AM +0200, Roger Pau Monné wrote:
> > > > > On Thu, Sep 10, 2020 at 05:57:23AM +0200, Marek Marczykowski-Górecki wrote:
> > > > > > Hi,
> > > > > >
> > > > > > After updating from Xen 4.13 to Xen 4.14 I have troubles starting any
> > > > > > HVM: just after hvmloader saying "Invoking SeaBIOS" I get "(XEN) MMIO
> > > > > > emulation failed (1): d29v0 32bit @ 0008:fffeedf d -> "
> > > > > >
> > > > > > I come to a situation where seemingly the same domU started via xl
> > > > > > works, while when started via libvirt it crashes. This seems to be
> > > > > > related to xl setting b_info.{acpi,apic}, while libvirt setting
> > > > > > b_info.u.hvm.{acpi,apic}. Modifying libvirt to use the former fixes the
> > > > > > issue.
> > > > >
> > > > > Could you print the values of the involved fields at the end of
> > > > > libxl__domain_build_info_setdefault in both cases?
> > > > >
> > > > > I'm not able to spot what changed between 4.13 and 4.14 that could
> > > > > alter the behavior, but knowing the values at that point might make
> > > > > it easier.
> > > >
> > > > Sure, will do.
> > > > It may be also something else: maybe it acpi/apic settings were broken
> > > > before, but did not results in a domU crash this way.
> > > > FWIW when looking into /var/lib/xen/*-libxl-json I clearly see
> > > > difference between b_info.{acpi,apic} and b_info.u.hvm.{acpi,apic}.
> > >
> > > I think libxl__domain_build_info_setdefault should check whether
> > > b_info.u.hvm.{acpi,apic} is set and copy those into b_info.{acpi,apic}
> > > if those are not set?
> >
> > Looking at libxl__domain_build_info_setdefault this is not the case.
> > Instead there is libxl__acpi_defbool_val which looks at both.
> > Oh, and there is no similar thing for apic -> b_info.u.hvm.apic is
> > ignored!
> >
> > > Toolstack people is more likely to have an opinion here, or to help
> > > debug the issue.
>
> Ok, The crash reported initially was caused by a different thing: using
> seabios.bin instead of seabios-256k.bin (should that really cause the
> crash? shouldn't 128k seabios build work too?). But in any case, I think
> the b_info.u.hvm.{acpi,apic} is also not in a good shape.

Does 128K SeaBIOS have Xen support enabled?

Roger.

On Mon, Sep 14, 2020 at 05:19:56PM +0200, Roger Pau Monné wrote:
> On Sun, Sep 13, 2020 at 01:12:39PM +0200, Marek Marczykowski-Górecki wrote:
> > Ok, The crash reported initially was caused by a different thing: using
> > seabios.bin instead of seabios-256k.bin (should that really cause the
> > crash? shouldn't 128k seabios build work too?). But in any case, I think
> > the b_info.u.hvm.{acpi,apic} is also not in a good shape.
>
> Does 128K SeaBIOS have Xen support enabled?

This is a very good question. No:
https://src.fedoraproject.org/rpms/seabios/blob/f32/f/config.seabios-128k#_7

--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?