Mailing List Archive

On 8/5/2018 1:43 PM, Chris Apsey wrote:
> Trying to enable some alternate (non-x86) architectures on xenial +
> queens.  I can load up images and set the property correctly according
> to the supported values
> (https://docs.openstack.org/nova/queens/configuration/config.html) in
> image_properties_default_architecture.  From what I can tell, the
> scheduler works correctly and instances are only scheduled on nodes that
> have the correct qemu binary installed.  However, when the instance
> request lands on this node, it always starts it with qemu-system-x86_64
> rather than qemu-system-arm, qemu-system-ppc, etc.  If I manually set
> the correct binary, everything works as expected.
>
> Am I missing something here, or is this a bug in nova-compute?

image_properties_default_architecture is only used in the scheduler
filter to pick a compute host, it doesn't do anything about the qemu
binary used in nova-compute. mnaser added the config option so maybe he
can share what he's done on his computes.

Do you have qemu-system-x86_64 on non-x86 systems? Seems like a
package/deploy issue since I'd expect x86 packages shouldn't install on
a ppc system and vice versa, and only one qemu package should provide
the binary.

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Hey Matt,

We don't actually have any non-x86 hardware at the moment - we're just
looking to run certain workloads in qemu full emulation mode sans KVM
extensions (we know there is a huge performance hit - it's just for a
few very specific things). The hosts I'm talking about are normal
intel-based compute nodes with several different qemu packages installed
(arm, ppc, mips, x86_64 w/ kvm extensions, etc.).

Is nova designed to work in this kind of scenario? It seems like many
pieces are there, but they're just not quite tied together quite right,
or there is some config option I'm missing.

Thanks!

---
v/r

Chris Apsey
bitskrieg@bitskrieg.net
https://www.bitskrieg.net

On 2018-08-07 09:32 AM, Matt Riedemann wrote:
> On 8/5/2018 1:43 PM, Chris Apsey wrote:
>> Trying to enable some alternate (non-x86) architectures on xenial +
>> queens.  I can load up images and set the property correctly according
>> to the supported values
>> (https://docs.openstack.org/nova/queens/configuration/config.html) in
>> image_properties_default_architecture.  From what I can tell, the
>> scheduler works correctly and instances are only scheduled on nodes
>> that have the correct qemu binary installed.  However, when the
>> instance request lands on this node, it always starts it with
>> qemu-system-x86_64 rather than qemu-system-arm, qemu-system-ppc, etc. 
>> If I manually set the correct binary, everything works as expected.
>>
>> Am I missing something here, or is this a bug in nova-compute?
>
> image_properties_default_architecture is only used in the scheduler
> filter to pick a compute host, it doesn't do anything about the qemu
> binary used in nova-compute. mnaser added the config option so maybe
> he can share what he's done on his computes.
>
> Do you have qemu-system-x86_64 on non-x86 systems? Seems like a
> package/deploy issue since I'd expect x86 packages shouldn't install
> on a ppc system and vice versa, and only one qemu package should
> provide the binary.

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

On 8/7/2018 8:54 AM, Chris Apsey wrote:
> We don't actually have any non-x86 hardware at the moment - we're just
> looking to run certain workloads in qemu full emulation mode sans KVM
> extensions (we know there is a huge performance hit - it's just for a
> few very specific things).  The hosts I'm talking about are normal
> intel-based compute nodes with several different qemu packages installed
> (arm, ppc, mips, x86_64 w/ kvm extensions, etc.).
>
> Is nova designed to work in this kind of scenario?  It seems like many
> pieces are there, but they're just not quite tied together quite right,
> or there is some config option I'm missing.

As far as I know, nova doesn't make anything arch-specific for QEMU.
Nova will execute some qemu commands like qemu-img but as far as the
virt driver, it goes through the libvirt-python API bindings which wrap
over libvirtd which interfaces with QEMU. I would expect that if you're
on an x86_64 arch host, that you can't have non-x86_64 packages
installed on there (or they are noarch packages). Like, I don't know how
your packaging works (are these rpms or debs, or other?) but how do you
have ppc packages installed on an x86 system?

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Matt,

qemu-system-arm, qemu-system-ppc64, etc. in our environment are all x86
packages, but they perform system-mode emulation (via dynamic
instruction translation) for those target environments. So, you run
qemu-system-ppc64 on an x86 host in order to get a ppc64-emulated VM.
Our use case is specifically directed at reverse engineering binaries
and fuzzing for vulnerabilities inside of those architectures for things
that aren't built for x86, but there are others.

If you were to apt-get install qemu-system and then hit autocomplete,
you'd get a list of archiectures that qemu can emulate on x86 hardware -
that's what we're trying to do incorporate. We still want to run normal
qemu-x86 with KVM virtualization extensions, but we ALSO want to run the
other emulators without the KVM virtualization extensions in order to
have more choice for target environments.

So to me, openstack would interpret this by checking to see if a target
host supports the architecture specified in the image (it does this
correctly), then it would choose the correct qemu-system-xx for spawning
the instance based on the architecture flag of the image, which it
currently does not (it always choose qemu-system-x86_64).

Does that make sense?

Chris



---
v/r

Chris Apsey
bitskrieg@bitskrieg.net
https://www.bitskrieg.net

On 2018-08-08 03:07 PM, Matt Riedemann wrote:
> On 8/7/2018 8:54 AM, Chris Apsey wrote:
>> We don't actually have any non-x86 hardware at the moment - we're just
>> looking to run certain workloads in qemu full emulation mode sans KVM
>> extensions (we know there is a huge performance hit - it's just for a
>> few very specific things).  The hosts I'm talking about are normal
>> intel-based compute nodes with several different qemu packages
>> installed (arm, ppc, mips, x86_64 w/ kvm extensions, etc.).
>>
>> Is nova designed to work in this kind of scenario?  It seems like many
>> pieces are there, but they're just not quite tied together quite
>> right, or there is some config option I'm missing.
>
> As far as I know, nova doesn't make anything arch-specific for QEMU.
> Nova will execute some qemu commands like qemu-img but as far as the
> virt driver, it goes through the libvirt-python API bindings which
> wrap over libvirtd which interfaces with QEMU. I would expect that if
> you're on an x86_64 arch host, that you can't have non-x86_64 packages
> installed on there (or they are noarch packages). Like, I don't know
> how your packaging works (are these rpms or debs, or other?) but how
> do you have ppc packages installed on an x86 system?

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

On 8/8/2018 2:42 PM, Chris Apsey wrote:
> qemu-system-arm, qemu-system-ppc64, etc. in our environment are all x86
> packages, but they perform system-mode emulation (via dynamic
> instruction translation) for those target environments.  So, you run
> qemu-system-ppc64 on an x86 host in order to get a ppc64-emulated VM.
> Our use case is specifically directed at reverse engineering binaries
> and fuzzing for vulnerabilities inside of those architectures for things
> that aren't built for x86, but there are others.
>
> If you were to apt-get install qemu-system and then hit autocomplete,
> you'd get a list of archiectures that qemu can emulate on x86 hardware -
> that's what we're trying to do incorporate.  We still want to run normal
> qemu-x86 with KVM virtualization extensions, but we ALSO want to run the
> other emulators without the KVM virtualization extensions in order to
> have more choice for target environments.
>
> So to me, openstack would interpret this by checking to see if a target
> host supports the architecture specified in the image (it does this
> correctly), then it would choose the correct qemu-system-xx for spawning
> the instance based on the architecture flag of the image, which it
> currently does not (it always choose qemu-system-x86_64).
>
> Does that make sense?

OK yeah now I'm following you - running ppc guests on an x86 host
(virt_type=qemu rather than kvm right?).

I would have thought the hw_architecture image property was used for
this somehow to configure the arch in the guest xml properly, like it's
used in a few places [1][2][3].

See [4], I'd think we'd set the guest.arch but don't see that happening.
We do set the guest.os_type though [5].

[1]
https://github.com/openstack/nova/blob/c18b1c1bd646d7cefa3d3e4b25ce59460d1a6ebc/nova/virt/libvirt/driver.py#L4649
[2]
https://github.com/openstack/nova/blob/c18b1c1bd646d7cefa3d3e4b25ce59460d1a6ebc/nova/virt/libvirt/driver.py#L4927
[3]
https://github.com/openstack/nova/blob/c18b1c1bd646d7cefa3d3e4b25ce59460d1a6ebc/nova/virt/libvirt/blockinfo.py#L257
[4] https://libvirt.org/formatcaps.html#elementGuest
[5]
https://github.com/openstack/nova/blob/c18b1c1bd646d7cefa3d3e4b25ce59460d1a6ebc/nova/virt/libvirt/driver.py#L5196

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Exactly. And I agree, it seems like hw_architecture should dictate
which emulator is chosen, but as you mentioned its currently not. I'm
not sure if this is a bug and it's supposed to 'just work', or just
something that was never fully implemented (intentionally) and would be
more of a feature request/suggestion for a later version. The docs are
kind of sparse in this area.

What are your thoughts? I can open a bug if you think the scope is
reasonable.

---
v/r

Chris Apsey
bitskrieg@bitskrieg.net
https://www.bitskrieg.net

On 2018-08-08 06:40 PM, Matt Riedemann wrote:
> On 8/8/2018 2:42 PM, Chris Apsey wrote:
>> qemu-system-arm, qemu-system-ppc64, etc. in our environment are all
>> x86 packages, but they perform system-mode emulation (via dynamic
>> instruction translation) for those target environments.  So, you run
>> qemu-system-ppc64 on an x86 host in order to get a ppc64-emulated VM.
>> Our use case is specifically directed at reverse engineering binaries
>> and fuzzing for vulnerabilities inside of those architectures for
>> things that aren't built for x86, but there are others.
>>
>> If you were to apt-get install qemu-system and then hit autocomplete,
>> you'd get a list of archiectures that qemu can emulate on x86 hardware
>> - that's what we're trying to do incorporate.  We still want to run
>> normal qemu-x86 with KVM virtualization extensions, but we ALSO want
>> to run the other emulators without the KVM virtualization extensions
>> in order to have more choice for target environments.
>>
>> So to me, openstack would interpret this by checking to see if a
>> target host supports the architecture specified in the image (it does
>> this correctly), then it would choose the correct qemu-system-xx for
>> spawning the instance based on the architecture flag of the image,
>> which it currently does not (it always choose qemu-system-x86_64).
>>
>> Does that make sense?
>
> OK yeah now I'm following you - running ppc guests on an x86 host
> (virt_type=qemu rather than kvm right?).
>
> I would have thought the hw_architecture image property was used for
> this somehow to configure the arch in the guest xml properly, like
> it's used in a few places [1][2][3].
>
> See [4], I'd think we'd set the guest.arch but don't see that
> happening. We do set the guest.os_type though [5].
>
> [1]
> https://github.com/openstack/nova/blob/c18b1c1bd646d7cefa3d3e4b25ce59460d1a6ebc/nova/virt/libvirt/driver.py#L4649
> [2]
> https://github.com/openstack/nova/blob/c18b1c1bd646d7cefa3d3e4b25ce59460d1a6ebc/nova/virt/libvirt/driver.py#L4927
> [3]
> https://github.com/openstack/nova/blob/c18b1c1bd646d7cefa3d3e4b25ce59460d1a6ebc/nova/virt/libvirt/blockinfo.py#L257
> [4] https://libvirt.org/formatcaps.html#elementGuest
> [5]
> https://github.com/openstack/nova/blob/c18b1c1bd646d7cefa3d3e4b25ce59460d1a6ebc/nova/virt/libvirt/driver.py#L5196

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

On 8/9/2018 6:03 AM, Chris Apsey wrote:
> Exactly.  And I agree, it seems like hw_architecture should dictate
> which emulator is chosen, but as you mentioned its currently not.  I'm
> not sure if this is a bug and it's supposed to 'just work', or just
> something that was never fully implemented (intentionally) and would be
> more of a feature request/suggestion for a later version.  The docs are
> kind of sparse in this area.
>
> What are your thoughts?  I can open a bug if you think the scope is
> reasonable.

I'm not sure if this is a bug or a feature, or if there are reasons why
it's never been done. I'm gonna have to rope in Kashyap and danpb since
they'd likely know more.

Dan/Kaskyap: tl;dr why doesn't the nova libvirt driver, configured for
qemu, set the guest.arch based on the hw_architecture image property so
that you can run ppc guests in an x86 host?

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

This sounds promising and there seems to be a feasible way to do this, but
it also sounds like a decent amount of effort and would be a new feature in
a future release rather than a bugfix - am I correct in that assessment?



On August 9, 2018 13:30:31 "Daniel P. Berrangé" <berrange@redhat.com> wrote:

> On Thu, Aug 09, 2018 at 12:14:56PM -0500, Matt Riedemann wrote:
>> On 8/9/2018 6:03 AM, Chris Apsey wrote:
>>> Exactly. And I agree, it seems like hw_architecture should dictate
>>> which emulator is chosen, but as you mentioned its currently not. I'm
>>> not sure if this is a bug and it's supposed to 'just work', or just
>>> something that was never fully implemented (intentionally) and would be
>>> more of a feature request/suggestion for a later version. The docs are
>>> kind of sparse in this area.
>>>
>>> What are your thoughts? I can open a bug if you think the scope is
>>> reasonable.
>>
>> I'm not sure if this is a bug or a feature, or if there are reasons why it's
>> never been done. I'm gonna have to rope in Kashyap and danpb since they'd
>> likely know more.
>>
>> Dan/Kaskyap: tl;dr why doesn't the nova libvirt driver, configured for qemu,
>> set the guest.arch based on the hw_architecture image property so that you
>> can run ppc guests in an x86 host?
>
> Yes, it should do exactly that IMHO !
>
> The main caveat is that a hell of alot of code in libvirt assumes that
> guest arch == host arch. ie when building guest XML there's lots of code
> that looks at caps.host.cpu.arch to decide how to configure the guest.
> This all needs fixing to look at the guest.arch value instead, having
> set that from hw_architecture prop.
>
> Nova libvirt driver is already reporting that it is capable of running
> guest with multiple arches (the _get_instance_capaiblities method in
> nova/virt/libvirt/driver.py).
>
> The only other thing is that you likely want to distinguish between
> hosts that can do PPC64 via KVM vs those that can only do it via
> emulation, so you don't get guests randomly placed on slow vs fast
> hosts. Some kind of scheduler filter / weighting can do that based
> on info already reported from the compute host I expect.
>
>
> Regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o- https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|




_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

On 8/11/2018 12:50 AM, Chris Apsey wrote:
> This sounds promising and there seems to be a feasible way to do this,
> but it also sounds like a decent amount of effort and would be a new
> feature in a future release rather than a bugfix - am I correct in that
> assessment?

Yes I'd say it's a blueprint and not a bug fix - it's not something we'd
backport to stable branches upstream, for example.

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators