Mailing List Archive

[PATCH v4 00/18] VM forking
The following series implements VM forking for Intel HVM guests to allow for
the fast creation of identical VMs without the assosciated high startup costs
of booting or restoring the VM from a savefile.

JIRA issue: https://xenproject.atlassian.net/browse/XEN-89

The fork operation is implemented as part of the "xl fork-vm" command:
xl fork-vm -C <config_file_for_fork> -Q <qemu_save_file> <parent_domid>

By default a fully functional fork is created. The user is in charge however to
create the appropriate config file for the fork and to generate the QEMU save
file before the fork-vm call is made. The config file needs to give the
fork a new name at minimum but other settings may also require changes.

The interface also allows to split the forking into two steps:
xl fork-vm --launch-dm no \
-p <parent_domid>
xl fork-vm --launch-dm late \
-C <config_file_for_fork> \
-Q <qemu_save_file> \
<fork_domid>

The split creation model is useful when the VM needs to be created as fast as
possible. The forked VM can be unpaused without the device model being launched
to be monitored and accessed via VMI. Note however that without its device
model running (depending on what is executing in the VM) it is bound to
misbehave or even crash when its trying to access devices that would be
emulated by QEMU. We anticipate that for certain use-cases this would be an
acceptable situation, in case for example when fuzzing is performed of code
segments that don't access such devices.

Launching the device model requires the QEMU Xen savefile to be generated
manually from the parent VM. This can be accomplished simply by connecting to
its QMP socket and issuing the "xen-save-devices-state" command. For example
using the standard tool socat these commands can be used to generate the file:
socat - UNIX-CONNECT:/var/run/xen/qmp-libxl-<parent_domid>
{ "execute": "qmp_capabilities" }
{ "execute": "xen-save-devices-state", \
"arguments": { "filename": "/path/to/save/qemu_state", \
"live": false} }

At runtime the forked VM starts running with an empty p2m which gets lazily
populated when the VM generates EPT faults, similar to how altp2m views are
populated. If the memory access is a read-only access, the p2m entry is
populated with a memory shared entry with its parent. For write memory accesses
or in case memory sharing wasn't possible (for example in case a reference is
held by a third party), a new page is allocated and the page contents are
copied over from the parent VM. Forks can be further forked if needed, thus
allowing for further memory savings.

A VM fork reset hypercall is also added that allows the fork to be reset to the
state it was just after a fork, also accessible via xl:
xl fork-vm --fork-reset -p <fork_domid>

This is an optimization for cases where the forks are very short-lived and run
without a device model, so resetting saves some time compared to creating a
brand new fork provided the fork has not aquired a lot of memory. If the fork
has a lot of memory deduplicated it is likely going to be faster to create a
new fork from scratch and asynchronously destroying the old one.

The series has been tested with both Linux and Windows VMs and functions as
expected. VM forking time has been measured to be 0.0007s, device model launch
to be around 1s depending largely on the number of devices being emulated. Fork
resets have been measured to be 0.0001s under the optimal circumstances.

Patches 1-2 implement changes to existing internal Xen APIs to make VM forking
possible.

Patches 3-14 are code-cleanups and adjustments of to Xen memory sharing
subsystem with no functional changes.

Patch 15 adds the hypervisor-side code implementing VM forking.

Patch 16 is integration of mem_access with forked VMs.

Patch 17 implements the VM fork reset operation hypervisor side bits.

Patch 18 adds the toolstack-side code implementing VM forking and reset.

Tamas K Lengyel (18):
x86/hvm: introduce hvm_copy_context_and_params
xen/x86: Make hap_get_allocation accessible
x86/mem_sharing: make get_two_gfns take locks conditionally
x86/mem_sharing: drop flags from mem_sharing_unshare_page
x86/mem_sharing: don't try to unshare twice during page fault
x86/mem_sharing: define mem_sharing_domain to hold some scattered
variables
x86/mem_sharing: Use INVALID_MFN and p2m_is_shared in
relinquish_shared_pages
x86/mem_sharing: Make add_to_physmap static and shorten name
x86/mem_sharing: Convert MEM_SHARING_DESTROY_GFN to a bool
x86/mem_sharing: Replace MEM_SHARING_DEBUG with gdprintk
x86/mem_sharing: ASSERT that p2m_set_entry succeeds
x86/mem_sharing: Enable mem_sharing on first memop
x86/mem_sharing: Skip xen heap pages in memshr nominate
x86/mem_sharing: check page type count earlier
xen/mem_sharing: VM forking
xen/mem_access: Use __get_gfn_type_access in set_mem_access
x86/mem_sharing: reset a fork
xen/tools: VM forking toolstack side

docs/man/xl.1.pod.in | 36 +++
tools/libxc/include/xenctrl.h | 13 +
tools/libxc/xc_memshr.c | 22 ++
tools/libxl/libxl.h | 7 +
tools/libxl/libxl_create.c | 237 +++++++++-----
tools/libxl/libxl_dm.c | 2 +-
tools/libxl/libxl_dom.c | 83 +++--
tools/libxl/libxl_internal.h | 1 +
tools/libxl/libxl_types.idl | 1 +
tools/xl/xl.h | 5 +
tools/xl/xl_cmdtable.c | 12 +
tools/xl/xl_saverestore.c | 96 ++++++
tools/xl/xl_vmcontrol.c | 8 +
xen/arch/x86/hvm/hvm.c | 271 ++++++++++------
xen/arch/x86/mm/hap/hap.c | 3 +-
xen/arch/x86/mm/mem_access.c | 5 +-
xen/arch/x86/mm/mem_sharing.c | 501 +++++++++++++++++++++++-------
xen/arch/x86/mm/p2m.c | 16 +-
xen/common/memory.c | 2 +-
xen/drivers/passthrough/pci.c | 3 +-
xen/include/asm-x86/hap.h | 1 +
xen/include/asm-x86/hvm/domain.h | 6 +-
xen/include/asm-x86/hvm/hvm.h | 2 +
xen/include/asm-x86/mem_sharing.h | 43 ++-
xen/include/asm-x86/p2m.h | 14 +-
xen/include/public/memory.h | 6 +
xen/include/xen/sched.h | 1 +
27 files changed, 1058 insertions(+), 339 deletions(-)

--
2.20.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v4 00/18] VM forking [ In reply to ]
On 08.01.2020 18:13, Tamas K Lengyel wrote:
> Tamas K Lengyel (18):
> x86/hvm: introduce hvm_copy_context_and_params
> xen/x86: Make hap_get_allocation accessible
> x86/mem_sharing: make get_two_gfns take locks conditionally
> x86/mem_sharing: drop flags from mem_sharing_unshare_page
> x86/mem_sharing: don't try to unshare twice during page fault
> x86/mem_sharing: define mem_sharing_domain to hold some scattered
> variables
> x86/mem_sharing: Use INVALID_MFN and p2m_is_shared in
> relinquish_shared_pages
> x86/mem_sharing: Make add_to_physmap static and shorten name
> x86/mem_sharing: Convert MEM_SHARING_DESTROY_GFN to a bool

I've looked at these patches, and I think 2-4 and 7-9 could go
in right away (6 has a small question pending, but may otherwise
also be ready), if you were to give (or delegate) your ack that
they would need afaict. Thoughts?

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v4 00/18] VM forking [ In reply to ]
On Thu, Jan 16, 2020 at 8:47 AM Jan Beulich <jbeulich@suse.com> wrote:
>
> On 08.01.2020 18:13, Tamas K Lengyel wrote:
> > Tamas K Lengyel (18):
> > x86/hvm: introduce hvm_copy_context_and_params
> > xen/x86: Make hap_get_allocation accessible
> > x86/mem_sharing: make get_two_gfns take locks conditionally
> > x86/mem_sharing: drop flags from mem_sharing_unshare_page
> > x86/mem_sharing: don't try to unshare twice during page fault
> > x86/mem_sharing: define mem_sharing_domain to hold some scattered
> > variables
> > x86/mem_sharing: Use INVALID_MFN and p2m_is_shared in
> > relinquish_shared_pages
> > x86/mem_sharing: Make add_to_physmap static and shorten name
> > x86/mem_sharing: Convert MEM_SHARING_DESTROY_GFN to a bool
>
> I've looked at these patches, and I think 2-4 and 7-9 could go
> in right away (6 has a small question pending, but may otherwise
> also be ready), if you were to give (or delegate) your ack that
> they would need afaict. Thoughts?
>

Not sure I understand your question. My understanding is that since
I'm the maintainer of the code being changed by these patches I just
need a "reviewed-by" from someone in the community and no outstanding
issue on them. Provided this is v4 now of the series and no issues
were raised so far for these particular patches they can be merged
with your Reviewed-by.

Thanks,
Tamas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v4 00/18] VM forking [ In reply to ]
On 16.01.2020 17:24, Tamas K Lengyel wrote:
> On Thu, Jan 16, 2020 at 8:47 AM Jan Beulich <jbeulich@suse.com> wrote:
>>
>> On 08.01.2020 18:13, Tamas K Lengyel wrote:
>>> Tamas K Lengyel (18):
>>> x86/hvm: introduce hvm_copy_context_and_params
>>> xen/x86: Make hap_get_allocation accessible
>>> x86/mem_sharing: make get_two_gfns take locks conditionally
>>> x86/mem_sharing: drop flags from mem_sharing_unshare_page
>>> x86/mem_sharing: don't try to unshare twice during page fault
>>> x86/mem_sharing: define mem_sharing_domain to hold some scattered
>>> variables
>>> x86/mem_sharing: Use INVALID_MFN and p2m_is_shared in
>>> relinquish_shared_pages
>>> x86/mem_sharing: Make add_to_physmap static and shorten name
>>> x86/mem_sharing: Convert MEM_SHARING_DESTROY_GFN to a bool
>>
>> I've looked at these patches, and I think 2-4 and 7-9 could go
>> in right away (6 has a small question pending, but may otherwise
>> also be ready), if you were to give (or delegate) your ack that
>> they would need afaict. Thoughts?
>>
>
> Not sure I understand your question. My understanding is that since
> I'm the maintainer of the code being changed by these patches I just
> need a "reviewed-by" from someone in the community and no outstanding
> issue on them.

Please note that my previous mail was _to_ George, with you only
_cc_-ed. Hence the question was to George, not you. (It is a
common issue which I keep mentioning on meetings that the
distinction of To and Cc is often not being honored, albeit
typically more by senders than recipients.)

> Provided this is v4 now of the series and no issues
> were raised so far for these particular patches they can be merged
> with your Reviewed-by.

I don't think so, under the current (sufficiently) common
understanding of the rules. See George's proposal to change to a
model like what you imply:
https://lists.xenproject.org/archives/html/xen-devel/2020-01/msg00885.html

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v4 00/18] VM forking [ In reply to ]
On Fri, Jan 17, 2020 at 10:12:14AM +0100, Jan Beulich wrote:
> Please note that my previous mail was _to_ George, with you only
> _cc_-ed. Hence the question was to George, not you. (It is a
> common issue which I keep mentioning on meetings that the
> distinction of To and Cc is often not being honored, albeit
> typically more by senders than recipients.)

Tip: Jan, you could also have started the sentence by "George, " in
addition to properly setting the "To:", it would help a lot I think.

Teaching people about setting properly "To:", and reading it before
reading the email is a lost fight I think. Even so it can be useful to
filter email which needs a response.

Cheers,

--
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v4 00/18] VM forking [ In reply to ]
On Fri, Jan 17, 2020 at 4:15 AM Anthony PERARD
<anthony.perard@citrix.com> wrote:
>
> On Fri, Jan 17, 2020 at 10:12:14AM +0100, Jan Beulich wrote:
> > Please note that my previous mail was _to_ George, with you only
> > _cc_-ed. Hence the question was to George, not you. (It is a
> > common issue which I keep mentioning on meetings that the
> > distinction of To and Cc is often not being honored, albeit
> > typically more by senders than recipients.)
>
> Tip: Jan, you could also have started the sentence by "George, " in
> addition to properly setting the "To:", it would help a lot I think.
>
> Teaching people about setting properly "To:", and reading it before
> reading the email is a lost fight I think. Even so it can be useful to
> filter email which needs a response.

Yea, +1 for that, it would make addressed questions more apparent.
Gmail (which is what I use) doesn't break out the email header by
default with separate lines for to: and cc:, all recipients are in a
single line with no distinction between them.

Tamas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v4 00/18] VM forking [ In reply to ]
> > Provided this is v4 now of the series and no issues
> > were raised so far for these particular patches they can be merged
> > with your Reviewed-by.
>
> I don't think so, under the current (sufficiently) common
> understanding of the rules. See George's proposal to change to a
> model like what you imply:
> https://lists.xenproject.org/archives/html/xen-devel/2020-01/msg00885.html
>

Ah OK, I though that was already agreed upon. I would certainly prefer
that model to speed things up and reduce the hassle to work with code
noone else maintains then me.

Tamas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel