Mailing List Archive: Re: [MirageOS-devel] xen dev summit talk slides: Branch Consistency for Xenstore

Re: [MirageOS-devel] xen dev summit talk slides: Branch Consistency for Xenstore

Aug 19, 2014, 6:44 AM

Post #1 of 5 (1537 views)

On 17 Aug 2014, at 11:01, Anil Madhavapeddy <anil@recoil.org> wrote:

> I'm giving a talk at the Xen Developer Summit at Linuxcon tomorrow on the work that we've (primarily Dave Scott and Thomas Gazagnaire) on improving the Xenstore via Irmin features.
>
> Slides here: http://decks.openmirage.org/xendevsummit14#/
> Repo: https://github.com/mirage/mirage-decks as usual
>

[+xen-api-devel] Talk went well, quick notes here:

- git workflow very popular. Lots of people twigged onto the maintainability benefits of `git bisect` automation in particular.

- questions about why Xenstore transactions are still necessary in the modern world. Can replace with consensus protocols instead? Maybe time for an ABI bump to deprecate the ancient xenstore protocol.

- space usage is a concern -- building an RRD-style constant size library to maintain progressive history would be a big win.

- *excellent* talk from Felipe Huici (CCed) from NEC about building much denser VM workloads, and he observed that Xenstored/xenconsoled are a big bottleneck at ~10000 VMs. Are your slides available Felipe? Some sub notes:

- we could write a mirage xenconsoled to log to irmin and drain guest console rings much faster.
- a distributed xenstored+irmin would allow significantly more scalability than attempting to build a serially fast version.
- felipe has the beginning of a simple c++ xenstored that doesn't implement the full semantics, but is enough for MiniOS.
- it may be useful to negotiate a xenstore v2 protocol and use that for new guests. It could use a simple fixed-length binary protocol (protobuf-style?), and eliminate the need for transactions perhaps.

-anil
_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

Re: [MirageOS-devel] xen dev summit talk slides: Branch Consistency for Xenstore [ In reply to ]

anil at recoil

Aug 19, 2014, 6:47 AM

Post #2 of 5 (1505 views)

Permalink

On 19 Aug 2014, at 08:44, Anil Madhavapeddy <anil@recoil.org> wrote:

> On 17 Aug 2014, at 11:01, Anil Madhavapeddy <anil@recoil.org> wrote:
>
>> I'm giving a talk at the Xen Developer Summit at Linuxcon tomorrow on the work that we've (primarily Dave Scott and Thomas Gazagnaire) on improving the Xenstore via Irmin features.
>>
>> Slides here: http://decks.openmirage.org/xendevsummit14#/
>> Repo: https://github.com/mirage/mirage-decks as usual
>>
>
> [+xen-api-devel] Talk went well, quick notes here:
>
> - git workflow very popular. Lots of people twigged onto the maintainability benefits of `git bisect` automation in particular.
>
> - questions about why Xenstore transactions are still necessary in the modern world. Can replace with consensus protocols instead? Maybe time for an ABI bump to deprecate the ancient xenstore protocol.
>
> - space usage is a concern -- building an RRD-style constant size library to maintain progressive history would be a big win.
>
> - *excellent* talk from Felipe Huici (CCed) from NEC about building much denser VM workloads, and he observed that Xenstored/xenconsoled are a big bottleneck at ~10000 VMs. Are your slides available Felipe? Some sub notes:
>
> - we could write a mirage xenconsoled to log to irmin and drain guest console rings much faster.
> - a distributed xenstored+irmin would allow significantly more scalability than attempting to build a serially fast version.
> - felipe has the beginning of a simple c++ xenstored that doesn't implement the full semantics, but is enough for MiniOS.
> - it may be useful to negotiate a xenstore v2 protocol and use that for new guests. It could use a simple fixed-length binary protocol (protobuf-style?), and eliminate the need for transactions perhaps.

sent 'send' too quickly. The most important point:

- xenstore need consistent indices. Using domain names in the current stack results in a linear search through all domids -- very slow with 10000 VMs! Irmin could maintain a branch that updates a 'name -> domid' that is consistent within the transaction, and do constant time lookups for the toolstack.

-anil

_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

Re: [MirageOS-devel] xen dev summit talk slides: Branch Consistency for Xenstore [ In reply to ]

thomas at gazagnaire

Aug 19, 2014, 9:26 AM

Post #3 of 5 (1498 views)

Permalink

>>> I'm giving a talk at the Xen Developer Summit at Linuxcon tomorrow on the work that we've (primarily Dave Scott and Thomas Gazagnaire) on improving the Xenstore via Irmin features.
>>>
>>> Slides here: http://decks.openmirage.org/xendevsummit14#/
>>> Repo: https://github.com/mirage/mirage-decks as usual
>>>
>>
>> [+xen-api-devel] Talk went well, quick notes here:
>>
>> - git workflow very popular. Lots of people twigged onto the maintainability benefits of `git bisect` automation in particular.
>>
>> - questions about why Xenstore transactions are still necessary in the modern world. Can replace with consensus protocols instead? Maybe time for an ABI bump to deprecate the ancient xenstore protocol.
>>
>> - space usage is a concern -- building an RRD-style constant size library to maintain progressive history would be a big win.

A note on that: during his internship, Benjamin Farinier wrote a very simple backend for Irmin which maps Irmin objects to the OCaml heap directly[1], which means that 1/ no SHA1 needs to be computed (so it's fast) and 2/ we picky-bag the OCaml GC to collect Irmin objects. I think this is a good base for an in-memory store, we just need to understand how to expose it efficiently to the Git backend (ie. you don't want to recompute all the SHA1 on every export).

But yes, having an RRD-style history is the way to go.

[1] https://github.com/BFarinier/irmin/blob/master/examples/objAO.ml#L7

>> - *excellent* talk from Felipe Huici (CCed) from NEC about building much denser VM workloads, and he observed that Xenstored/xenconsoled are a big bottleneck at ~10000 VMs. Are your slides available Felipe? Some sub notes:
>>
>> - we could write a mirage xenconsoled to log to irmin and drain guest console rings much faster.
>> - a distributed xenstored+irmin would allow significantly more scalability than attempting to build a serially fast version.
>> - felipe has the beginning of a simple c++ xenstored that doesn't implement the full semantics, but is enough for MiniOS.
>> - it may be useful to negotiate a xenstore v2 protocol and use that for new guests. It could use a simple fixed-length binary protocol (protobuf-style?), and eliminate the need for transactions perhaps.
>
> sent 'send' too quickly. The most important point:
>
> - xenstore need consistent indices. Using domain names in the current stack results in a linear search through all domids -- very slow with 10000 VMs! Irmin could maintain a branch that updates a 'name -> domid' that is consistent within the transaction, and do constant time lookups for the toolstack.

Irmin use persistent map to store sub-paths, so we already have logarithmic lookup (initially it was stored in a list, but my initial benchmarks pointed that as a bottleneck very quickly). Using maps is very efficient in practice.

Thomas

>
> -anil
>
>
> _______________________________________________
> MirageOS-devel mailing list
> MirageOS-devel@lists.xenproject.org
> http://lists.xenproject.org/cgi-bin/mailman/listinfo/mirageos-devel

_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

Re: [MirageOS-devel] xen dev summit talk slides: Branch Consistency for Xenstore [ In reply to ]

Felipe.Huici at neclab

Aug 20, 2014, 2:05 AM

Post #4 of 5 (1500 views)

Permalink

Hi Anil, all,

As a clarification, the talk was given by my colleague Filipe Manco, in
cc, not me. Iâ€™ve temporarily placed the slides at:

http://www0.cs.ucl.ac.uk/staff/f.huici/presentations/clickos-xendevsummit14
.pdf

They should appear on the Xen summit briefly, and more permanently on our
website at:

https://cnp.neclab.eu

Thanks,

â€” Felipe

============================================================
Dr. Felipe Huici
Senior Researcher, Networked Systems and Data
Analytics Group
NEC Laboratories Europe, Network Research Division
Kurfuerstenanlage 36, D-69115 Heidelberg
Tel. +49
(0)6221 4342-241
Fax: +49
(0)6221 4342-155

e-mail:
felipe.huici@neclab.eu
============================================================
NEC Europe Limited Registered Office: NEC House, 1
Victoria Road, London W3 6BL Registered in England 2832014

On 8/19/14, 3:47 PM, "Anil Madhavapeddy" <anil@recoil.org> wrote:

>On 19 Aug 2014, at 08:44, Anil Madhavapeddy <anil@recoil.org> wrote:
>
>> On 17 Aug 2014, at 11:01, Anil Madhavapeddy <anil@recoil.org> wrote:
>>
>>> I'm giving a talk at the Xen Developer Summit at Linuxcon tomorrow on
>>>the work that we've (primarily Dave Scott and Thomas Gazagnaire) on
>>>improving the Xenstore via Irmin features.
>>>
>>> Slides here: http://decks.openmirage.org/xendevsummit14#/
>>> Repo: https://github.com/mirage/mirage-decks as usual
>>>
>>
>> [+xen-api-devel] Talk went well, quick notes here:
>>
>> - git workflow very popular. Lots of people twigged onto the
>>maintainability benefits of `git bisect` automation in particular.
>>
>> - questions about why Xenstore transactions are still necessary in the
>>modern world. Can replace with consensus protocols instead? Maybe time
>>for an ABI bump to deprecate the ancient xenstore protocol.
>>
>> - space usage is a concern -- building an RRD-style constant size
>>library to maintain progressive history would be a big win.
>>
>> - *excellent* talk from Felipe Huici (CCed) from NEC about building
>>much denser VM workloads, and he observed that Xenstored/xenconsoled are
>>a big bottleneck at ~10000 VMs. Are your slides available Felipe? Some
>>sub notes:
>>
>> - we could write a mirage xenconsoled to log to irmin and drain guest
>>console rings much faster.
>> - a distributed xenstored+irmin would allow significantly more
>>scalability than attempting to build a serially fast version.
>> - felipe has the beginning of a simple c++ xenstored that doesn't
>>implement the full semantics, but is enough for MiniOS.
>> - it may be useful to negotiate a xenstore v2 protocol and use that for
>>new guests. It could use a simple fixed-length binary protocol
>>(protobuf-style?), and eliminate the need for transactions perhaps.
>
>sent 'send' too quickly. The most important point:
>
>- xenstore need consistent indices. Using domain names in the current
>stack results in a linear search through all domids -- very slow with
>10000 VMs! Irmin could maintain a branch that updates a 'name -> domid'
>that is consistent within the transaction, and do constant time lookups
>for the toolstack.
>
>-anil
>

_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

Re: [MirageOS-devel] xen dev summit talk slides: Branch Consistency for Xenstore [ In reply to ]

anil at recoil

Aug 20, 2014, 6:18 AM

Post #5 of 5 (1492 views)

Permalink

Thanks Felipe! (Huici and Manco :-)

Terrible wifi at the dev summit this year, so it was difficult to share more
information in real time.

best,
Anil

On 20 Aug 2014, at 04:05, Felipe Huici <Felipe.Huici@neclab.eu> wrote:

> Hi Anil, all,
>
> As a clarification, the talk was given by my colleague Filipe Manco, in
> cc, not me. I’ve temporarily placed the slides at:
>
> http://www0.cs.ucl.ac.uk/staff/f.huici/presentations/clickos-xendevsummit14
> .pdf
>
> They should appear on the Xen summit briefly, and more permanently on our
> website at:
>
> https://cnp.neclab.eu
>
> Thanks,
>
> — Felipe
>
> ============================================================
> Dr. Felipe Huici
> Senior Researcher, Networked Systems and Data
> Analytics Group
> NEC Laboratories Europe, Network Research Division
> Kurfuerstenanlage 36, D-69115 Heidelberg
> Tel. +49
> (0)6221 4342-241
> Fax: +49
> (0)6221 4342-155
>
> e-mail:
> felipe.huici@neclab.eu
> ============================================================
> NEC Europe Limited Registered Office: NEC House, 1
> Victoria Road, London W3 6BL Registered in England 2832014
>
>
>
>
> On 8/19/14, 3:47 PM, "Anil Madhavapeddy" <anil@recoil.org> wrote:
>
>> On 19 Aug 2014, at 08:44, Anil Madhavapeddy <anil@recoil.org> wrote:
>>
>>> On 17 Aug 2014, at 11:01, Anil Madhavapeddy <anil@recoil.org> wrote:
>>>
>>>> I'm giving a talk at the Xen Developer Summit at Linuxcon tomorrow on
>>>> the work that we've (primarily Dave Scott and Thomas Gazagnaire) on
>>>> improving the Xenstore via Irmin features.
>>>>
>>>> Slides here: http://decks.openmirage.org/xendevsummit14#/
>>>> Repo: https://github.com/mirage/mirage-decks as usual
>>>>
>>>
>>> [+xen-api-devel] Talk went well, quick notes here:
>>>
>>> - git workflow very popular. Lots of people twigged onto the
>>> maintainability benefits of `git bisect` automation in particular.
>>>
>>> - questions about why Xenstore transactions are still necessary in the
>>> modern world. Can replace with consensus protocols instead? Maybe time
>>> for an ABI bump to deprecate the ancient xenstore protocol.
>>>
>>> - space usage is a concern -- building an RRD-style constant size
>>> library to maintain progressive history would be a big win.
>>>
>>> - *excellent* talk from Felipe Huici (CCed) from NEC about building
>>> much denser VM workloads, and he observed that Xenstored/xenconsoled are
>>> a big bottleneck at ~10000 VMs. Are your slides available Felipe? Some
>>> sub notes:
>>>
>>> - we could write a mirage xenconsoled to log to irmin and drain guest
>>> console rings much faster.
>>> - a distributed xenstored+irmin would allow significantly more
>>> scalability than attempting to build a serially fast version.
>>> - felipe has the beginning of a simple c++ xenstored that doesn't
>>> implement the full semantics, but is enough for MiniOS.
>>> - it may be useful to negotiate a xenstore v2 protocol and use that for
>>> new guests. It could use a simple fixed-length binary protocol
>>> (protobuf-style?), and eliminate the need for transactions perhaps.
>>
>> sent 'send' too quickly. The most important point:
>>
>> - xenstore need consistent indices. Using domain names in the current
>> stack results in a linear search through all domids -- very slow with
>> 10000 VMs! Irmin could maintain a branch that updates a 'name -> domid'
>> that is consistent within the transaction, and do constant time lookups
>> for the toolstack.
>>
>> -anil
>>
>

_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api