Mailing List Archive

On 02/06/14 15:48, Anil Madhavapeddy wrote:

> Thanks for the RRD library link -- I hadnt seen this before and it looks very useful for Xen tracing in MirageOS! A few questions while reading through the source:
>
> - Rrd_io just exposes a single Resource_closed exception and no other values. That's perhaps easier to encode in the return value?

Heh, I'm not sure what I was thinking there...I agree :)

>
> - Rrd_reader is functorized on its transport mechanism (for Gnt selection). This should mean that the library can compile as a stub domain on Mirage, as well as using the /dev/gnt* interfaces on Linux. Is the current usecase to run in dom0 or a separate RRD domain?

The most likely use case at the moment would be to run an rrdd plugin (writer) in a domU with rrdd (reader) running in dom0. However, I've tried to keep the library as generic as possible rather than building it for any particular use case - it works quite nicely for domU-domU communication. Check out the two programs under util/ if you want to have a play.

>
> - Have you considered using vchan to let other tools capture the rrd stream more easily?

I hadn't, but I think that could be made to work. At the moment the protocols expect the shared-memory resource to be represented as a cstruct so it would need a bit of tweaking to work with a socket-style interface instead.

I also wanted to keep the library free of dependencies to any particular handshake mechanism (e.g. xenstore), so I think the right interface would be to start with a ready-initialised vchan server and turn that into a reader (or conversely turn a vchan client into a writer).

John

_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

On 2 Jun 2014, at 18:23, John Else <john.else@citrix.com> wrote:

> On 02/06/14 15:48, Anil Madhavapeddy wrote:
>
>> Thanks for the RRD library link -- I hadnt seen this before and it looks very useful for Xen tracing in MirageOS! A few questions while reading through the source:
>>
>> - Rrd_io just exposes a single Resource_closed exception and no other values. That's perhaps easier to encode in the return value?
>
> Heh, I'm not sure what I was thinking there...I agree :)
>
>>
>> - Rrd_reader is functorized on its transport mechanism (for Gnt selection). This should mean that the library can compile as a stub domain on Mirage, as well as using the /dev/gnt* interfaces on Linux. Is the current usecase to run in dom0 or a separate RRD domain?
>
> The most likely use case at the moment would be to run an rrdd plugin (writer) in a domU with rrdd (reader) running in dom0. However, I've tried to keep the library as generic as possible rather than building it for any particular use case - it works quite nicely for domU-domU communication. Check out the two programs under util/ if you want to have a play.

Will do, thanks! I really like the idea of MirageOS apps maintaining an RRD scoreboard of ongoing stats like netstat and so on.

One thing that would really help with using these libraries outside of Xapi is to reduce the dependency on the Stdext modules, and to make the core libraries more standalone. For instance, in rrd-transport, most of the logic except for the final "to_fd" functions looks like pure OCaml, and so would be useful to have as a standalone library.

>> - Have you considered using vchan to let other tools capture the rrd stream more easily?
>
> I hadn't, but I think that could be made to work. At the moment the protocols expect the shared-memory resource to be represented as a cstruct so it would need a bit of tweaking to work with a socket-style interface instead.
>
> I also wanted to keep the library free of dependencies to any particular handshake mechanism (e.g. xenstore), so I think the right interface would be to start with a ready-initialised vchan server and turn that into a reader (or conversely turn a vchan client into a writer).

Yep -- Jon Ludlam and I were thinking of installing netcat-style vchan binaries to make this sort of interface easier from shell scripts. I'll have a better idea of the MirageOS end of things once I finish refactoring the Conduit [1] library, which hides away the transport mechanism details.

[1] https://github.com/mirage/ocaml-conduit

best,
Anil
_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

On 2 Jun 2014, at 18:23, John Else <john.else@citrix.com> wrote:

> I hadn't, but I think that could be made to work. At the moment the protocols expect the shared-memory resource to be represented as a cstruct so it would need a bit of tweaking to work with a socket-style interface instead.

Actually thinking about it, I think a Cstruct interface is all you need, since vchan provides this too.

A socket interface would somewhat defeat the point of an RRD service due to all the copies and allocation to read/write data from it...

-anil
_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

On 02/06/14 18:23, John Else wrote:
>
>> - Have you considered using vchan to let other tools capture the rrd stream more easily?
> I hadn't, but I think that could be made to work. At the moment the protocols expect the shared-memory resource to be represented as a cstruct so it would need a bit of tweaking to work with a socket-style interface instead.
>
> I also wanted to keep the library free of dependencies to any particular handshake mechanism (e.g. xenstore), so I think the right interface would be to start with a ready-initialised vchan server and turn that into a reader (or conversely turn a vchan client into a writer).
A nice aspect of the current simple shared-memory design is that it
desynchronises reader and writer, so the writer can simply update its
page without worrying that the reader has died and isn't going to drain
the queue. It also means that the reader degrades nicely as the number
of data sources increases - it can simply sample less frequently. This
works best for DERIVE type data sources where the writer maintains a
monotonically increasing count of which the interesting thing is the
derivative (e.g. CPU time, or VIF bytes transmitted).

I think we should strive for the principle that observing the stats
should have as little impact as possible on the thing being observed, so
removing synchronisation points seems like a good idea. We clearly want
to avoid the Consciousness Causes Collapse problem :-)

Jon


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api