Mailing List Archive

On 06/02/2014 03:59 PM, Jon Ludlam wrote:
> Hi all,
>
> It's about time we had a release!
>
> There has been a huge amount of effort been spent over the last year or
> so since we joined the Linux Foundation, and we're almost at the point
> where the master branch of the xapi-project repositories can be used to
> manage VMs in several different environments. So the main goal of this
> release is that it works without source change on XenServer, CentOS 6.5
> and Ubuntu 14.04. Note that this will be a source only release, as with
> standard xen-project releases.
>
> What we need to do is agree as a community what we need to do before we
> hit the release button. We had a good starting discussion at the Xen
> Hackathon last week, which I'll summarise here:
>
> * Needs to pass 'exercise.sh' tests on Ubuntu 14.04 and CentOS 6.5 (via
> buildroot packages).
> * On XenServer, we can exploit the Citrix internal tests (nightly and
> stress), which need to pass with 'reasonable' rates.
> * All OCaml libraries must be available in opam - either through
> xapi-project/opam-repo-dev or (preferably) the standard opam repository.
> * Need to be able to bypass xcp-networkd taking over the management of
> the networking on the host.
> * Documentation! We need at least installation, basic usage, and how to
> contribute, including build instructions.
>
> So, if anyone else has any suggestions for what needs to be done before
> release, please reply and we'll gather together a list of requirements.
>
> Also, we'll need to track the todo list. Anybody have any preference for
> where to do this? wiki.xenproject.org? Github issues?
>
>
I don't want to sound to skeptical, but after about 5 year of deep work
with xapi, I was really disappointed by Citrix VS opensource
relationship. XenServer was and is not libre. There is source code but
there is no way to rebuild original ISO from those sources. Decisions
were made purely in-house sole by Citrix, and published opensource
version (xapi packages) were broken beyond repair (and were removed from
repository).

Does something really changed here? I see lot of problems in xapi VS
community relationship and Citrix is kinda not opensource company
(Presentation Server, World of Windows and so on). Xensource was bit
unexpected addition to this tight and cozy wold of Citrix and Microsoft,
and source publication is not made xapi libre software. Only 'open
source', but not libre.



_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

Hi Jon,

XO team reporting :)

I think for a better community visibility, GitHub is the right way.
You can tag any issue with anything you like, and thus create simple
yet powerful workflows. Using the Xen wiki will sound more "closed".

As we discussed in the Hackathon, documentation is also really
important to catch people and really impulse a community. When I'll
have time, I'll probably create issues or create requests for this
doc.

Cheers,

--
Olivier
Xen Orchestra Project Leader
http://xen-orchestra.com

On Mon, Jun 2, 2014 at 2:59 PM, Jon Ludlam
<jonathan.ludlam@eu.citrix.com> wrote:
> Hi all,
>
> It's about time we had a release!
>
> There has been a huge amount of effort been spent over the last year or
> so since we joined the Linux Foundation, and we're almost at the point
> where the master branch of the xapi-project repositories can be used to
> manage VMs in several different environments. So the main goal of this
> release is that it works without source change on XenServer, CentOS 6.5
> and Ubuntu 14.04. Note that this will be a source only release, as with
> standard xen-project releases.
>
> What we need to do is agree as a community what we need to do before we
> hit the release button. We had a good starting discussion at the Xen
> Hackathon last week, which I'll summarise here:
>
> * Needs to pass 'exercise.sh' tests on Ubuntu 14.04 and CentOS 6.5 (via
> buildroot packages).
> * On XenServer, we can exploit the Citrix internal tests (nightly and
> stress), which need to pass with 'reasonable' rates.
> * All OCaml libraries must be available in opam - either through
> xapi-project/opam-repo-dev or (preferably) the standard opam repository.
> * Need to be able to bypass xcp-networkd taking over the management of
> the networking on the host.
> * Documentation! We need at least installation, basic usage, and how to
> contribute, including build instructions.
>
> So, if anyone else has any suggestions for what needs to be done before
> release, please reply and we'll gather together a list of requirements.
>
> Also, we'll need to track the todo list. Anybody have any preference for
> where to do this? wiki.xenproject.org? Github issues?
>
> Cheers,
>
> Jon
>
>
> _______________________________________________
> Xen-api mailing list
> Xen-api@lists.xen.org
> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

Hi George,

On 2 Jun 2014, at 14:13, George Shuklin <george.shuklin@gmail.com> wrote:

> On 06/02/2014 03:59 PM, Jon Ludlam wrote:
>> Hi all,
>>
>> It's about time we had a release!
>>
>> There has been a huge amount of effort been spent over the last year or
>> so since we joined the Linux Foundation, and we're almost at the point
>> where the master branch of the xapi-project repositories can be used to
>> manage VMs in several different environments. So the main goal of this
>> release is that it works without source change on XenServer, CentOS 6.5
>> and Ubuntu 14.04. Note that this will be a source only release, as with
>> standard xen-project releases.
>>
>> What we need to do is agree as a community what we need to do before we
>> hit the release button. We had a good starting discussion at the Xen
>> Hackathon last week, which I'll summarise here:
>>
>> * Needs to pass 'exercise.sh' tests on Ubuntu 14.04 and CentOS 6.5 (via
>> buildroot packages).
>> * On XenServer, we can exploit the Citrix internal tests (nightly and
>> stress), which need to pass with 'reasonable' rates.
>> * All OCaml libraries must be available in opam - either through
>> xapi-project/opam-repo-dev or (preferably) the standard opam repository.
>> * Need to be able to bypass xcp-networkd taking over the management of
>> the networking on the host.
>> * Documentation! We need at least installation, basic usage, and how to
>> contribute, including build instructions.
>>
>> So, if anyone else has any suggestions for what needs to be done before
>> release, please reply and we'll gather together a list of requirements.
>>
>> Also, we'll need to track the todo list. Anybody have any preference for
>> where to do this? wiki.xenproject.org? Github issues?
>>
>>
> I don't want to sound to skeptical, but after about 5 year of deep work with xapi, I was really disappointed by Citrix VS opensource relationship. XenServer was and is not libre. There is source code but there is no way to rebuild original ISO from those sources. Decisions were made purely in-house sole by Citrix, and published opensource version (xapi packages) were broken beyond repair (and were removed from repository).

I agree that the inability to rebuild the ISO from sources is a big problem. We’ve been working on this slowly over the past year and have now reached a point where the ‘master’ branch of xapi (and associated tools) should build on a modern Linux distro. Since there are a set of libraries we depend on we’ve created this repo:

https://github.com/xenserver/buildroot

— it contains “spec” files for all these dependencies. If you fire up a CentOS 6 or an Ubuntu trusty box you should be able to clone that repo and type

./configure.sh
make
make install

I’ve switched over to using this as my primary development environment now since it’s the most convenient way (especially combined with a client hypervisor like virtualbox on a laptop). Note that it builds the latest bleeding-edge version of the code, so it’s not production-ready yet.

I also agree that our previous attempt to publish open source packages weren’t really usable. The old version of the code had so many dependencies on a customised CentOS distro that the packages needed a very large patch queue to build. The patch queue created effectively a fork of the code that we couldn’t afford to maintain. Merging these patches into the ‘master’ branch has taken about a year but is done now (more or less).

Therefore I think one of the “xapi 2” release criteria should definitely be “all the portability patches have been merged” so that clean packages can be created.

I hope this helps,
Dave Scott


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

On 2 Jun 2014, at 14:13, George Shuklin <george.shuklin@gmail.com> wrote:

> I don't want to sound to skeptical, but after about 5 year of deep work with xapi, I was really disappointed by Citrix VS opensource relationship. XenServer was and is not libre. There is source code but there is no way to rebuild original ISO from those sources. Decisions were made purely in-house sole by Citrix, and published opensource version (xapi packages) were broken beyond repair (and were removed from repository).

As another former contributor but now external to Citrix for a long time, I was very encouraged by the discussions at the Xen hackathon last week. There's been a lot of behind-the-scenes cleanup to move Xapi towards working on an unmodified set of RPMs/Debs that indicate that this upstreaming effort will be much more sustainable than previous iterations.

One key thing for me would be an OPAM submission of Xapi such that it could all be rebuilt easily from the public repo. Once that's done, I believe that many of the other issues (local patches for LVM and so on) are addressed by having storage plugins that are designed to work with open-source distros (such as the local-directory ffs plugin instead of the NFS SR).

best,
Anil
_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

On 02/06/14 14:13, George Shuklin wrote:
> I don't want to sound to skeptical, but after about 5 year of deep
> work with xapi, I was really disappointed by Citrix VS opensource
> relationship. XenServer was and is not libre. There is source code but
> there is no way to rebuild original ISO from those sources. Decisions
> were made purely in-house sole by Citrix, and published opensource
> version (xapi packages) were broken beyond repair (and were removed
> from repository).
>
Yes, building the ISO was and still is awkward for non-Citrix people.
This isn't a release of the ISO though, this is just a release of some
of the tools on the ISO, and the biggest focus is on making it work
_properly_ outside the context of XenServer. As Dave mentioned,
buildroot is proof that the source can work in both a Debian-like and
CentOS-like environment.

> Does something really changed here? I see lot of problems in xapi VS
> community relationship and Citrix is kinda not opensource company
> (Presentation Server, World of Windows and so on). Xensource was bit
> unexpected addition to this tight and cozy wold of Citrix and
> Microsoft, and source publication is not made xapi libre software.
> Only 'open source', but not libre.
>
Yes, something has changed. For example, the problem back when we got
xapi into Debian was that we had to get _much_ too involved in the
packaging process - it wasn't simply a case of grabbing the source
tarballs, building and packaging them - it effectively meant that a few
of us had to spend several months making the thing work at all, and
those patches ended up in the debian source package, rotting gently
while the master branch moved on. There was no way back then that xapi
could ever have been packaged up from source by anyone other than us.

This is _very_ different from now. We've spent a long time splitting
repositories up, adding standard build system files, removing hard-coded
paths, splitting things up into more sensible smaller chunks and
generalizing the code. We've split two massive monolithic repositories
(xen-api and xen-api-libs) into about 30 smaller more sensible ones. By
the time of the release, the installation workflow should for these
individual repositories should simply be 'clone from github', then
'configure', 'make', 'make install', and you should be there, and I
think it's entirely reasonable that someone with a working Xen
installation could package it up successfully.

Of course, simply building is not enough. The buildroot repository
demonstrates that, with a minimum of patching (we've still got patches
for xcp-sm, vncterm and opasswd, but these will be fixed before the
release) viable packages can be created that work well enough to run VMs
to OpenStack's satisfaction.

I'm aware that this doesn't address all of your concerns. But I believe
it's a really good first step, and hopefully our next steps will all go
in the same direction too :-)

Jon


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

On 02/06/14 14:21, Olivier Lambert wrote:
> Hi Jon,
>
> XO team reporting :)
>
> I think for a better community visibility, GitHub is the right way.
> You can tag any issue with anything you like, and thus create simple
> yet powerful workflows. Using the Xen wiki will sound more "closed".

Good points. My only concern was that we're talking about a release of
lots of different repositories, so using the issues will be a bit more
awkward - I think we'll still need a central point of coordination, but
perhaps that could just be an issue in xen-api.

> As we discussed in the Hackathon, documentation is also really
> important to catch people and really impulse a community. When I'll
> have time, I'll probably create issues or create requests for this
> doc.

Brilliant, that'll be really useful.

One thing that came up recently is that the API documentation is quite
tricky to find online, and is only released on XenServer releases.
Perhaps we ought to get the dev version hosted somewhere other than
docs.vmd.citrix.com?

Jon

_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

We face (at a really smaller scale!) the same problem. As XO is
modular, we've got different repo (xo-web, xo-cli, xo-server). We
choose to create a "meta" repository called "xo"
(https://github.com/vatesfr/xo) with links to the others, and also the
documentation. It's our central point for non-specific issues/reports.
In your case, a central repo with the doc can be great: anyone
(thinking of me) can clone and push doc stuff without the fear to
break code.

Indeed, having links or doc in this repo could help people wanting to
do some nice hacks on the top of the XAPI.

On Mon, Jun 2, 2014 at 5:34 PM, Jon Ludlam
<jonathan.ludlam@eu.citrix.com> wrote:
> On 02/06/14 14:21, Olivier Lambert wrote:
>> Hi Jon,
>>
>> XO team reporting :)
>>
>> I think for a better community visibility, GitHub is the right way.
>> You can tag any issue with anything you like, and thus create simple
>> yet powerful workflows. Using the Xen wiki will sound more "closed".
>
> Good points. My only concern was that we're talking about a release of
> lots of different repositories, so using the issues will be a bit more
> awkward - I think we'll still need a central point of coordination, but
> perhaps that could just be an issue in xen-api.
>
>> As we discussed in the Hackathon, documentation is also really
>> important to catch people and really impulse a community. When I'll
>> have time, I'll probably create issues or create requests for this
>> doc.
>
> Brilliant, that'll be really useful.
>
> One thing that came up recently is that the API documentation is quite
> tricky to find online, and is only released on XenServer releases.
> Perhaps we ought to get the dev version hosted somewhere other than
> docs.vmd.citrix.com?
>
> Jon

_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

On 06/02/2014 04:41 PM, Dave Scott wrote:
> Hi George,
>
> On 2 Jun 2014, at 14:13, George Shuklin <george.shuklin@gmail.com> wrote:
>
>> On 06/02/2014 03:59 PM, Jon Ludlam wrote:
>>> Hi all,
>>>
>>> It's about time we had a release!
>>>
>>> There has been a huge amount of effort been spent over the last year or
>>> so since we joined the Linux Foundation, and we're almost at the point
>>> where the master branch of the xapi-project repositories can be used to
>>> manage VMs in several different environments. So the main goal of this
>>> release is that it works without source change on XenServer, CentOS 6.5
>>> and Ubuntu 14.04. Note that this will be a source only release, as with
>>> standard xen-project releases.
>>>
>>> What we need to do is agree as a community what we need to do before we
>>> hit the release button. We had a good starting discussion at the Xen
>>> Hackathon last week, which I'll summarise here:
>>>
>>> * Needs to pass 'exercise.sh' tests on Ubuntu 14.04 and CentOS 6.5 (via
>>> buildroot packages).
>>> * On XenServer, we can exploit the Citrix internal tests (nightly and
>>> stress), which need to pass with 'reasonable' rates.
>>> * All OCaml libraries must be available in opam - either through
>>> xapi-project/opam-repo-dev or (preferably) the standard opam repository.
>>> * Need to be able to bypass xcp-networkd taking over the management of
>>> the networking on the host.
>>> * Documentation! We need at least installation, basic usage, and how to
>>> contribute, including build instructions.
>>>
>>> So, if anyone else has any suggestions for what needs to be done before
>>> release, please reply and we'll gather together a list of requirements.
>>>
>>> Also, we'll need to track the todo list. Anybody have any preference for
>>> where to do this? wiki.xenproject.org? Github issues?
>>>
>>>
>> I don't want to sound to skeptical, but after about 5 year of deep work with xapi, I was really disappointed by Citrix VS opensource relationship. XenServer was and is not libre. There is source code but there is no way to rebuild original ISO from those sources. Decisions were made purely in-house sole by Citrix, and published opensource version (xapi packages) were broken beyond repair (and were removed from repository).
> I agree that the inability to rebuild the ISO from sources is a big problem. We’ve been working on this slowly over the past year and have now reached a point where the ‘master’ branch of xapi (and associated tools) should build on a modern Linux distro. Since there are a set of libraries we depend on we’ve created this repo:
>
> https://github.com/xenserver/buildroot
>
> — it contains “spec” files for all these dependencies. If you fire up a CentOS 6 or an Ubuntu trusty box you should be able to clone that repo and type
>
> ./configure.sh
> make
> make install
>
> I’ve switched over to using this as my primary development environment now since it’s the most convenient way (especially combined with a client hypervisor like virtualbox on a laptop). Note that it builds the latest bleeding-edge version of the code, so it’s not production-ready yet.
>
> I also agree that our previous attempt to publish open source packages weren’t really usable. The old version of the code had so many dependencies on a customised CentOS distro that the packages needed a very large patch queue to build. The patch queue created effectively a fork of the code that we couldn’t afford to maintain. Merging these patches into the ‘master’ branch has taken about a year but is done now (more or less).
>
> Therefore I think one of the “xapi 2” release criteria should definitely be “all the portability patches have been merged” so that clean packages can be created.
>
>

Glad to hear that.

But still, how about problems with LVM patches? As far as I know there
is a bug ugly patch over lvm2 tools in XenServer. This patch will never
be accepted to upstream, and lack of that patch kills completely
LVMoISCSI SR. I never saw source of ISO installator and patches for
Centos/RHEL installers for elilo.

And, the main question: governance. Personally I think XenServer made a
horrible mistake when adopted VHD format - and because Citrix is
completely control of XenServer development, there is no chances to undo
that. And Citrix is more interested in VDIs than everything else...


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

On 06/02/2014 06:26 PM, Jon Ludlam wrote:
> On 02/06/14 14:13, George Shuklin wrote:
>> I don't want to sound to skeptical, but after about 5 year of deep
>> work with xapi, I was really disappointed by Citrix VS opensource
>> relationship. XenServer was and is not libre. There is source code but
>> there is no way to rebuild original ISO from those sources. Decisions
>> were made purely in-house sole by Citrix, and published opensource
>> version (xapi packages) were broken beyond repair (and were removed
>> from repository).
>>
> Yes, building the ISO was and still is awkward for non-Citrix people.
> This isn't a release of the ISO though, this is just a release of some
> of the tools on the ISO, and the biggest focus is on making it work
> _properly_ outside the context of XenServer. As Dave mentioned,
> buildroot is proof that the source can work in both a Debian-like and
> CentOS-like environment.
All problems with xapi and Xenserver is that XenServer is 'tightly
build' bunch of scripts and patched programs, outside of xapi. If you
looks to execution path for real operations with VMs, you can see, that
right after nice and cool ocaml code in xapi, there is an extremely
strange code on python ( os.popen('ls') for the file list is strong
example of 'strange'). But right after python code (/opt/xensource/sm)
there is much much more problematic set of shell scripts for many
host-wide operations (/opt/xensource/scripts).

It is not core part and it was very poorly maintained outside XenServer.
I filed more than 10 bugs of completely broken functions (like host
rename) - they are clearly no 'xapi', but xapi without them is cripple.

if this has been changed, it really nice. I really hope so.

>> Does something really changed here? I see lot of problems in xapi VS
>> community relationship and Citrix is kinda not opensource company
>> (Presentation Server, World of Windows and so on). Xensource was bit
>> unexpected addition to this tight and cozy wold of Citrix and
>> Microsoft, and source publication is not made xapi libre software.
>> Only 'open source', but not libre.
>>
> Yes, something has changed. For example, the problem back when we got
> xapi into Debian was that we had to get _much_ too involved in the
> packaging process - it wasn't simply a case of grabbing the source
> tarballs, building and packaging them - it effectively meant that a few
> of us had to spend several months making the thing work at all, and
> those patches ended up in the debian source package, rotting gently
> while the master branch moved on. There was no way back then that xapi
> could ever have been packaged up from source by anyone other than us.
>
> This is _very_ different from now. We've spent a long time splitting
> repositories up, adding standard build system files, removing hard-coded
> paths, splitting things up into more sensible smaller chunks and
> generalizing the code. We've split two massive monolithic repositories
> (xen-api and xen-api-libs) into about 30 smaller more sensible ones. By
> the time of the release, the installation workflow should for these
> individual repositories should simply be 'clone from github', then
> 'configure', 'make', 'make install', and you should be there, and I
> think it's entirely reasonable that someone with a working Xen
> installation could package it up successfully.
>
> Of course, simply building is not enough. The buildroot repository
> demonstrates that, with a minimum of patching (we've still got patches
> for xcp-sm, vncterm and opasswd, but these will be fixed before the
> release) viable packages can be created that work well enough to run VMs
> to OpenStack's satisfaction.
>
> I'm aware that this doesn't address all of your concerns. But I believe
> it's a really good first step, and hopefully our next steps will all go
> in the same direction too :-)
>
Is buildroot cover tests for shell scripts? If you compare neutron ovs
plugin code to xensource scripts - ovs got very neat tests and constant
configuration checking. Xensource scripts do not check configuration,
they blindly change configuration with expectation of perfectly sane
environment (inside OVS config). This is on of 'tighly build' part of
xenserver, which cause huge pain in case of slightest changes in host
environment.

I don't want to be rude, but xapi is too 'api-centric' and just ignore
all 'dirty' (in CS meaning of 'dirty') operations like disk
initialization, volume manipulation and so on. And it passes all those
operations to 'dirty' languages like python and bash to handle dirty
work. And they do it dirty (pun intended).

I think that part is much more important for wide adoption than
perfection of xapi itself.



_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

On 02/06/14 17:11, George Shuklin wrote:
> On 06/02/2014 06:26 PM, Jon Ludlam wrote:
>> On 02/06/14 14:13, George Shuklin wrote:
>>> I don't want to sound to skeptical, but after about 5 year of deep
>>> work with xapi, I was really disappointed by Citrix VS opensource
>>> relationship. XenServer was and is not libre. There is source code but
>>> there is no way to rebuild original ISO from those sources. Decisions
>>> were made purely in-house sole by Citrix, and published opensource
>>> version (xapi packages) were broken beyond repair (and were removed
>>> from repository).
>>>
>> Yes, building the ISO was and still is awkward for non-Citrix people.
>> This isn't a release of the ISO though, this is just a release of some
>> of the tools on the ISO, and the biggest focus is on making it work
>> _properly_ outside the context of XenServer. As Dave mentioned,
>> buildroot is proof that the source can work in both a Debian-like and
>> CentOS-like environment.
> All problems with xapi and Xenserver is that XenServer is 'tightly
> build' bunch of scripts and patched programs, outside of xapi. If you
> looks to execution path for real operations with VMs, you can see,
> that right after nice and cool ocaml code in xapi, there is an
> extremely strange code on python ( os.popen('ls') for the file list is
> strong example of 'strange'). But right after python code
> (/opt/xensource/sm) there is much much more problematic set of shell
> scripts for many host-wide operations (/opt/xensource/scripts).
>
> It is not core part and it was very poorly maintained outside
> XenServer. I filed more than 10 bugs of completely broken functions
> (like host rename) - they are clearly no 'xapi', but xapi without them
> is cripple.
>
> if this has been changed, it really nice. I really hope so.
>

This is interesting. We've been working under the assumption that bits
of the API just won't work when you're not in a XenServer context - for
example, the patch management APIs - they just wouldn't make sense.
However, the VM, storage and most of the networking APIs are intended to
be first class and should work correctly. Then there's a grey area where
it's not clear whether people will expect things to work or not - host
rename is a good example. Should it work?


>>> Does something really changed here? I see lot of problems in xapi VS
>>> community relationship and Citrix is kinda not opensource company
>>> (Presentation Server, World of Windows and so on). Xensource was bit
>>> unexpected addition to this tight and cozy wold of Citrix and
>>> Microsoft, and source publication is not made xapi libre software.
>>> Only 'open source', but not libre.
>>>
>> Yes, something has changed. For example, the problem back when we got
>> xapi into Debian was that we had to get _much_ too involved in the
>> packaging process - it wasn't simply a case of grabbing the source
>> tarballs, building and packaging them - it effectively meant that a few
>> of us had to spend several months making the thing work at all, and
>> those patches ended up in the debian source package, rotting gently
>> while the master branch moved on. There was no way back then that xapi
>> could ever have been packaged up from source by anyone other than us.
>>
>> This is _very_ different from now. We've spent a long time splitting
>> repositories up, adding standard build system files, removing hard-coded
>> paths, splitting things up into more sensible smaller chunks and
>> generalizing the code. We've split two massive monolithic repositories
>> (xen-api and xen-api-libs) into about 30 smaller more sensible ones. By
>> the time of the release, the installation workflow should for these
>> individual repositories should simply be 'clone from github', then
>> 'configure', 'make', 'make install', and you should be there, and I
>> think it's entirely reasonable that someone with a working Xen
>> installation could package it up successfully.
>>
>> Of course, simply building is not enough. The buildroot repository
>> demonstrates that, with a minimum of patching (we've still got patches
>> for xcp-sm, vncterm and opasswd, but these will be fixed before the
>> release) viable packages can be created that work well enough to run VMs
>> to OpenStack's satisfaction.
>>
>> I'm aware that this doesn't address all of your concerns. But I believe
>> it's a really good first step, and hopefully our next steps will all go
>> in the same direction too :-)
>>
> Is buildroot cover tests for shell scripts? If you compare neutron ovs
> plugin code to xensource scripts - ovs got very neat tests and
> constant configuration checking. Xensource scripts do not check
> configuration, they blindly change configuration with expectation of
> perfectly sane environment (inside OVS config). This is on of 'tighly
> build' part of xenserver, which cause huge pain in case of slightest
> changes in host environment.
>
Buildroot is just example packaging configuration. The tests are from
OpenStack, and are tests for the functionality OpenStack wants.
> I don't want to be rude, but xapi is too 'api-centric' and just
> ignore all 'dirty' (in CS meaning of 'dirty') operations like disk
> initialization, volume manipulation and so on. And it passes all those
> operations to 'dirty' languages like python and bash to handle dirty
> work. And they do it dirty (pun intended).
>
Which isn't entirely unreasonable - it at least gives an opportunity to
make quick 'dirty' fixes when the environment has changed.

You're dead right that XenServer is, and has always been, a 'tightly
coupled' system. It's an embedded system, and that's the mindset; it has
always had complete control of the system. However, it would be a
mistake to try and make that work in a general linux environment. The
bits I would like to see work are the sorts of things that Xen
Orchestra, CloudStack, OpenStack, Vagrant, & co all want to do -
install, start & stop VMs, configure their networking, snapshot their
disks, migrate and so on. Part of that is indeed fixing things like the
network and storage scripts so that they are more tolerant and careful,
and there is definitely work to do on that front. We've got a start, for
example 'ffs', a storage backend that is much easier to run on an
already-installed system, but there's still plenty of work to do. The
intention here is first to make the master branches at least work, so
that any work to make them work _well_ can be easily upstreamed.

Jon


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

Jon,
you should consider a blog post on blog.xenproject.org for increased
visibility
Lars

On 02/06/2014 13:59, Jon Ludlam wrote:
> Hi all,
>
> It's about time we had a release!
>
> There has been a huge amount of effort been spent over the last year or
> so since we joined the Linux Foundation, and we're almost at the point
> where the master branch of the xapi-project repositories can be used to
> manage VMs in several different environments. So the main goal of this
> release is that it works without source change on XenServer, CentOS 6.5
> and Ubuntu 14.04. Note that this will be a source only release, as with
> standard xen-project releases.
>
> What we need to do is agree as a community what we need to do before we
> hit the release button. We had a good starting discussion at the Xen
> Hackathon last week, which I'll summarise here:
>
> * Needs to pass 'exercise.sh' tests on Ubuntu 14.04 and CentOS 6.5 (via
> buildroot packages).
> * On XenServer, we can exploit the Citrix internal tests (nightly and
> stress), which need to pass with 'reasonable' rates.
> * All OCaml libraries must be available in opam - either through
> xapi-project/opam-repo-dev or (preferably) the standard opam repository.
> * Need to be able to bypass xcp-networkd taking over the management of
> the networking on the host.
> * Documentation! We need at least installation, basic usage, and how to
> contribute, including build instructions.
>
> So, if anyone else has any suggestions for what needs to be done before
> release, please reply and we'll gather together a list of requirements.
>
> Also, we'll need to track the todo list. Anybody have any preference for
> where to do this? wiki.xenproject.org? Github issues?
>
> Cheers,
>
> Jon
>
>
> _______________________________________________
> Xen-api mailing list
> Xen-api@lists.xen.org
> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

On 03/06/14 11:19, Lars Kurth wrote:
> Jon,
> you should consider a blog post on blog.xenproject.org for increased
> visibility
> Lars
>

Good idea. I'll start drafting...

Jon


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

Good suggestions. I've created this:

https://github.com/xapi-project/xapi-project

and in particular, this:

https://github.com/xapi-project/xapi-project/issues/1

A bit sparse at the moment, but we'll fill it in as we think of issues!

Jon

On 02/06/14 16:40, Olivier Lambert wrote:
> We face (at a really smaller scale!) the same problem. As XO is
> modular, we've got different repo (xo-web, xo-cli, xo-server). We
> choose to create a "meta" repository called "xo"
> (https://github.com/vatesfr/xo) with links to the others, and also the
> documentation. It's our central point for non-specific issues/reports.
> In your case, a central repo with the doc can be great: anyone
> (thinking of me) can clone and push doc stuff without the fear to
> break code.
>
> Indeed, having links or doc in this repo could help people wanting to
> do some nice hacks on the top of the XAPI.
>
> On Mon, Jun 2, 2014 at 5:34 PM, Jon Ludlam
> <jonathan.ludlam@eu.citrix.com> wrote:
>> On 02/06/14 14:21, Olivier Lambert wrote:
>>> Hi Jon,
>>>
>>> XO team reporting :)
>>>
>>> I think for a better community visibility, GitHub is the right way.
>>> You can tag any issue with anything you like, and thus create simple
>>> yet powerful workflows. Using the Xen wiki will sound more "closed".
>> Good points. My only concern was that we're talking about a release of
>> lots of different repositories, so using the issues will be a bit more
>> awkward - I think we'll still need a central point of coordination, but
>> perhaps that could just be an issue in xen-api.
>>
>>> As we discussed in the Hackathon, documentation is also really
>>> important to catch people and really impulse a community. When I'll
>>> have time, I'll probably create issues or create requests for this
>>> doc.
>> Brilliant, that'll be really useful.
>>
>> One thing that came up recently is that the API documentation is quite
>> tricky to find online, and is only released on XenServer releases.
>> Perhaps we ought to get the dev version hosted somewhere other than
>> docs.vmd.citrix.com?
>>
>> Jon


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

On 06/03/2014 11:25 AM, Jon Ludlam wrote:
>>> Yes, building the ISO was and still is awkward for non-Citrix people.
>>> This isn't a release of the ISO though, this is just a release of some
>>> of the tools on the ISO, and the biggest focus is on making it work
>>> _properly_ outside the context of XenServer. As Dave mentioned,
>>> buildroot is proof that the source can work in both a Debian-like and
>>> CentOS-like environment.
>> All problems with xapi and Xenserver is that XenServer is 'tightly
>> build' bunch of scripts and patched programs, outside of xapi. If you
>> looks to execution path for real operations with VMs, you can see,
>> that right after nice and cool ocaml code in xapi, there is an
>> extremely strange code on python ( os.popen('ls') for the file list is
>> strong example of 'strange'). But right after python code
>> (/opt/xensource/sm) there is much much more problematic set of shell
>> scripts for many host-wide operations (/opt/xensource/scripts).
>>
>> It is not core part and it was very poorly maintained outside
>> XenServer. I filed more than 10 bugs of completely broken functions
>> (like host rename) - they are clearly no 'xapi', but xapi without them
>> is cripple.
>>
>> if this has been changed, it really nice. I really hope so.
>>
> This is interesting. We've been working under the assumption that bits
> of the API just won't work when you're not in a XenServer context - for
> example, the patch management APIs - they just wouldn't make sense.
> However, the VM, storage and most of the networking APIs are intended to
> be first class and should work correctly. Then there's a grey area where
> it's not clear whether people will expect things to work or not - host
> rename is a good example. Should it work?
>
Last time I saw xapi outside XenServer, it did not work. As well, as
most of host-specific things.

As I say, main problem is that xapi is very pure and strict inside and
require same strictness from running environment. There is no
'discovery', no 'adaptation' , no 'correction'. If environment does not
match xapi expectation, everything is wrong. And xapi will not take
(mostly) attempts to fix it.

Hostname, network settings for management network, iscsi error recovery
- all that is isolated by rather thick layer of shell scripts without
any agility it it.

It sound like bulling the poor xapi, but I've got rather painfull
experience with production under high load, and there is many dark IO
corners around glorious api model.

Simple examples: if iscsi drive change it name due logon/logoff, there
is no way to reconnect everything back without host reboot. If remove
host has no enough memory to accept VM and migration starts (and fails),
VM will leave in broken state. If vm dies unexpectedly during shutdown
initiation, tapdisk will leave VDI locked endlessly (with actual opened
fd and running tapdisk process, without corresponding domain).

List is very, very, very large. Most of it has no relation to API, but
caused by misscommunication between xen, linux drivers, udev and shell
scripts managing them.
> Buildroot is just example packaging configuration. The tests are from
> OpenStack, and are tests for the functionality OpenStack wants.
Oh, ok. Is it cover all python scripts for storage management?
>> I don't want to be rude, but xapi is too 'api-centric' and just
>> ignore all 'dirty' (in CS meaning of 'dirty') operations like disk
>> initialization, volume manipulation and so on. And it passes all those
>> operations to 'dirty' languages like python and bash to handle dirty
>> work. And they do it dirty (pun intended).
>>
> Which isn't entirely unreasonable - it at least gives an opportunity to
> make quick 'dirty' fixes when the environment has changed.
>
> You're dead right that XenServer is, and has always been, a 'tightly
> coupled' system. It's an embedded system, and that's the mindset; it has
> always had complete control of the system. However, it would be a
> mistake to try and make that work in a general linux environment. The
> bits I would like to see work are the sorts of things that Xen
> Orchestra, CloudStack, OpenStack, Vagrant, & co all want to do -
> install, start & stop VMs, configure their networking, snapshot their
> disks, migrate and so on. Part of that is indeed fixing things like the
> network and storage scripts so that they are more tolerant and careful,
> and there is definitely work to do on that front. We've got a start, for
> example 'ffs', a storage backend that is much easier to run on an
> already-installed system, but there's still plenty of work to do. The
> intention here is first to make the master branches at least work, so
> that any work to make them work _well_ can be easily upstreamed.
One more important notice. Xapi is really RPC hungry. In large
installation it cause serious CPU load on master just to control all
slaves and almost nothing can happens without master approval. And xapi
is kinda 'commander and worker' same time.

But most of cloud orchestration do not want too smart host control
software. xapi is just too big and complicated compare to libvirtd/qemu
(just example). I think, for purposes of underlaying
host-hypervisor-management software xapi should gave up some of it
abstraction and complexity toward the simplicity. For example, for
openstack all sm code is just source of headaches (espesially with
randomly changing relationship between vdi uuid, files and attached
drives to virtual machine during snapshot manipulation). Same for
HA-stuff. Openstack do not need 'highly available pool master'. It does
not need even the master of the pool. Same for all smart stuff around
VPP, templating and so on.

I think that overcomplicated part cause problems for 'short path' and
one of the reasons xapi is not loved by openstack/qemu community.


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

On 03/06/2014 14:26, Jon Ludlam wrote:
> On 03/06/14 11:19, Lars Kurth wrote:
>> Jon,
>> you should consider a blog post on blog.xenproject.org for increased
>> visibility
>> Lars
>>
> Good idea. I'll start drafting...
>
> Jon
Jon,

we brought this up at the Advisory Board meeting. If you want you can
have a press release also. However, I need to have a rough idea on
timing for the release and would need some lead time. Mirage OS 2.0 is
planned to release at OSCON.
When do you think we will release XAPI 2.0 and will there be some clear
downloading instructions (e.g. a tarball that can be uploaded via
http://xenproject.org/downloads.html). It would also be a great
opportunity to clean up some of the past XCP confusion. There has been a
community decision to move historical XCP binaries to xenserver.org and
I am waiting for Russell and Tim to do the actual migration (I believe
pretty much everything is prepared). If we could do this before the XAPI
release that would be great.

Best Regards
Lars

_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

Lars,

Tim has turned on the XenServer wiki and it is populated with the old XCP wiki pages. I still have to do XCP cleanup on the Xen Project wiki, tentatively planned for next week.

Don't delay any plans on our account; it will be cleaned up very soon.

Russ Pavlicek
Xen Project Evangelist, Citrix Systems
Home Office: +1-301-829-5327
Mobile: +1-240-397-0199
UK VoIP: +44 1223 852 894
________________________________________
From: Lars Kurth [lars.kurth.xen@gmail.com] on behalf of Lars Kurth [lars.kurth@xen.org]
Sent: Wednesday, June 18, 2014 7:32 AM
To: Jon Ludlam; xen-api@lists.xen.org; Tim Mackey; Russell Pavlicek; Sarah Conway
Subject: Re: [Xen-API] Xapi Project 2.0 release

On 03/06/2014 14:26, Jon Ludlam wrote:
> On 03/06/14 11:19, Lars Kurth wrote:
>> Jon,
>> you should consider a blog post on blog.xenproject.org for increased
>> visibility
>> Lars
>>
> Good idea. I'll start drafting...
>
> Jon
Jon,

we brought this up at the Advisory Board meeting. If you want you can
have a press release also. However, I need to have a rough idea on
timing for the release and would need some lead time. Mirage OS 2.0 is
planned to release at OSCON.
When do you think we will release XAPI 2.0 and will there be some clear
downloading instructions (e.g. a tarball that can be uploaded via
http://xenproject.org/downloads.html). It would also be a great
opportunity to clean up some of the past XCP confusion. There has been a
community decision to move historical XCP binaries to xenserver.org and
I am waiting for Russell and Tim to do the actual migration (I believe
pretty much everything is prepared). If we could do this before the XAPI
release that would be great.

Best Regards
Lars

_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

I also need to turn on the downloads, but had the pages staged. I'll do that when I get back on Monday


Sent via the Samsung GALAXY S®4, an AT&T 4G LTE smartphone


-------- Original message --------
From: Russell Pavlicek
Date:06/18/2014 13:41 (GMT+00:00)
To: lars.kurth@xen.org,Jon Ludlam ,xen-api@lists.xen.org,Tim Mackey ,Sarah Conway
Subject: RE: [Xen-API] Xapi Project 2.0 release

Lars,

Tim has turned on the XenServer wiki and it is populated with the old XCP wiki pages. I still have to do XCP cleanup on the Xen Project wiki, tentatively planned for next week.

Don't delay any plans on our account; it will be cleaned up very soon.

Russ Pavlicek
Xen Project Evangelist, Citrix Systems
Home Office: +1-301-829-5327
Mobile: +1-240-397-0199
UK VoIP: +44 1223 852 894
________________________________________
From: Lars Kurth [lars.kurth.xen@gmail.com] on behalf of Lars Kurth [lars.kurth@xen.org]
Sent: Wednesday, June 18, 2014 7:32 AM
To: Jon Ludlam; xen-api@lists.xen.org; Tim Mackey; Russell Pavlicek; Sarah Conway
Subject: Re: [Xen-API] Xapi Project 2.0 release

On 03/06/2014 14:26, Jon Ludlam wrote:
> On 03/06/14 11:19, Lars Kurth wrote:
>> Jon,
>> you should consider a blog post on blog.xenproject.org for increased
>> visibility
>> Lars
>>
> Good idea. I'll start drafting...
>
> Jon
Jon,

we brought this up at the Advisory Board meeting. If you want you can
have a press release also. However, I need to have a rough idea on
timing for the release and would need some lead time. Mirage OS 2.0 is
planned to release at OSCON.
When do you think we will release XAPI 2.0 and will there be some clear
downloading instructions (e.g. a tarball that can be uploaded via
http://xenproject.org/downloads.html). It would also be a great
opportunity to clean up some of the past XCP confusion. There has been a
community decision to move historical XCP binaries to xenserver.org and
I am waiting for Russell and Tim to do the actual migration (I believe
pretty much everything is prepared). If we could do this before the XAPI
release that would be great.

Best Regards
Lars

Our of curiosity, will the package still be called xcp-xapi? I'm working on some XenProject hypervisor vs XenServer stuff in CloudStack and want to make certain I doc things correctly.


Sent via the Samsung GALAXY S®4, an AT&T 4G LTE smartphone


-------- Original message --------
From: Russell Pavlicek
Date:06/18/2014 14:41 (GMT+01:00)
To: lars.kurth@xen.org,Jon Ludlam ,xen-api@lists.xen.org,Tim Mackey ,Sarah Conway
Subject: RE: [Xen-API] Xapi Project 2.0 release

Lars,

Tim has turned on the XenServer wiki and it is populated with the old XCP wiki pages. I still have to do XCP cleanup on the Xen Project wiki, tentatively planned for next week.

Don't delay any plans on our account; it will be cleaned up very soon.

Russ Pavlicek
Xen Project Evangelist, Citrix Systems
Home Office: +1-301-829-5327
Mobile: +1-240-397-0199
UK VoIP: +44 1223 852 894
________________________________________
From: Lars Kurth [lars.kurth.xen@gmail.com] on behalf of Lars Kurth [lars.kurth@xen.org]
Sent: Wednesday, June 18, 2014 7:32 AM
To: Jon Ludlam; xen-api@lists.xen.org; Tim Mackey; Russell Pavlicek; Sarah Conway
Subject: Re: [Xen-API] Xapi Project 2.0 release

On 03/06/2014 14:26, Jon Ludlam wrote:
> On 03/06/14 11:19, Lars Kurth wrote:
>> Jon,
>> you should consider a blog post on blog.xenproject.org for increased
>> visibility
>> Lars
>>
> Good idea. I'll start drafting...
>
> Jon
Jon,

we brought this up at the Advisory Board meeting. If you want you can
have a press release also. However, I need to have a rough idea on
timing for the release and would need some lead time. Mirage OS 2.0 is
planned to release at OSCON.
When do you think we will release XAPI 2.0 and will there be some clear
downloading instructions (e.g. a tarball that can be uploaded via
http://xenproject.org/downloads.html). It would also be a great
opportunity to clean up some of the past XCP confusion. There has been a
community decision to move historical XCP binaries to xenserver.org and
I am waiting for Russell and Tim to do the actual migration (I believe
pretty much everything is prepared). If we could do this before the XAPI
release that would be great.

Best Regards
Lars

I'm fairly certain we'll be dropping the XCP prefix wherever possible.

Jon

On 19/06/14 10:42, Tim Mackey wrote:
> Our of curiosity, will the package still be called xcp-xapi? I'm
> working on some XenProject hypervisor vs XenServer stuff in CloudStack
> and want to make certain I doc things correctly.
>
>
> Sent via the Samsung GALAXY S®4, an AT&T 4G LTE smartphone
>
>
> -------- Original message --------
> From: Russell Pavlicek
> Date:06/18/2014 14:41 (GMT+01:00)
> To: lars.kurth@xen.org,Jon Ludlam ,xen-api@lists.xen.org,Tim Mackey
> ,Sarah Conway
> Subject: RE: [Xen-API] Xapi Project 2.0 release
>
> Lars,
>
> Tim has turned on the XenServer wiki and it is populated with the old
> XCP wiki pages. I still have to do XCP cleanup on the Xen Project
> wiki, tentatively planned for next week.
>
> Don't delay any plans on our account; it will be cleaned up very soon.
>
> Russ Pavlicek
> Xen Project Evangelist, Citrix Systems
> Home Office: +1-301-829-5327
> Mobile: +1-240-397-0199
> UK VoIP: +44 1223 852 894
> ________________________________________
> From: Lars Kurth [lars.kurth.xen@gmail.com] on behalf of Lars Kurth
> [lars.kurth@xen.org]
> Sent: Wednesday, June 18, 2014 7:32 AM
> To: Jon Ludlam; xen-api@lists.xen.org; Tim Mackey; Russell Pavlicek;
> Sarah Conway
> Subject: Re: [Xen-API] Xapi Project 2.0 release
>
> On 03/06/2014 14:26, Jon Ludlam wrote:
> > On 03/06/14 11:19, Lars Kurth wrote:
> >> Jon,
> >> you should consider a blog post on blog.xenproject.org for increased
> >> visibility
> >> Lars
> >>
> > Good idea. I'll start drafting...
> >
> > Jon
> Jon,
>
> we brought this up at the Advisory Board meeting. If you want you can
> have a press release also. However, I need to have a rough idea on
> timing for the release and would need some lead time. Mirage OS 2.0 is
> planned to release at OSCON.
> When do you think we will release XAPI 2.0 and will there be some clear
> downloading instructions (e.g. a tarball that can be uploaded via
> http://xenproject.org/downloads.html). It would also be a great
> opportunity to clean up some of the past XCP confusion. There has been a
> community decision to move historical XCP binaries to xenserver.org and
> I am waiting for Russell and Tim to do the actual migration (I believe
> pretty much everything is prepared). If we could do this before the XAPI
> release that would be great.
>
> Best Regards
> Lars
>
>
> _______________________________________________
> Xen-api mailing list
> Xen-api@lists.xen.org
> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api