Mailing List Archive: [openstack-ansible] How to manage system upgrades ?

[openstack-ansible] How to manage system upgrades ?

Jul 27, 2018, 1:34 AM

Post #1 of 5 (637 views)

Hello !

Would be great to have a playbook to upgrade system parts of an
OpenStack Cloud !
With OpenStack Ansible : LXC containers and hosts.

It would be awesome to do a controlled rolling reboot of hosts when
needed...

Different conditions to check :
- for controllers : check galera status...
- for compute nodes : disable compute node and live-evacuate
instances...
- for storage : with Ceph : set no out...

I know, I can do it and contribute, but perhaps someone already has
something similar ?
It could be hosted in one of these projects :
- https://git.openstack.org/cgit/openstack/openstack-ansible-ops
-
https://git.openstack.org/cgit/openstack/ansible-role-openstack-operations

(Why already to project for the same goal ?)

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-ansible] How to manage system upgrades ? [ In reply to ]

christian.zunker at codecentric

Jul 27, 2018, 4:58 AM

Post #2 of 5 (637 views)

Permalink

Hi Gilles,

sounds like a good idea. We've just written a script for live evacuate,
which we can contribute after some refactoring.

Gilles Mocellin <gilles.mocellin@nuagelibre.org> schrieb am Fr., 27. Juli
2018 um 10:44 Uhr:

> Hello !
>
> Would be great to have a playbook to upgrade system parts of an
> OpenStack Cloud !
> With OpenStack Ansible : LXC containers and hosts.
>
> It would be awesome to do a controlled rolling reboot of hosts when
> needed...
>
> Different conditions to check :
> - for controllers : check galera status...
> - for compute nodes : disable compute node and live-evacuate
> instances...
> - for storage : with Ceph : set no out...
>
> I know, I can do it and contribute, but perhaps someone already has
> something similar ?
> It could be hosted in one of these projects :
> - https://git.openstack.org/cgit/openstack/openstack-ansible-ops
> -
> https://git.openstack.org/cgit/openstack/ansible-role-openstack-operations
>
> (Why already to project for the same goal ?)
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

Re: [openstack-ansible] How to manage system upgrades ? [ In reply to ]

mriedemos at gmail

Jul 30, 2018, 8:52 AM

Post #3 of 5 (634 views)

Permalink

On 7/27/2018 3:34 AM, Gilles Mocellin wrote:
> - for compute nodes : disable compute node and live-evacuate instances...

To be clear, what do you mean exactly by "live-evacuate"? I assume you
mean live migration of all instances off each (disabled) compute node
*before* you upgrade it. I wanted to ask because "evacuate" as a server
operation is something else entirely (it's rebuild on another host which
is definitely disruptive to the workload on that server).

http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/

--

Thanks,

Matt

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-ansible] How to manage system upgrades ? [ In reply to ]

christian.zunker at codecentric

Aug 1, 2018, 12:58 AM

Post #4 of 5 (633 views)

Permalink

Hi Matt,

you are right. That's at least what I understand under 'live-evacuate'. The
compute is still working, all VMs on it are running.
Our VMs can all be live-migrated. We wrote a script to live-migrate all VMs
to other computes, to do hardware maintenance on the evacuated
compute server.

I know, this is not going to work for every setup, but could be starting
point, for what Gilles described.

Matt Riedemann <mriedemos@gmail.com> schrieb am Mo., 30. Juli 2018 um
20:20 Uhr:

> On 7/27/2018 3:34 AM, Gilles Mocellin wrote:
> > - for compute nodes : disable compute node and live-evacuate instances...
>
> To be clear, what do you mean exactly by "live-evacuate"? I assume you
> mean live migration of all instances off each (disabled) compute node
> *before* you upgrade it. I wanted to ask because "evacuate" as a server
> operation is something else entirely (it's rebuild on another host which
> is definitely disruptive to the workload on that server).
>
>
> http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/
>
> --
>
> Thanks,
>
> Matt
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

Re: [openstack-ansible] How to manage system upgrades ? [ In reply to ]

gilles.mocellin at nuagelibre

Aug 5, 2018, 12:27 AM

Post #5 of 5 (629 views)

Permalink

Le lundi 30 juillet, Matt Riedemann écrivit :

> On 7/27/2018 3:34 AM, Gilles Mocellin wrote:
> > - for compute nodes : disable compute node and live-evacuate instances...
>
> To be clear, what do you mean exactly by "live-evacuate"? I assume you mean
> live migration of all instances off each (disabled) compute node *before*
> you upgrade it. I wanted to ask because "evacuate" as a server operation is
> something else entirely (it's rebuild on another host which is definitely
> disruptive to the workload on that server).
>
> http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/
>
> --
>
> Thanks,
>
> Matt

[Sorry for the delay, my mail was blocked]

Ah yes, I know the difference. Of course, I mean live migrate all instances on
the node before disabling it, and upgrade / reboot.
So, the following nova command match :

$ nova host-evacuate-live

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators