Mailing List Archive

planned Phabricator upgrade tomorrow
Tomorrow we have planned a major upgrade of the Phabricator prod server.

Starting from around 00:00 UTC (after Evening SWAT) we want to switch from
phab1001 to phab1003 as the production server.

This will involve:

- switching from jessie to stretch
- switching from mod_php to php-fpm
- switching from PHP 5 to PHP 7.2
- upgrading apache from 2.4.10 to 2.4.25

We are also hoping it will finally fix the httpd memory leaks we have been
seeing on the current server for a long time and which require periodic
service restarts.

In case we run into any unexpected issue we will simply fall back to
phab1001
and keep running as before.

https://phabricator.wikimedia.org/T221389
https://phabricator.wikimedia.org/T182832
https://phabricator.wikimedia.org/T190568
https://phabricator.wikimedia.org/T151070

Cheers,

Daniel and Mukunda

--
Daniel Zahn <dzahn@wikimedia.org>
Operations Engineer
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: planned Phabricator upgrade tomorrow [ In reply to ]
Hi, thanks for the notice. Can you specify whether "tomorrow" refers to 22
May UTC or 23 May UTC, and how long the maintenance window is likely to be?
Some of us may want to read or write a Phabricator
<strike>complaint</strike> task during the maintenance window, and if we
know when to expect Phabricator may be temporarily unavailable then we can
plan accordingly.

Pine

( https://meta.wikimedia.org/wiki/User:Pine )





On Tue, May 21, 2019, 17:52 Daniel Zahn <dzahn@wikimedia.org> wrote:

> Tomorrow we have planned a major upgrade of the Phabricator prod server.
>
> Starting from around 00:00 UTC (after Evening SWAT) we want to switch from
> phab1001 to phab1003 as the production server.
>
> This will involve:
>
> - switching from jessie to stretch
> - switching from mod_php to php-fpm
> - switching from PHP 5 to PHP 7.2
> - upgrading apache from 2.4.10 to 2.4.25
>
> We are also hoping it will finally fix the httpd memory leaks we have been
> seeing on the current server for a long time and which require periodic
> service restarts.
>
> In case we run into any unexpected issue we will simply fall back to
> phab1001
> and keep running as before.
>
> https://phabricator.wikimedia.org/T221389
> https://phabricator.wikimedia.org/T182832
> https://phabricator.wikimedia.org/T190568
> https://phabricator.wikimedia.org/T151070
>
> Cheers,
>
> Daniel and Mukunda
>
> --
> Daniel Zahn <dzahn@wikimedia.org>
> Operations Engineer
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: planned Phabricator upgrade tomorrow [ In reply to ]
To answer the first part of your question, Pine, Daniel's email was sent at
01:52 UTC on the 22nd. So tomorrow would refer to 00:00 on the 23rd.

-- Lewis Cawte (Lcawte)

On Wed, 22 May 2019 at 07:26, Pine W <wiki.pine@gmail.com> wrote:

> Hi, thanks for the notice. Can you specify whether "tomorrow" refers to 22
> May UTC or 23 May UTC, and how long the maintenance window is likely to be?
> Some of us may want to read or write a Phabricator
> <strike>complaint</strike> task during the maintenance window, and if we
> know when to expect Phabricator may be temporarily unavailable then we can
> plan accordingly.
>
> Pine
>
> ( https://meta.wikimedia.org/wiki/User:Pine )
>
>
>
>
>
> On Tue, May 21, 2019, 17:52 Daniel Zahn <dzahn@wikimedia.org> wrote:
>
> > Tomorrow we have planned a major upgrade of the Phabricator prod server.
> >
> > Starting from around 00:00 UTC (after Evening SWAT) we want to switch
> from
> > phab1001 to phab1003 as the production server.
> >
> > This will involve:
> >
> > - switching from jessie to stretch
> > - switching from mod_php to php-fpm
> > - switching from PHP 5 to PHP 7.2
> > - upgrading apache from 2.4.10 to 2.4.25
> >
> > We are also hoping it will finally fix the httpd memory leaks we have
> been
> > seeing on the current server for a long time and which require periodic
> > service restarts.
> >
> > In case we run into any unexpected issue we will simply fall back to
> > phab1001
> > and keep running as before.
> >
> > https://phabricator.wikimedia.org/T221389
> > https://phabricator.wikimedia.org/T182832
> > https://phabricator.wikimedia.org/T190568
> > https://phabricator.wikimedia.org/T151070
> >
> > Cheers,
> >
> > Daniel and Mukunda
> >
> > --
> > Daniel Zahn <dzahn@wikimedia.org>
> > Operations Engineer
> > _______________________________________________
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: planned Phabricator upgrade tomorrow [ In reply to ]
Indeed, routine Phabricator maintenance is scheduled to take place every
Wednesday at 0:00 UTC.
This maintenance window is not always utilized as we only make changes once
or twice each month.
When there is a maintenance task, typical downtime is less than 5 minutes.
When nothing needs to be done for a given week,
I try to at least !log the absence of any maintenance. Realistically though
I often forget to do that.

This week the planned work involves a lot of changes so expect a few
minutes of downtime while we switch everything over.
Then we will take a few more minutes to validate that the new system is
working as expected, if not we will roll back to the old one.

Optimistically I'd guess that the whole process takes 10 to 15 minutes with
downtime closer to 5 minutes.

On Wed, May 22, 2019 at 2:02 AM Lewis Cawte via Wikitech-l <
wikitech-l@lists.wikimedia.org> wrote:

> To answer the first part of your question, Pine, Daniel's email was sent at
> 01:52 UTC on the 22nd. So tomorrow would refer to 00:00 on the 23rd.
>
> -- Lewis Cawte (Lcawte)
>
> On Wed, 22 May 2019 at 07:26, Pine W <wiki.pine@gmail.com> wrote:
>
> > Hi, thanks for the notice. Can you specify whether "tomorrow" refers to
> 22
> > May UTC or 23 May UTC, and how long the maintenance window is likely to
> be?
> > Some of us may want to read or write a Phabricator
> > <strike>complaint</strike> task during the maintenance window, and if we
> > know when to expect Phabricator may be temporarily unavailable then we
> can
> > plan accordingly.
> >
> > Pine
> >
> > ( https://meta.wikimedia.org/wiki/User:Pine )
> >
> >
> >
> >
> >
> > On Tue, May 21, 2019, 17:52 Daniel Zahn <dzahn@wikimedia.org> wrote:
> >
> > > Tomorrow we have planned a major upgrade of the Phabricator prod
> server.
> > >
> > > Starting from around 00:00 UTC (after Evening SWAT) we want to switch
> > from
> > > phab1001 to phab1003 as the production server.
> > >
> > > This will involve:
> > >
> > > - switching from jessie to stretch
> > > - switching from mod_php to php-fpm
> > > - switching from PHP 5 to PHP 7.2
> > > - upgrading apache from 2.4.10 to 2.4.25
> > >
> > > We are also hoping it will finally fix the httpd memory leaks we have
> > been
> > > seeing on the current server for a long time and which require periodic
> > > service restarts.
> > >
> > > In case we run into any unexpected issue we will simply fall back to
> > > phab1001
> > > and keep running as before.
> > >
> > > https://phabricator.wikimedia.org/T221389
> > > https://phabricator.wikimedia.org/T182832
> > > https://phabricator.wikimedia.org/T190568
> > > https://phabricator.wikimedia.org/T151070
> > >
> > > Cheers,
> > >
> > > Daniel and Mukunda
> > >
> > > --
> > > Daniel Zahn <dzahn@wikimedia.org>
> > > Operations Engineer
> > > _______________________________________________
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > _______________________________________________
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: planned Phabricator upgrade tomorrow [ In reply to ]
OK, thanks for the info.

Pine
( https://meta.wikimedia.org/wiki/User:Pine )

On Wed, May 22, 2019, 03:01 Mukunda Modell <mmodell@wikimedia.org> wrote:

> Indeed, routine Phabricator maintenance is scheduled to take place every
> Wednesday at 0:00 UTC.
> This maintenance window is not always utilized as we only make changes once
> or twice each month.
> When there is a maintenance task, typical downtime is less than 5 minutes.
> When nothing needs to be done for a given week,
> I try to at least !log the absence of any maintenance. Realistically though
> I often forget to do that.
>
> This week the planned work involves a lot of changes so expect a few
> minutes of downtime while we switch everything over.
> Then we will take a few more minutes to validate that the new system is
> working as expected, if not we will roll back to the old one.
>
> Optimistically I'd guess that the whole process takes 10 to 15 minutes with
> downtime closer to 5 minutes.
>
> On Wed, May 22, 2019 at 2:02 AM Lewis Cawte via Wikitech-l <
> wikitech-l@lists.wikimedia.org> wrote:
>
> > To answer the first part of your question, Pine, Daniel's email was sent
> at
> > 01:52 UTC on the 22nd. So tomorrow would refer to 00:00 on the 23rd.
> >
> > -- Lewis Cawte (Lcawte)
> >
> > On Wed, 22 May 2019 at 07:26, Pine W <wiki.pine@gmail.com> wrote:
> >
> > > Hi, thanks for the notice. Can you specify whether "tomorrow" refers to
> > 22
> > > May UTC or 23 May UTC, and how long the maintenance window is likely to
> > be?
> > > Some of us may want to read or write a Phabricator
> > > <strike>complaint</strike> task during the maintenance window, and if
> we
> > > know when to expect Phabricator may be temporarily unavailable then we
> > can
> > > plan accordingly.
> > >
> > > Pine
> > >
> > > ( https://meta.wikimedia.org/wiki/User:Pine )
> > >
> > >
> > >
> > >
> > >
> > > On Tue, May 21, 2019, 17:52 Daniel Zahn <dzahn@wikimedia.org> wrote:
> > >
> > > > Tomorrow we have planned a major upgrade of the Phabricator prod
> > server.
> > > >
> > > > Starting from around 00:00 UTC (after Evening SWAT) we want to switch
> > > from
> > > > phab1001 to phab1003 as the production server.
> > > >
> > > > This will involve:
> > > >
> > > > - switching from jessie to stretch
> > > > - switching from mod_php to php-fpm
> > > > - switching from PHP 5 to PHP 7.2
> > > > - upgrading apache from 2.4.10 to 2.4.25
> > > >
> > > > We are also hoping it will finally fix the httpd memory leaks we have
> > > been
> > > > seeing on the current server for a long time and which require
> periodic
> > > > service restarts.
> > > >
> > > > In case we run into any unexpected issue we will simply fall back to
> > > > phab1001
> > > > and keep running as before.
> > > >
> > > > https://phabricator.wikimedia.org/T221389
> > > > https://phabricator.wikimedia.org/T182832
> > > > https://phabricator.wikimedia.org/T190568
> > > > https://phabricator.wikimedia.org/T151070
> > > >
> > > > Cheers,
> > > >
> > > > Daniel and Mukunda
> > > >
> > > > --
> > > > Daniel Zahn <dzahn@wikimedia.org>
> > > > Operations Engineer
> > > > _______________________________________________
> > > > Wikitech-l mailing list
> > > > Wikitech-l@lists.wikimedia.org
> > > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > > _______________________________________________
> > > Wikitech-l mailing list
> > > Wikitech-l@lists.wikimedia.org
> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> > _______________________________________________
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: planned Phabricator upgrade tomorrow [ In reply to ]
It took a bit longer than expected but this is now done and we are
read-write again and serving Phabricator
from the new server phab1003.
_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l