Mailing List Archive

How we deploy code
Last week, I spoke to a few of my Wikimedia Foundation colleagues about how
we deploy code—I completely botched it.

At the end of the conversation, I was pretty sure I'd only succeeded in
making a complex process more opaque. I decided to write a blog to redeem
myself: How We Deploy Code
<https://phabricator.wikimedia.org/phame/post/view/253/how_we_deploy_code/>

My goal was to write a very high-level overview of the process we use to
deploy code to Wikimedia production.

Hopefully, this is helpful.

<3
– Tyler Cipriani (he/him)
Engineering Manager, Release Engineering
Wikimedia Foundation
Re: How we deploy code [ In reply to ]
This is great, thanks Tyler! \o/

On Mon, Sep 27, 2021 at 3:25 PM Tyler Cipriani <tcipriani@wikimedia.org>
wrote:

> Last week, I spoke to a few of my Wikimedia Foundation colleagues about
> how we deploy code—I completely botched it.
>
> At the end of the conversation, I was pretty sure I'd only succeeded in
> making a complex process more opaque. I decided to write a blog to redeem
> myself: How We Deploy Code
> <https://phabricator.wikimedia.org/phame/post/view/253/how_we_deploy_code/>
>
> My goal was to write a very high-level overview of the process we use to
> deploy code to Wikimedia production.
>
> Hopefully, this is helpful.
>
> <3
> – Tyler Cipriani (he/him)
> Engineering Manager, Release Engineering
> Wikimedia Foundation
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



--
| Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E |
| Dir. Engineering Productivity A18D 1138 8E47 FAC8 1C7D |
Re: How we deploy code [ In reply to ]
So succinct! Great read.

It reads like the Inverted pyramid[1]:

- The gist
- Important details
- The hairy details that you couldn't understand without the gist and
important details


1. https://en.wikipedia.org/wiki/Inverted_pyramid_(journalism)

On Mon, Sep 27, 2021 at 4:04 PM Greg Grossmeier <greg@wikimedia.org> wrote:

> This is great, thanks Tyler! \o/
>
> On Mon, Sep 27, 2021 at 3:25 PM Tyler Cipriani <tcipriani@wikimedia.org>
> wrote:
>
>> Last week, I spoke to a few of my Wikimedia Foundation colleagues about
>> how we deploy code—I completely botched it.
>>
>> At the end of the conversation, I was pretty sure I'd only succeeded in
>> making a complex process more opaque. I decided to write a blog to redeem
>> myself: How We Deploy Code
>> <https://phabricator.wikimedia.org/phame/post/view/253/how_we_deploy_code/>
>>
>> My goal was to write a very high-level overview of the process we use to
>> deploy code to Wikimedia production.
>>
>> Hopefully, this is helpful.
>>
>> <3
>> – Tyler Cipriani (he/him)
>> Engineering Manager, Release Engineering
>> Wikimedia Foundation
>> _______________________________________________
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
>
>
> --
> | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E |
> | Dir. Engineering Productivity A18D 1138 8E47 FAC8 1C7D |
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Re: How we deploy code [ In reply to ]
Great, really nice! I like. <3

I wonder if it makes sense to sneak in the aspect of canary servers.
Whenever I talk to other devs about our deployment processes they are most
amazed by that concept.

Am Di., 28. Sept. 2021 um 01:13 Uhr schrieb Aaron Halfaker <
aaron.halfaker@gmail.com>:

> So succinct! Great read.
>
> It reads like the Inverted pyramid[1]:
>
> - The gist
> - Important details
> - The hairy details that you couldn't understand without the gist and
> important details
>
>
> 1. https://en.wikipedia.org/wiki/Inverted_pyramid_(journalism)
>
> On Mon, Sep 27, 2021 at 4:04 PM Greg Grossmeier <greg@wikimedia.org>
> wrote:
>
>> This is great, thanks Tyler! \o/
>>
>> On Mon, Sep 27, 2021 at 3:25 PM Tyler Cipriani <tcipriani@wikimedia.org>
>> wrote:
>>
>>> Last week, I spoke to a few of my Wikimedia Foundation colleagues about
>>> how we deploy code—I completely botched it.
>>>
>>> At the end of the conversation, I was pretty sure I'd only succeeded in
>>> making a complex process more opaque. I decided to write a blog to redeem
>>> myself: How We Deploy Code
>>> <https://phabricator.wikimedia.org/phame/post/view/253/how_we_deploy_code/>
>>>
>>> My goal was to write a very high-level overview of the process we use to
>>> deploy code to Wikimedia production.
>>>
>>> Hopefully, this is helpful.
>>>
>>> <3
>>> – Tyler Cipriani (he/him)
>>> Engineering Manager, Release Engineering
>>> Wikimedia Foundation
>>> _______________________________________________
>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>
>>
>>
>> --
>> | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E |
>> | Dir. Engineering Productivity A18D 1138 8E47 FAC8 1C7D |
>> _______________________________________________
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



--
Christoph Jauera
Software Developer

Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Phone: +49 (0)30 219 158 26-0
https://wikimedia.de

Imagine a world in which every single human being can freely share in the
sum of all knowledge. Help us to achieve our vision!
https://spenden.wikimedia.de

Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
Re: How we deploy code [ In reply to ]
On Thu, Sep 30, 2021 at 9:38 AM Christoph Jauera <
christoph.jauera@wikimedia.de> wrote:

> I wonder if it makes sense to sneak in the aspect of canary servers.
> Whenever I talk to other devs about our deployment processes they are most
> amazed by that concept.
>

That's a great thought! During deployment training[0] I always forget to
mention how many safeguards there are on deployments. It's (thankfully)
much harder to break production than it used to be.

For future iterations of this series, I think :)

[0]: <https://wikitech.wikimedia.org/wiki/Deployments/Training>

Am Di., 28. Sept. 2021 um 01:13 Uhr schrieb Aaron Halfaker <
> aaron.halfaker@gmail.com>:
>
>> So succinct! Great read.
>>
>> It reads like the Inverted pyramid[1]:
>>
>> - The gist
>> - Important details
>> - The hairy details that you couldn't understand without the gist and
>> important details
>>
>>
>> 1. https://en.wikipedia.org/wiki/Inverted_pyramid_(journalism)
>>
>> On Mon, Sep 27, 2021 at 4:04 PM Greg Grossmeier <greg@wikimedia.org>
>> wrote:
>>
>>> This is great, thanks Tyler! \o/
>>>
>>> On Mon, Sep 27, 2021 at 3:25 PM Tyler Cipriani <tcipriani@wikimedia.org>
>>> wrote:
>>>
>>>> Last week, I spoke to a few of my Wikimedia Foundation colleagues about
>>>> how we deploy code—I completely botched it.
>>>>
>>>> At the end of the conversation, I was pretty sure I'd only
>>>> succeeded in making a complex process more opaque. I decided to write a
>>>> blog to redeem myself: How We Deploy Code
>>>> <https://phabricator.wikimedia.org/phame/post/view/253/how_we_deploy_code/>
>>>>
>>>> My goal was to write a very high-level overview of the process we use
>>>> to deploy code to Wikimedia production.
>>>>
>>>> Hopefully, this is helpful.
>>>>
>>>> <3
>>>> – Tyler Cipriani (he/him)
>>>> Engineering Manager, Release Engineering
>>>> Wikimedia Foundation
>>>> _______________________________________________
>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>
>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>
>>>
>>>
>>> --
>>> | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E |
>>> | Dir. Engineering Productivity A18D 1138 8E47 FAC8 1C7D |
>>> _______________________________________________
>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>
>> _______________________________________________
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
>
>
> --
> Christoph Jauera
> Software Developer
>
> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
> Phone: +49 (0)30 219 158 26-0
> https://wikimedia.de
>
> Imagine a world in which every single human being can freely share in the
> sum of all knowledge. Help us to achieve our vision!
> https://spenden.wikimedia.de
>
> Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V.
> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
> der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
> Körperschaften I Berlin, Steuernummer 27/029/42207.
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Re: How we deploy code [ In reply to ]
Great read! Thanks!

As someone on a team that mostly deploys non-Mediawiki code, I appreciate
the mention that this code also exists and does not follow the same process!

Big thanks!

MrG

On Thu, 30 Sept 2021 at 23:06, Tyler Cipriani <tcipriani@wikimedia.org>
wrote:

> On Thu, Sep 30, 2021 at 9:38 AM Christoph Jauera <
> christoph.jauera@wikimedia.de> wrote:
>
>> I wonder if it makes sense to sneak in the aspect of canary servers.
>> Whenever I talk to other devs about our deployment processes they are most
>> amazed by that concept.
>>
>
> That's a great thought! During deployment training[0] I always forget to
> mention how many safeguards there are on deployments. It's (thankfully)
> much harder to break production than it used to be.
>
> For future iterations of this series, I think :)
>
> [0]: <https://wikitech.wikimedia.org/wiki/Deployments/Training>
>
> Am Di., 28. Sept. 2021 um 01:13 Uhr schrieb Aaron Halfaker <
>> aaron.halfaker@gmail.com>:
>>
>>> So succinct! Great read.
>>>
>>> It reads like the Inverted pyramid[1]:
>>>
>>> - The gist
>>> - Important details
>>> - The hairy details that you couldn't understand without the gist
>>> and important details
>>>
>>>
>>> 1. https://en.wikipedia.org/wiki/Inverted_pyramid_(journalism)
>>>
>>> On Mon, Sep 27, 2021 at 4:04 PM Greg Grossmeier <greg@wikimedia.org>
>>> wrote:
>>>
>>>> This is great, thanks Tyler! \o/
>>>>
>>>> On Mon, Sep 27, 2021 at 3:25 PM Tyler Cipriani <tcipriani@wikimedia.org>
>>>> wrote:
>>>>
>>>>> Last week, I spoke to a few of my Wikimedia Foundation colleagues
>>>>> about how we deploy code—I completely botched it.
>>>>>
>>>>> At the end of the conversation, I was pretty sure I'd only
>>>>> succeeded in making a complex process more opaque. I decided to write a
>>>>> blog to redeem myself: How We Deploy Code
>>>>> <https://phabricator.wikimedia.org/phame/post/view/253/how_we_deploy_code/>
>>>>>
>>>>> My goal was to write a very high-level overview of the process we use
>>>>> to deploy code to Wikimedia production.
>>>>>
>>>>> Hopefully, this is helpful.
>>>>>
>>>>> <3
>>>>> – Tyler Cipriani (he/him)
>>>>> Engineering Manager, Release Engineering
>>>>> Wikimedia Foundation
>>>>> _______________________________________________
>>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>>
>>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>>
>>>>
>>>>
>>>> --
>>>> | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E |
>>>> | Dir. Engineering Productivity A18D 1138 8E47 FAC8 1C7D |
>>>> _______________________________________________
>>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>>
>>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>>
>>> _______________________________________________
>>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>>
>>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>>
>>
>>
>> --
>> Christoph Jauera
>> Software Developer
>>
>> Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
>> Phone: +49 (0)30 219 158 26-0
>> https://wikimedia.de
>>
>> Imagine a world in which every single human being can freely share in the
>> sum of all knowledge. Help us to achieve our vision!
>> https://spenden.wikimedia.de
>>
>> Wikimedia Deutschland – Gesellschaft zur Förderung Freien Wissens e. V.
>> Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
>> der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
>> Körperschaften I Berlin, Steuernummer 27/029/42207.
>> _______________________________________________
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>>
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



--
*Guillaume Lederrey* (he/him)
Engineering Manager
Wikimedia Foundation <https://wikimediafoundation.org/>
Re: How we deploy code [ In reply to ]
Tyler, thanks for the process document, very helpful:

Here is another type of release challenge:
Database table alteration is a whole additional minefield I am dealing
with. Preservation of existing data and migration of existing data into
the added fields, dealing with 3 database engines, ensuring existing code
can function pre and post alteration if needed and ensuring that a feature
needing the table alteration is released in phases. Tack on doing this as
an extension not core code. Interesting dependencies are how 3rd party
deployments will be affected, how replicated databases are affected, how to
test the non manually updated table alteration scripts and the one time
executing data migration code and how the UI will work during the various
release phases. Security risks need assessment, performance of the new code
needs assessment, adding appropriate indexes to reduce database load of new
queries needs assessment.
I think I am close to having figured out the path and consulted with Sam
and Manuel, and others have chimed in such as Alex and Subbu.
Best,
Shannon

On Mon, Sep 27, 2021 at 3:25 PM Tyler Cipriani <tcipriani@wikimedia.org>
wrote:

> Last week, I spoke to a few of my Wikimedia Foundation colleagues about
> how we deploy code—I completely botched it.
>
> At the end of the conversation, I was pretty sure I'd only succeeded in
> making a complex process more opaque. I decided to write a blog to redeem
> myself: How We Deploy Code
> <https://phabricator.wikimedia.org/phame/post/view/253/how_we_deploy_code/>
>
> My goal was to write a very high-level overview of the process we use to
> deploy code to Wikimedia production.
>
> Hopefully, this is helpful.
>
> <3
> – Tyler Cipriani (he/him)
> Engineering Manager, Release Engineering
> Wikimedia Foundation
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/