Mailing List Archive

Why does the train start on Tuesday?
Hi all

A few questions to provoke discussion/share knowledge better:
* Why does the train run Tue,Wed, Thur rather than Mon,Tue,Wed
* Why do we only have 2 group 1 Wikipedia's (Catalan and Hebrew)
* Should there be a backport window Friday mornings for certain changes?

Longer spiel:

A few weeks ago a change I made led to a small but noticeable UI
regression. The site was perfectly usable, but looked noticeably off. It
was in a more obscure part of the UI so we missed it during QA/code review.

Late Wednesday a ticket was reported against Wikimedia commons, but I only
became aware of it late Thursday when the regression rolled out to English
Wikipedia. A village pump discussion was started and several duplicate
tickets were created. While the site could still be used it didn't look
great and upset the experience of many editors.

Once aware of the problem, the issue was easy to fix. A patch was written
on Friday.

I understand Friday backports are possible, but my team tend to use them as
a last resort in fear of creating more work for my fellow maintainers over
weekend periods. As a result, given the site was still usable, the fix
wasn't backported until the first available backport window on Monday. This
is unfortunately a regular pattern, particularly for small UI regressions.

We addressed the issue on Monday, but I got feedback from several users
that this particular issue took too long to get backported. I mentioned the
no Friday deploy policy. One user asked me why we don't run the train
Monday-Wednesday and to be honest I wasn't sure. I couldn't find anything
on https://wikitech.wikimedia.org/wiki/Deployments/Train.

My team tries to avoid big changes on Mondays as Monday merged patches are
more likely to have issues since they don't always get the time to go
through QA during the week by our dedicated QA engineer.

So... Why don't we run the train Monday-Wednesday? Having a Thursday buffer
during which we can more comfortably backport any issues not caught in
testing, particularly UI bugs would be extremely helpful to my team and I
don't think we'd lose much by losing the Monday to rush last-minute changes.

Assuming there are good reasons for Tuesday-Thursday train, I think there
is another problem with our deploy process which is the size of group 1.
Given the complexity of our interfaces (several skins, gadgets, multiple
special pages, user preferences, gadgets, multiple extensions, and
different user rights), generally, many obscure UI bugs get missed in QA by
people who don't use the software every day and have a clear mental model
of how it looks and behaves. My team mostly works on visible user interface
changes and we rely heavily on Catalan and Hebrew Wikipedia users - our
only group 1 wikis to notice errors with UI before they go out to a wider
audience. Given the size of those audiences, that often doesn't work, and
it's often group 2 wikis that make us aware of issues. If we are going to
keep the existing train of Tue-Thur, I think it's essential we have at
least one larger Wikipedia in our group 1 deploy to give us better
protection against UI regressions living over the weekend. My understanding
is for some reason this is not a decision release engineering can make, but
one that requires an on-wiki RFC by the editors themselves. Is that
correct? While I can understand the reluctance of editors to experience
bugs, I'd argue that it's better to have a bug for a day than to have it
for an entire weekend, and definitely something we need to think more
deeply about.
Re: Why does the train start on Tuesday? [ In reply to ]
??????? ??? ??, 22 ????? 2021 ?-23:04 ??? ?Jon Robson?? <?
jrobson@wikimedia.org??>:?

> Hi all
>
> A few questions to provoke discussion/share knowledge better:
> * Why does the train run Tue,Wed, Thur rather than Mon,Tue,Wed
> * Why do we only have 2 group 1 Wikipedia's (Catalan and Hebrew)
>

It was my idea, kind of.

Earlier, all Wikipedias were on Thursday. Wikipedias are a bit different
because they have some extensions other sites don't, such as Content
Translation, which interests me, and some others. It happened a few times
that a bad change was rolled out to Wikipedias on Thursday and it was
difficult to roll it back until the next week. Moving all Wikipedias or
just English a day earlier looked risky, so I proposed moving just two
Wikipedias that are fairly active and have a community that is known to be
friendly to early adoption and bug reporting. Everyone in these Wikipedias
and in Ops thought it was a good idea and it was done.

I have a vague recollection that it helped catch a few bugs early on
Wednesday and get them fixed before the Thursday train, although I don't
remember the details.

This is not written in stone, and I don't think anyone in these two
Wikipedias will strongly object to changing some things, although if anyone
seriously wants to change it, it should be brought up on Village pumps
there.
Re: Why does the train start on Tuesday? [ In reply to ]
I'm not a train conductor, and I don't know anything about the
decision-making process that went into the current train schedule. But I'll
note that Mondays tend to be observed as holidays, which could cause
complications for train scheduling. By my quick count from officewiki,
there are 9 US holidays on Mondays in 2021 (not counting the end-of-year
holiday).

I've been on teams that switched sprint start dates from Monday to Tuesday
because we found the number of Monday holidays disruptive to our cadence.

Bill Pirkle
Software Engineer
www.wikimediafoundation.org


On Tue, Jun 22, 2021 at 3:04 PM Jon Robson <jrobson@wikimedia.org> wrote:

> Hi all
>
> A few questions to provoke discussion/share knowledge better:
> * Why does the train run Tue,Wed, Thur rather than Mon,Tue,Wed
> * Why do we only have 2 group 1 Wikipedia's (Catalan and Hebrew)
> * Should there be a backport window Friday mornings for certain changes?
>
> Longer spiel:
>
> A few weeks ago a change I made led to a small but noticeable UI
> regression. The site was perfectly usable, but looked noticeably off. It
> was in a more obscure part of the UI so we missed it during QA/code review.
>
> Late Wednesday a ticket was reported against Wikimedia commons, but I only
> became aware of it late Thursday when the regression rolled out to English
> Wikipedia. A village pump discussion was started and several duplicate
> tickets were created. While the site could still be used it didn't look
> great and upset the experience of many editors.
>
> Once aware of the problem, the issue was easy to fix. A patch was written
> on Friday.
>
> I understand Friday backports are possible, but my team tend to use them
> as a last resort in fear of creating more work for my fellow maintainers
> over weekend periods. As a result, given the site was still usable, the fix
> wasn't backported until the first available backport window on Monday. This
> is unfortunately a regular pattern, particularly for small UI regressions.
>
> We addressed the issue on Monday, but I got feedback from several users
> that this particular issue took too long to get backported. I mentioned the
> no Friday deploy policy. One user asked me why we don't run the train
> Monday-Wednesday and to be honest I wasn't sure. I couldn't find anything
> on https://wikitech.wikimedia.org/wiki/Deployments/Train.
>
> My team tries to avoid big changes on Mondays as Monday merged patches are
> more likely to have issues since they don't always get the time to go
> through QA during the week by our dedicated QA engineer.
>
> So... Why don't we run the train Monday-Wednesday? Having a Thursday
> buffer during which we can more comfortably backport any issues not caught
> in testing, particularly UI bugs would be extremely helpful to my team and
> I don't think we'd lose much by losing the Monday to rush last-minute
> changes.
>
> Assuming there are good reasons for Tuesday-Thursday train, I think there
> is another problem with our deploy process which is the size of group 1.
> Given the complexity of our interfaces (several skins, gadgets, multiple
> special pages, user preferences, gadgets, multiple extensions, and
> different user rights), generally, many obscure UI bugs get missed in QA by
> people who don't use the software every day and have a clear mental model
> of how it looks and behaves. My team mostly works on visible user interface
> changes and we rely heavily on Catalan and Hebrew Wikipedia users - our
> only group 1 wikis to notice errors with UI before they go out to a wider
> audience. Given the size of those audiences, that often doesn't work, and
> it's often group 2 wikis that make us aware of issues. If we are going to
> keep the existing train of Tue-Thur, I think it's essential we have at
> least one larger Wikipedia in our group 1 deploy to give us better
> protection against UI regressions living over the weekend. My understanding
> is for some reason this is not a decision release engineering can make, but
> one that requires an on-wiki RFC by the editors themselves. Is that
> correct? While I can understand the reluctance of editors to experience
> bugs, I'd argue that it's better to have a bug for a day than to have it
> for an entire weekend, and definitely something we need to think more
> deeply about.
>
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Re: Why does the train start on Tuesday? [ In reply to ]
On Tue, Jun 22, 2021 at 3:03 PM Jon Robson <jrobson@wikimedia.org> wrote:

> A few questions to provoke discussion/share knowledge better:
> * Why does the train run Tue,Wed, Thur rather than Mon,Tue,Wed
>

I'd note here that the standard security deployment window is Monday
between 21:00 and 23:00 UTC. That date and time is not a hard requirement
by any means, but having such a window exist early in the week, prior to
the start of the train, has worked out well for a few reasons. It's both
convenient and less risky to only deploy security patches to a single wmf
production branch, which is the case most Mondays. It's also less risky
having the space to monitor patches and roll them back or re-patch during
the week, as opposed to say, on a Friday, with substantially reduced
coverage going into most weekends. Of course there are times when critical
security issues need to be dealt with on a Friday or even over the weekend,
but in general, the Security Team likes to avoid this. Moving the train to
a Mon, Tue, Wed cadence would imply the security window be moved to the
previous Friday or possibly Thursday, which is doable, but not desired for
the aforementioned reasons.

--
Scott Bassett
sbassett@wikimedia.org
Re: Why does the train start on Tuesday? [ In reply to ]
Jon, I think you're misunderstanding the point of the "No Deployment on
Friday" policy.

Let's look from far, Why is a work day a no-deploy day? Why are we limiting
ourselves to a four-day week while we can use our full potential? The
reason is that if we deploy something to production on Friday and if it
breaks production, then there is no one around to fix the issue over the
weekend. So far it's rather obvious.

In other words, the requirement for having a time to deploy changes is to
have some buffer until the weekend *to fix issues caused by changes
deployed*. That buffer is Friday. It's made a no deployment day so you
could push urgent fixes (including train blockers). Of course, the fix
should not be too big to cause issues later.

Fridays are not "no deployment day" in the same of sense Sunday is a "no
deployment day". It's the buffer to fix UBN issues, not a long weekend. If
you're fixing an UBN issue, then please go ahead and deploy on Friday. The
ultimate goal of RelEng policies is to have major issues live in production
for the shortest possible time. Refusing to deploy a major fix on a Friday
does the exact opposite of that.

Regarding Catalan and Hebrew Wikipedia, the other Amir said it well, I
don't think I have much to add beside the fact that I have personally seen
them finding major issues before they hit all Wikipedia languages many
times, more than I can count.

HTH

On Tue, Jun 22, 2021 at 11:11 PM Scott Bassett <sbassett@wikimedia.org>
wrote:

> On Tue, Jun 22, 2021 at 3:03 PM Jon Robson <jrobson@wikimedia.org> wrote:
>
>> A few questions to provoke discussion/share knowledge better:
>> * Why does the train run Tue,Wed, Thur rather than Mon,Tue,Wed
>>
>
> I'd note here that the standard security deployment window is Monday
> between 21:00 and 23:00 UTC. That date and time is not a hard requirement
> by any means, but having such a window exist early in the week, prior to
> the start of the train, has worked out well for a few reasons. It's both
> convenient and less risky to only deploy security patches to a single wmf
> production branch, which is the case most Mondays. It's also less risky
> having the space to monitor patches and roll them back or re-patch during
> the week, as opposed to say, on a Friday, with substantially reduced
> coverage going into most weekends. Of course there are times when critical
> security issues need to be dealt with on a Friday or even over the weekend,
> but in general, the Security Team likes to avoid this. Moving the train to
> a Mon, Tue, Wed cadence would imply the security window be moved to the
> previous Friday or possibly Thursday, which is doable, but not desired for
> the aforementioned reasons.
>
> --
> Scott Bassett
> sbassett@wikimedia.org
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/



--
Amir (he/him)
Re: Why does the train start on Tuesday? [ In reply to ]
Thanks for all the input so far.

On Tue, Jun 22, 2021 at 2:41 PM Amir Sarabadani <ladsgroup@gmail.com> wrote:

> Jon, I think you're misunderstanding the point of the "No Deployment on
> Friday" policy.
>

I don't think I'm misunderstanding the policy? I'm talking explicitly about
high priority issues UI regressions, not unbreak now (ie. site is still
functional but styled incorrectly e.g. imagine a link is the wrong color).
I've used Friday deployments historically, but only for really really bad
issues.

To give an example, if an icon is visually overlapping text I don't
consider that an UBN, I consider that unfortunate. If the icon overlap is
on the edit icon and that's not clickable, definitely UBN and I'm happy to
go the extra lengths to get a Friday patch out.

I understand the Friday is a buffer, but it's not a great buffer,
particularly now at the Wikimedia foundation most people observe "Silent
Fridays", and many people in teams that are involved in decision making
work in different timezones. Side note what constitutes a UBN UI regression
is being discussed on the talk page [1].

> Regarding Catalan and Hebrew Wikipedia, the other Amir said it well, I
don't think I have much to add beside the fact that I have personally seen
them finding major issues before they hit all Wikipedia languages many
times, more than I can count.

Apologies if I didn't make myself clear, but it seems I didn't given both
Amir's comments. I am very happy that we have these, and my question was *not
*why do we have them, but rather* why do we only have 2*. I want more of
them and every time I've asked why we don't have more, I've been told it's
a community decision and that seems odd to me.


[1] https://wikitech.wikimedia.org/wiki/Talk:Deployments/Holding_the_train
Re: Why does the train start on Tuesday? [ In reply to ]
> Regarding Catalan and Hebrew Wikipedia, the other Amir said it well, I
don't think I have much to add beside the fact that I have personally seen
them finding major issues before they hit all Wikipedia languages many
times, more than I can count.

>
> Apologies if I didn't make myself clear, but it seems I didn't given both
> Amir's comments. I am very happy that we have these, and my question was *not
> *why do we have them, but rather* why do we only have 2*. I want more of
> them and every time I've asked why we don't have more, I've been told it's
> a community decision and that seems odd to me.
>

Anyone is welcome to correct me if I'm wrong, but I'm quite sure that there
is no community decision to have *only* two. I suggested these two back
then, because I felt that it is *enough*, I can read these two languages
and I thought these two communities would not be opposed to it, and I even
happened to be right. (Hebrew also represents RTL languages well.)

So there is probably no blocker to have more than two. You should probably
ask the relevant wikis similarly to how I did back then, get an agreement
from them and from Rel. Eng. and SRE, and that's it.

How did I ask them? I just raised it myself with my personal account in the
Village pumps, there were a few positive responses, and that was it, it
worked and no one complained. But don't take my word for it :) You can do
the same or maybe you can go through a more formal process. Up to you.
Which languages to pick? - I'm also not sure. Maybe some wikis that
represent special features, like the Growth extensions, or language
converter, etc. Also up to you.
Re: Why does the train start on Tuesday? [ In reply to ]
On Wed, Jun 23, 2021 at 12:10 AM Jon Robson <jrobson@wikimedia.org> wrote:

> I understand the Friday is a buffer, but it's not a great buffer,
> particularly now
>

I don't have any suggestions, as I am not super-familiar with the current
situation, but I can understand (and suffered) from some hidden bugs (not
immediately obvious) taking some time to surface and/or reach the right
people, but I have some questions:

* How often are issues surfaced in the group0 -> group1 vs group1 ->
group2, are there any stats to back the need for a change there?
* Without changing the actual deploying days or the frequency, would there
be any benefit of shifting the deploy over multiple weeks? (random example
Tu: group1->group2, (new branch) We: group0, Th: group0-> group1) or would
that make things worse?
* You mention commons. I am guessing Commons, and Wikidata, to some extent-
are both large sites with a lot of visibility but also very different from
the core features that are similar to most other wikis, but the test
version of those on group0 may not be enough to catch all issues. Is there
something that could be improved specifically for those sites?
* Can we do something to improve the speed from "a user notices an issue
with the site" to "the right team/owner is aware of it and acts on it"?

--
Jaime Crespo
<http://wikimedia.org>
Re: Why does the train start on Tuesday? [ In reply to ]
On Tue, 22 Jun 2021 at 23:10, Jon Robson <jrobson@wikimedia.org> wrote:

> Apologies if I didn't make myself clear, but it seems I didn't given both
> Amir's comments. I am very happy that we have these, and my question was *not
> *why do we have them, but rather* why do we only have 2*. I want more of
> them and every time I've asked why we don't have more, I've been told it's
> a community decision and that seems odd to me.
>

It's a community decision in the sense that it wasn't something the WMF
decided for them, it was those communities decided it for themselves.
They're open to it probably as a result of being generally less
conservative when it comes to changes, and more willing to try things out
early. But, it's not a community decision in the sense that you, as staff,
can't get involved and try to recruit more wikis!

So, if other wikis are asked if they want to be in group 1, understand the
advantages and disadvantages of that, and they said yes, I doubt anyone
would raise any objections. Having more wikis in that group to test things
on early would definitely be a benefit.

Amir's suggestion to contact wikis that have been open to Growth features,
like Czech and Korean (I think?), sounds like a great one, especially if
there are community ambassadors that could help explain what it means to be
in group 1.

Dan
Re: Why does the train start on Tuesday? [ In reply to ]
(Late reply: I was out the week this was sent, then another week of
vacation happened)

On Wed, Jun 23, 2021 at 2:59 AM Jaime Crespo <jcrespo@wikimedia.org> wrote:

> * How often are issues surfaced in the group0 -> group1 vs group1 ->
> group2, are there any stats to back the need for a change there?
>

The closest number I have to issues/group is the count of new "blocker"
tasks filed in phabricator per group.

Progressive rollout to each group gives us more confidence in the code
being deployed, so for each group we should see progressively fewer
blockers.

????????*Group vs blocker discovery over the past 153 trains**:*
[image: README_10_0.png]

- *Before group0*: 233
- *Group0*: 180
- *Group1*: 230
- *Group2*: 91

"Before group0" means that before we've rolled out the train to any wiki in
wikiprod, there's a blocker on the train task (just like today
1.37.0-wmf.14 is not deployed anywhere, but there's a blocker on the train
task: https://phabricator.wikimedia.org/T281155 ).

If we want each group to have progressively fewer blockers for each group
then the data shows that group0 is too small and/or group1 is too big.
There are other considerations. Deployers have a lot of work to do on
group0 day vs group1 day: so making group0 bigger/more useful for
developers makes the lives of deployers harder.

* Without changing the actual deploying days or the frequency, would there
> be any benefit of shifting the deploy over multiple weeks? (random example
> Tu: group1->group2, (new branch) We: group0, Th: group0-> group1) or would
> that make things worse?
>

I wonder what impact this change would have on blocker reports. For
instance, is it a function of the time left in the week that group2
surfaces relatively few blockers?


> * You mention commons. I am guessing Commons, and Wikidata, to some
> extent- are both large sites with a lot of visibility but also very
> different from the core features that are similar to most other wikis, but
> the test version of those on group0 may not be enough to catch all issues.
> Is there something that could be improved specifically for those sites?
>

This is a subset of a question I've been asking folks: why does the train
give us confidence? What does a train give us that a testing environment
like beta or a local environment can't give us? I think some of the magic
of train is the amount of traffic, but if that were the case then
artificial traffic should suffice. I think the other aspect of the train is
Hyrum's law[0]—all observable properties of a system are hammered with
traffic: even observable properties that were not built intentionally.


> * Can we do something to improve the speed from "a user notices an issue
> with the site" to "the right team/owner is aware of it and acts on it"?
>

Or can we do something to improve how many issues users notice? :)

Thanks for all of these great questions.
– Tyler

[0]: <https://www.hyrumslaw.com/>
Re: Why does the train start on Tuesday? [ In reply to ]
On Mon, 12 Jul 2021 at 19:26, Tyler Cipriani <tcipriani@wikimedia.org>
wrote:

> (Late reply: I was out the week this was sent, then another week of
> vacation happened)
>
>
>
>
>> * Can we do something to improve the speed from "a user notices an issue
>> with the site" to "the right team/owner is aware of it and acts on it"?
>>
>
> Or can we do something to improve how many issues users notice? :)
>
>
As someone who's been around for a long time as an editor, I can say
honestly that having most of the issues addressed before they hit the
really big projects has resulted in a huge improvement. The train really
works, and the only challenge I really see is what Jon mentions in his
original post. Some of those issues aren't really that significant in the
great scheme of things, but there's a big leap when something takes two
business days to fix from the Tuesday deployment and two business days to
fix from the Thursday deployment.

It's not always possible for even the best developer and the best testing
systems to catch an issue that will be spotted by a hands-on user, several
of whom are much more familiar with the purpose, expected outcomes and
change impact on extensions than the people who have written them or QA'd
them. That's why there will always be plenty of issues that are identified
by users, and it is in no way a problem that a small number of them
(compared to what we saw 10-15 years ago) get through to the end of the
train before being identified as needing to be addressed (for different
values of "addressed").

Other people on this list are in a better position to understand the
implications of backporting on Fridays, but I think it would be worthwhile
to give serious consideration to expanding the list of what could be
included in a Friday backport. Limiting it to essentially "breaks the site"
or "major impact to accessibility of the content" doesn't really include
most of the noticeable user experience issues (for either editors or
readers), and leaving those sitting for a minimum of half a week is
suboptimal.

Risker/Anne
Re: Why does the train start on Tuesday? [ In reply to ]
This makes me wonder if there could ever be a way to allow some requests to
be on one group and other requests for the same wiki could be another group.

e.g. users could opt-in to being a canary or some users could be randomly
selected.

When would this not work? If a newer version required schema change then
have to stop using old version after applying new schema? but that isn't
very common? any other complications?

-Jeremyb
Re: Why does the train start on Tuesday? [ In reply to ]
You can already opt in to testing undployed code by using the
WikimediaDebug[1] browser extension (available for firefox[2] and chrome[3])

Maybe we should add an option to force a request to be served from the
latest branch instead of the one assigned in wikiversions.json.

[1] https://wikitech.wikimedia.org/wiki/WikimediaDebug
[2] https://addons.mozilla.org/en-US/firefox/addon/wikimedia-debug-header/
[3]
https://chrome.google.com/webstore/detail/wikimediadebug/binmakecefompkjggiklgjenddjoifbb


On Tue, Jul 13, 2021 at 12:21 AM Jeremy Baron <jeremy@tuxmachine.com> wrote:

> This makes me wonder if there could ever be a way to allow some requests
> to be on one group and other requests for the same wiki could be another
> group.
>
> e.g. users could opt-in to being a canary or some users could be randomly
> selected.
>
> When would this not work? If a newer version required schema change then
> have to stop using old version after applying new schema? but that isn't
> very common? any other complications?
>
> -Jeremyb
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Re: Why does the train start on Tuesday? [ In reply to ]
Thanks all for the interesting discussion.
I think the most immediate actionable here is expanding group 1 wikis, so
I'm looking into that here https://phabricator.wikimedia.org/T286664.
Ideally, it's my belief that 2 top ten wikis that are not English would
give us the visibility of problems with UI changes that we need.

Some thoughts that come to mind:
* What if group 0 and 1 were merged? How would that impact the train? From
the data for the last 153 trains it seems those usually capture the
majority of our issues. It wouldn't help capturing issues in group 2, but
it would give more of a buffer to fix them.
* What if group 0 happened after the security window on a Monday? Would
that be too stressful/not feasible for those involved?

For me, regarding backporting on Fridays, ideally, as an engineer I
appreciate a buffer going into the weekend to reduce any anxiety around my
work having incapacitated our editors in some way. Deploying code on
Friday's only increases that anxiety in some cases.



On Tue, Jun 22, 2021 at 3:09 PM Jon Robson <jrobson@wikimedia.org> wrote:

> Thanks for all the input so far.
>
> On Tue, Jun 22, 2021 at 2:41 PM Amir Sarabadani <ladsgroup@gmail.com>
> wrote:
>
>> Jon, I think you're misunderstanding the point of the "No Deployment on
>> Friday" policy.
>>
>
> I don't think I'm misunderstanding the policy? I'm talking explicitly
> about high priority issues UI regressions, not unbreak now (ie. site is
> still functional but styled incorrectly e.g. imagine a link is the wrong
> color). I've used Friday deployments historically, but only for really
> really bad issues.
>
> To give an example, if an icon is visually overlapping text I don't
> consider that an UBN, I consider that unfortunate. If the icon overlap is
> on the edit icon and that's not clickable, definitely UBN and I'm happy to
> go the extra lengths to get a Friday patch out.
>
> I understand the Friday is a buffer, but it's not a great buffer,
> particularly now at the Wikimedia foundation most people observe "Silent
> Fridays", and many people in teams that are involved in decision making
> work in different timezones. Side note what constitutes a UBN UI regression
> is being discussed on the talk page [1].
>
> > Regarding Catalan and Hebrew Wikipedia, the other Amir said it well, I
> don't think I have much to add beside the fact that I have personally seen
> them finding major issues before they hit all Wikipedia languages many
> times, more than I can count.
>
> Apologies if I didn't make myself clear, but it seems I didn't given both
> Amir's comments. I am very happy that we have these, and my question was *not
> *why do we have them, but rather* why do we only have 2*. I want more of
> them and every time I've asked why we don't have more, I've been told it's
> a community decision and that seems odd to me.
>
>
> [1] https://wikitech.wikimedia.org/wiki/Talk:Deployments/Holding_the_train
>
Re: Why does the train start on Tuesday? [ In reply to ]
On Mon, Jul 12, 2021 at 10:47 PM Risker <risker.wp@gmail.com> wrote:

> On Mon, 12 Jul 2021 at 19:26, Tyler Cipriani <tcipriani@wikimedia.org>
> wrote:
>
>> * Can we do something to improve the speed from "a user notices an issue
>>> with the site" to "the right team/owner is aware of it and acts on it"?
>>>
>>
>> Or can we do something to improve how many issues users notice? :)
>>
>>
> As someone who's been around for a long time as an editor, I can say
> honestly that having most of the issues addressed before they hit the
> really big projects has resulted in a huge improvement. The train really
> works, and the only challenge I really see is what Jon mentions in his
> original post. Some of those issues aren't really that significant in the
> great scheme of things, but there's a big leap when something takes two
> business days to fix from the Tuesday deployment and two business days to
> fix from the Thursday deployment.
>
> It's not always possible for even the best developer and the best testing
> systems to catch an issue that will be spotted by a hands-on user, several
> of whom are much more familiar with the purpose, expected outcomes and
> change impact on extensions than the people who have written them or QA'd
> them. That's why there will always be plenty of issues that are identified
> by users, and it is in no way a problem that a small number of them
> (compared to what we saw 10-15 years ago) get through to the end of the
> train before being identified as needing to be addressed (for different
> values of "addressed").
>

Thank you for this response! The train existed before I started thinking
about MediaWiki-software deployment. The impression that it has had a
positive impact on the number of problems seen by users is important
information. Your response is a fantastic answer to a different question
I wonder about a lot: why does the train process give us confidence in the
code being released?

The next part of that question is: are there ways we can gain this
confidence with less disruption? I'd be interested in trying to catalog the
types of problems that are only spotted by hands-on users in the interest
of seeing if patterns emerge.

Thank you again!
– Tyler