How’d we do in our strive for operational excellence last month? Read on to
find out!
Incidents
We've had quite an eventful month, with 8 documented incidents in
September. That's the highest since last year (Feb 2020) and one of the
three worst months of the last five years.
- 2021-09-01 partial Parsoid outage
<https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-01_partial_parsoid_outage>
- Impact: For 9 hours, 10% of Parsoid requests to parse/save pages
were failing on all wikis. Little to no end-user impact apart from minor
due to RESTBase retries.
- 2021-09-04 appserver latency
<https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-04_appserver_latency>
- Impact: For 37 minutes, MW backends were slow with 2% of requests
receiving errors. This affected all wikis through logged-in
users, bots/API
queries, and some page views from unregistered users (e.g. pages
that were
recently edited or expired from CDN cache).
- 2021-09-06 Wikifeeds
<https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-06_Wikifeeds>
- Impact: For 3 days, the Wikifeeds API failed ~1% of requests (e.g.
5 of 500 req/s).
- 2021-09-12 Esams upload
<https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-12-Esams-upload>
- Impact: For 20 minutes, images were unavailable for people in
Europe, affecting all wikis.
- 2021-09-13 CirrusSearch restart
<https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-13_cirrussearch_restart>
- Impact: For ~2 hours, search was unavailable on Wikipedia from all
regions. Search suggestions were missing or slow, and the search results
page errored with "Try again later".
- 2021-09-18 appserver latency
<https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-18_appserver_latency>
- Impact: For ~10 minutes, MW backends were slow or unavailable for
all wikis.
- 2021-09-26 appserver latency
<https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-26_appserver_latency>
- Impact: For ~15 minutes, MW backends were slow or unavailable for
all wikis.
- 2021-09-29 eqiad kubernetes
<https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-29_eqiad-kubernetes>
- Impact: For 2 minutes, MW backends were affected by a Kubernetes
issue (via Kask sessionstore). 1500 edit attempts failed (8% of
POSTs), and
logged-in pageviews were slowed down, often taking several seconds.
Remember to review and schedule Incident Follow-up work
<https://phabricator.wikimedia.org/project/view/4758/> in Phabricator,
which are preventive measures and tech debt mitigations written down after
an incident is concluded.
See also Incident graphs <https://codepen.io/Krinkle/full/wbYMZK>.
Trends
The month of September saw 24 new production error reports of which 11 have
since been resolved, and today, three to six weeks later, 13 remain open
and have thus carried over to the next month. This is about average,
although it makes it no less sad that we continue to introduce (and carry
over) more errors than we rectify in the same time frame.
On the other hand, last month we did have a healthy focus on some of the
older reports. The workboard stood at 301 unresolved errors last month. Of
those, 16 were resolved. With the 13 new errors from September, this
reduces the total slightly, to 298 open tasks.
Figure 1: Unresolved error reports by month.
<https://phabricator.wikimedia.org/phame/post/view/259/production_excellence_36_september_2021/#trends>
For the month-over-month numbers, refer to the spreadsheet data
<https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml>
.
Did you know
- ???? The default *"system error" page now includes a request ID*. T291192
<https://phabricator.wikimedia.org/T291192>
- ???? To zoom in and find your team's error reports, *use the appropriate
"Filter" link in the sidebar* of the workboard
<https://phabricator.wikimedia.org/tag/wikimedia-production-error/>.
Outstanding errors
Take a look at the workboard and look for tasks that could use your help.
? https://phabricator.wikimedia.org/tag/wikimedia-production-error/
Last few months in review:
Jan 2021 (50 issues
<https://phabricator.wikimedia.org/maniphest/query/gn7TOpf2LdVE/#R>) 3
left. *Unchanged.*
Feb 2021 (20 issues
<https://phabricator.wikimedia.org/maniphest/query/xQxnXZys4q97/#R>) 5 > 4
left.
Mar 2021 (48 issues
<https://phabricator.wikimedia.org/maniphest/query/To0edISjsA9s/#R>) 10 > 9
left.
Apr 2021 (42 issues
<https://phabricator.wikimedia.org/maniphest/query/ORxSVxnJBlLc/#R>) 17 >
10 left.
May 2021 (54 issues
<https://phabricator.wikimedia.org/maniphest/query/9y.PWGoGgWbK/#R>) 20 >
17 left.
Jun 2021 (26 issues
<https://phabricator.wikimedia.org/maniphest/query/DlpqBkLj0aP4/#R>) 10 > 9
left.
Jul 2021 (31 issues
<https://phabricator.wikimedia.org/maniphest/query/qQAV178rYaJ_/#R>) 12
left. *Unchanged.*
Aug 2021 (46 issues
<https://phabricator.wikimedia.org/maniphest/query/i1wawBd5GKVY/#R>) 17 >
12 left.
Sep 2021 (24 issues
<https://phabricator.wikimedia.org/maniphest/query/BA8dqsGwaE_a/#R>) 13
unresolved issues remaining.
------------------------------
Tally
301 issues open, as of Excellence #35 (August 2021)
<https://phabricator.wikimedia.org/phame/post/view/248/production_excellence_35_august_2021/>
-16 issues closed, of the previous 301 open issues.
+13 new issues that survived September 2021.
298 issues open, as of today (19 Oct 2021).
Thanks!
Thank you to everyone who helped by reporting, investigating, or resolving
problems in Wikimedia production. Thanks!
Until next time,
– Timo Tijhof
???? Share or read later via
<https://phabricator.wikimedia.org/phame/post/view/240/>
https://phabricator.wikimedia.org/phame/post/view/259/
<https://phabricator.wikimedia.org/phame/post/view/259/production_excellence_36_september_2021/>
find out!
Incidents
We've had quite an eventful month, with 8 documented incidents in
September. That's the highest since last year (Feb 2020) and one of the
three worst months of the last five years.
- 2021-09-01 partial Parsoid outage
<https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-01_partial_parsoid_outage>
- Impact: For 9 hours, 10% of Parsoid requests to parse/save pages
were failing on all wikis. Little to no end-user impact apart from minor
due to RESTBase retries.
- 2021-09-04 appserver latency
<https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-04_appserver_latency>
- Impact: For 37 minutes, MW backends were slow with 2% of requests
receiving errors. This affected all wikis through logged-in
users, bots/API
queries, and some page views from unregistered users (e.g. pages
that were
recently edited or expired from CDN cache).
- 2021-09-06 Wikifeeds
<https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-06_Wikifeeds>
- Impact: For 3 days, the Wikifeeds API failed ~1% of requests (e.g.
5 of 500 req/s).
- 2021-09-12 Esams upload
<https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-12-Esams-upload>
- Impact: For 20 minutes, images were unavailable for people in
Europe, affecting all wikis.
- 2021-09-13 CirrusSearch restart
<https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-13_cirrussearch_restart>
- Impact: For ~2 hours, search was unavailable on Wikipedia from all
regions. Search suggestions were missing or slow, and the search results
page errored with "Try again later".
- 2021-09-18 appserver latency
<https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-18_appserver_latency>
- Impact: For ~10 minutes, MW backends were slow or unavailable for
all wikis.
- 2021-09-26 appserver latency
<https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-26_appserver_latency>
- Impact: For ~15 minutes, MW backends were slow or unavailable for
all wikis.
- 2021-09-29 eqiad kubernetes
<https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-29_eqiad-kubernetes>
- Impact: For 2 minutes, MW backends were affected by a Kubernetes
issue (via Kask sessionstore). 1500 edit attempts failed (8% of
POSTs), and
logged-in pageviews were slowed down, often taking several seconds.
Remember to review and schedule Incident Follow-up work
<https://phabricator.wikimedia.org/project/view/4758/> in Phabricator,
which are preventive measures and tech debt mitigations written down after
an incident is concluded.
See also Incident graphs <https://codepen.io/Krinkle/full/wbYMZK>.
Trends
The month of September saw 24 new production error reports of which 11 have
since been resolved, and today, three to six weeks later, 13 remain open
and have thus carried over to the next month. This is about average,
although it makes it no less sad that we continue to introduce (and carry
over) more errors than we rectify in the same time frame.
On the other hand, last month we did have a healthy focus on some of the
older reports. The workboard stood at 301 unresolved errors last month. Of
those, 16 were resolved. With the 13 new errors from September, this
reduces the total slightly, to 298 open tasks.
Figure 1: Unresolved error reports by month.
<https://phabricator.wikimedia.org/phame/post/view/259/production_excellence_36_september_2021/#trends>
For the month-over-month numbers, refer to the spreadsheet data
<https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml>
.
Did you know
- ???? The default *"system error" page now includes a request ID*. T291192
<https://phabricator.wikimedia.org/T291192>
- ???? To zoom in and find your team's error reports, *use the appropriate
"Filter" link in the sidebar* of the workboard
<https://phabricator.wikimedia.org/tag/wikimedia-production-error/>.
Outstanding errors
Take a look at the workboard and look for tasks that could use your help.
? https://phabricator.wikimedia.org/tag/wikimedia-production-error/
Last few months in review:
Jan 2021 (50 issues
<https://phabricator.wikimedia.org/maniphest/query/gn7TOpf2LdVE/#R>) 3
left. *Unchanged.*
Feb 2021 (20 issues
<https://phabricator.wikimedia.org/maniphest/query/xQxnXZys4q97/#R>) 5 > 4
left.
Mar 2021 (48 issues
<https://phabricator.wikimedia.org/maniphest/query/To0edISjsA9s/#R>) 10 > 9
left.
Apr 2021 (42 issues
<https://phabricator.wikimedia.org/maniphest/query/ORxSVxnJBlLc/#R>) 17 >
10 left.
May 2021 (54 issues
<https://phabricator.wikimedia.org/maniphest/query/9y.PWGoGgWbK/#R>) 20 >
17 left.
Jun 2021 (26 issues
<https://phabricator.wikimedia.org/maniphest/query/DlpqBkLj0aP4/#R>) 10 > 9
left.
Jul 2021 (31 issues
<https://phabricator.wikimedia.org/maniphest/query/qQAV178rYaJ_/#R>) 12
left. *Unchanged.*
Aug 2021 (46 issues
<https://phabricator.wikimedia.org/maniphest/query/i1wawBd5GKVY/#R>) 17 >
12 left.
Sep 2021 (24 issues
<https://phabricator.wikimedia.org/maniphest/query/BA8dqsGwaE_a/#R>) 13
unresolved issues remaining.
------------------------------
Tally
301 issues open, as of Excellence #35 (August 2021)
<https://phabricator.wikimedia.org/phame/post/view/248/production_excellence_35_august_2021/>
-16 issues closed, of the previous 301 open issues.
+13 new issues that survived September 2021.
298 issues open, as of today (19 Oct 2021).
Thanks!
Thank you to everyone who helped by reporting, investigating, or resolving
problems in Wikimedia production. Thanks!
Until next time,
– Timo Tijhof
???? Share or read later via
<https://phabricator.wikimedia.org/phame/post/view/240/>
https://phabricator.wikimedia.org/phame/post/view/259/
<https://phabricator.wikimedia.org/phame/post/view/259/production_excellence_36_september_2021/>