How’d we do in our strive for operational excellence last month? Read on to find out!
Incidents
One documented incident last month (Incident graphs <https://codepen.io/Krinkle/full/wbYMZK>).
2021-12-03 mx <https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-12-03_mx>
Impact: A portion of outgoing email from wikimedia.org was delivered with a delay of upto 24 hours. This affected staff Gmail, and Znuny/Phabricator notifications. No mail was lost, it was eventually delivered.
Incident follow-up
Remember to review and schedule Incident Follow-up work <https://phabricator.wikimedia.org/project/view/4758/> in Phabricator. These are preventive measures and tech debt mitigations written down after an incident. Read about past incidents at Incident status <https://wikitech.wikimedia.org/wiki/Incident_status> on Wikitech.
Recently resolved incident follow-up:
Create paging alert for high MX queues <https://phabricator.wikimedia.org/T297144>.
Filed in December after the mail delivery incident, resolved later that month by Keith (Herron).
Limit db execution time of expensive MW special pages <https://phabricator.wikimedia.org/T297708>.
Filed in December after various incidents due to high DB/appserver load, carried out by Amir (Ladsgroup).
Trends
In December we reported 22 new errors in December <https://phabricator.wikimedia.org/maniphest/query/DhZaBJ5PI1NA/#R>, of which 5 have since been resolved, and 17 remain open and have carried over to January. From the 298 issues previously carried over, we also resolved 17, thus the workboard still adds up to 298 in total.
In previous editions, we sometimes looked at the breakdown of tasks that remained unresolved. This time, I'd like to draw attention to the throughput and distribution of tasks that did get resolved.
Production errors resolved in the month of December, by team and component (query <https://phabricator.wikimedia.org/maniphest/query/vIEXYsei8lwE/#R>):
* Community-Tech (2): GlobalPreferences (1), CodeMirror (1).
* DBA: DjVuHandler (1).
* Editing-team: DiscussionTools (1).
* Fundraising Tech: CentralNotice (1).
* Growth-Team (8): GrowthExperiments (6), Image-Suggestions (1), StructuredDiscussions (1).
* Language-Team: UniversalLanguageSelector (1).
* Parsoid (1).
* Product-Infrastructure: TemplateStyles (1).
* Readers-Web (2).
* Structured-Data (2).
* Wikidata team: Wikidata-Page-Banner (1).
* Missing steward (1): MediaWiki-Logevents (T289806: Thanks Umherirrender!).
For the month-over-month numbers, refer to the spreadsheet data <https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml>.
Outstanding errors
Oldest unresolved errors:
* (June 2020) WikibaseClient: RuntimeException in wblistentityusage API. T254334 <https://phabricator.wikimedia.org/T254334>
* (June 2020) WikibaseClient: Deadlock in EntityUsageTable::addUsages method. T255706 <https://phabricator.wikimedia.org/T255706>
Take a look at the workboard and look for tasks that could use your help.
? https://phabricator.wikimedia.org/tag/wikimedia-production-error/
Thanks!
Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!
Until next time,
– Timo Tijhof
???? Share or read later via https://phabricator.wikimedia.org/phame/post/view/265/
Incidents
One documented incident last month (Incident graphs <https://codepen.io/Krinkle/full/wbYMZK>).
2021-12-03 mx <https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-12-03_mx>
Impact: A portion of outgoing email from wikimedia.org was delivered with a delay of upto 24 hours. This affected staff Gmail, and Znuny/Phabricator notifications. No mail was lost, it was eventually delivered.
Incident follow-up
Remember to review and schedule Incident Follow-up work <https://phabricator.wikimedia.org/project/view/4758/> in Phabricator. These are preventive measures and tech debt mitigations written down after an incident. Read about past incidents at Incident status <https://wikitech.wikimedia.org/wiki/Incident_status> on Wikitech.
Recently resolved incident follow-up:
Create paging alert for high MX queues <https://phabricator.wikimedia.org/T297144>.
Filed in December after the mail delivery incident, resolved later that month by Keith (Herron).
Limit db execution time of expensive MW special pages <https://phabricator.wikimedia.org/T297708>.
Filed in December after various incidents due to high DB/appserver load, carried out by Amir (Ladsgroup).
Trends
In December we reported 22 new errors in December <https://phabricator.wikimedia.org/maniphest/query/DhZaBJ5PI1NA/#R>, of which 5 have since been resolved, and 17 remain open and have carried over to January. From the 298 issues previously carried over, we also resolved 17, thus the workboard still adds up to 298 in total.
In previous editions, we sometimes looked at the breakdown of tasks that remained unresolved. This time, I'd like to draw attention to the throughput and distribution of tasks that did get resolved.
Production errors resolved in the month of December, by team and component (query <https://phabricator.wikimedia.org/maniphest/query/vIEXYsei8lwE/#R>):
* Community-Tech (2): GlobalPreferences (1), CodeMirror (1).
* DBA: DjVuHandler (1).
* Editing-team: DiscussionTools (1).
* Fundraising Tech: CentralNotice (1).
* Growth-Team (8): GrowthExperiments (6), Image-Suggestions (1), StructuredDiscussions (1).
* Language-Team: UniversalLanguageSelector (1).
* Parsoid (1).
* Product-Infrastructure: TemplateStyles (1).
* Readers-Web (2).
* Structured-Data (2).
* Wikidata team: Wikidata-Page-Banner (1).
* Missing steward (1): MediaWiki-Logevents (T289806: Thanks Umherirrender!).
For the month-over-month numbers, refer to the spreadsheet data <https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml>.
Outstanding errors
Oldest unresolved errors:
* (June 2020) WikibaseClient: RuntimeException in wblistentityusage API. T254334 <https://phabricator.wikimedia.org/T254334>
* (June 2020) WikibaseClient: Deadlock in EntityUsageTable::addUsages method. T255706 <https://phabricator.wikimedia.org/T255706>
Take a look at the workboard and look for tasks that could use your help.
? https://phabricator.wikimedia.org/tag/wikimedia-production-error/
Thanks!
Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!
Until next time,
– Timo Tijhof
???? Share or read later via https://phabricator.wikimedia.org/phame/post/view/265/