How’d we do in our strive for operational excellence last month? Read on to find out!
Incidents
By golly, we've had quite the month! 10 documented incidents, which is more than three times the two-year median of 3. The last time we experienced ten or more incidents in one month, was June 2019 when we had eleven (Incident graphs <https://codepen.io/Krinkle/full/wbYMZK>, Excellence monthly of June 2019 <https://phabricator.wikimedia.org/phame/post/view/163/production_excellence_12_june_2019/>).
I'd like to draw your attention to something positive. As you read the below, take note of incidents that did *not* impact public services, and did *not* have lasting impact or data loss. For example, the Apache incident <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-24_Failed_Apache_restart> benefited from PyBal's automatic health-based depooling. The deployment server incident <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-02_deployment> recovered without loss thanks to Bacula. The Etcd incident <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-01_etcd> impact was limited by serving stale data. And, the Hadoop incident <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-31_Analytics_Data_Lake_-_Hadoop_Namenode_failure> recovered by resuming from Kafka right where it left off.
2022-05-01 etcd <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-01_etcd>
Impact: For 2 hours, Conftool could not sync Etcd data between our core data centers. Puppet and some other internal services were unavailable or out of sync. The issue was isolated, with no impact on public services.
2022-05-02 deployment server <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-02_deployment>
Impact: For 4 hours, we could not update or deploy MediaWiki and other services, due to corruption on the active deployment server. No impact on public services.
2022-05-05 site outage <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-05_Wikimedia_full_site_outage>
Impact: For 20 minutes, all wikis were unreachable for logged-in users and non-cached pages. This was due to a GlobalBlocks schema change causing significant slowdown in a frequent database query.
2022-05-09 Codfw confctl <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-09_confctl>
Impact: For 5 minutes, all web traffic routed to Codfw received error responses. This affected central USA and South America (local time after midnight). The cause was human error and lack of CLI parameter validation.
2022-05-09 exim-bdat-errors <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-09_exim-bdat-errors>
Impact: During five days, about 14,000 incoming emails from Gmail users to wikimedia.org were rejected and returned to sender.
2022-05-21 varnish cache busting <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-21_varnish_cache_busting>
Impact: For 2 minutes, all wikis and services behind our CDN were unavailable to all users.
2022-05-24 failed Apache restart <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-24_Failed_Apache_restart>
Impact: For 35 minutes, numerous internal services that use Apache on the backend were down. This included Kibana (logstash) and Matomo (piwik). For 20 of those minutes, there was also reduced MediaWiki server capacity, but no measurable end-user impact for wiki traffic.
2022-05-25 de.wikipedia.org <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-25_de.wikipedia.org>
Impact: For 6 minutes, a portion of logged-in users and non-cached pages experienced a slower response or an error. This was due to increased load on one of the databases.
2022-05-26 m1 database hardware <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-26_Database_hardware_failure>
Impact: For 12 minutes, internal services hosted on the m1 database (e.g. Etherpad) were unavailable or at reduced capacity.
2022-05-31 Analytics Hadoop failure <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-31_Analytics_Data_Lake_-_Hadoop_Namenode_failure>
Impact: For 1 hour, all HDFS writes and reads were failing. After recovery, ingestion from Kafka resumed and caught up. No data loss or other lasting impact on the Data Lake.
Incident follow-up
Recently completed incident follow-up:
Invalid confctl selector should either error out or select nothing <https://phabricator.wikimedia.org/T308100>
Filed by Amir (@Ladsgroup <https://phabricator.wikimedia.org/p/Ladsgroup/>) after the confctl incident this past month. Giuseppe (@Joe <https://phabricator.wikimedia.org/p/Joe/>) implemented CLI parameter validation to prevent human error from causing a similar outage in the future.
Backup opensearch dashboards data <https://phabricator.wikimedia.org/T237224>
Filed back in 2019 by Filippo (@fgiunchedi <https://phabricator.wikimedia.org/p/fgiunchedi/>). The OpenSearch homepage dashboard (at logstash.wikimedia.org) was accidentally deleted last month. Bryan (@bd808 <https://phabricator.wikimedia.org/p/bd808/>) tracked down its content and re-created it. Cole (@colewhite <https://phabricator.wikimedia.org/p/colewhite/>) and Jaime (@jcrespo <https://phabricator.wikimedia.org/p/jcrespo/>) worked out a strategy and set up automated backups going forward.
Remember to review and schedule Incident Follow-up work <https://phabricator.wikimedia.org/project/view/4758/> in Phabricator! These are preventive measures and tech debt mitigations written down after an incident is concluded. Read more about past incidents at Incident status on Wikitech.
????*Did you know?*: The form on the *Incident status <https://wikitech.wikimedia.org/wiki/Incident_status>* page now includes a date, to more easily create backdated reports.
Trends
In May we discovered 28 new production errors <https://phabricator.wikimedia.org/maniphest/query/z7vLwJdXtLu2/#R>, of which 20 remain unresolved and have come with us to June.
Last month the workboard totalled 292 tasks still open from prior months. Since the last edition, we completed 11 tasks from previous months, gained 11 additional errors from May (some of May was counted in last month), and have 7 fresh errors in the current month of June. As of today, the workboard houses 299 open production error tasks (spreadsheet and graph <https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml>, phab report <https://phabricator.wikimedia.org/project/reports/1055/>).
Take a look at the workboard and look for tasks that could use your help.
? https://phabricator.wikimedia.org/tag/wikimedia-production-error/
Thanks!
Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!
Until next time,
– Timo Tijhof
???? Share or read later via https://phabricator.wikimedia.org/phame/post/view/285/
Incidents
By golly, we've had quite the month! 10 documented incidents, which is more than three times the two-year median of 3. The last time we experienced ten or more incidents in one month, was June 2019 when we had eleven (Incident graphs <https://codepen.io/Krinkle/full/wbYMZK>, Excellence monthly of June 2019 <https://phabricator.wikimedia.org/phame/post/view/163/production_excellence_12_june_2019/>).
I'd like to draw your attention to something positive. As you read the below, take note of incidents that did *not* impact public services, and did *not* have lasting impact or data loss. For example, the Apache incident <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-24_Failed_Apache_restart> benefited from PyBal's automatic health-based depooling. The deployment server incident <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-02_deployment> recovered without loss thanks to Bacula. The Etcd incident <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-01_etcd> impact was limited by serving stale data. And, the Hadoop incident <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-31_Analytics_Data_Lake_-_Hadoop_Namenode_failure> recovered by resuming from Kafka right where it left off.
2022-05-01 etcd <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-01_etcd>
Impact: For 2 hours, Conftool could not sync Etcd data between our core data centers. Puppet and some other internal services were unavailable or out of sync. The issue was isolated, with no impact on public services.
2022-05-02 deployment server <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-02_deployment>
Impact: For 4 hours, we could not update or deploy MediaWiki and other services, due to corruption on the active deployment server. No impact on public services.
2022-05-05 site outage <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-05_Wikimedia_full_site_outage>
Impact: For 20 minutes, all wikis were unreachable for logged-in users and non-cached pages. This was due to a GlobalBlocks schema change causing significant slowdown in a frequent database query.
2022-05-09 Codfw confctl <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-09_confctl>
Impact: For 5 minutes, all web traffic routed to Codfw received error responses. This affected central USA and South America (local time after midnight). The cause was human error and lack of CLI parameter validation.
2022-05-09 exim-bdat-errors <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-09_exim-bdat-errors>
Impact: During five days, about 14,000 incoming emails from Gmail users to wikimedia.org were rejected and returned to sender.
2022-05-21 varnish cache busting <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-21_varnish_cache_busting>
Impact: For 2 minutes, all wikis and services behind our CDN were unavailable to all users.
2022-05-24 failed Apache restart <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-24_Failed_Apache_restart>
Impact: For 35 minutes, numerous internal services that use Apache on the backend were down. This included Kibana (logstash) and Matomo (piwik). For 20 of those minutes, there was also reduced MediaWiki server capacity, but no measurable end-user impact for wiki traffic.
2022-05-25 de.wikipedia.org <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-25_de.wikipedia.org>
Impact: For 6 minutes, a portion of logged-in users and non-cached pages experienced a slower response or an error. This was due to increased load on one of the databases.
2022-05-26 m1 database hardware <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-26_Database_hardware_failure>
Impact: For 12 minutes, internal services hosted on the m1 database (e.g. Etherpad) were unavailable or at reduced capacity.
2022-05-31 Analytics Hadoop failure <https://wikitech.wikimedia.org/wiki/Incidents/2022-05-31_Analytics_Data_Lake_-_Hadoop_Namenode_failure>
Impact: For 1 hour, all HDFS writes and reads were failing. After recovery, ingestion from Kafka resumed and caught up. No data loss or other lasting impact on the Data Lake.
Incident follow-up
Recently completed incident follow-up:
Invalid confctl selector should either error out or select nothing <https://phabricator.wikimedia.org/T308100>
Filed by Amir (@Ladsgroup <https://phabricator.wikimedia.org/p/Ladsgroup/>) after the confctl incident this past month. Giuseppe (@Joe <https://phabricator.wikimedia.org/p/Joe/>) implemented CLI parameter validation to prevent human error from causing a similar outage in the future.
Backup opensearch dashboards data <https://phabricator.wikimedia.org/T237224>
Filed back in 2019 by Filippo (@fgiunchedi <https://phabricator.wikimedia.org/p/fgiunchedi/>). The OpenSearch homepage dashboard (at logstash.wikimedia.org) was accidentally deleted last month. Bryan (@bd808 <https://phabricator.wikimedia.org/p/bd808/>) tracked down its content and re-created it. Cole (@colewhite <https://phabricator.wikimedia.org/p/colewhite/>) and Jaime (@jcrespo <https://phabricator.wikimedia.org/p/jcrespo/>) worked out a strategy and set up automated backups going forward.
Remember to review and schedule Incident Follow-up work <https://phabricator.wikimedia.org/project/view/4758/> in Phabricator! These are preventive measures and tech debt mitigations written down after an incident is concluded. Read more about past incidents at Incident status on Wikitech.
????*Did you know?*: The form on the *Incident status <https://wikitech.wikimedia.org/wiki/Incident_status>* page now includes a date, to more easily create backdated reports.
Trends
In May we discovered 28 new production errors <https://phabricator.wikimedia.org/maniphest/query/z7vLwJdXtLu2/#R>, of which 20 remain unresolved and have come with us to June.
Last month the workboard totalled 292 tasks still open from prior months. Since the last edition, we completed 11 tasks from previous months, gained 11 additional errors from May (some of May was counted in last month), and have 7 fresh errors in the current month of June. As of today, the workboard houses 299 open production error tasks (spreadsheet and graph <https://docs.google.com/spreadsheets/d/e/2PACX-1vTrUCAI10hIroYDU-i5_8s7pony8M71ATXrFRiXXV7t5-tITZYrTRLGch-3iJbmeG41ZMcj1vGfzZ70/pubhtml>, phab report <https://phabricator.wikimedia.org/project/reports/1055/>).
Take a look at the workboard and look for tasks that could use your help.
? https://phabricator.wikimedia.org/tag/wikimedia-production-error/
Thanks!
Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!
Until next time,
– Timo Tijhof
???? Share or read later via https://phabricator.wikimedia.org/phame/post/view/285/