Hi everyone, I just enabled canary events for the mentioned streams.
Please comment here or on the task if you encounter any issues.
Thank you!
-Andrew Otto & the WMF Data Engineering team
<https://wikitech.wikimedia.org/wiki/Data_Engineering>
On Tue, Nov 7, 2023 at 1:57?PM Andrew Otto <otto@wikimedia.org> wrote:
> tl;dr
>
> Ignore this email if you do not use MediaWiki event streams.
>
> On Monday December 11 2023, all MediaWiki related event streams will have artificial
> canary events
> <https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams#Canary_Events>
> injected into them. If you use any of these streams, you should discard
> thesee canary events.
>
> Add code to your consumers that discards events where meta.domain ==
> "canary".
> Canary Events
>
> At WMF, we use artificial 'canary' AKA 'heartbeat' events
> <https://wikitech.wikimedia.org/wiki/Event_Platform/Stream_Configuration#canary_events_enabled>
> to differentiate between a broken event stream and an empty event stream.
> Canary events should be produced at least once an hour. If there are no
> events in a stream for an hour, then something is likely broken with that
> stream.
>
> These artificial canary events can be identified by the fact that their
> meta.domain field is set to "canary". If you use any of the streams
> listed below, you will need to add code that discards any events where meta.domain
> == "canary".
>
> Back in 2020, we began producing canary events into all new streams, but
> we never got around to enabling these for streams that already existed. We
> needed to ensure that all consumers of these streams filtered out the
> canary events. We're just finally getting around to enabling canary events
> for all streams.
>
> We will enable canary event production
> <https://phabricator.wikimedia.org/T266798> for the following streams on
> Monday, December 11th, 2023:
>
> - mediawiki.recentchange
>
> - mediawiki.page-create
>
> - mediawiki.page-delete
>
> - mediawiki.page-links-change
>
> - mediawiki.page-move
>
> - mediawiki.page-properties-change
>
> - mediawiki.page-restrictions-change
>
> - mediawiki.page-suppress
>
> - mediawiki.page-undelete
>
> - mediawiki.revision-create
>
> - mediawiki.revision-visibility-change
>
> - mediawiki.user-blocks-change
>
> - mediawiki.centralnotice.campaign-change
>
> - mediawiki.centralnotice.campaign-create
>
> - mediawiki.centralnotice.campaign-delete
>
> If you consume any of these streams, either external to WMF networks using
> EventStreams, or internally using Kafka, please ensure that your consumer
> logic discards events where meta.domain == "canary" before this date.
> (Note that not all of these streams are exposed publicly at
> stream.wikimedia.org <https://stream.wikimedia.org/?doc#/streams>.)
>
> Thank you,
>
> -Andrew Otto & the WMF Data Engineering team
> <https://wikitech.wikimedia.org/wiki/Data_Engineering>
>
> References
>
> - T266798 - Enable canary events for all MediaWiki streams
> <https://phabricator.wikimedia.org/T266798>
>
> - T251609 - Automate ingestion and refinement into Hive of event data
> from Kafka using stream configs and canary/heartbeat events
> <https://phabricator.wikimedia.org/T251609>
>
>
Please comment here or on the task if you encounter any issues.
Thank you!
-Andrew Otto & the WMF Data Engineering team
<https://wikitech.wikimedia.org/wiki/Data_Engineering>
On Tue, Nov 7, 2023 at 1:57?PM Andrew Otto <otto@wikimedia.org> wrote:
> tl;dr
>
> Ignore this email if you do not use MediaWiki event streams.
>
> On Monday December 11 2023, all MediaWiki related event streams will have artificial
> canary events
> <https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams#Canary_Events>
> injected into them. If you use any of these streams, you should discard
> thesee canary events.
>
> Add code to your consumers that discards events where meta.domain ==
> "canary".
> Canary Events
>
> At WMF, we use artificial 'canary' AKA 'heartbeat' events
> <https://wikitech.wikimedia.org/wiki/Event_Platform/Stream_Configuration#canary_events_enabled>
> to differentiate between a broken event stream and an empty event stream.
> Canary events should be produced at least once an hour. If there are no
> events in a stream for an hour, then something is likely broken with that
> stream.
>
> These artificial canary events can be identified by the fact that their
> meta.domain field is set to "canary". If you use any of the streams
> listed below, you will need to add code that discards any events where meta.domain
> == "canary".
>
> Back in 2020, we began producing canary events into all new streams, but
> we never got around to enabling these for streams that already existed. We
> needed to ensure that all consumers of these streams filtered out the
> canary events. We're just finally getting around to enabling canary events
> for all streams.
>
> We will enable canary event production
> <https://phabricator.wikimedia.org/T266798> for the following streams on
> Monday, December 11th, 2023:
>
> - mediawiki.recentchange
>
> - mediawiki.page-create
>
> - mediawiki.page-delete
>
> - mediawiki.page-links-change
>
> - mediawiki.page-move
>
> - mediawiki.page-properties-change
>
> - mediawiki.page-restrictions-change
>
> - mediawiki.page-suppress
>
> - mediawiki.page-undelete
>
> - mediawiki.revision-create
>
> - mediawiki.revision-visibility-change
>
> - mediawiki.user-blocks-change
>
> - mediawiki.centralnotice.campaign-change
>
> - mediawiki.centralnotice.campaign-create
>
> - mediawiki.centralnotice.campaign-delete
>
> If you consume any of these streams, either external to WMF networks using
> EventStreams, or internally using Kafka, please ensure that your consumer
> logic discards events where meta.domain == "canary" before this date.
> (Note that not all of these streams are exposed publicly at
> stream.wikimedia.org <https://stream.wikimedia.org/?doc#/streams>.)
>
> Thank you,
>
> -Andrew Otto & the WMF Data Engineering team
> <https://wikitech.wikimedia.org/wiki/Data_Engineering>
>
> References
>
> - T266798 - Enable canary events for all MediaWiki streams
> <https://phabricator.wikimedia.org/T266798>
>
> - T251609 - Automate ingestion and refinement into Hive of event data
> from Kafka using stream configs and canary/heartbeat events
> <https://phabricator.wikimedia.org/T251609>
>
>