Mailing List Archive

Discord: Sever outages and Increased API errors
Discord seems to be having issues. It is showing on Down Detector. The splash page says:



The Server Status page shows:


Mostly FYI.

--
TTFN,
patrick
Re: Discord: Sever outages and Increased API errors [ In reply to ]
We should be back now. As you can probably expect we've had a very large
amount of growth recently and hit a file descriptor limit on a central
system that caused some cascading failures.

On Mon, Mar 16, 2020 at 2:49 PM Patrick W. Gilmore via Outages <
outages@outages.org> wrote:

> Discord seems to be having issues. It is showing on Down Detector. The
> splash page says:
>
> The Server Status page shows:
>
> Mostly FYI.
>
> --
> TTFN,
> patrick
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
Re: Discord: Sever outages and Increased API errors [ In reply to ]
If anyone is interested we published a post mortem on our status page for
this incident. https://status.discordapp.com/incidents/62gt9cgjwdgf

On Mon, Mar 16, 2020 at 2:49 PM Patrick W. Gilmore via Outages <
outages@outages.org> wrote:

> Discord seems to be having issues. It is showing on Down Detector. The
> splash page says:
>
> The Server Status page shows:
>
> Mostly FYI.
>
> --
> TTFN,
> patrick
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
Re: Discord: Sever outages and Increased API errors [ In reply to ]
I am sufficiently interested that I'm seriously considering writing a book on this topic actually. That was a very nice write-up. Do I correctly understand, then, that the root cause was the broken golang library function / routine?

(For the moment, we will pass on the topic of the irony of having to come up with an alternate name for your own language because otherwise people won't be able to find things about it... in the search engine you run that you wrote it for :-) Way to go Google.)

On March 20, 2020 11:54:08 PM EDT, Gary Josack via Outages <outages@outages.org> wrote:
>If anyone is interested we published a post mortem on our status page
>for
>this incident. https://status.discordapp.com/incidents/62gt9cgjwdgf
>
>On Mon, Mar 16, 2020 at 2:49 PM Patrick W. Gilmore via Outages <
>outages@outages.org> wrote:
>
>> Discord seems to be having issues. It is showing on Down Detector.
>The
>> splash page says:
>>
>> The Server Status page shows:
>>
>> Mostly FYI.
>>
>> --
>> TTFN,
>> patrick
>>
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: [EXTERNAL] Re: Discord: Sever outages and Increased API errors [ In reply to ]
Whenever you decide to publish such a book, I’ve got your title and cover art ready...

Put me down for a pre-order.

-Brad

[A close up of a bird Description automatically generated]



From: Outages <outages-bounces@outages.org> on behalf of Jay Ashworth via Outages <outages@outages.org>
Reply-To: Jay Ashworth <jra@baylink.com>
Date: Friday, March 20, 2020 at 9:07 PM
To: Gary Josack <gary@byoteki.com>, Gary Josack via Outages <outages@outages.org>, "Patrick W. Gilmore" <patrick@ianai.net>
Cc: outages <outages@outages.org>
Subject: [EXTERNAL] Re: [outages] Discord: Sever outages and Increased API errors

I am sufficiently interested that I'm seriously considering writing a book on this topic actually. That was a very nice write-up. Do I correctly understand, then, that the root cause was the broken golang library function / routine?

(For the moment, we will pass on the topic of the irony of having to come up with an alternate name for your own language because otherwise people won't be able to find things about it... in the search engine you run that you wrote it for :-) Way to go Google.)
On March 20, 2020 11:54:08 PM EDT, Gary Josack via Outages <outages@outages.org> wrote:
If anyone is interested we published a post mortem on our status page for this incident. https://status.discordapp.com/incidents/62gt9cgjwdgf<https://urldefense.com/v3/__https:/status.discordapp.com/incidents/62gt9cgjwdgf__;!!PIZeeW5wscynRQ!9IlozgcLDbWntVNJYRLbLSjlkY1RnEa2iRitGoMKZBTwtilpUibKlSbLxIv4M9w-XQ$>

On Mon, Mar 16, 2020 at 2:49 PM Patrick W. Gilmore via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
Discord seems to be having issues. It is showing on Down Detector. The splash page says:
[https://files.slack.com/files-pri/T2F7AJRMY-FV5C2QE4A/image.png]
[cid:170fb3917e5a2da56b91]

The Server Status page shows:
[cid:170fb3917e5774fd4222]

Mostly FYI.

--
TTFN,
patrick

_______________________________________________
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages<https://urldefense.com/v3/__https:/puck.nether.net/mailman/listinfo/outages__;!!PIZeeW5wscynRQ!9IlozgcLDbWntVNJYRLbLSjlkY1RnEa2iRitGoMKZBTwtilpUibKlSbLxIs82I34TA$>

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: Discord: Sever outages and Increased API errors [ In reply to ]
>> Nodes announce themselves to etcd by issuing an HTTP PUT, etcd client
uses Python's builtin HTTP client, and sends a PUT request (along with the
Content-Length header) in one packet, and the request body in another
packet. We determined that the connection was reset after sending the first
packet, but before the second packet could be sent.

Looks like service discovery poisoning, linked to python lib. Is it so?
(Even when its not DNS, it looks very much like DNS)

On Fri, Mar 20, 2020 at 11:05 PM Jay Ashworth via Outages <
outages@outages.org> wrote:

> I am sufficiently interested that I'm seriously considering writing a book
> on this topic actually. That was a very nice write-up. Do I correctly
> understand, then, that the root cause was the broken golang library
> function / routine?
>
> (For the moment, we will pass on the topic of the irony of having to come
> up with an alternate name for your own language because otherwise people
> won't be able to find things about it... in the search engine you run that
> you wrote it for :-) Way to go Google.)
>
> On March 20, 2020 11:54:08 PM EDT, Gary Josack via Outages <
> outages@outages.org> wrote:
>>
>> If anyone is interested we published a post mortem on our status page for
>> this incident. https://status.discordapp.com/incidents/62gt9cgjwdgf
>>
>> On Mon, Mar 16, 2020 at 2:49 PM Patrick W. Gilmore via Outages <
>> outages@outages.org> wrote:
>>
>>> Discord seems to be having issues. It is showing on Down Detector. The
>>> splash page says:
>>>
>>> The Server Status page shows:
>>>
>>> Mostly FYI.
>>>
>>> --
>>> TTFN,
>>> patrick
>>>
>>> _______________________________________________
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>


--
Andrei Hurynovich
Ops&IT @ Charlesworth Research, Chicago
Re: [EXTERNAL] Re: Discord: Sever outages and Increased API errors [ In reply to ]
Yeah, but that's an owl, with a 360-degree neck. Shouldn't it have been facing away from the camera? :-)

On March 22, 2020 2:03:04 AM EDT, "Chapman, Brad (NBCUniversal)" <Brad.Chapman@nbcuni.com> wrote:
>Whenever you decide to publish such a book, I’ve got your title and
>cover art ready...
>
>Put me down for a pre-order.
>
>-Brad
>
>[A close up of a bird Description automatically generated]
>
>
>
>From: Outages <outages-bounces@outages.org> on behalf of Jay Ashworth
>via Outages <outages@outages.org>
>Reply-To: Jay Ashworth <jra@baylink.com>
>Date: Friday, March 20, 2020 at 9:07 PM
>To: Gary Josack <gary@byoteki.com>, Gary Josack via Outages
><outages@outages.org>, "Patrick W. Gilmore" <patrick@ianai.net>
>Cc: outages <outages@outages.org>
>Subject: [EXTERNAL] Re: [outages] Discord: Sever outages and Increased
>API errors
>
>I am sufficiently interested that I'm seriously considering writing a
>book on this topic actually. That was a very nice write-up. Do I
>correctly understand, then, that the root cause was the broken golang
>library function / routine?
>
>(For the moment, we will pass on the topic of the irony of having to
>come up with an alternate name for your own language because otherwise
>people won't be able to find things about it... in the search engine
>you run that you wrote it for :-) Way to go Google.)
>On March 20, 2020 11:54:08 PM EDT, Gary Josack via Outages
><outages@outages.org> wrote:
>If anyone is interested we published a post mortem on our status page
>for this incident.
>https://status.discordapp.com/incidents/62gt9cgjwdgf<https://urldefense.com/v3/__https:/status.discordapp.com/incidents/62gt9cgjwdgf__;!!PIZeeW5wscynRQ!9IlozgcLDbWntVNJYRLbLSjlkY1RnEa2iRitGoMKZBTwtilpUibKlSbLxIv4M9w-XQ$>
>
>On Mon, Mar 16, 2020 at 2:49 PM Patrick W. Gilmore via Outages
><outages@outages.org<mailto:outages@outages.org>> wrote:
>Discord seems to be having issues. It is showing on Down Detector. The
>splash page says:
>[https://files.slack.com/files-pri/T2F7AJRMY-FV5C2QE4A/image.png]
>[cid:170fb3917e5a2da56b91]
>
>The Server Status page shows:
>[cid:170fb3917e5774fd4222]
>
>Mostly FYI.
>
>--
>TTFN,
>patrick
>
>_______________________________________________
>Outages mailing list
>Outages@outages.org<mailto:Outages@outages.org>
>https://puck.nether.net/mailman/listinfo/outages<https://urldefense.com/v3/__https:/puck.nether.net/mailman/listinfo/outages__;!!PIZeeW5wscynRQ!9IlozgcLDbWntVNJYRLbLSjlkY1RnEa2iRitGoMKZBTwtilpUibKlSbLxIs82I34TA$>
>
>--
>Sent from my Android device with K-9 Mail. Please excuse my brevity.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.