Mailing List Archive

also to forgot to mention that DownDetector is showing an increase in
Frontier complaints.


On 7/31/23 12:22 PM, William Kern via Outages wrote:
> We have seen flapping all morning reaching our AWS East Coast servers
> from our Thousand Oaks location. They drop off for a minute or so and
> then come back.
>
> It seems to occur at 3-5 minute intervals for several cycles, then
> everything is fine for 30 minutes or so. Then the alarms start to come
> in again so wash rinse and repeat.
>
> The issue is definately not local, as our local monitoring to our peer
> connections are fine as are our connections to AWS WestCoast servers.
> The only complaint I got so far was from a customer on the East cost
> trying to VPN to a server here and getting booted out a lot.
>
> Traceroutes imply there may be an issue with Frontier as the few times
> I have caught the problem while logged into the remote East Coast
> server the path outbound from AWS East goes to Dallas and then seems
> to have issue reaching Los Angeles. I also show those traceroute pages
> moving around a bit between those locations.
>
> The connection from here to the the East Cost show weirdness involving
> Charter so maybe the issue is on the reverse trip.
>
> this is pretty typical.
>
>  8  lag-401.dllstx976iw-bcr00.netops.charter.com (66.109.5.229) 36.002
> ms  36.316 ms lag-16.dllstx976iw-bcr00.netops.charter.com
> (66.109.6.1)  36.451 ms
>  9  lag-0.pr3.dfw10.netops.charter.com (66.109.5.121)  39.187 ms
> lag-302.pr3.dfw10.netops.charter.com (209.18.43.77)  42.279 ms 45.338 ms
> 10  99.83.71.242 (99.83.71.242)  35.427 ms 99.83.71.240
> (99.83.71.240)  36.389 ms 99.82.176.170 (99.82.176.170)  35.339 ms
> 11  * 150.222.206.169 (150.222.206.169)  34.071 ms *
> 12  * * 15.230.48.42 (15.230.48.42)  33.717 ms
>
> Finally its not within AWS as our WestCoast servers can maintain a
> steady ping with the affected East Coast sites.
>
> So is anyone else seeing weirdness on cross country trips?
>
>
> William Kern
>
> PixelGate
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

We are seeing similar issues from Santa Monica trying to reach our sites in the DC area today. I made a ticket with our east coast ISP Allied and they mentioned that AT&T was having peering issues with some ISPs in California. We are BGP'd with AT&T and Frontier and usually our outbound traffic rides on AT&T and inbound traffic picks whichever route is better between the two ISPs. I moved traffic from AT&T to Frontier and then our others sites started to drop so I backed off. I'll try removing the Frontier routes later and see if things improve.

Jose Gomez

-----Original Message-----
From: Outages <outages-bounces@outages.org> On Behalf Of William Kern via Outages
Sent: Monday, July 31, 2023 12:23 PM
To: outages@outages.org
Subject: [outages] flapping issues reaching East Coast AWS

We have seen flapping all morning reaching our AWS East Coast servers from our Thousand Oaks location. They drop off for a minute or so and then come back.

It seems to occur at 3-5 minute intervals for several cycles, then everything is fine for 30 minutes or so. Then the alarms start to come in again so wash rinse and repeat.

The issue is definately not local, as our local monitoring to our peer connections are fine as are our connections to AWS WestCoast servers.
The only complaint I got so far was from a customer on the East cost trying to VPN to a server here and getting booted out a lot.

Traceroutes imply there may be an issue with Frontier as the few times I have caught the problem while logged into the remote East Coast server the path outbound from AWS East goes to Dallas and then seems to have issue reaching Los Angeles. I also show those traceroute pages moving around a bit between those locations.

The connection from here to the the East Cost show weirdness involving Charter so maybe the issue is on the reverse trip.

this is pretty typical.

 8  lag-401.dllstx976iw-bcr00.netops.charter.com (66.109.5.229) 36.002 ms  36.316 ms lag-16.dllstx976iw-bcr00.netops.charter.com (66.109.6.1)
36.451 ms
 9  lag-0.pr3.dfw10.netops.charter.com (66.109.5.121)  39.187 ms lag-302.pr3.dfw10.netops.charter.com (209.18.43.77)  42.279 ms 45.338 ms
10  99.83.71.242 (99.83.71.242)  35.427 ms 99.83.71.240 (99.83.71.240)
36.389 ms 99.82.176.170 (99.82.176.170)  35.339 ms
11  * 150.222.206.169 (150.222.206.169)  34.071 ms *
12  * * 15.230.48.42 (15.230.48.42)  33.717 ms

Finally its not within AWS as our WestCoast servers can maintain a steady ping with the affected East Coast sites.

So is anyone else seeing weirdness on cross country trips?


William Kern

PixelGate

_______________________________________________
Outages mailing list
Outages@outages.org
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwIGaQ&c=UtbViGLMoQq17uDUqpM_9A&r=UuApoUz30cXKbJIUtc8kezBFqg4H3SVzFz95ksh7T7M&m=p_NqNaE60PttM3wrSQvpWgTX5jNSyUB04sP6Z1mRGHOtGfr76RMTdXCd8Wve4_Ow&s=kDELerzTlOJQypprUN7vkLHLcTQNzksKVnAjBEF6Pac&e=
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

This should be resolved on the Frontier side at this time. Would you mind checking again and let me know how things look?

Regards,
Jeff

> On Jul 31, 2023, at 12:24 PM, William Kern via Outages <outages@outages.org> wrote:
>
> ?also to forgot to mention that DownDetector is showing an increase in Frontier complaints.
>
>
>> On 7/31/23 12:22 PM, William Kern via Outages wrote:
>> We have seen flapping all morning reaching our AWS East Coast servers from our Thousand Oaks location. They drop off for a minute or so and then come back.
>>
>> It seems to occur at 3-5 minute intervals for several cycles, then everything is fine for 30 minutes or so. Then the alarms start to come in again so wash rinse and repeat.
>>
>> The issue is definately not local, as our local monitoring to our peer connections are fine as are our connections to AWS WestCoast servers. The only complaint I got so far was from a customer on the East cost trying to VPN to a server here and getting booted out a lot.
>>
>> Traceroutes imply there may be an issue with Frontier as the few times I have caught the problem while logged into the remote East Coast server the path outbound from AWS East goes to Dallas and then seems to have issue reaching Los Angeles. I also show those traceroute pages moving around a bit between those locations.
>>
>> The connection from here to the the East Cost show weirdness involving Charter so maybe the issue is on the reverse trip.
>>
>> this is pretty typical.
>>
>> 8 lag-401.dllstx976iw-bcr00.netops.charter.com (66.109.5.229) 36.002 ms 36.316 ms lag-16.dllstx976iw-bcr00.netops.charter.com (66.109.6.1) 36.451 ms
>> 9 lag-0.pr3.dfw10.netops.charter.com (66.109.5.121) 39.187 ms lag-302.pr3.dfw10.netops.charter.com (209.18.43.77) 42.279 ms 45.338 ms
>> 10 99.83.71.242 (99.83.71.242) 35.427 ms 99.83.71.240 (99.83.71.240) 36.389 ms 99.82.176.170 (99.82.176.170) 35.339 ms
>> 11 * 150.222.206.169 (150.222.206.169) 34.071 ms *
>> 12 * * 15.230.48.42 (15.230.48.42) 33.717 ms
>>
>> Finally its not within AWS as our WestCoast servers can maintain a steady ping with the affected East Coast sites.
>>
>> So is anyone else seeing weirdness on cross country trips?
>>
>>
>> William Kern
>>
>> PixelGate
>>
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

the last monit failure/success pair was 14:34/36 Pacific

I would have expected more warnings around 15:00.

Because its 15:16 now, we seem to have at least gone longer than we did
all morning. So that is promising!

I'll respond back if I see another failure.

-bill


On 7/31/23 14:55, Jeff Richmond wrote:
> This should be resolved on the Frontier side at this time. Would you mind checking again and let me know how things look?
>
> Regards,
> Jeff
>
>> On Jul 31, 2023, at 12:24 PM, William Kern via Outages <outages@outages.org> wrote:
>>
>> ?also to forgot to mention that DownDetector is showing an increase in Frontier complaints.
>>
>>
>>> On 7/31/23 12:22 PM, William Kern via Outages wrote:
>>> We have seen flapping all morning reaching our AWS East Coast servers from our Thousand Oaks location. They drop off for a minute or so and then come back.
>>>
>>> It seems to occur at 3-5 minute intervals for several cycles, then everything is fine for 30 minutes or so. Then the alarms start to come in again so wash rinse and repeat.
>>>
>>> The issue is definately not local, as our local monitoring to our peer connections are fine as are our connections to AWS WestCoast servers. The only complaint I got so far was from a customer on the East cost trying to VPN to a server here and getting booted out a lot.
>>>
>>> Traceroutes imply there may be an issue with Frontier as the few times I have caught the problem while logged into the remote East Coast server the path outbound from AWS East goes to Dallas and then seems to have issue reaching Los Angeles. I also show those traceroute pages moving around a bit between those locations.
>>>
>>> The connection from here to the the East Cost show weirdness involving Charter so maybe the issue is on the reverse trip.
>>>
>>> this is pretty typical.
>>>
>>> 8 lag-401.dllstx976iw-bcr00.netops.charter.com (66.109.5.229) 36.002 ms 36.316 ms lag-16.dllstx976iw-bcr00.netops.charter.com (66.109.6.1) 36.451 ms
>>> 9 lag-0.pr3.dfw10.netops.charter.com (66.109.5.121) 39.187 ms lag-302.pr3.dfw10.netops.charter.com (209.18.43.77) 42.279 ms 45.338 ms
>>> 10 99.83.71.242 (99.83.71.242) 35.427 ms 99.83.71.240 (99.83.71.240) 36.389 ms 99.82.176.170 (99.82.176.170) 35.339 ms
>>> 11 * 150.222.206.169 (150.222.206.169) 34.071 ms *
>>> 12 * * 15.230.48.42 (15.230.48.42) 33.717 ms
>>>
>>> Finally its not within AWS as our WestCoast servers can maintain a steady ping with the affected East Coast sites.
>>>
>>> So is anyone else seeing weirdness on cross country trips?
>>>
>>>
>>> William Kern
>>>
>>> PixelGate
>>>
>>> _______________________________________________
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Ok great, thanks for the update. I am away at the moment so I don’t have all the details but am happy to ping the ops folks if you see any further degradation.

Thanks,
Jeff

> On Jul 31, 2023, at 3:18 PM, William Kern <wkern@pixelgate.net> wrote:
>
> ?the last monit failure/success pair was 14:34/36 Pacific
>
> I would have expected more warnings around 15:00.
>
> Because its 15:16 now, we seem to have at least gone longer than we did all morning. So that is promising!
>
> I'll respond back if I see another failure.
>
> -bill
>
>
>> On 7/31/23 14:55, Jeff Richmond wrote:
>> This should be resolved on the Frontier side at this time. Would you mind checking again and let me know how things look?
>>
>> Regards,
>> Jeff
>>
>>>> On Jul 31, 2023, at 12:24 PM, William Kern via Outages <outages@outages.org> wrote:
>>>
>>> ?also to forgot to mention that DownDetector is showing an increase in Frontier complaints.
>>>
>>>
>>>> On 7/31/23 12:22 PM, William Kern via Outages wrote:
>>>> We have seen flapping all morning reaching our AWS East Coast servers from our Thousand Oaks location. They drop off for a minute or so and then come back.
>>>>
>>>> It seems to occur at 3-5 minute intervals for several cycles, then everything is fine for 30 minutes or so. Then the alarms start to come in again so wash rinse and repeat.
>>>>
>>>> The issue is definately not local, as our local monitoring to our peer connections are fine as are our connections to AWS WestCoast servers. The only complaint I got so far was from a customer on the East cost trying to VPN to a server here and getting booted out a lot.
>>>>
>>>> Traceroutes imply there may be an issue with Frontier as the few times I have caught the problem while logged into the remote East Coast server the path outbound from AWS East goes to Dallas and then seems to have issue reaching Los Angeles. I also show those traceroute pages moving around a bit between those locations.
>>>>
>>>> The connection from here to the the East Cost show weirdness involving Charter so maybe the issue is on the reverse trip.
>>>>
>>>> this is pretty typical.
>>>>
>>>> 8 lag-401.dllstx976iw-bcr00.netops.charter.com (66.109.5.229) 36.002 ms 36.316 ms lag-16.dllstx976iw-bcr00.netops.charter.com (66.109.6.1) 36.451 ms
>>>> 9 lag-0.pr3.dfw10.netops.charter.com (66.109.5.121) 39.187 ms lag-302.pr3.dfw10.netops.charter.com (209.18.43.77) 42.279 ms 45.338 ms
>>>> 10 99.83.71.242 (99.83.71.242) 35.427 ms 99.83.71.240 (99.83.71.240) 36.389 ms 99.82.176.170 (99.82.176.170) 35.339 ms
>>>> 11 * 150.222.206.169 (150.222.206.169) 34.071 ms *
>>>> 12 * * 15.230.48.42 (15.230.48.42) 33.717 ms
>>>>
>>>> Finally its not within AWS as our WestCoast servers can maintain a steady ping with the affected East Coast sites.
>>>>
>>>> So is anyone else seeing weirdness on cross country trips?
>>>>
>>>>
>>>> William Kern
>>>>
>>>> PixelGate
>>>>
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages@outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>> _______________________________________________
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

On Mon, 31 Jul 2023 15:23:59 -0700
Jeff Richmond via Outages <outages@outages.org> wrote:

> Ok great, thanks for the update. I am away at the moment so I don’t
> have all the details but am happy to ping the ops folks if you see
> any further degradation.

Frontier.com has an AAAA RR, but it does not appear to be reachable and
functioning. Not sure if this is indicative of anything bigger than
that, but it would be nice if it either worded or the RR was removed.

John
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages