Mailing List Archive: Twelve99 / AWS usw2 significant loss

Twelve99 / AWS usw2 significant loss

Jan 26, 2024, 12:19 AM

Post #1 of 4 (83 views)

Hi Folks,

Cross posting to outages and nanog.

Can anybody from Twelve99 / AWS investigate high loss and latency around SJO between Twelve99 and AWS?

Traces below:

88.99.88.67 to 216.147.3.209:

My traceroute [v0.94]
phil (10.88.10.10) -> 216.147.3.209 2024-01-26T08:16:16+0000
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 10.88.10.254 0.0% 176 0.2 0.1 0.1 0.3 0.1
2. fe-fw-01.lavin.me.uk 0.0% 176 0.4 0.4 0.2 0.7 0.1
3. 10.69.69.1 0.0% 176 0.5 0.4 0.2 0.7 0.1
4. static.225.8.4.46.clients.your-server.de 0.0% 176 0.8 1.0 0.5 8.2 1.0
5. core22.fsn1.hetzner.com 0.6% 176 0.8 2.0 0.7 39.0 4.6
6. static.213-239-254-234.clients.your-server.de 0.0% 176 3.2 3.8 3.1 36.5 3.3
7. nug-b1-link.ip.twelve99.net 0.0% 176 3.3 3.5 3.1 24.1 1.6
8. hbg-bb2-link.ip.twelve99.net 86.9% 175 18.9 18.9 18.7 19.2 0.1
9. ldn-bb2-link.ip.twelve99.net 92.0% 175 30.5 30.6 30.4 30.8 0.1
10. nyk-bb1-link.ip.twelve99.net 4.6% 175 99.5 99.5 99.3 100.1 0.2
11. sjo-b23-link.ip.twelve99.net 56.3% 175 296.8 306.0 289.7 315.0 5.5
12. amazon-ic-366608.ip.twelve99-cust.net 80.5% 175 510.0 513.5 500.7 539.7 8.4
13. (waiting for reply)
…...
26. 216.147.3.209 59.1% 150 534.2 534.4 513.5 549.7 8.1

44.236.47.236 to 178.63.26.145:

jcasc-rtc01 (10.96.54.240) -> hc-ping.com 2024-01-26T08:10:24+0000
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. ip-10-96-50-153.us-west-2.compute.internal 0.0% 267 0.2 0.2 0.2 0.4 0.0
2. ec2-34-221-151-225.us-west-2.compute.amazonaw 0.0% 267 6.8 15.9 0.7 114.1 24.1
3. (waiting for reply)
4. (waiting for reply)
5. (waiting for reply)
6. 241.0.2.5 0.0% 267 0.4 0.4 0.3 1.2 0.1
7. 242.0.42.195 0.0% 267 1.2 1.7 0.5 23.9 3.0
8. 240.2.144.12 0.0% 267 5.2 5.2 5.2 27.4 1.4
9. amazon-ic-375418.ip.twelve99-cust.net 0.0% 267 5.3 5.2 5.2 6.1 0.1
10. port-b4-link.ip.twelve99.net 0.0% 267 5.3 5.5 4.9 35.5 3.0
11. port-b3-link.ip.twelve99.net 0.0% 267 5.8 5.9 5.6 11.8 0.5
12. palo-b24-link.ip.twelve99.net 4.9% 267 21.1 21.5 21.0 58.4 3.1
13. sjo-b23-link.ip.twelve99.net 0.0% 266 21.4 22.7 21.3 86.2 6.5
14. nyk-bb1-link.ip.twelve99.net 58.1% 266 432.7 422.7 407.2 438.5 6.5
15. ldn-bb2-link.ip.twelve99.net 98.1% 266 485.6 485.4 481.6 491.1 3.9
16. hbg-bb2-link.ip.twelve99.net 92.5% 266 504.1 499.8 489.8 510.1 5.9
17. nug-b1-link.ip.twelve99.net 55.5% 266 523.5 519.6 504.4 561.7 7.6
18. hetzner-ic-340780.ip.twelve99-cust.net 53.6% 266 524.4 519.2 506.0 545.5 6.9
19. core22.fsn1.hetzner.com 70.2% 266 521.7 519.2 498.5 531.7 6.6
20. static.213-239-254-150.clients.your-server.de 33.2% 266 382.4 375.4 364.9 396.5 4.1
21. static.145.26.63.178.clients.your-server.de 62.0% 266 529.9 518.4 506.9 531.3 6.1

Thanks
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: Twelve99 / AWS usw2 significant loss [ In reply to ]

outages at outages

Jan 26, 2024, 12:45 AM

Post #2 of 4 (83 views)

Permalink

Thanks, ytti. I have raised a case with AWS but I expect it to be as unproductive as usual. In this type of issue, there is often a friendly Engineer keeping an eye on NANOG list or IRC channel who can mitigate it. Hoping that to be the case this time, also.

> On 26 Jan 2024, at 08:40, Saku Ytti <saku@ytti.fi> wrote:
>
> On Fri, 26 Jan 2024 at 10:23, Phil Lavin via NANOG <nanog@nanog.org> wrote:
>
>
>> 88.99.88.67 to 216.147.3.209:
>> Host Loss% Snt Last Avg Best Wrst StDev
>> 1. 10.88.10.254 0.0% 176 0.2 0.1 0.1 0.3 0.1
>> 7. nug-b1-link.ip.twelve99.net 0.0% 176 3.3 3.5 3.1 24.1 1.6
>> 8. hbg-bb2-link.ip.twelve99.net 86.9% 175 18.9 18.9 18.7 19.2 0.1
>> 9. ldn-bb2-link.ip.twelve99.net 92.0% 175 30.5 30.6 30.4 30.8 0.1
>> 10. nyk-bb1-link.ip.twelve99.net 4.6% 175 99.5 99.5 99.3 100.1 0.2
>> 11. sjo-b23-link.ip.twelve99.net 56.3% 175 296.8 306.0 289.7 315.0 5.5
>> 12. amazon-ic-366608.ip.twelve99-cust.net 80.5% 175 510.0 513.5 500.7 539.7 8.4
>
> This implies the problem is not on this path, because #10 is not
> experiencing it, possibly because it happens to return a packet via
> another option, but certainly shows the problem didn't happen in this
> direction yet at #10, but because #8 and #9 saw it, they already saw
> it on the other direction.
>
>
>> 44.236.47.236 to 178.63.26.145:
>> Host Loss% Snt Last Avg Best Wrst StDev
>> 1. ip-10-96-50-153.us-west-2.compute.internal 0.0% 267 0.2 0.2 0.2 0.4 0.0
>> 11. port-b3-link.ip.twelve99.net 0.0% 267 5.8 5.9 5.6 11.8 0.5
>> 12. palo-b24-link.ip.twelve99.net 4.9% 267 21.1 21.5 21.0 58.4 3.1
>> 13. sjo-b23-link.ip.twelve99.net 0.0% 266 21.4 22.7 21.3 86.2 6.5
>> 14. nyk-bb1-link.ip.twelve99.net 58.1% 266 432.7 422.7 407.2 438.5 6.5
>> 15. ldn-bb2-link.ip.twelve99.net 98.1% 266 485.6 485.4 481.6 491.1 3.9
>> 16. hbg-bb2-link.ip.twelve99.net 92.5% 266 504.1 499.8 489.8 510.1 5.9
>> 17. nug-b1-link.ip.twelve99.net 55.5% 266 523.5 519.6 504.4 561.7 7.6
>> 18. hetzner-ic-340780.ip.twelve99-cust.net 53.6% 266 524.4 519.2 506.0 545.5 6.9
>> 19. core22.fsn1.hetzner.com 70.2% 266 521.7 519.2 498.5 531.7 6.6
>> 20. static.213-239-254-150.clients.your-server.de 33.2% 266 382.4 375.4 364.9 396.5 4.1
>> 21. static.145.26.63.178.clients.your-server.de 62.0% 266 529.9 518.4 506.9 531.3 6.1
>
> This suggests the congestion point is from sjo to nyk, in 1299, not AWS at all.
>
> You could try to fix SPORT/DPORT, and do several SPORT options, to see
> if loss goes away with some, to determine if all LAG members are full
> or just one.
>
>
> At any rate, this seems business as usual, sometimes internet is very
> lossy, you should contact your service provider, which I guess is AWS
> here, so they can contact their service provider or 1299.
>
> --
> ++ytti

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: Twelve99 / AWS usw2 significant loss [ In reply to ]

outages at outages

Jan 26, 2024, 12:59 AM

Post #3 of 4 (83 views)

Permalink

I see the AWS->me path has just changed to Cogent. me->AWS path still appears to be Twelve99. Loss and latency has now subsided.

Thanks

> On 26 Jan 2024, at 08:19, Phil Lavin <phil.lavin@vonage.com> wrote:
>
> Hi Folks,
>
> Cross posting to outages and nanog.
>
> Can anybody from Twelve99 / AWS investigate high loss and latency around SJO between Twelve99 and AWS?
>
> Traces below:
>
> 88.99.88.67 to 216.147.3.209:
>
> My traceroute [v0.94]
> phil (10.88.10.10) -> 216.147.3.209 2024-01-26T08:16:16+0000
> Keys: Help Display mode Restart statistics Order of fields quit
> Packets Pings
> Host Loss% Snt Last Avg Best Wrst StDev
> 1. 10.88.10.254 0.0% 176 0.2 0.1 0.1 0.3 0.1
> 2. fe-fw-01.lavin.me.uk 0.0% 176 0.4 0.4 0.2 0.7 0.1
> 3. 10.69.69.1 0.0% 176 0.5 0.4 0.2 0.7 0.1
> 4. static.225.8.4.46.clients.your-server.de 0.0% 176 0.8 1.0 0.5 8.2 1.0
> 5. core22.fsn1.hetzner.com 0.6% 176 0.8 2.0 0.7 39.0 4.6
> 6. static.213-239-254-234.clients.your-server.de 0.0% 176 3.2 3.8 3.1 36.5 3.3
> 7. nug-b1-link.ip.twelve99.net 0.0% 176 3.3 3.5 3.1 24.1 1.6
> 8. hbg-bb2-link.ip.twelve99.net 86.9% 175 18.9 18.9 18.7 19.2 0.1
> 9. ldn-bb2-link.ip.twelve99.net 92.0% 175 30.5 30.6 30.4 30.8 0.1
> 10. nyk-bb1-link.ip.twelve99.net 4.6% 175 99.5 99.5 99.3 100.1 0.2
> 11. sjo-b23-link.ip.twelve99.net 56.3% 175 296.8 306.0 289.7 315.0 5.5
> 12. amazon-ic-366608.ip.twelve99-cust.net 80.5% 175 510.0 513.5 500.7 539.7 8.4
> 13. (waiting for reply)
> …...
> 26. 216.147.3.209 59.1% 150 534.2 534.4 513.5 549.7 8.1
>
> 44.236.47.236 to 178.63.26.145:
>
> jcasc-rtc01 (10.96.54.240) -> hc-ping.com 2024-01-26T08:10:24+0000
> Keys: Help Display mode Restart statistics Order of fields quit
> Packets Pings
> Host Loss% Snt Last Avg Best Wrst StDev
> 1. ip-10-96-50-153.us-west-2.compute.internal 0.0% 267 0.2 0.2 0.2 0.4 0.0
> 2. ec2-34-221-151-225.us-west-2.compute.amazonaw 0.0% 267 6.8 15.9 0.7 114.1 24.1
> 3. (waiting for reply)
> 4. (waiting for reply)
> 5. (waiting for reply)
> 6. 241.0.2.5 0.0% 267 0.4 0.4 0.3 1.2 0.1
> 7. 242.0.42.195 0.0% 267 1.2 1.7 0.5 23.9 3.0
> 8. 240.2.144.12 0.0% 267 5.2 5.2 5.2 27.4 1.4
> 9. amazon-ic-375418.ip.twelve99-cust.net 0.0% 267 5.3 5.2 5.2 6.1 0.1
> 10. port-b4-link.ip.twelve99.net 0.0% 267 5.3 5.5 4.9 35.5 3.0
> 11. port-b3-link.ip.twelve99.net 0.0% 267 5.8 5.9 5.6 11.8 0.5
> 12. palo-b24-link.ip.twelve99.net 4.9% 267 21.1 21.5 21.0 58.4 3.1
> 13. sjo-b23-link.ip.twelve99.net 0.0% 266 21.4 22.7 21.3 86.2 6.5
> 14. nyk-bb1-link.ip.twelve99.net 58.1% 266 432.7 422.7 407.2 438.5 6.5
> 15. ldn-bb2-link.ip.twelve99.net 98.1% 266 485.6 485.4 481.6 491.1 3.9
> 16. hbg-bb2-link.ip.twelve99.net 92.5% 266 504.1 499.8 489.8 510.1 5.9
> 17. nug-b1-link.ip.twelve99.net 55.5% 266 523.5 519.6 504.4 561.7 7.6
> 18. hetzner-ic-340780.ip.twelve99-cust.net 53.6% 266 524.4 519.2 506.0 545.5 6.9
> 19. core22.fsn1.hetzner.com 70.2% 266 521.7 519.2 498.5 531.7 6.6
> 20. static.213-239-254-150.clients.your-server.de 33.2% 266 382.4 375.4 364.9 396.5 4.1
> 21. static.145.26.63.178.clients.your-server.de 62.0% 266 529.9 518.4 506.9 531.3 6.1
>
>
> Thanks

_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: Twelve99 / AWS usw2 significant loss [ In reply to ]

outages at outages

Jan 26, 2024, 4:43 PM

Post #4 of 4 (79 views)

Permalink

Seems like the destination is in Hetzner, they could also raise it with
Twelve99 or prepend routes to use an alternate path.

On Fri, Jan 26, 2024 at 7:46?PM Phil Lavin via Outages <outages@outages.org>
wrote:

> Thanks, ytti. I have raised a case with AWS but I expect it to be as
> unproductive as usual. In this type of issue, there is often a friendly
> Engineer keeping an eye on NANOG list or IRC channel who can mitigate it.
> Hoping that to be the case this time, also.
>
>
> > On 26 Jan 2024, at 08:40, Saku Ytti <saku@ytti.fi> wrote:
> >
> > On Fri, 26 Jan 2024 at 10:23, Phil Lavin via NANOG <nanog@nanog.org>
> wrote:
> >
> >
> >> 88.99.88.67 to 216.147.3.209:
> >> Host Loss% Snt Last
> Avg Best Wrst StDev
> >> 1. 10.88.10.254 0.0% 176 0.2
> 0.1 0.1 0.3 0.1
> >> 7. nug-b1-link.ip.twelve99.net 0.0% 176 3.3
> 3.5 3.1 24.1 1.6
> >> 8. hbg-bb2-link.ip.twelve99.net 86.9% 175 18.9
> 18.9 18.7 19.2 0.1
> >> 9. ldn-bb2-link.ip.twelve99.net 92.0% 175 30.5
> 30.6 30.4 30.8 0.1
> >> 10. nyk-bb1-link.ip.twelve99.net 4.6% 175
> 99.5 99.5 99.3 100.1 0.2
> >> 11. sjo-b23-link.ip.twelve99.net 56.3% 175 296.8
> 306.0 289.7 315.0 5.5
> >> 12. amazon-ic-366608.ip.twelve99-cust.net 80.5% 175 510.0
> 513.5 500.7 539.7 8.4
> >
> > This implies the problem is not on this path, because #10 is not
> > experiencing it, possibly because it happens to return a packet via
> > another option, but certainly shows the problem didn't happen in this
> > direction yet at #10, but because #8 and #9 saw it, they already saw
> > it on the other direction.
> >
> >
> >> 44.236.47.236 to 178.63.26.145:
> >> Host Loss% Snt Last
> Avg Best Wrst StDev
> >> 1. ip-10-96-50-153.us-west-2.compute.internal 0.0% 267 0.2
> 0.2 0.2 0.4 0.0
> >> 11. port-b3-link.ip.twelve99.net 0.0% 267 5.8
> 5.9 5.6 11.8 0.5
> >> 12. palo-b24-link.ip.twelve99.net 4.9% 267 21.1
> 21.5 21.0 58.4 3.1
> >> 13. sjo-b23-link.ip.twelve99.net 0.0% 266 21.4
> 22.7 21.3 86.2 6.5
> >> 14. nyk-bb1-link.ip.twelve99.net 58.1% 266 432.7
> 422.7 407.2 438.5 6.5
> >> 15. ldn-bb2-link.ip.twelve99.net 98.1% 266 485.6
> 485.4 481.6 491.1 3.9
> >> 16. hbg-bb2-link.ip.twelve99.net 92.5% 266 504.1
> 499.8 489.8 510.1 5.9
> >> 17. nug-b1-link.ip.twelve99.net 55.5% 266 523.5
> 519.6 504.4 561.7 7.6
> >> 18. hetzner-ic-340780.ip.twelve99-cust.net 53.6% 266 524.4
> 519.2 506.0 545.5 6.9
> >> 19. core22.fsn1.hetzner.com 70.2% 266 521.7
> 519.2 498.5 531.7 6.6
> >> 20. static.213-239-254-150.clients.your-server.de 33.2% 266 382.4
> 375.4 364.9 396.5 4.1
> >> 21. static.145.26.63.178.clients.your-server.de 62.0% 266 529.9
> 518.4 506.9 531.3 6.1
> >
> > This suggests the congestion point is from sjo to nyk, in 1299, not AWS
> at all.
> >
> > You could try to fix SPORT/DPORT, and do several SPORT options, to see
> > if loss goes away with some, to determine if all LAG members are full
> > or just one.
> >
> >
> > At any rate, this seems business as usual, sometimes internet is very
> > lossy, you should contact your service provider, which I guess is AWS
> > here, so they can contact their service provider or 1299.
> >
> > --
> > ++ytti
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>