Yesterday evening, today again.
[quote]
"We have identified what seems to be the sequence of events. We are
currently working on confirming the hypothesis in the lab, while the
platform is stabilised, before we proceed to any further changes/actions.
We have confirmed that Juniper propagated LACP packets from a customer
to the rest of the platform. Of course, this shouldn't happen which
points to a bug. This causes customer LACP LAGs to be torn down and,
potential, pseudowires to get destroyed and rebuild.
In consequence, this leads to full buffers and resource starvation which
leads to RSVP messages timeout/errors. Then, RSVP PathError messages are
sent (aggressively) and trigger another Juniper bug, which sends
PathError messages to both the head-end PEs and new RSVP Path messages
to the tail-end PEs, without any back-off timeouts.
As a cascade effect, this causes issues to SLXes as well."
Update 15:20:
We have finished the call with our vendor and we can confirm that the
root cause is indeed triggered by (wrong) propagation of LACP packets
from Juniper PE equipment.
As a mitigation solution, we will be deploying ACL entries to filter out
LACP packets on all non-LACP interfaces on Juniper boxes.
At the same time, we will reboot Juniper core-glo-205, to refresh its
runtime state and proceed will load balancing all traffic to both cores.
Please note that the platform is currently stable and we are working on
installing the failsafes to avoid any reoccurrence.
[/quote]
On 23-11-2023 15:57, Eric Kuhnke wrote:
> https://www.ams-ix.net/ams/documentation/total-stats
>
> https://www.ams-ix.net/ams/outage-on-amsterdam-peering-platform
>
>