Mailing List Archive: Link down affecting BGP peer

Link down affecting BGP peer

May 5, 2022, 9:04 AM

Post #1 of 8 (1101 views)

I have 4 individual links defined as part of a Bundle-ether (IOS-XR
5.3.3 on ASR9010):

interface TenGigE0/2/0/1
bundle id 2 mode active
flow-control bidirectional
carrier-delay up 100 down 4000
! They are all part of a bundle...
interface Bundle-Ether2
mtu 9192
bundle minimum-active links 2

When I shut off just 1 of these 4 links - the bundle stays up yet
certain BGP sessions flap for about 5 seconds - different peers
depending on which of the 4 links gets turned down.

My BGP config:
router bgp 378
rpki server x.139.197.151
transport tcp port 8282
refresh-time 600
!
bgp log neighbor changes detail
address-family ipv4 unicast
bgp dampening 5 750 3000 10
bgp attribute-download
!
neighbor x.x.125.1
remote-as xxxx5
address-family ipv4 unicast
send-community-ebgp
soft-reconfiguration inbound

What could be causing the bgp peer to flap even though the LAG stays up?

Thanks,
Hank

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: Link down affecting BGP peer [ In reply to ]

dudepron at gmail

May 5, 2022, 10:46 AM

Post #2 of 8 (1099 views)

Permalink

Are the sessions that bounced hashed to use the failed/turned off link?

On Thu, May 5, 2022 at 12:07 PM Hank Nussbacher <hank@interall.co.il> wrote:

> I have 4 individual links defined as part of a Bundle-ether (IOS-XR
> 5.3.3 on ASR9010):
>
> interface TenGigE0/2/0/1
> bundle id 2 mode active
> flow-control bidirectional
> carrier-delay up 100 down 4000
> ! They are all part of a bundle...
> interface Bundle-Ether2
> mtu 9192
> bundle minimum-active links 2
>
> When I shut off just 1 of these 4 links - the bundle stays up yet
> certain BGP sessions flap for about 5 seconds - different peers
> depending on which of the 4 links gets turned down.
>
> My BGP config:
> router bgp 378
> rpki server x.139.197.151
> transport tcp port 8282
> refresh-time 600
> !
> bgp log neighbor changes detail
> address-family ipv4 unicast
> bgp dampening 5 750 3000 10
> bgp attribute-download
> !
> neighbor x.x.125.1
> remote-as xxxx5
> address-family ipv4 unicast
> send-community-ebgp
> soft-reconfiguration inbound
>
> What could be causing the bgp peer to flap even though the LAG stays up?
>
> Thanks,
> Hank
>
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: Link down affecting BGP peer [ In reply to ]

mark at tinka

May 5, 2022, 10:04 PM

Post #3 of 8 (1096 views)

Permalink

On 5/5/22 18:04, Hank Nussbacher wrote:

>
> What could be causing the bgp peer to flap even though the LAG stays up?

Could be that the session flows are hashed to the 1 member link that you
fail.

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: Link down affecting BGP peer [ In reply to ]

tedpelas at gmail

May 5, 2022, 10:04 PM

Post #4 of 8 (1096 views)

Permalink

Try configuring MicroBFD/BoB rather than using Logic bundle/BLB, see more details at https://community.cisco.com/t5/service-providers-blogs/bfd-over-logical-bundle-blb-implementation-on-ncs5500-platforms/ba-p/3309345

/Ted

> On 5 May 2022, at 18:07, Hank Nussbacher <hank@interall.co.il> wrote:
>
> I have 4 individual links defined as part of a Bundle-ether (IOS-XR 5.3.3 on ASR9010):
>
> interface TenGigE0/2/0/1
> bundle id 2 mode active
> flow-control bidirectional
> carrier-delay up 100 down 4000
> ! They are all part of a bundle...
> interface Bundle-Ether2
> mtu 9192
> bundle minimum-active links 2
>
> When I shut off just 1 of these 4 links - the bundle stays up yet certain BGP sessions flap for about 5 seconds - different peers depending on which of the 4 links gets turned down.
>
> My BGP config:
> router bgp 378
> rpki server x.139.197.151
> transport tcp port 8282
> refresh-time 600
> !
> bgp log neighbor changes detail
> address-family ipv4 unicast
> bgp dampening 5 750 3000 10
> bgp attribute-download
> !
> neighbor x.x.125.1
> remote-as xxxx5
> address-family ipv4 unicast
> send-community-ebgp
> soft-reconfiguration inbound
>
> What could be causing the bgp peer to flap even though the LAG stays up?
>
> Thanks,
> Hank
>
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: Link down affecting BGP peer [ In reply to ]

mark at tinka

May 6, 2022, 5:22 AM

Post #5 of 8 (1095 views)

Permalink

On 5/6/22 07:04, Ted Pelas Johansson wrote:

> Try configuring MicroBFD/BoB rather than using Logic bundle/BLB, see more details at https://community.cisco.com/t5/service-providers-blogs/bfd-over-logical-bundle-blb-implementation-on-ncs5500-platforms/ba-p/3309345

Is this a p2p or LAN?

Mark.
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: Link down affecting BGP peer [ In reply to ]

ikiris at gmail

May 18, 2022, 7:36 AM

Post #6 of 8 (971 views)

Permalink

Is it about 4000 msec of flapping? Perhaps the bundle member in question is
delaying to go down for some reason?

On Thu, May 5, 2022, 11:07 Hank Nussbacher <hank@interall.co.il> wrote:

> I have 4 individual links defined as part of a Bundle-ether (IOS-XR
> 5.3.3 on ASR9010):
>
> interface TenGigE0/2/0/1
> bundle id 2 mode active
> flow-control bidirectional
> carrier-delay up 100 down 4000
> ! They are all part of a bundle...
> interface Bundle-Ether2
> mtu 9192
> bundle minimum-active links 2
>
> When I shut off just 1 of these 4 links - the bundle stays up yet
> certain BGP sessions flap for about 5 seconds - different peers
> depending on which of the 4 links gets turned down.
>
> My BGP config:
> router bgp 378
> rpki server x.139.197.151
> transport tcp port 8282
> refresh-time 600
> !
> bgp log neighbor changes detail
> address-family ipv4 unicast
> bgp dampening 5 750 3000 10
> bgp attribute-download
> !
> neighbor x.x.125.1
> remote-as xxxx5
> address-family ipv4 unicast
> send-community-ebgp
> soft-reconfiguration inbound
>
> What could be causing the bgp peer to flap even though the LAG stays up?
>
> Thanks,
> Hank
>
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: Link down affecting BGP peer [ In reply to ]

hank at interall

May 19, 2022, 12:21 AM

Post #7 of 8 (969 views)

Permalink

On 18/05/2022 17:36, Blake Dunlap wrote:

Others have explained this. Basically, a BGP peer gets locked onto one
of the LAG links and will migrate to another link in the event that the
specific link goes down. This is normal behavior.

-Hank

> Is it about 4000 msec of flapping? Perhaps the bundle member in question
> is delaying to go down for some reason?
>
> On Thu, May 5, 2022, 11:07 Hank Nussbacher <hank@interall.co.il
> <mailto:hank@interall.co.il>> wrote:
>
> I have 4 individual links defined as part of a Bundle-ether (IOS-XR
> 5.3.3 on ASR9010):
>
> interface TenGigE0/2/0/1
>   bundle id 2 mode active
>   flow-control bidirectional
>   carrier-delay up 100 down 4000
> ! They are all part of a bundle...
> interface Bundle-Ether2
>   mtu 9192
>   bundle minimum-active links 2
>
> When I shut off just 1 of these 4 links - the bundle stays up yet
> certain BGP sessions flap for about 5 seconds - different peers
> depending on which of the 4 links gets turned down.
>
> My BGP config:
> router bgp 378
>   rpki server x.139.197.151
>    transport tcp port 8282
>    refresh-time 600
>   !
>   bgp log neighbor changes detail
>   address-family ipv4 unicast
>    bgp dampening 5 750 3000 10
>    bgp attribute-download
> !
>   neighbor x.x.125.1
>    remote-as xxxx5
>    address-family ipv4 unicast
>     send-community-ebgp
>     soft-reconfiguration inbound
>
> What could be causing the bgp peer to flap even though the LAG stays up?
>
> Thanks,
> Hank
>
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> <mailto:cisco-nsp@puck.nether.net>
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> <https://puck.nether.net/mailman/listinfo/cisco-nsp>
> archive at http://puck.nether.net/pipermail/cisco-nsp/
> <http://puck.nether.net/pipermail/cisco-nsp/>
>

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: Link down affecting BGP peer [ In reply to ]

saku at ytti

May 19, 2022, 3:30 AM

Post #8 of 8 (969 views)

Permalink

On Thu, 19 May 2022 at 10:27, Hank Nussbacher <hank@interall.co.il> wrote:

> Others have explained this. Basically, a BGP peer gets locked onto one
> of the LAG links and will migrate to another link in the event that the
> specific link goes down. This is normal behavior.

I'm not sure about normal behaviour and certainly objectively broken.

Even though ultimately some physical interfaces serialise those BGP
packets out, the fast external fallover should be tracking the
aggregate interface, not some member. What should happen when a member
comes down is that the hash=>interface table has one interface less,
so the packet is now hashed out to some of the remaining interfaces.

We can accept flaps if we don't know the physical interface is down,
while it is down. Like if the carrier-delay down is higher than bgp
keepalive or if the interface is blackholing for whatever reason.
Other than that, no, it's broken.

--
++ytti
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/