Mailing List Archive: ARP Pending Entries queue being maxed out

ARP Pending Entries queue being maxed out

warwick at frogfoot

Sep 9, 2014, 3:45 AM

Post #1 of 7 (6822 views)

Hello all

We have a client with a two unit X450 stack with about 400 layer 3 VLAN
interfaces and we are finding that the ARP Pending Entries queue
increases to whatever the upper limit is set to. We are also seeing
that an apparently random selection of hosts connected to the switches
becomes occasionally becomes unreachable, which would make sense given
that the switch is unable to resolve ARP for them.

This started being noticable at around 350 layer 3 VLAN but increasing
the Pending Entries limit seemed to keep it in check for a while,
although I guess that might have just lowered the number of complaints.

Does anyone know if there is some known limitation with the X450 or XOS
12.4.1.7 that can cause this behaviour? Can the issue be resolved by
using a different firmware version?

Any other ideas for mitigating this problem?

The output of 'show iparp" gives the following statistics:
==========================================================
Dynamic Entries : 347 Static Entries : 0
Pending Entries : 2048
In Request : 34990997 In Response : 590791
Out Request : 31413049 Out Response : 20684269
Failed Requests : 5137454
Proxy Answered : 2583974
Rx Error : 4 Dup IP Addr : a.b.c.177
Rejected Count : 485915 Rejected IP : 169.254.135.15
Rejected Port : 1:21 Rejected IF : CLIENT134

Max ARP entries : 8192 Max ARP pending entries : 2048
ARP address check: Enabled ARP refresh : Enabled
Timeout : 20 minutes ARP Sender-Mac Learning : Disabled
==========================================================

Regards
Warwick

--
Warwick Duncan
Frogfoot Networks ISP
http://www.frogfoot.com/
+27.21.448.7225
_______________________________________________
extreme-nsp mailing list
extreme-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/extreme-nsp

Re: ARP Pending Entries queue being maxed out [ In reply to ]

jarek.kasjaniuk at gmail

Sep 9, 2014, 8:42 AM

Post #2 of 7 (6738 views)

Hello

did you try to increase arp timeout?
Maybe you can add static arp?

Regards
--
Jarek Kasjaniuk
(sent from mobile phone)

Re: ARP Pending Entries queue being maxed out [ In reply to ]

Sep 10, 2014, 12:10 AM

Post #3 of 7 (6743 views)

On Tue, 9 Sep 2014, Warwick Duncan wrote:

> Hello all
>
> We have a client with a two unit X450 stack with about 400 layer 3 VLAN
> interfaces and we are finding that the ARP Pending Entries queue
> increases to whatever the upper limit is set to. We are also seeing
> that an apparently random selection of hosts connected to the switches
> becomes occasionally becomes unreachable, which would make sense given
> that the switch is unable to resolve ARP for them.
>
> This started being noticable at around 350 layer 3 VLAN but increasing
> the Pending Entries limit seemed to keep it in check for a while,
> although I guess that might have just lowered the number of complaints.
>
> Does anyone know if there is some known limitation with the X450 or XOS
> 12.4.1.7 that can cause this behaviour? Can the issue be resolved by
> using a different firmware version?
>
> Any other ideas for mitigating this problem?

How many active and pending arp entries do you have?

--
Mikael Abrahamsson email: swmike@swm.pp.se
_______________________________________________
extreme-nsp mailing list
extreme-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/extreme-nsp

Re: ARP Pending Entries queue being maxed out [ In reply to ]

warwick at frogfoot

Sep 10, 2014, 2:53 AM

Post #4 of 7 (6755 views)

Hi Jarek

On Tue, Sep 09, 2014 at 05:42:10PM +0200, Jaros??aw Kasjaniuk wrote:
> did you try to increase arp timeout?

We haven't tried it but it is one of the few possibilities that we came
up with. It's good to hear our thinking be confirmed.

> Maybe you can add static arp?

That's something we didn't consider, we'll definitely give it some
thought.

Thanks!

Regards
Warwick

--
Warwick Duncan
Frogfoot Networks ISP
http://www.frogfoot.com/
+27.21.448.7225
_______________________________________________
extreme-nsp mailing list
extreme-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/extreme-nsp

Re: ARP Pending Entries queue being maxed out [ In reply to ]

warwick at frogfoot

Sep 10, 2014, 2:55 AM

Post #5 of 7 (6749 views)

Hi Mikael

On Wed, Sep 10, 2014 at 09:10:22AM +0200, Mikael Abrahamsson wrote:
[..]
> How many active and pending arp entries do you have?

The last set of numbers we pulled was
- Dynamic Entries : 347
- Static Entries : 0
- Pending Entries : 2048

We've increased the maximum allowed value for Pending Entries as high as
it can go.

Regards
Warwick

--
Warwick Duncan
Frogfoot Networks ISP
http://www.frogfoot.com/
+27.21.448.7225
_______________________________________________
extreme-nsp mailing list
extreme-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/extreme-nsp

Re: ARP Pending Entries queue being maxed out [ In reply to ]

Sep 10, 2014, 3:37 AM

Post #6 of 7 (6749 views)

On Wed, 10 Sep 2014, Warwick Duncan wrote:

> Hi Mikael
>
> On Wed, Sep 10, 2014 at 09:10:22AM +0200, Mikael Abrahamsson wrote:
> [..]
>> How many active and pending arp entries do you have?
>
> The last set of numbers we pulled was
> - Dynamic Entries : 347
> - Static Entries : 0
> - Pending Entries : 2048
>
> We've increased the maximum allowed value for Pending Entries as high as
> it can go.

How big are your subnets? From the look of it, it sounds like you have
fairly large subnets with traffic destined for IPs that do not exist
(scanning traffic?). This has been a problem before, I know people 10
years ago at Interop used large subnets out of the 45.0.0.0/8 they had,
and they had to lessen the size of the subnets because of scanning traffic
making the ARP engine in ExOS go haywire. So either try to remove this
unsolicited traffic to IPs not in use, or make the subnets smaller.

Either you do this, or you lower the pending timeout, I don't know if this
is possible, I don't have access to the gear you're using.

--
Mikael Abrahamsson email: swmike@swm.pp.se
_______________________________________________
extreme-nsp mailing list
extreme-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/extreme-nsp

Re: ARP Pending Entries queue being maxed out [ In reply to ]

warwick at frogfoot

Sep 10, 2014, 5:07 AM

Post #7 of 7 (6746 views)

Hi Mikael

On Wed, Sep 10, 2014 at 12:37:29PM +0200, Mikael Abrahamsson wrote:
> How big are your subnets? From the look of it, it sounds like you

There are ~300 directly attached /24s plus another ~150 smaller that are
mostly /29 and /30.

> have fairly large subnets with traffic destined for IPs that do not
> exist (scanning traffic?). This has been a problem before, I know
> people 10 years ago at Interop used large subnets out of the
> 45.0.0.0/8 they had, and they had to lessen the size of the subnets
> because of scanning traffic making the ARP engine in ExOS go
> haywire. So either try to remove this unsolicited traffic to IPs not
> in use, or make the subnets smaller.

That makes a lot of sense.

> Either you do this, or you lower the pending timeout, I don't know
> if this is possible, I don't have access to the gear you're using.

As far as I can tell it isn't, we can only modify the timeout for cached
entries in this version of XOS (12.4.1.7). More recent versions seem to
have a few more options in this regard but we can't upgrade this
hardware much beyond where it already is.

Regards
Warwick

--
Warwick Duncan
Frogfoot Networks ISP
http://www.frogfoot.com/
+27.21.448.7225
_______________________________________________
extreme-nsp mailing list
extreme-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/extreme-nsp