Thanks Joseph and Eldon. That seems to be the issue.
On the XMR in question, I currently have 126 IPs listed in the default-vrf. The others have even more.
I am wondering though. Let's say I had 29 IPs on my box with this rACL. Then provisioning add three more customers. What happens? That would take you over the limit. Would the entire rACL be removed or just not applied to the three new interfaces? Then again, if the box was rebooted what would then happen?
Thanks
Darren
http://www.mellowd.co.uk/ccie From: Joseph.Kennedy@purchase.edu
To: darrenoc@outlook.com
CC: esk-puck.nether.net@esk.cs.usu.edu; foundry-nsp@puck.nether.net
Subject: Re: [f-nsp] exceed configured CAM size, larger partition size required
Date: Sun, 26 Jan 2014 19:51:40 +0000
You’ll want to do a "show cam-partition usage” and look under the [Session] section to see the "Receive ACL" CAM usage. The more interface IP’s you have on your XMR16, the more you'll need to increase your receive-cam value accordingly. Rule-based ACL CAM
is not affected by rACL statements. The two are taken out of the same overall pool but they are exclusive.
Eldon is correct. Every rACL entry you specify is being added to CAM for every interface IP address as a destination. You can calculate your required receive-cam usage as approximately (rACL size) x (total number of interface IP’s). You can count entries
from "show ip interfaces | inc default-vrf” and multiply by 33. Anything over 30 interface IP's on your XMR16 would mean you have to increase your receive-cam value to at least 2048. Yes VE’s and loopbacks count. Based on IP address not physical interface
per-se.
Also see the note from the guide(since you said you’re using the Multi-service 4 cam-profile) just incase you have an exceptionally large number of interface IP’s or plan to increase your rACL size considerably.
From the 5.4.00a Config Guide:
NOTES: The following limitations apply when the <number> variable has a maximum limit of 16384.
• The 16K Receive ACL CAM partition is not supported on the cam profiles such as IPv6,
Multi-service 3, and Multi-service 4.
• Depending on the configuration, any of the IPv4 ACL sub-partitions such as IP Source
Guard, Broadcast ACL, IP Multicast, and OpenFlow should be decreased to allow the
creation of the 16K rACL partition.
--JK
On Jan 26, 2014, at 11:00 AM, Darren O'Connor <darrenoc@outlook.com> wrote:
Hi all. I'll answer all three questions altogether.
All line cards are the same. It's a mixture of copper and sfp 20 and 24 port modules, as well as a mix of 2X10Gb and 4X10Gb cards. All XMR
The receive-cam values are currently the same on all our XMRs:
show default values | include receive-cam
receive-cam 1024 16384 1024 1024 1024 No
The scrubbed ACLs in question are as follows:
access-list 180 permit ospf any any
access-list 180 permit vrrp any any
access-list 180 permit tcp any any eq bgp
access-list 180 permit tcp any any eq 3784
access-list 180 permit tcp any any eq 3785
access-list 180 permit udp any any eq 3784
access-list 180 permit udp any any eq 3785
access-list 180 permit tcp any any eq 639
access-list 180 permit esp any any
access-list 180 permit mpls-in-ip any any
access-list 180 permit rsvp any any
access-list 180 permit pim any any
access-list 180 permit tcp any any eq 646
access-list 180 permit udp any any eq 646
access-list 180 permit tcp any any eq 646
access-list 180 permit udp any any eq 646
access-list 180 permit tcp any eq 646 any
access-list 180 permit udp any eq 646 any
!
access-list 181 permit icmp any any
!
access-list 182 deny ip any any
!
access-list 199 permit ip 1.1.106.128 0.0.0.7 any
access-list 199 permit ip 1.1.106.152 0.0.0.7 any
access-list 199 permit ip 1.1.111.80 0.0.0.15 any
access-list 199 permit ip 1.1.90.224 0.0.0.31 any
access-list 199 permit ip 1.1.107.104 0.0.0.7 any
access-list 199 permit ip 1.1.94.240 0.0.0.15 any
access-list 199 permit ip 1.1.224.128 0.0.0.31 any
access-list 199 permit ip 1.1.225.128 0.0.0.31 any
access-list 199 permit ip 1.1.226.0 0.0.0.127 any
access-list 199 permit ip 1.1.239.240 0.0.0.15 any
access-list 199 permit ip 1.1.24.96 0.0.0.15 any
access-list 199 permit ip 1.1.22.32 0.0.0.31 any
access-list 199 permit ip host 1.1.60.78 any
policy-map 1m-management
cir 1042432 cbs 1250
ip receive access-list 199 sequence 1
ip receive access-list 181 sequence 10 policy-map 1m-management strict-acl
ip receive access-list 180 sequence 20
ip receive access-list 182 sequence 200
There is but a handful of ACLs applied to an interface here and there. Hardly any. The above ACLs are to protect the XMR CPU to only allow what is needed to reach the CPU of course.
Thanks
Darren
http://www.mellowd.co.uk/ccie > From: Joseph.Kennedy@purchase.edu
> To: esk-puck.nether.net@esk.cs.usu.edu; darrenoc@outlook.com
> CC: foundry-nsp@puck.nether.net
> Subject: RE: [f-nsp] exceed configured CAM size, larger partition size required
> Date: Fri, 24 Jan 2014 21:40:03 +0000
>
> IP receive cam is for rACL's aka ip receive ACL's. Basically you would use the "ip receive ..." command to bind an ACL to all your interfaces in the default VRF and limit traffic directed to those interfaces . If you look at your cam-partition usage it shouldn't
use more than a single entry unless you've explicitly configured it. If you aren't using the ACL in question for transit traffic but rather to protect your router IP's I would say you should switch to rACL's (unless you are running VRF's). You might even consider
cam-sharing if the restrictions don't apply to you(like doesn't work with ve's).
>
> Darren, do you mind showing us your ACL(with IP ranges omitted or changed if necessary) and your current cam-partition usage for a typical line card with XMR4 vs XMR16? You can limit the usage stats to the [Session] sections.
>
> --JK
>
> -----Original Message-----
> From: foundry-nsp [mailto:foundry-nsp-bounces@puck.nether.net] On Behalf Of Eldon Koyle
> Sent: Friday, January 24, 2014 3:54 PM
> To: 'Darren O'Connor'
> Cc: foundry-nsp@puck.nether.net
> Subject: Re: [f-nsp] exceed configured CAM size, larger partition size required
>
> You might check the receive-cam value. It is documented in the security guide. The default is 1024 on the MLX, not sure if it is different for XMR. It says this is "the maximum number of ACL CAM entries that are allowed"; 1024 sounds a bit low if you are
applying an ACL on every interface.
>
> Assuming that each line in an ACL consumes one such entry and each interface requires its own copy of the ACL, you could apply a 20-line ACL to 51 interfaces before you ran out. I could be interpreting this wrong, though.
>
> MLXe8-M# show default values | inc receive-cam
> receive-cam 1024 16384 1024 1024 1024 No
>
> --
> Eldon Koyle
> --
> BOFH excuse #350:
> paradigm shift...without a clutch
>
>
> On Jan 24 19:20+0000, Kennedy, Joseph wrote:
> > Are you using the same line cards in the XMR4 as you are in the XMR8's and XMR16's?
> >
> > Are you using tcp/udp port ranges in the ACL in question? (I believe
> > every tcp/udp port in the range may require its own CAM entry)
> >
> > --JK
> >
> > -----Original Message-----
> > From: foundry-nsp [mailto:foundry-nsp-bounces@puck.nether.net] On
> > Behalf Of Darren O'Connor
> > Sent: Friday, January 24, 2014 1:05 PM
> > To: Eldon Koyle
> > Cc: foundry-nsp@puck.nether.net
> > Subject: Re: [f-nsp] exceed configured CAM size, larger partition size
> > required
> >
> > ACL applied is only 20 lines long. I have just a handful of ACLs applied elsewhere.
> >
> > yes the XMR16 has a load more ports, but that should not cause an issue due to the tiny ACLs I'm using.
> >
> > same tcam profile used on all boxes: CAM partitioning profile:
> > multi-service-4
> >
> > system max:
> >
> > sh run | inc system-max
> > system-max vlan 4095
> > system-max ip-cache 768000
> > system-max ip-route 768000
> > system-max virtual-interface 4095
> > system-max ipv6-cache 32000
> > system-max ipv6-route 32000
> > system-max lsp-out-acl-cam 1000
> >
> >
> > Thanks
> > Darren
> > http://www.mellowd.co.uk/ccie
> >
> >
> >
> >
> > > Date: Fri, 24 Jan 2014 09:58:06 -0700
> > > From: ekoyle@gmail.com
> > > To: darrenoc@outlook.com
> > > CC: foundry-nsp@puck.nether.net
> > > Subject: Re: [f-nsp] exceed configured CAM size, larger partition
> > > size required
> > >
> > > Which cam-partition profile are you using? How long are your ACLs?
> > > I'm guessing your XMR16 has a lot more ports than any of your XMR4s,
> > > so that could explain why you are having issues there.
> > >
> > > --
> > > Eldon Koyle
> > >
> > > On Jan 24 9:01+0000, Darren O'Connor wrote:
> > > > Most interfaces have a single IP, some have 2. No more than that
> > > >
> > > > Thanks
> > > > Darren
> > > > http://www.mellowd.co.uk/ccie
> > > >
> > > >
> > > >
> > > > Subject: Re: [f-nsp] exceed configured CAM size, larger partition
> > > > size required
> > > > From: mtindle@he.net
> > > > Date: Thu, 23 Jan 2014 15:54:25 -0800
> > > > CC: foundry-nsp@puck.nether.net
> > > > To: darrenoc@outlook.com
> > > >
> > > > Check if you have a lot of IP addresses configured on interfaces. The rACL has to be applied for each inbound IP address the router could be listening on. The limited CAM size for rACLs can have an impact if there are a lot of IPs and the ACL is long.
> > > > Regards,Mike
> > > >
> > > > On Jan 23, 2014, at 8:14 AM, Darren O'Connor <darrenoc@outlook.com> wrote:
> > > >
> > > >
> > > > Last weekend we added new receive ACLs to our XMRs. All our XMRs (4, 8, and 16) have identical TCAM profiles set up.
> > > >
> > > > I had applied the new receive ACL to 4 XMR4s with no problems. When applying it to an XMR16 in-band I lost connection to the box. Going through OOB I removed and re-added the ACL. I was shown this error:
> > > >
> > > > Port 16/1, IP Receive ACL 199 exceed configured CAM size, larger partition size required.
> > > > Unbinding IP Receive ACL 199
> > > > Port 2/1, IP Receive ACL 199 exceed configured CAM size, larger partition size required.
> > > > Unbinding IP Receive ACL 199
> > > > Port 9/1, IP Receive ACL 199 exceed configured CAM size, larger partition size required.
> > > > Unbinding IP Receive ACL 199
> > > > Port 6/1, IP Receive ACL 199 exceed configured CAM size, larger partition size required.
> > > > Unbinding IP Receive ACL 199
> > > > Port 5/1, IP Receive ACL 199 exceed configured CAM size, larger partition size required.
> > > > Unbinding IP Receive ACL 199
> > > > Port 3/1, IP Receive ACL 199 exceed configured CAM size, larger partition size required.
> > > > Unbinding IP Receive ACL 199
> > > > Port 1/1, IP Receive ACL 199 exceed configured CAM size, larger partition size required.
> > > > Unbinding IP Receive ACL 199
> > > > Port 4/1, IP Receive ACL 199 exceed configured CAM size, larger partition size required.
> > > > Unbinding IP Receive ACL 199
> > > >
> > > > Port 16/3, IP Receive ACL 180 exceed configured CAM size, larger partition size required.
> > > > Unbinding IP Receive ACL 180
> > > >
> > > >
> > > > Odd, as mentioned all my cam-partitions are identical across all boxes. After this happened I did not try and add it to any other box as it was too disruptive.
> > > >
> > > > Any ideas why I would get this? Currently on 5.4d and was upgrading to 5.4e on the night.
> > > >
> > > >
> > > > Thanks
> > > > Darren
> > > > http://www.mellowd.co.uk/ccie
> > > >
> > > >
> > > > _______________________________________________
> > > > foundry-nsp mailing list
> > > > foundry-nsp@puck.nether.net
> > > > http://puck.nether.net/mailman/listinfo/foundry-nsp
> > > >
> > > > *----------- H U R R I C A N E - E L E C T R I C ---------->>
> > > > | Mike Tindle | Senior Network Engineer |
mtindle@he.net ASN 6939
> > > > | | http://www.he.net | 510-580-4126
> > > > *--------------------------------------------------->>
> > > >
> > > >
> > > >
> > >
> > > > _______________________________________________
> > > > foundry-nsp mailing list
> > > > foundry-nsp@puck.nether.net
> > > > http://puck.nether.net/mailman/listinfo/foundry-nsp
> > >
> >
> >
> > _______________________________________________
> > foundry-nsp mailing list
> > foundry-nsp@puck.nether.net
> > http://puck.nether.net/mailman/listinfo/foundry-nsp
> >
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp@puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp