Mailing List Archive

Hi Gunther,
the problem wasnt just the cam filling, once it was full instead of ageing out
an entry and installing a new entry for the requested route it is matching
anything else in the CAM.

eg
i sent a packet to 10.10.10.10

i have a route 10.10.10.0/24 to go out of port 1 - this is not in the CAM
i have a route 10.10.0.0/16 to go out of port 2 - this is in the CAM

we expect to incur a cache miss and install the /24, instead the cache matches
the /16 and the packet is sent the wrong way


in Kristian's case, he was seeing matches to 0.0.0.0/0 to null0 matching where
no CAM entry was found and the packet was dropped when there was a valid route
in the routing table

Steve


On Mon, 6 Mar 2006, Gunther Stammwitz wrote:

> Hello Kristian,
>
> That sounds horrible! Didn't you open a case with TAC/technical-support?
> Which management module exactly are you using? Is it ironcore or jetcore and
> what are your settings regarding the cam-partition, the supernet-aggregation
> and the ip net-aggregate?
>
>
> I have seen some strange behaviour too on our jetcore-module:
> When enabling the ip supernet aggregate-command which the cpuload jumped
> from 10 to 20 or even 25% so we disabled it again. The interestint thing was
> that the cam was freed up to ~50% or something like that.
>
> The next try was to enable ip net-aggregate and the load jumped from 10% to
> 99 which was rather bad and caused a lot of packet loss and so on
> (Trouble!). The solutions was to use ip net-aggregate X where X is a number
> of let's say 10 or 30 so that the aggregation-process won't try to compres
> the cam every one second which is the default behaviour but rather every 10
> or maybe even 30 seconds.
> This reduced the load to 5% and we can't even notice the aggregation-process
> that should run from time to time. No packetloss no nothing
>
> Our current cam-usage is looking like this:
> #show cam ip 1/1 stat
> CAM IP statistics: free entries total entries
> level1: 8670 28672
> level2: 1147 2048
> level3: 1960 2047
> I'm not really sure that it was before but I think to remember that without
> any aggregation at all there were about 20000 entries (layer1) free but I'm
> not sure. We're using bgp with 3 full views.
>
> On another router that I'm managing there's a ironcore mgmt-card, 2 full
> bgp-views and some peers and most of the traffic is following the default
> route and the usage is looking like this:
> show cam ip 1/1 stat
> CAM IP statistics: free entries total entries
> level1: 4072 8192
> level2: 1726 2048
> level3: 2020 2047
>
>
> What do the others say: does aggregation help and does the cam usually fill
> up in an isp enviorment?
> What about situation with gameserver-providers or voip which causes a lot of
> connections?
>
> Gunther
>
>

On Monday 06 March 2006 16:48, Gunther Stammwitz wrote:
> What do the others say: does aggregation help and does the cam
> usually fill up in an isp enviorment?

I do not use aggregation yet but this might be also interesting: I
see a lot of /32 CAM entries for destinations that are reachable
through a supernet which the router learned via OSPF from two neighbors
via two Ethernet links (= 4 equal paths):


(System: NI4GMR/B2P08000)

#sh ip route 212.79.18.100
Total number of IP routes: 180834
Destination NetMask Gateway Port Cost Type
212.79.0.0 255.255.192.0 212.79.49.135 1/4 20 O
212.79.0.0 255.255.192.0 *212.79.49.131 1/4 20 O
212.79.0.0 255.255.192.0 212.79.48.3 1/3 20 O
212.79.0.0 255.255.192.0 212.79.48.5 1/3 20 O
#sh cam ip 1/1 | inc 212.79.0.0
(nothing)

but:

#sh cam ip 1/1 | inc 212.79.18
1 8374 212.79.18.198/32 00e0.1e7e.4101 37 1 ether 1/4
1 8642 212.79.18.53/32 0050.0b3b.5400 120 1 ether 1/4
1 9142 212.79.18.94/32 00e0.1e7e.4101 30 1 ether 1/4
1 10131 212.79.18.44/32 0050.0b3b.5470 52 1 ether 1/3
1 10601 212.79.18.68/32 00e0.1e7e.4101 7 1 ether 1/4
1 11572 212.79.18.100/32 00e0.1e7e.4102 125 1 ether 1/3


First I thought this could be caused by the load balancing but then I
found an other network (reachable via 2 equal paths) that have a /24
CAM entry:


#sh ip route 212.79.56.156
Total number of IP routes: 180855
Destination NetMask Gateway Port Cost Type
212.79.56.0 255.255.255.0 212.79.48.4 1/3 20 O
212.79.56.0 255.255.255.0 *212.79.49.129 1/4 20 O
Gateway: 212.79.49.129
CAM Entry Flag: 0000000300000000H

#sh cam ip 1/1 | inc 212.79.56
1 10933 212.79.56.0/24 0001.6482.5c00 0 1 ether 1/4


Reading the 'Changing CAM Partitions' document on the Foundry website
does not really enlighten me - especially "Example 2" looks weird for
me. In my opinion the /32's would make sense for directly connected
systems or host routes only.... or have I missed something?


--
Gerald

On Mon, 6 Mar 2006, Gerald Krause wrote:

> On Monday 06 March 2006 16:48, Gunther Stammwitz wrote:
> > What do the others say: does aggregation help and does the cam
> > usually fill up in an isp enviorment?
>
> I do not use aggregation yet but this might be also interesting: I
> see a lot of /32 CAM entries for destinations that are reachable
> through a supernet which the router learned via OSPF from two neighbors
> via two Ethernet links (= 4 equal paths):
>
>
> (System: NI4GMR/B2P08000)
>
> #sh ip route 212.79.18.100
> Total number of IP routes: 180834
> Destination NetMask Gateway Port Cost Type
> 212.79.0.0 255.255.192.0 212.79.49.135 1/4 20 O
> 212.79.0.0 255.255.192.0 *212.79.49.131 1/4 20 O
> 212.79.0.0 255.255.192.0 212.79.48.3 1/3 20 O
> 212.79.0.0 255.255.192.0 212.79.48.5 1/3 20 O
> #sh cam ip 1/1 | inc 212.79.0.0
> (nothing)
>
> but:
>
> #sh cam ip 1/1 | inc 212.79.18
> 1 8374 212.79.18.198/32 00e0.1e7e.4101 37 1 ether 1/4
> 1 8642 212.79.18.53/32 0050.0b3b.5400 120 1 ether 1/4
> 1 9142 212.79.18.94/32 00e0.1e7e.4101 30 1 ether 1/4
> 1 10131 212.79.18.44/32 0050.0b3b.5470 52 1 ether 1/3
> 1 10601 212.79.18.68/32 00e0.1e7e.4101 7 1 ether 1/4
> 1 11572 212.79.18.100/32 00e0.1e7e.4102 125 1 ether 1/3
>

i dont see this behaviour on both routers with and without aggregation enabled,
i checked against routes learned in ospf with 2 or 3 next hops of the same cost

however i do observe /32s being added where the route is to discard (by having a
static 0.0.0.0/0 to null0) - this may explain why my CAM is getting so full

> Reading the 'Changing CAM Partitions' document on the Foundry website
> does not really enlighten me - especially "Example 2" looks weird for
> me. In my opinion the /32's would make sense for directly connected
> systems or host routes only.... or have I missed something?

yeah i have no idea why they would need to use /32s when simply adding the /30
and not applying aggregation would be better

Steve

On Monday 06 March 2006 22:05, Stephen J. Wilcox wrote:
> On Mon, 6 Mar 2006, Gerald Krause wrote:
> > On Monday 06 March 2006 16:48, Gunther Stammwitz wrote:
> > > What do the others say: does aggregation help and does the cam
> > > usually fill up in an isp enviorment?
> >
> > I do not use aggregation yet but this might be also interesting:
> > I see a lot of /32 CAM entries for destinations that are reachable
> > through a supernet which the router learned via OSPF from two
> > neighbors via two Ethernet links (= 4 equal paths):
>
> i dont see this behaviour on both routers with and without
> aggregation enabled, i checked against routes learned in ospf with 2
> or 3 next hops of the same cost
>
> however i do observe /32s being added where the route is to discard
> (by having a static 0.0.0.0/0 to null0) - this may explain why my CAM
> is getting so full

I also have a 0/0->null0 route in my config and after some deeper
inspection I realize that all my /32s seems to be not reachable hosts
(not configured on any other device - only the grounding routes
via /18->null0 on the two other routers exist).

The only hint I have so far is that I see this behavior only on a
certain /18 (other /18s have proper CAM entries) and that this /18 is
scanned very often for open ports. So the source of the problem could
be related to the amount of different destinations foreign systems try
to reach in our network.

But this makes the situation not clearer to me. I simply would expect
the NI forwarding all incomming packets towards the 4 next hops through
4 /18 CAM entries regardless if the host is reachable in the end or
not.

> > Reading the 'Changing CAM Partitions' document on the Foundry
> > website does not really enlighten me - especially "Example 2" looks
> > weird for me. In my opinion the /32's would make sense for directly
> > connected systems or host routes only.... or have I missed
> > something?
>
> yeah i have no idea why they would need to use /32s when simply
> adding the /30 and not applying aggregation would be better

Ack and I'am notably astonished about their conclusion "adding
50.50.50.0/30 ... results in ... 50.1.1.1/32" without any comments -
wtf?!? I'll like their drugs ;-)


--
Gerald

On Monday 06 March 2006 23:48, Gerald Krause wrote:
> But this makes the situation not clearer to me. I simply would
> expect the NI forwarding all incomming packets towards the 4 next
> hops through 4 /18 CAM entries regardless if the host is reachable in

Ok, "through 4 /18 CAM entries" is likely wrong - "through 1
(changing?) /18 CAM entry" may it describe better.


--
Gerald

On Mon, Mar 06, 2006 at 11:48:23PM +0100, Gerald Krause wrote:
> On Monday 06 March 2006 22:05, Stephen J. Wilcox wrote:
> > On Mon, 6 Mar 2006, Gerald Krause wrote:
> > > On Monday 06 March 2006 16:48, Gunther Stammwitz wrote:
> > > > What do the others say: does aggregation help and does the cam
> > > > usually fill up in an isp enviorment?
> > >
> > > I do not use aggregation yet but this might be also interesting:
> > > I see a lot of /32 CAM entries for destinations that are reachable
> > > through a supernet which the router learned via OSPF from two
> > > neighbors via two Ethernet links (= 4 equal paths):
> >
> > i dont see this behaviour on both routers with and without
> > aggregation enabled, i checked against routes learned in ospf with 2
> > or 3 next hops of the same cost
> >
> > however i do observe /32s being added where the route is to discard
> > (by having a static 0.0.0.0/0 to null0) - this may explain why my CAM
> > is getting so full
>
> I also have a 0/0->null0 route in my config and after some deeper
> inspection I realize that all my /32s seems to be not reachable hosts
> (not configured on any other device - only the grounding routes
> via /18->null0 on the two other routers exist).
>
> The only hint I have so far is that I see this behavior only on a
> certain /18 (other /18s have proper CAM entries) and that this /18 is
> scanned very often for open ports. So the source of the problem could
> be related to the amount of different destinations foreign systems try
> to reach in our network.
>
> But this makes the situation not clearer to me. I simply would expect
> the NI forwarding all incomming packets towards the 4 next hops through
> 4 /18 CAM entries regardless if the host is reachable in the end or
> not.
checkout ip hw-drop-def-in-hardware or something
like that..
>
> > > Reading the 'Changing CAM Partitions' document on the Foundry
> > > website does not really enlighten me - especially "Example 2" looks
> > > weird for me. In my opinion the /32's would make sense for directly
> > > connected systems or host routes only.... or have I missed
> > > something?
> >
> > yeah i have no idea why they would need to use /32s when simply
> > adding the /30 and not applying aggregation would be better
>
> Ack and I'am notably astonished about their conclusion "adding
> 50.50.50.0/30 ... results in ... 50.1.1.1/32" without any comments -
> wtf?!? I'll like their drugs ;-)
>
>
> --
> Gerald
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp at puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp

On Tuesday 07 March 2006 07:55, Kristian Larsson wrote:
>> But this makes the situation not clearer to me. I simply would expect
>> the NI forwarding all incomming packets towards the 4 next hops
>> through
>> 4 /18 CAM entries regardless if the host is reachable in the end or
>> not.
> checkout ip hw-drop-def-in-hardware or something
> like that..

Ok, I found "ip hw-drop-on-def-route" - but even this command will
prevent the NI from creating /32 CAM 'drop' entries and force packet
drops to be processed in software: why the system has to drop packets
for this /18 at all when the network is not directly connected? The
*next* hops (RTR1/2) should take care of this, not the NI:

+----SW1----+------+
| | |
UPLINK----NI400 RTR1 RTR2
| | |
+----SW2----+------+


NI400: Static: 0/0 -> null0, OSPF: 212.79/18 -> RTR1/2
RTR1/2: Static: 212.79/18 -> null0, OSPF: 0/0 -> NI


In my opinion 'unreachable hosts & dropping packets' could/should not be
the source of the problem that I'am face (seeing a lot of 212.79.x.x/32
CAM entries on the NI). I really don't get the point at the moment.


--
Gerald

On Tue, 7 Mar 2006, Gerald Krause wrote:

> On Tuesday 07 March 2006 07:55, Kristian Larsson wrote:
> >> But this makes the situation not clearer to me. I simply would expect
> >> the NI forwarding all incomming packets towards the 4 next hops
> >> through
> >> 4 /18 CAM entries regardless if the host is reachable in the end or
> >> not.
> > checkout ip hw-drop-def-in-hardware or something
> > like that..
>
> Ok, I found "ip hw-drop-on-def-route" - but even this command will
> prevent the NI from creating /32 CAM 'drop' entries and force packet
> drops to be processed in software: why the system has to drop packets
> for this /18 at all when the network is not directly connected? The
> *next* hops (RTR1/2) should take care of this, not the NI:
>
> +----SW1----+------+
> | | |
> UPLINK----NI400 RTR1 RTR2
> | | |
> +----SW2----+------+
>
>
> NI400: Static: 0/0 -> null0, OSPF: 212.79/18 -> RTR1/2
> RTR1/2: Static: 212.79/18 -> null0, OSPF: 0/0 -> NI
>
>
> In my opinion 'unreachable hosts & dropping packets' could/should not be
> the source of the problem that I'am face (seeing a lot of 212.79.x.x/32
> CAM entries on the NI). I really don't get the point at the moment.

ditto.

from the cam:
1 19 12.211.79.164/32 000c.db59.0e00 81 100 ether 1/1

from the routing table:
12.128.0.0/9 <x.x.x.x>57 v10 B

from bgp:
*>i 12.128.0.0/9 <y.y.y.y>251 5 100 0 xxx xxx i


the route for y.y.y.y:
<y.y.y.y>/32 <x.x.x.x>57 v10 5 O
<y.y.y.y>/32 *<z.z.z.z>120 v11 5 O
Gateway: <z.z.z.z>120
CIDX0: 22846[hw 9921 | 0x026c1 Right]

from the cam for y.y.y.y:
1 22846 <y.y.y.y>/32 000c.db59.0e00 102 101 ether 1/2


In summary I have a /9 in teh routing table, it is set to route to a particular
next hop in BGP, this next hop has two OSPF paths. I dont see the /9 in the cam
but i do see /32s.

Perhaps this is caused by a load balancing phenemenon - it sees conflicting host
routes in the cam and therefore does not add the aggregate

I'm investigating some more....

Steve

On Tuesday 07 March 2006 23:42, Stephen J. Wilcox wrote:
> In summary I have a /9 in teh routing table, it is set to route to a
> particular next hop in BGP, this next hop has two OSPF paths. I dont
> see the /9 in the cam but i do see /32s.
>
> Perhaps this is caused by a load balancing phenemenon - it sees
> conflicting host routes in the cam and therefore does not add the
> aggregate

This was my first impression too but now I found other /32 CAM entries
for destinations without loadbalanced next hops which breaks this
theorie from my point of view:

#sh cam ip 1/1 | inc 12.216
1 8377 12.216.34.147/32 0050.e28c.9156 83 1 ether 1/1

--

#sh ip cache 12.216.34.147
Total number of cache entries: 90822
D:Dynamic P:Permanent F:Forward U:Us C:Complex Filter
W:Wait ARP I:ICMP Deny K:Drop R:Fragment S:Snap Encap
IP Address Next Hop MAC Type Port
12.216.34.147 84.233.185.1 0050.e28c.9156 DF 1/1

CAM Entry Flag: 0000000000000000H

--

#sh ip route 12.216.34.147
Total number of IP routes: 180397
Destination NetMask Gateway Port Cost Type
12.128.0.0 255.128.0.0 84.233.185.1 1/1 B

--


#sh ip bgp 12.216.34.147
Network Next Hop Metric LocPrf Weight
*> 12.128.0.0/9 84.233.185.1 100 100
*i 12.128.0.0/9 194.97.172.109 0 100 0
*i 12.128.0.0/9 194.221.220.128 0 100 0

--

#sh ip route 84.233.185.1
Total number of IP routes: 180402
Destination NetMask Gateway Port Cost Type
84.233.185.0 255.255.255.252 0.0.0.0 1/1 1 D


> I'm investigating some more....

Same here, still struggle with it.


--
Gerald

Ok, reading the foundry doc's again very carefully, I understad at least
one of the problems - it relates to "overlapping networks":

My two IP transfernets between the NI and RTR1/2 ...


+----SW1----+------+
| | |
UPLINK----NI400 RTR1 RTR2
| | |
+----SW2----+------+

...are subnets of the mysterious /18, so the /18 is a supernet of
directly connected networks from the NI view, e.g.:

- directly connected: 212.79.x/24 (via SW1)
- directly connected: 212.79.y/24 (via SW2)
- supernet route: 212.79/18 -> RTR1/2

The document

http://www.foundrynet.com/services/documentation/diag/CAM-partitioning.html

describes now what will happen in this situation: per default the router
will create /32 CAM entries to every needed destination within the
supernet. But the solution the document suggest makes me not happy
because it seems that the only way to get rid of the bunch of /32s is
the way via software/cpu forwarding for some prefixes.


--
Gerald