Mailing List Archive

"Low pool memory, Pool(2-Global) below 5%" on Brocade MLX box
Dear all,

recently we connected another upstream to a Brocade MLX box.

We noticed that the MLX was not learning all routers from this BGP
peer both on the IPv4 and IPv6 session. Looking for the root cause we
found the following entries in the log of the box:
Mar 7 11:46:59:W:System: Low pool memory, Pool(2-Global) below 5%,
total memory(1073741824), available memory(53534720) on MP
Mar 7 11:46:58:W:System: Low physical memory, Pool(2-Global) below
5%, total memory(1073741824), available memory(53235712) on MP

I'm a bit worried because the system has not that many BGP sessions
active (around 8 while 4 peers send a full table that is reduces to
around 390k routes in the RIB/FIB using prefix lists).

After removing the BGP peers the free memory went to arund 7%, however
prior to the configuration we have never had any warning regarding the
usual 10% threshold).

Looking at the memory histogram i can see that most memory is consumed
by the following processes:
bgp 1221 106342047
bgp_io 3925 473596101

Any idea how to reduce the memory footprint of bgp_io or to free up
more space by adjusting some system-max variables? Or any other
options than replacing the box with some new gear (luckily this will
happen end of 2016).

Thanks,
Chris

For the recordds, the modules we have in place and a sh ve of the system:
SSH@CRBR13-F#sh module
Module Status
Ports Starting MAC
M1 (left ):NI-MLX-MR Management Module Active
M2 (right):NI-MLX-MR Management Module Standby(Ready State)
F1: NI-X-SF Switch Fabric Module Active
F2: NI-X-SF Switch Fabric Module Active
F3: NI-X-SF Switch Fabric Module Active
S1: NI-MLX-10Gx4 4-port 10GbE Module
CARD_STATE_UP 4 001b.ed17.d000
S2: NI-MLX-10Gx4 4-port 10GbE Module
CARD_STATE_UP 4 001b.ed17.d030
S3: NI-MLX-10Gx4 4-port 10GbE Module
CARD_STATE_UP 4 001b.ed17.d060
S4: NI-MLX-10Gx4 4-port 10GbE Module
CARD_STATE_UP 4 001b.ed17.d090
S5: NI-MLX-10Gx4 4-port 10GbE Module
CARD_STATE_UP 4 001b.ed17.d0c0

SSH@CRBR13-F#sh ve
System Mode: MLX
Chassis: NetIron 8-slot (Serial #: A60532F06T, Part #: 35549-000C)
NI-X-SF Switch Fabric Module 1 (Serial #: SA38070273, Part #: 35523-300C)
FE 1: Type fe200, Version 2
FE 3: Type fe200, Version 2
Switch Fabric Module 1 Up Time is 286 days 23 hours 8 minutes 55 seconds
NI-X-SF Switch Fabric Module 2 (Serial #: SA13070172, Part #: 35523-300C)
FE 1: Type fe200, Version 2
FE 3: Type fe200, Version 2
Switch Fabric Module 2 Up Time is 286 days 23 hours 8 minutes 55 seconds
NI-X-SF Switch Fabric Module 3 (Serial #: SA13080287, Part #: 35523-301A)
FE 1: Type fe200, Version 2
FE 3: Type fe200, Version 2
Switch Fabric Module 3 Up Time is 286 days 23 hours 8 minutes 55 seconds
==========================================================================
SL M1: NI-MLX-MR Management Module Active (Serial #: S10526F06N, Part
#: 35524-103J):
Boot : Version 5.6.0T165 Copyright (c) 1996-2014 Brocade
Communications Systems, Inc.
Compiled on Sep 20 2013 at 16:42:38 labeled as xmprm05600
(516258 bytes) from boot flash
Monitor : Version 5.7.0T165 Copyright (c) 1996-2014 Brocade
Communications Systems, Inc.
Compiled on May 5 2014 at 10:41:24 labeled as xmb05700
(537647 bytes) from code flash
IronWare : Version 5.7.0cT163 Copyright (c) 1996-2014 Brocade
Communications Systems, Inc.
Compiled on Feb 25 2015 at 20:44:00 labeled as xmr05700c
(9651771 bytes) from Primary
Board ID : 00 MBRIDGE Revision : 37
916 MHz Power PC processor 7447A (version 8003/0101) 166 MHz bus
512 KB Boot Flash (AM29LV040B), 32 MB Code Flash (MT28F128J3)
1024 MB DRAM INSTALLED
1024 MB DRAM ADDRESSABLE
Active Management uptime is 286 days 23 hours 8 minutes 55 seconds
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp
Re: "Low pool memory, Pool(2-Global) below 5%" on Brocade MLX box [ In reply to ]
Could you also provide the output of 'show default values'?

You may need to change your cam-partition profile.

--
Eldon

On Tue, Mar 8, 2016 at 8:13 AM, Chris Hellkvist via foundry-nsp
<foundry-nsp@puck.nether.net> wrote:
> Dear all,
>
> recently we connected another upstream to a Brocade MLX box.
>
> We noticed that the MLX was not learning all routers from this BGP
> peer both on the IPv4 and IPv6 session. Looking for the root cause we
> found the following entries in the log of the box:
> Mar 7 11:46:59:W:System: Low pool memory, Pool(2-Global) below 5%,
> total memory(1073741824), available memory(53534720) on MP
> Mar 7 11:46:58:W:System: Low physical memory, Pool(2-Global) below
> 5%, total memory(1073741824), available memory(53235712) on MP
>
> I'm a bit worried because the system has not that many BGP sessions
> active (around 8 while 4 peers send a full table that is reduces to
> around 390k routes in the RIB/FIB using prefix lists).
>
> After removing the BGP peers the free memory went to arund 7%, however
> prior to the configuration we have never had any warning regarding the
> usual 10% threshold).
>
> Looking at the memory histogram i can see that most memory is consumed
> by the following processes:
> bgp 1221 106342047
> bgp_io 3925 473596101
>
> Any idea how to reduce the memory footprint of bgp_io or to free up
> more space by adjusting some system-max variables? Or any other
> options than replacing the box with some new gear (luckily this will
> happen end of 2016).
>
> Thanks,
> Chris
>
> For the recordds, the modules we have in place and a sh ve of the system:
> SSH@CRBR13-F#sh module
> Module Status
> Ports Starting MAC
> M1 (left ):NI-MLX-MR Management Module Active
> M2 (right):NI-MLX-MR Management Module Standby(Ready State)
> F1: NI-X-SF Switch Fabric Module Active
> F2: NI-X-SF Switch Fabric Module Active
> F3: NI-X-SF Switch Fabric Module Active
> S1: NI-MLX-10Gx4 4-port 10GbE Module
> CARD_STATE_UP 4 001b.ed17.d000
> S2: NI-MLX-10Gx4 4-port 10GbE Module
> CARD_STATE_UP 4 001b.ed17.d030
> S3: NI-MLX-10Gx4 4-port 10GbE Module
> CARD_STATE_UP 4 001b.ed17.d060
> S4: NI-MLX-10Gx4 4-port 10GbE Module
> CARD_STATE_UP 4 001b.ed17.d090
> S5: NI-MLX-10Gx4 4-port 10GbE Module
> CARD_STATE_UP 4 001b.ed17.d0c0
>
> SSH@CRBR13-F#sh ve
> System Mode: MLX
> Chassis: NetIron 8-slot (Serial #: A60532F06T, Part #: 35549-000C)
> NI-X-SF Switch Fabric Module 1 (Serial #: SA38070273, Part #: 35523-300C)
> FE 1: Type fe200, Version 2
> FE 3: Type fe200, Version 2
> Switch Fabric Module 1 Up Time is 286 days 23 hours 8 minutes 55 seconds
> NI-X-SF Switch Fabric Module 2 (Serial #: SA13070172, Part #: 35523-300C)
> FE 1: Type fe200, Version 2
> FE 3: Type fe200, Version 2
> Switch Fabric Module 2 Up Time is 286 days 23 hours 8 minutes 55 seconds
> NI-X-SF Switch Fabric Module 3 (Serial #: SA13080287, Part #: 35523-301A)
> FE 1: Type fe200, Version 2
> FE 3: Type fe200, Version 2
> Switch Fabric Module 3 Up Time is 286 days 23 hours 8 minutes 55 seconds
> ==========================================================================
> SL M1: NI-MLX-MR Management Module Active (Serial #: S10526F06N, Part
> #: 35524-103J):
> Boot : Version 5.6.0T165 Copyright (c) 1996-2014 Brocade
> Communications Systems, Inc.
> Compiled on Sep 20 2013 at 16:42:38 labeled as xmprm05600
> (516258 bytes) from boot flash
> Monitor : Version 5.7.0T165 Copyright (c) 1996-2014 Brocade
> Communications Systems, Inc.
> Compiled on May 5 2014 at 10:41:24 labeled as xmb05700
> (537647 bytes) from code flash
> IronWare : Version 5.7.0cT163 Copyright (c) 1996-2014 Brocade
> Communications Systems, Inc.
> Compiled on Feb 25 2015 at 20:44:00 labeled as xmr05700c
> (9651771 bytes) from Primary
> Board ID : 00 MBRIDGE Revision : 37
> 916 MHz Power PC processor 7447A (version 8003/0101) 166 MHz bus
> 512 KB Boot Flash (AM29LV040B), 32 MB Code Flash (MT28F128J3)
> 1024 MB DRAM INSTALLED
> 1024 MB DRAM ADDRESSABLE
> Active Management uptime is 286 days 23 hours 8 minutes 55 seconds
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp@puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp
Re: "Low pool memory, Pool(2-Global) below 5%" on Brocade MLX box [ In reply to ]
Sure:

SSH@CRBR13-F> sh cam-partition brief

CAM partitioning profile: multi-service-2

Slot 1 XPP20SP 0:
# of CAM device = 3
Total CAM Size = 524288 entries (36Mbits)

IP: Raw Size 458752, User Size 458752(0 reserved)
Subpartition 0: Raw Size 2048, User Size 2048, (0 reserved)
Subpartition 1: Raw Size 411289, User Size 411289, (0 reserved)
Subpartition 2: Raw Size 37892, User Size 37892, (0 reserved)
Subpartition 3: Raw Size 5741, User Size 5741, (0 reserved)
Subpartition 4: Raw Size 1112, User Size 1112, (0 reserved)

IPv6: Raw Size 32768, User Size 16384(0 reserved)
Subpartition 0: Raw Size 2048, User Size 1024, (0 reserved)
Subpartition 1: Raw Size 27204, User Size 13602, (0 reserved)
Subpartition 2: Raw Size 2332, User Size 1166, (0 reserved)
Subpartition 3: Raw Size 512, User Size 256, (0 reserved)
Subpartition 4: Raw Size 384, User Size 192, (0 reserved)

SSH@CRBR13-F>sh default values
sys log buffers:50 mac age time:300 sec telnet sessions:5

ip arp age:10 min bootp relay max hops:4 ip ttl:64 hops
ip addr per intf:24

when multicast enabled :
igmp group memb.: 260 sec igmp query: 125 sec

when ospf enabled :
ospf dead:40 sec ospf hello:10 sec ospf retrans:5 sec
ospf transit delay:1 sec

when bgp enabled :
bgp local pref.:100 bgp keep alive:60 sec bgp hold:180 sec
bgp metric:10 bgp local as:1 bgp cluster id:0
bgp ext. distance:20 bgp int. distance:200 bgp local distance:200

when IS-IS enabled :
isis hello interval:10 sec isis hello multiplier:3
isis port metric:10 isis priority:64
isis csnp-interval:10 sec isis default-metric:10
isis distance:115 isis lsp-gen-interval:10 sec
isis lsp-interval:33 msec isis lsp-refresh-interval:900 sec
isis max-lsp-lifetime:1200 sec isis maximum-paths:4
isis retransmit-interval:5 sec isis spf-interval:5 sec

filter change update delay:10 sec

System Parameters Default Maximum Current Actual
Bootup Revertible
mac 32768 1048576 32768 32768
32768 Yes
vlan 512 4095 1024 1024
1024 No
spanning-tree 32 128 32 32
32 No
rstp 32 128 32 32
32 No
ip-arp 8192 65536 40000 40000
40000 No
ip-cache 102400 524288 440000 440000
440000 Yes
ip-route 102400 524288 440000 440000
440000 Yes
ip-subnet-port 24 128 24 24
24 No
virtual-interface 255 4095 800 800
800 No
vpls-mac 2048 262144 2048 2048
2048 Yes
vpls-num 512 4096 512 512
512 No
session-limit 8192 40960 8192 8192
8192 Yes
ip-filter-sys 4096 102400 4096 4096
4096 No
mgmt-port-acl-size 20 100 20 20
20 No
l2-acl-table-entries 64 256 64 64
64 No
ipv6-cache 32768 114688 16000 16000
16000 Yes
ipv6-route 32768 114688 16000 16000
16000 Yes
ip-vrf-route 5120 262144 5120 5120
5120 Yes
receive-cam 1024 16384 1024 1024
1024 No
ip-tunnels 256 8192 256 256
256 No
lsp-out-acl-cam 0 16384 0 0 0
No
trunk-num 128 256 128 128
128 No
config-file-size 8388608 32505856 8388608 8388608
8388608 No
ifl-cam 8192 81920 8192 8192
8192 No
ip-source-guard-cam 0 131072 0 0 0
No
ipv4-mcast-cam 4096 32768 4096 4096
4096 No
ipv6-mcast-cam 1024 8192 1024 1024
1024 No
mcast-vpls-cam 0 2048 0 0 0
Yes
ip-vrf 400 400 400 400
400 Yes
ipv6-vrf-route 128 16384 128 128
128 Yes
subnet-broadcast-acl-cam 0 4096 0 0 0
No
trunk-num-100g 8 64 8 8 8
No
openflow-flow-entries 0 65536 0 0 0
Yes
openflow-pvlan-entries 0 2048 0 0 0
Yes
openflow-unprotectedvlan 0 4096 0 0 0
Yes
ipv6-receive-cam 0 8192 0 0 0
No
ecmp-pram-block-size 32 32 32 32
32 Yes
tvf-lag-lb-fid-pool 0 2048 0 0 0
Yes
tvf-lag-lb-fid-group 4 16 4 4 4
Yes
openflow-group-select-bu 8 64 8 8 8
Yes

Openflow Slot Parameters slot Default
Maximum Actual Revertible
np-openflow-flow-entries : layer2or3 1 0
32768 0 No
np-openflow-flow-entries : layer2or3 2 0
32768 0 No
np-openflow-flow-entries : layer2or3 3 0
32768 0 No
np-openflow-flow-entries : layer2or3 4 0
32768 0 No
np-openflow-flow-entries : layer2or3 5 0
32768 0 No
np-openflow-flow-entries : layer2or3 6 0
32768 0 No
np-openflow-flow-entries : layer2or3 7 0
32768 0 No
np-openflow-flow-entries : layer2or3 8 0
32768 0 No

np-openflow-flow-entries : layer23ipv4 1 0
32768 0 No
np-openflow-flow-entries : layer23ipv4 2 0
32768 0 No
np-openflow-flow-entries : layer23ipv4 3 0
32768 0 No
np-openflow-flow-entries : layer23ipv4 4 0
32768 0 No
np-openflow-flow-entries : layer23ipv4 5 0
32768 0 No
np-openflow-flow-entries : layer23ipv4 6 0
32768 0 No
np-openflow-flow-entries : layer23ipv4 7 0
32768 0 No
np-openflow-flow-entries : layer23ipv4 8 0
32768 0 No

2016-03-08 18:00 GMT+01:00 Eldon Koyle <ekoyle+puck.nether.net@gmail.com>:
> Could you also provide the output of 'show default values'?
>
> You may need to change your cam-partition profile.
>
> --
> Eldon
>
> On Tue, Mar 8, 2016 at 8:13 AM, Chris Hellkvist via foundry-nsp
> <foundry-nsp@puck.nether.net> wrote:
>> Dear all,
>>
>> recently we connected another upstream to a Brocade MLX box.
>>
>> We noticed that the MLX was not learning all routers from this BGP
>> peer both on the IPv4 and IPv6 session. Looking for the root cause we
>> found the following entries in the log of the box:
>> Mar 7 11:46:59:W:System: Low pool memory, Pool(2-Global) below 5%,
>> total memory(1073741824), available memory(53534720) on MP
>> Mar 7 11:46:58:W:System: Low physical memory, Pool(2-Global) below
>> 5%, total memory(1073741824), available memory(53235712) on MP
>>
>> I'm a bit worried because the system has not that many BGP sessions
>> active (around 8 while 4 peers send a full table that is reduces to
>> around 390k routes in the RIB/FIB using prefix lists).
>>
>> After removing the BGP peers the free memory went to arund 7%, however
>> prior to the configuration we have never had any warning regarding the
>> usual 10% threshold).
>>
>> Looking at the memory histogram i can see that most memory is consumed
>> by the following processes:
>> bgp 1221 106342047
>> bgp_io 3925 473596101
>>
>> Any idea how to reduce the memory footprint of bgp_io or to free up
>> more space by adjusting some system-max variables? Or any other
>> options than replacing the box with some new gear (luckily this will
>> happen end of 2016).
>>
>> Thanks,
>> Chris
>>
>> For the recordds, the modules we have in place and a sh ve of the system:
>> SSH@CRBR13-F#sh module
>> Module Status
>> Ports Starting MAC
>> M1 (left ):NI-MLX-MR Management Module Active
>> M2 (right):NI-MLX-MR Management Module Standby(Ready State)
>> F1: NI-X-SF Switch Fabric Module Active
>> F2: NI-X-SF Switch Fabric Module Active
>> F3: NI-X-SF Switch Fabric Module Active
>> S1: NI-MLX-10Gx4 4-port 10GbE Module
>> CARD_STATE_UP 4 001b.ed17.d000
>> S2: NI-MLX-10Gx4 4-port 10GbE Module
>> CARD_STATE_UP 4 001b.ed17.d030
>> S3: NI-MLX-10Gx4 4-port 10GbE Module
>> CARD_STATE_UP 4 001b.ed17.d060
>> S4: NI-MLX-10Gx4 4-port 10GbE Module
>> CARD_STATE_UP 4 001b.ed17.d090
>> S5: NI-MLX-10Gx4 4-port 10GbE Module
>> CARD_STATE_UP 4 001b.ed17.d0c0
>>
>> SSH@CRBR13-F#sh ve
>> System Mode: MLX
>> Chassis: NetIron 8-slot (Serial #: A60532F06T, Part #: 35549-000C)
>> NI-X-SF Switch Fabric Module 1 (Serial #: SA38070273, Part #: 35523-300C)
>> FE 1: Type fe200, Version 2
>> FE 3: Type fe200, Version 2
>> Switch Fabric Module 1 Up Time is 286 days 23 hours 8 minutes 55 seconds
>> NI-X-SF Switch Fabric Module 2 (Serial #: SA13070172, Part #: 35523-300C)
>> FE 1: Type fe200, Version 2
>> FE 3: Type fe200, Version 2
>> Switch Fabric Module 2 Up Time is 286 days 23 hours 8 minutes 55 seconds
>> NI-X-SF Switch Fabric Module 3 (Serial #: SA13080287, Part #: 35523-301A)
>> FE 1: Type fe200, Version 2
>> FE 3: Type fe200, Version 2
>> Switch Fabric Module 3 Up Time is 286 days 23 hours 8 minutes 55 seconds
>> ==========================================================================
>> SL M1: NI-MLX-MR Management Module Active (Serial #: S10526F06N, Part
>> #: 35524-103J):
>> Boot : Version 5.6.0T165 Copyright (c) 1996-2014 Brocade
>> Communications Systems, Inc.
>> Compiled on Sep 20 2013 at 16:42:38 labeled as xmprm05600
>> (516258 bytes) from boot flash
>> Monitor : Version 5.7.0T165 Copyright (c) 1996-2014 Brocade
>> Communications Systems, Inc.
>> Compiled on May 5 2014 at 10:41:24 labeled as xmb05700
>> (537647 bytes) from code flash
>> IronWare : Version 5.7.0cT163 Copyright (c) 1996-2014 Brocade
>> Communications Systems, Inc.
>> Compiled on Feb 25 2015 at 20:44:00 labeled as xmr05700c
>> (9651771 bytes) from Primary
>> Board ID : 00 MBRIDGE Revision : 37
>> 916 MHz Power PC processor 7447A (version 8003/0101) 166 MHz bus
>> 512 KB Boot Flash (AM29LV040B), 32 MB Code Flash (MT28F128J3)
>> 1024 MB DRAM INSTALLED
>> 1024 MB DRAM ADDRESSABLE
>> Active Management uptime is 286 days 23 hours 8 minutes 55 seconds
>> _______________________________________________
>> foundry-nsp mailing list
>> foundry-nsp@puck.nether.net
>> http://puck.nether.net/mailman/listinfo/foundry-nsp
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp
Re: "Low pool memory, Pool(2-Global) below 5%" on Brocade MLX box [ In reply to ]
I thought 5.6 was the last supported release for the 1st gen hardware
(-MR management module, 10Gx4 cards, etc.), although I'm not seeing
anything about the MR module so maybe it is still supported.

From 5.7 release notes:

Note: Gen 1 Module SKUs are not supported beyond R05.6:00
• NI-MLX-1Gx20-GC
• NI-MLX-1Gx20-SFP
• NI-XMR-1Gx20-GC
• NI-XMR-1Gx20-SFP
• NI-MLX-10Gx2
• NI-MLX-10Gx4

I don't know if that is part of your problem, but is there a feature
you needed that wasn't available in 05600g? If not, you might
consider downgrading to see if there is a difference there.

It is possible that there is a memory leak somewhere in 5.7 or
something like that.

The MR modules only had 1 GB of ram, the MR2 has 4 GB. You could
upgrade to a single MR2 management module in this device (this is what
they use in the MLXe chassis, if you are sticking with brocade for
your replacement).

--
Eldon


On Tue, Mar 8, 2016 at 10:17 AM, Chris Hellkvist
<chris.hellkvist@googlemail.com> wrote:
> Sure:
>
> SSH@CRBR13-F> sh cam-partition brief
>
> CAM partitioning profile: multi-service-2
>
> Slot 1 XPP20SP 0:
> # of CAM device = 3
> Total CAM Size = 524288 entries (36Mbits)
>
> IP: Raw Size 458752, User Size 458752(0 reserved)
> Subpartition 0: Raw Size 2048, User Size 2048, (0 reserved)
> Subpartition 1: Raw Size 411289, User Size 411289, (0 reserved)
> Subpartition 2: Raw Size 37892, User Size 37892, (0 reserved)
> Subpartition 3: Raw Size 5741, User Size 5741, (0 reserved)
> Subpartition 4: Raw Size 1112, User Size 1112, (0 reserved)
>
> IPv6: Raw Size 32768, User Size 16384(0 reserved)
> Subpartition 0: Raw Size 2048, User Size 1024, (0 reserved)
> Subpartition 1: Raw Size 27204, User Size 13602, (0 reserved)
> Subpartition 2: Raw Size 2332, User Size 1166, (0 reserved)
> Subpartition 3: Raw Size 512, User Size 256, (0 reserved)
> Subpartition 4: Raw Size 384, User Size 192, (0 reserved)
>
> SSH@CRBR13-F>sh default values
> sys log buffers:50 mac age time:300 sec telnet sessions:5
>
> ip arp age:10 min bootp relay max hops:4 ip ttl:64 hops
> ip addr per intf:24
>
> when multicast enabled :
> igmp group memb.: 260 sec igmp query: 125 sec
>
> when ospf enabled :
> ospf dead:40 sec ospf hello:10 sec ospf retrans:5 sec
> ospf transit delay:1 sec
>
> when bgp enabled :
> bgp local pref.:100 bgp keep alive:60 sec bgp hold:180 sec
> bgp metric:10 bgp local as:1 bgp cluster id:0
> bgp ext. distance:20 bgp int. distance:200 bgp local distance:200
>
> when IS-IS enabled :
> isis hello interval:10 sec isis hello multiplier:3
> isis port metric:10 isis priority:64
> isis csnp-interval:10 sec isis default-metric:10
> isis distance:115 isis lsp-gen-interval:10 sec
> isis lsp-interval:33 msec isis lsp-refresh-interval:900 sec
> isis max-lsp-lifetime:1200 sec isis maximum-paths:4
> isis retransmit-interval:5 sec isis spf-interval:5 sec
>
> filter change update delay:10 sec
>
> System Parameters Default Maximum Current Actual
> Bootup Revertible
> mac 32768 1048576 32768 32768
> 32768 Yes
> vlan 512 4095 1024 1024
> 1024 No
> spanning-tree 32 128 32 32
> 32 No
> rstp 32 128 32 32
> 32 No
> ip-arp 8192 65536 40000 40000
> 40000 No
> ip-cache 102400 524288 440000 440000
> 440000 Yes
> ip-route 102400 524288 440000 440000
> 440000 Yes
> ip-subnet-port 24 128 24 24
> 24 No
> virtual-interface 255 4095 800 800
> 800 No
> vpls-mac 2048 262144 2048 2048
> 2048 Yes
> vpls-num 512 4096 512 512
> 512 No
> session-limit 8192 40960 8192 8192
> 8192 Yes
> ip-filter-sys 4096 102400 4096 4096
> 4096 No
> mgmt-port-acl-size 20 100 20 20
> 20 No
> l2-acl-table-entries 64 256 64 64
> 64 No
> ipv6-cache 32768 114688 16000 16000
> 16000 Yes
> ipv6-route 32768 114688 16000 16000
> 16000 Yes
> ip-vrf-route 5120 262144 5120 5120
> 5120 Yes
> receive-cam 1024 16384 1024 1024
> 1024 No
> ip-tunnels 256 8192 256 256
> 256 No
> lsp-out-acl-cam 0 16384 0 0 0
> No
> trunk-num 128 256 128 128
> 128 No
> config-file-size 8388608 32505856 8388608 8388608
> 8388608 No
> ifl-cam 8192 81920 8192 8192
> 8192 No
> ip-source-guard-cam 0 131072 0 0 0
> No
> ipv4-mcast-cam 4096 32768 4096 4096
> 4096 No
> ipv6-mcast-cam 1024 8192 1024 1024
> 1024 No
> mcast-vpls-cam 0 2048 0 0 0
> Yes
> ip-vrf 400 400 400 400
> 400 Yes
> ipv6-vrf-route 128 16384 128 128
> 128 Yes
> subnet-broadcast-acl-cam 0 4096 0 0 0
> No
> trunk-num-100g 8 64 8 8 8
> No
> openflow-flow-entries 0 65536 0 0 0
> Yes
> openflow-pvlan-entries 0 2048 0 0 0
> Yes
> openflow-unprotectedvlan 0 4096 0 0 0
> Yes
> ipv6-receive-cam 0 8192 0 0 0
> No
> ecmp-pram-block-size 32 32 32 32
> 32 Yes
> tvf-lag-lb-fid-pool 0 2048 0 0 0
> Yes
> tvf-lag-lb-fid-group 4 16 4 4 4
> Yes
> openflow-group-select-bu 8 64 8 8 8
> Yes
>
> Openflow Slot Parameters slot Default
> Maximum Actual Revertible
> np-openflow-flow-entries : layer2or3 1 0
> 32768 0 No
> np-openflow-flow-entries : layer2or3 2 0
> 32768 0 No
> np-openflow-flow-entries : layer2or3 3 0
> 32768 0 No
> np-openflow-flow-entries : layer2or3 4 0
> 32768 0 No
> np-openflow-flow-entries : layer2or3 5 0
> 32768 0 No
> np-openflow-flow-entries : layer2or3 6 0
> 32768 0 No
> np-openflow-flow-entries : layer2or3 7 0
> 32768 0 No
> np-openflow-flow-entries : layer2or3 8 0
> 32768 0 No
>
> np-openflow-flow-entries : layer23ipv4 1 0
> 32768 0 No
> np-openflow-flow-entries : layer23ipv4 2 0
> 32768 0 No
> np-openflow-flow-entries : layer23ipv4 3 0
> 32768 0 No
> np-openflow-flow-entries : layer23ipv4 4 0
> 32768 0 No
> np-openflow-flow-entries : layer23ipv4 5 0
> 32768 0 No
> np-openflow-flow-entries : layer23ipv4 6 0
> 32768 0 No
> np-openflow-flow-entries : layer23ipv4 7 0
> 32768 0 No
> np-openflow-flow-entries : layer23ipv4 8 0
> 32768 0 No
>
> 2016-03-08 18:00 GMT+01:00 Eldon Koyle <ekoyle+puck.nether.net@gmail.com>:
>> Could you also provide the output of 'show default values'?
>>
>> You may need to change your cam-partition profile.
>>
>> --
>> Eldon
>>
>> On Tue, Mar 8, 2016 at 8:13 AM, Chris Hellkvist via foundry-nsp
>> <foundry-nsp@puck.nether.net> wrote:
>>> Dear all,
>>>
>>> recently we connected another upstream to a Brocade MLX box.
>>>
>>> We noticed that the MLX was not learning all routers from this BGP
>>> peer both on the IPv4 and IPv6 session. Looking for the root cause we
>>> found the following entries in the log of the box:
>>> Mar 7 11:46:59:W:System: Low pool memory, Pool(2-Global) below 5%,
>>> total memory(1073741824), available memory(53534720) on MP
>>> Mar 7 11:46:58:W:System: Low physical memory, Pool(2-Global) below
>>> 5%, total memory(1073741824), available memory(53235712) on MP
>>>
>>> I'm a bit worried because the system has not that many BGP sessions
>>> active (around 8 while 4 peers send a full table that is reduces to
>>> around 390k routes in the RIB/FIB using prefix lists).
>>>
>>> After removing the BGP peers the free memory went to arund 7%, however
>>> prior to the configuration we have never had any warning regarding the
>>> usual 10% threshold).
>>>
>>> Looking at the memory histogram i can see that most memory is consumed
>>> by the following processes:
>>> bgp 1221 106342047
>>> bgp_io 3925 473596101
>>>
>>> Any idea how to reduce the memory footprint of bgp_io or to free up
>>> more space by adjusting some system-max variables? Or any other
>>> options than replacing the box with some new gear (luckily this will
>>> happen end of 2016).
>>>
>>> Thanks,
>>> Chris
>>>
>>> For the recordds, the modules we have in place and a sh ve of the system:
>>> SSH@CRBR13-F#sh module
>>> Module Status
>>> Ports Starting MAC
>>> M1 (left ):NI-MLX-MR Management Module Active
>>> M2 (right):NI-MLX-MR Management Module Standby(Ready State)
>>> F1: NI-X-SF Switch Fabric Module Active
>>> F2: NI-X-SF Switch Fabric Module Active
>>> F3: NI-X-SF Switch Fabric Module Active
>>> S1: NI-MLX-10Gx4 4-port 10GbE Module
>>> CARD_STATE_UP 4 001b.ed17.d000
>>> S2: NI-MLX-10Gx4 4-port 10GbE Module
>>> CARD_STATE_UP 4 001b.ed17.d030
>>> S3: NI-MLX-10Gx4 4-port 10GbE Module
>>> CARD_STATE_UP 4 001b.ed17.d060
>>> S4: NI-MLX-10Gx4 4-port 10GbE Module
>>> CARD_STATE_UP 4 001b.ed17.d090
>>> S5: NI-MLX-10Gx4 4-port 10GbE Module
>>> CARD_STATE_UP 4 001b.ed17.d0c0
>>>
>>> SSH@CRBR13-F#sh ve
>>> System Mode: MLX
>>> Chassis: NetIron 8-slot (Serial #: A60532F06T, Part #: 35549-000C)
>>> NI-X-SF Switch Fabric Module 1 (Serial #: SA38070273, Part #: 35523-300C)
>>> FE 1: Type fe200, Version 2
>>> FE 3: Type fe200, Version 2
>>> Switch Fabric Module 1 Up Time is 286 days 23 hours 8 minutes 55 seconds
>>> NI-X-SF Switch Fabric Module 2 (Serial #: SA13070172, Part #: 35523-300C)
>>> FE 1: Type fe200, Version 2
>>> FE 3: Type fe200, Version 2
>>> Switch Fabric Module 2 Up Time is 286 days 23 hours 8 minutes 55 seconds
>>> NI-X-SF Switch Fabric Module 3 (Serial #: SA13080287, Part #: 35523-301A)
>>> FE 1: Type fe200, Version 2
>>> FE 3: Type fe200, Version 2
>>> Switch Fabric Module 3 Up Time is 286 days 23 hours 8 minutes 55 seconds
>>> ==========================================================================
>>> SL M1: NI-MLX-MR Management Module Active (Serial #: S10526F06N, Part
>>> #: 35524-103J):
>>> Boot : Version 5.6.0T165 Copyright (c) 1996-2014 Brocade
>>> Communications Systems, Inc.
>>> Compiled on Sep 20 2013 at 16:42:38 labeled as xmprm05600
>>> (516258 bytes) from boot flash
>>> Monitor : Version 5.7.0T165 Copyright (c) 1996-2014 Brocade
>>> Communications Systems, Inc.
>>> Compiled on May 5 2014 at 10:41:24 labeled as xmb05700
>>> (537647 bytes) from code flash
>>> IronWare : Version 5.7.0cT163 Copyright (c) 1996-2014 Brocade
>>> Communications Systems, Inc.
>>> Compiled on Feb 25 2015 at 20:44:00 labeled as xmr05700c
>>> (9651771 bytes) from Primary
>>> Board ID : 00 MBRIDGE Revision : 37
>>> 916 MHz Power PC processor 7447A (version 8003/0101) 166 MHz bus
>>> 512 KB Boot Flash (AM29LV040B), 32 MB Code Flash (MT28F128J3)
>>> 1024 MB DRAM INSTALLED
>>> 1024 MB DRAM ADDRESSABLE
>>> Active Management uptime is 286 days 23 hours 8 minutes 55 seconds
>>> _______________________________________________
>>> foundry-nsp mailing list
>>> foundry-nsp@puck.nether.net
>>> http://puck.nether.net/mailman/listinfo/foundry-nsp
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp
Re: "Low pool memory, Pool(2-Global) below 5%" on Brocade MLX box [ In reply to ]
Cheers,

2016-03-08 19:17 GMT+01:00 Eldon Koyle <ekoyle+puck.nether.net@gmail.com>:
> I thought 5.6 was the last supported release for the 1st gen hardware
> (-MR management module, 10Gx4 cards, etc.), although I'm not seeing
> anything about the MR module so maybe it is still supported.

you are right, to be honest i did not have this in mind while
upgrading the box to 5.7.

I did a re-check with another device that is running 5.6g and there we
have 59% free memory (but also some less peers).

So for now i'll give downgrading the box to 5.6g a try and see if this
helps to solve the issue; removing the soft-reconfig from the peers as
suggested from Joerg in another mail on the list did not free _any_
memory. I think there might be a memory leak hidden in 5.7 in
combination with 1st gen hw.

Thanks,
Chris
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp