We have a BigIron RX-4 running a collapsed setup in one of our colo facilities. It’s been doing well, despite its limitations on memory size, and we’re able to run BGP with 3 upstreams on v4 and v6, perform inter-VLAN routing, and handle basic Layer 2 switching in a single box.
About 3 months ago, we added a VLAN and a ve to correspond to it for an internal /28. We were unable to reach the assigned IP on the ve from outside of our network, which we ultimately discovered was due to a memory problem. The memory problem stemmed from improperly filtering one of our providers inbound v4 route table, which had suddenly grown by about 150k entries. That was resolved relatively easily.
Recently, it was time to add another VLAN and ve for another internal /28. This time, we did not have the same symptom, but found that a new host connected to module 3 could not communicate with any existing hosts on module 4, but could communicate with hosts on module 3. For the record, both are RX-BI-24C copper modules. We removed and added the VLAN and ve and even tried adding the new /28 to an existing vlan and ve.
We have checked free system memory, module software versions, and found nothing suspicious.
A full system reload has to be a last resort as this is a production VoIP site for us and even a five minute downtime requires significant planning. Does anyone have suggestions for troubleshooting or for reloading just a single module? We can function with just one of two modules, as all of our directly connected hosts are using XenServer active-active load balancing between them.
Thanks in advance for any guidance and assistance!
Output of show module:
show module
==========================================================================
*** BigIron RX-4 CHASSIS ***
Module Status Ports Starting MAC
M1 (left): RX-BI-MR2 Management Module (High Value) Active
M2 (right): RX-BI-MR2 Management Module (High Value) Standby (Ready State)
F1: RX-BI-SFM1 Switch Fabric Module OK
F2: RX-BI-SFM1 Switch Fabric Module OK
F3: RX-BI-SFM1 Switch Fabric Module OK
S1: RX-BI-24F 24-port 1 GbE Module CARD_STATE_UP 24 748e.f872.e200
S2: RX-BI-24F 24-port 1 GbE Module CARD_STATE_UP 24 748e.f872.e200
S3: RX-BI-24C 24-port 1 GbE Copper Module CARD_STATE_UP 24 748e.f872.e200
S4: RX-BI-24C 24-port 1 GbE Copper Module CARD_STATE_UP 24 748e.f872.e200
==========================================================================
Output of show memory
show memory
====================================================================
BigIron RX active MP slot M1 (left):
Total SDRAM: 2147483648 bytes
Available Memory: 1442529280 bytes
Free Physical Pages: 351724 pages
Malloc statistics: total 653736881
OS malloc count: 2764399318
OS malloc fail: 0
OS free count: 2764351754
OS free fail: 0
diff: 47564
====================================================================
BigIron RX standby MP slot M2 (right):
Total SDRAM: 2147483648 bytes
Available Memory: 1730822144 bytes
Free Physical Pages: 422094 pages
====================================================================
BigIron RX LP SL 1:
Total SDRAM: 536870912 bytes
Available Memory: 64344064 bytes
====================================================================
BigIron RX LP SL 2:
Total SDRAM: 536870912 bytes
Available Memory: 64262144 bytes
====================================================================
BigIron RX LP SL 3:
Total SDRAM: 536870912 bytes
Available Memory: 64258048 bytes
====================================================================
BigIron RX LP SL 4:
Total SDRAM: 536870912 bytes
Available Memory: 64274432 bytes
Output of show version
show version
==========================================================================
HW: BigIron RX Router
BigIron RX-4 CHASSIS (Serial #: H02718HM1A, Part #: 35550-001)
RX-BI-SFM1 Switch Fabric Module 1 (Serial #: S52606H0Y7, Part #: 35548-001G)
FE 1: Type fe200, Version 2
RX-BI-SFM1 Switch Fabric Module 2 (Serial #: S52612H13C, Part #: 35548-001G)
FE 1: Type fe200, Version 2
RX-BI-SFM1 Switch Fabric Module 3 (Serial #: S52612H13F, Part #: 35548-001G)
FE 1: Type fe200, Version 2
==========================================================================
SL M1: RX-BI-MR2 Management Module (High Value) Active (Serial #: S02610H136, Part #: 35524-005A):
Boot : Version 2.9.0dT145 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:20 labeled as rmb02900d
(446876 bytes) from boot flash
Monitor : Version 2.9.0dT145 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:20 labeled as rmb02900d
(446876 bytes) from code flash
IronWare : Version 2.9.0dT143 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:58:26 labeled as rmpr02900d
(4694103 bytes) from Primary
Board ID : 00 MBRIDGE Revision : 21
916 MHz Power PC processor (version 8003/0101) 166 MHz bus
512 KB Boot Flash (AM29LV040B), 32 MB Code Flash (MT28F128J3)
2048 MB DRAM
Active Management uptime is 180 days 8 hours 34 minutes 9 seconds
==========================================================================
SL M2: RX-BI-MR2 Management Module (High Value) Standby (Serial #: SA12062073, Part #: 31524-000A):
Boot : Version 2.9.0dT145 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:20 labeled as rmb02900d
(446876 bytes) from boot flash
Monitor : Version 2.9.0dT145 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:20 labeled as rmb02900d
(446876 bytes) from code flash
IronWare : Version 2.9.0dT143 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:58:26 labeled as rmpr02900d
(4694103 bytes) from Primary
Board ID : 00 MBRIDGE Revision : 21
916 MHz Power PC processor (version 8003/0101) 166 MHz bus
512 KB Boot Flash (AM29LV040B), 32 MB Code Flash (MT28F128J3)
2048 MB DRAM
Standby Management uptime is 180 days 8 hours 36 minutes 0 seconds
==========================================================================
SL 1: RX-BI-24F 24-port 1 GbE Module (Serial #: SA32081622, Part #: 35520-104C)
Boot : Version 2.9.0dT155 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:12 labeled as rlb02900d
(307574 bytes) from boot flash
Monitor : Version 2.9.0dT155 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:12 labeled as rlb02900d
(524288 bytes) from code flash
IronWare : Version 2.9.0dT157 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 01:02:18 labeled as rlp02900d
(2381284 bytes) from Primary
FAP 1 version: 3
FAP 2 version: 3
FAP 3 version: 0
FAP 4 version: 0
660 MHz Power PC processor 440GP (version 8020/0020) 330 MHz bus
512 KB Boot Flash (AM29LV040B), 16 MB Code Flash (MT28F640J3)
512 MB DRAM, 8 KB SRAM, 0 Bytes BRAM
LP Slot 1 uptime is 180 days 8 hours 29 minutes 53 seconds
==========================================================================
SL 2: RX-BI-24F 24-port 1 GbE Module (Serial #: S32625F0D3, Part #: 35520-104J)
Boot : Version 2.9.0dT155 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:12 labeled as rlb02900d
(307574 bytes) from boot flash
Monitor : Version 2.9.0dT155 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:12 labeled as rlb02900d
(524288 bytes) from code flash
IronWare : Version 2.9.0dT157 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 01:02:18 labeled as rlp02900d
(2381284 bytes) from Primary
FAP 1 version: 3
FAP 2 version: 3
FAP 3 version: 0
FAP 4 version: 0
660 MHz Power PC processor 440GP (version 8020/0020) 330 MHz bus
512 KB Boot Flash (AM29LV040B), 16 MB Code Flash (MT28F640J3)
512 MB DRAM, 8 KB SRAM, 0 Bytes BRAM
LP Slot 2 uptime is 180 days 8 hours 35 minutes 43 seconds
==========================================================================
SL 3: RX-BI-24C 24-port 1 GbE Copper Module (Serial #: SA31080246, Part #: 35521-003C)
Boot : Version 2.9.0dT155 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:12 labeled as rlb02900d
(307574 bytes) from boot flash
Monitor : Version 2.9.0dT155 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:12 labeled as rlb02900d
(524288 bytes) from code flash
IronWare : Version 2.9.0dT157 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 01:02:18 labeled as rlp02900d
(2381284 bytes) from Primary
FAP 1 version: 3
FAP 2 version: 3
FAP 3 version: 0
FAP 4 version: 0
660 MHz Power PC processor 440GP (version 8020/0020) 330 MHz bus
512 KB Boot Flash (AM29LV040B), 16 MB Code Flash (MT28F640J3)
512 MB DRAM, 8 KB SRAM, 0 Bytes BRAM
LP Slot 3 uptime is 180 days 8 hours 16 minutes 35 seconds
==========================================================================
SL 4: RX-BI-24C 24-port 1 GbE Copper Module (Serial #: SA15071060, Part #: 35521-002C)
Boot : Version 2.9.0dT155 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:12 labeled as rlb02900d
(307574 bytes) from boot flash
Monitor : Version 2.9.0dT155 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:12 labeled as rlb02900d
(524288 bytes) from code flash
IronWare : Version 2.9.0dT157 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 01:02:18 labeled as rlp02900d
(2381284 bytes) from Primary
FAP 1 version: 3
FAP 2 version: 3
FAP 3 version: 0
FAP 4 version: 0
660 MHz Power PC processor 440GP (version 8020/0020) 330 MHz bus
512 KB Boot Flash (AM29LV040B), 16 MB Code Flash (MT28F640J3)
512 MB DRAM, 8 KB SRAM, 0 Bytes BRAM
LP Slot 4 uptime is 180 days 8 hours 30 minutes 7 seconds
==========================================================================
All show version done
Derek Maxwell, Chief Technology Officer
derek.maxwell@chosentechgroup.com<mailto:derek.maxwell@chosentechgroup.com>
The Chosen Technology Group, LLC<http://www.chosentechgroup.com/>
5401 Tech Circle, Second Floor
Moorpark, California 93021
North America: (844) 283-0100<tel:8442830100> | Worldwide: (805) 283-0100<tel:8052830100>
Direct Dial: (805) 283-0140<tel:8052830140>
www.chosentechgroup.com<http://www.chosentechgroup.com/>
[cid:image001.jpg@01D30DBA.4BF12790]
[cid:image002.jpg@01D30DBA.4BF12790]<http://www.chosentechgroup.com/>
About 3 months ago, we added a VLAN and a ve to correspond to it for an internal /28. We were unable to reach the assigned IP on the ve from outside of our network, which we ultimately discovered was due to a memory problem. The memory problem stemmed from improperly filtering one of our providers inbound v4 route table, which had suddenly grown by about 150k entries. That was resolved relatively easily.
Recently, it was time to add another VLAN and ve for another internal /28. This time, we did not have the same symptom, but found that a new host connected to module 3 could not communicate with any existing hosts on module 4, but could communicate with hosts on module 3. For the record, both are RX-BI-24C copper modules. We removed and added the VLAN and ve and even tried adding the new /28 to an existing vlan and ve.
We have checked free system memory, module software versions, and found nothing suspicious.
A full system reload has to be a last resort as this is a production VoIP site for us and even a five minute downtime requires significant planning. Does anyone have suggestions for troubleshooting or for reloading just a single module? We can function with just one of two modules, as all of our directly connected hosts are using XenServer active-active load balancing between them.
Thanks in advance for any guidance and assistance!
Output of show module:
show module
==========================================================================
*** BigIron RX-4 CHASSIS ***
Module Status Ports Starting MAC
M1 (left): RX-BI-MR2 Management Module (High Value) Active
M2 (right): RX-BI-MR2 Management Module (High Value) Standby (Ready State)
F1: RX-BI-SFM1 Switch Fabric Module OK
F2: RX-BI-SFM1 Switch Fabric Module OK
F3: RX-BI-SFM1 Switch Fabric Module OK
S1: RX-BI-24F 24-port 1 GbE Module CARD_STATE_UP 24 748e.f872.e200
S2: RX-BI-24F 24-port 1 GbE Module CARD_STATE_UP 24 748e.f872.e200
S3: RX-BI-24C 24-port 1 GbE Copper Module CARD_STATE_UP 24 748e.f872.e200
S4: RX-BI-24C 24-port 1 GbE Copper Module CARD_STATE_UP 24 748e.f872.e200
==========================================================================
Output of show memory
show memory
====================================================================
BigIron RX active MP slot M1 (left):
Total SDRAM: 2147483648 bytes
Available Memory: 1442529280 bytes
Free Physical Pages: 351724 pages
Malloc statistics: total 653736881
OS malloc count: 2764399318
OS malloc fail: 0
OS free count: 2764351754
OS free fail: 0
diff: 47564
====================================================================
BigIron RX standby MP slot M2 (right):
Total SDRAM: 2147483648 bytes
Available Memory: 1730822144 bytes
Free Physical Pages: 422094 pages
====================================================================
BigIron RX LP SL 1:
Total SDRAM: 536870912 bytes
Available Memory: 64344064 bytes
====================================================================
BigIron RX LP SL 2:
Total SDRAM: 536870912 bytes
Available Memory: 64262144 bytes
====================================================================
BigIron RX LP SL 3:
Total SDRAM: 536870912 bytes
Available Memory: 64258048 bytes
====================================================================
BigIron RX LP SL 4:
Total SDRAM: 536870912 bytes
Available Memory: 64274432 bytes
Output of show version
show version
==========================================================================
HW: BigIron RX Router
BigIron RX-4 CHASSIS (Serial #: H02718HM1A, Part #: 35550-001)
RX-BI-SFM1 Switch Fabric Module 1 (Serial #: S52606H0Y7, Part #: 35548-001G)
FE 1: Type fe200, Version 2
RX-BI-SFM1 Switch Fabric Module 2 (Serial #: S52612H13C, Part #: 35548-001G)
FE 1: Type fe200, Version 2
RX-BI-SFM1 Switch Fabric Module 3 (Serial #: S52612H13F, Part #: 35548-001G)
FE 1: Type fe200, Version 2
==========================================================================
SL M1: RX-BI-MR2 Management Module (High Value) Active (Serial #: S02610H136, Part #: 35524-005A):
Boot : Version 2.9.0dT145 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:20 labeled as rmb02900d
(446876 bytes) from boot flash
Monitor : Version 2.9.0dT145 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:20 labeled as rmb02900d
(446876 bytes) from code flash
IronWare : Version 2.9.0dT143 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:58:26 labeled as rmpr02900d
(4694103 bytes) from Primary
Board ID : 00 MBRIDGE Revision : 21
916 MHz Power PC processor (version 8003/0101) 166 MHz bus
512 KB Boot Flash (AM29LV040B), 32 MB Code Flash (MT28F128J3)
2048 MB DRAM
Active Management uptime is 180 days 8 hours 34 minutes 9 seconds
==========================================================================
SL M2: RX-BI-MR2 Management Module (High Value) Standby (Serial #: SA12062073, Part #: 31524-000A):
Boot : Version 2.9.0dT145 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:20 labeled as rmb02900d
(446876 bytes) from boot flash
Monitor : Version 2.9.0dT145 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:20 labeled as rmb02900d
(446876 bytes) from code flash
IronWare : Version 2.9.0dT143 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:58:26 labeled as rmpr02900d
(4694103 bytes) from Primary
Board ID : 00 MBRIDGE Revision : 21
916 MHz Power PC processor (version 8003/0101) 166 MHz bus
512 KB Boot Flash (AM29LV040B), 32 MB Code Flash (MT28F128J3)
2048 MB DRAM
Standby Management uptime is 180 days 8 hours 36 minutes 0 seconds
==========================================================================
SL 1: RX-BI-24F 24-port 1 GbE Module (Serial #: SA32081622, Part #: 35520-104C)
Boot : Version 2.9.0dT155 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:12 labeled as rlb02900d
(307574 bytes) from boot flash
Monitor : Version 2.9.0dT155 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:12 labeled as rlb02900d
(524288 bytes) from code flash
IronWare : Version 2.9.0dT157 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 01:02:18 labeled as rlp02900d
(2381284 bytes) from Primary
FAP 1 version: 3
FAP 2 version: 3
FAP 3 version: 0
FAP 4 version: 0
660 MHz Power PC processor 440GP (version 8020/0020) 330 MHz bus
512 KB Boot Flash (AM29LV040B), 16 MB Code Flash (MT28F640J3)
512 MB DRAM, 8 KB SRAM, 0 Bytes BRAM
LP Slot 1 uptime is 180 days 8 hours 29 minutes 53 seconds
==========================================================================
SL 2: RX-BI-24F 24-port 1 GbE Module (Serial #: S32625F0D3, Part #: 35520-104J)
Boot : Version 2.9.0dT155 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:12 labeled as rlb02900d
(307574 bytes) from boot flash
Monitor : Version 2.9.0dT155 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:12 labeled as rlb02900d
(524288 bytes) from code flash
IronWare : Version 2.9.0dT157 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 01:02:18 labeled as rlp02900d
(2381284 bytes) from Primary
FAP 1 version: 3
FAP 2 version: 3
FAP 3 version: 0
FAP 4 version: 0
660 MHz Power PC processor 440GP (version 8020/0020) 330 MHz bus
512 KB Boot Flash (AM29LV040B), 16 MB Code Flash (MT28F640J3)
512 MB DRAM, 8 KB SRAM, 0 Bytes BRAM
LP Slot 2 uptime is 180 days 8 hours 35 minutes 43 seconds
==========================================================================
SL 3: RX-BI-24C 24-port 1 GbE Copper Module (Serial #: SA31080246, Part #: 35521-003C)
Boot : Version 2.9.0dT155 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:12 labeled as rlb02900d
(307574 bytes) from boot flash
Monitor : Version 2.9.0dT155 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:12 labeled as rlb02900d
(524288 bytes) from code flash
IronWare : Version 2.9.0dT157 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 01:02:18 labeled as rlp02900d
(2381284 bytes) from Primary
FAP 1 version: 3
FAP 2 version: 3
FAP 3 version: 0
FAP 4 version: 0
660 MHz Power PC processor 440GP (version 8020/0020) 330 MHz bus
512 KB Boot Flash (AM29LV040B), 16 MB Code Flash (MT28F640J3)
512 MB DRAM, 8 KB SRAM, 0 Bytes BRAM
LP Slot 3 uptime is 180 days 8 hours 16 minutes 35 seconds
==========================================================================
SL 4: RX-BI-24C 24-port 1 GbE Copper Module (Serial #: SA15071060, Part #: 35521-002C)
Boot : Version 2.9.0dT155 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:12 labeled as rlb02900d
(307574 bytes) from boot flash
Monitor : Version 2.9.0dT155 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 00:52:12 labeled as rlb02900d
(524288 bytes) from code flash
IronWare : Version 2.9.0dT157 Copyright (c) 1996-2010 Brocade Communications Systems, Inc.
Compiled on Nov 13 2014 at 01:02:18 labeled as rlp02900d
(2381284 bytes) from Primary
FAP 1 version: 3
FAP 2 version: 3
FAP 3 version: 0
FAP 4 version: 0
660 MHz Power PC processor 440GP (version 8020/0020) 330 MHz bus
512 KB Boot Flash (AM29LV040B), 16 MB Code Flash (MT28F640J3)
512 MB DRAM, 8 KB SRAM, 0 Bytes BRAM
LP Slot 4 uptime is 180 days 8 hours 30 minutes 7 seconds
==========================================================================
All show version done
Derek Maxwell, Chief Technology Officer
derek.maxwell@chosentechgroup.com<mailto:derek.maxwell@chosentechgroup.com>
The Chosen Technology Group, LLC<http://www.chosentechgroup.com/>
5401 Tech Circle, Second Floor
Moorpark, California 93021
North America: (844) 283-0100<tel:8442830100> | Worldwide: (805) 283-0100<tel:8052830100>
Direct Dial: (805) 283-0140<tel:8052830140>
www.chosentechgroup.com<http://www.chosentechgroup.com/>
[cid:image001.jpg@01D30DBA.4BF12790]
[cid:image002.jpg@01D30DBA.4BF12790]<http://www.chosentechgroup.com/>
Attached Files:
-
image001.jpg (4.90 KB)
-
image002.jpg (10.9 KB)