Hello @ll,
Recently i encountered a drop of all BGP sessions on ASR9K due to low memory.
All of the neighbors (around 40) status changed to OOM
RP/0/RSP0/CPU0:#sh bgp ipv4 unicast summary
Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 115785187 115785187 115785187 115785187 115785187 115785187
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
x.x.x.x 0 12345 4374395 186421 0 0 0 00:42:43 Idle (OOM)
The log stated:
RP/0/RSP0/CPU0:Jan 9 18:33:19.245 : wdsysmon[451]: %HA-HA_WD-4-TOP_MEMORY_USER_WARNING : 0: Process Name: bgp[1052], pid: 581965, Heap usage 810544 Kbytes, Virtual Shared memor
y usage: 56144 Kbytes.
RP/0/RSP0/CPU0:Jan 9 18:33:19.245 : wdsysmon[451]: %HA-HA_WD-4-TOP_MEMORY_USER_WARNING : 1: Process Name: mibd_entity[318], pid: 2167021, Heap usage 611188 Kbytes, Virtual Shar
ed memory usage: 26132 Kbytes.
RP/0/RSP0/CPU0:Jan 9 18:33:19.245 : wdsysmon[451]: %HA-HA_WD-4-TOP_MEMORY_USER_WARNING : 2: Process Name: ipv4_rib[1149], pid: 577855, Heap usage 312280 Kbytes, Virtual Shared
memory usage: 28516 Kbytes.
RP/0/RSP0/CPU0:Jan 9 18:33:19.245 : wdsysmon[451]: %HA-HA_WD-4-TOP_MEMORY_USER_WARNING : 3: Process Name: ipv6_rib[1150], pid: 577858, Heap usage 53604 Kbytes, Virtual Shared m
emory usage: 28584 Kbytes.
RP/0/RSP0/CPU0:Jan 9 18:33:19.245 : wdsysmon[451]: %HA-HA_WD-4-TOP_MEMORY_USER_WARNING : 4: Process Name: eth_server[60], pid: 53292, Heap usage 33420 Kbytes, Virtual Shared me
mory usage: 32568 Kbytes.
RP/0/RSP0/CPU0:Jan 9 18:33:20.251 : bgp[1052]: %ROUTING-BGP-5-LOWMEM_RESET : neighbor x.x.x.x reset due to SEVERE low memory condition - use 'clear bgp x.x.x.x' to
re-enable
RP/0/RSP0/CPU0:Jan 9 18:33:20.251 : bgp[1052]: %ROUTING-BGP-5-ADJCHANGE : neighbor x.x.x.x Down - Shutdown during SEVERE low memory condition (CEASE notification sent - o
ut of resource) (VRF: default) (AS: xxx)
RP/0/RSP1/CPU0:Jan 9 18:33:20.252 : bgp[1052]: %ROUTING-BGP-5-NBR_NSR_DISABLED_STANDBY : NSR disabled on neighbor x.x.x.x on standby RP due to Shutdown during SEVERE low
memory condition (VRF: default)
RP/0/RSP0/CPU0:Jan 9 18:33:34.226 : fib_mgr[221]: %OS-SHMWIN-2-ERROR_ENCOUNTERED : SHMWIN: Error encountered: System memory state is severe, please check the availability of th
e system memory
LC/0/2/CPU0:Jan 9 18:39:39.714 : fib_mgr[177]: %PLATFORM-PLAT_FIB-4-WARN_STR_PRM_OOR : The utilization of the object IPv4_LEAF_P drops below 83% on all NPs
LC/0/0/CPU0:Jan 9 18:45:10.210 : fib_mgr[177]: %PLATFORM-PLAT_FIB-4-WARN_STR_PRM_OOR : The utilization of the object IPv6_LEAF_P drops below 83% on all NPs
RP/0/RSP0/CPU0:(admin)#sh memory summary
Thu Jan 9 19:21:03.837 CEST
Physical Memory: 4096M total (565M available)
Application Memory : 3897M (565M available)
Image: 69M (bootram: 69M)
Reserved: 128M, IOMem: 2028M, flashfsys: 0
Total shared window: 191M
The IOS-XR installed is 5.3.4-SP10-1.0.0
While investigating we changed profile from L3 to L3XL inorder to be able to install more routes:
RP/0/RSP0/CPU0:#show cef platform resource summary loc 0/0/cpu0
Thu Jan 16 13:45:23.052 CEST
OBJECT USED MAX AVAILABLE
RPF_STRICT 0 65536 65536
IPv4_LEAF_P 780806 1314000 533194
IPv6_LEAF_P 389903 656500 266597
LEAF 826182 1460000 633818
TX_ADJ 826182 1460000 633818
NR_LDI 826182 1460000 633818
TE_NH_ADJ 0 32768 32768
RX_ADJ 28 32768 32740
R_LDI 13266 65536 52270
L2VPN_LDI 0 2048 2048
EXT_LSPA 0 0 0
IPv6_LL_LEAF_P 0 262144 262144
The problem is that we are unable to explain the value of 389903 concerning IPv6_LEAF_P. We tried dropping all IPv6 session but the value remained unchanged.
We decided to remove all BGP sessions:
RP/0/RSP0/CPU0:#sh cef platform resource summary location 0/0/CPU0
Fri Feb 14 07:25:27.917 CEST
OBJECT USED MAX AVAILABLE
RPF_STRICT 0 65536 65536
IPv4_LEAF_P 800 1314000 1313200
IPv6_LEAF_P 21 656500 656479 <—— Used=21. Value is correct
LEAF 40761 1460000 1419239
TX_ADJ 40761 1460000 1419239
NR_LDI 40761 1460000 1419239
TE_NH_ADJ 0 32768 32768
RX_ADJ 28 32768 32740
R_LDI 13265 65536 52271
L2VPN_LDI 0 2048 2048
EXT_LSPA 0 0 0
IPv6_LL_LEAF_P 0 262144 262144
RP/0/RSP0/CPU0:PA7#show cef platform resource location 0/0/CPU0
IPV6_LEAF_P usage is same on all NPs
NP: 0 struct 24: IPV6_LEAF_P (maps to ucode stru = 55)
Used Entries: 21 Max Entries: 656500
Adding one BGP IPv4 session:
RP/0/RSP0/CPU0:#sh cef platform resource summary location 0/0/CPU0
Fri Feb 14 07:34:57.943 CEST
OBJECT USED MAX AVAILABLE
RPF_STRICT 0 65536 65536
IPv4_LEAF_P 807653 1314000 506310
IPv6_LEAF_P 403345 656500 253155
LEAF 847658 1460000 612342
TX_ADJ 847658 1460000 612342
NR_LDI 847658 1460000 612342
TE_NH_ADJ 0 32768 32768
RX_ADJ 28 32768 32740
R_LDI 13266 65536 52270
L2VPN_LDI 0 2048 2048
EXT_LSPA 0 0 0
IPv6_LL_LEAF_P 0 262144 262144
But the following output states completely a different value:
RP/0/RSP0/CPU0:#show cef platform resource location 0/0/CPU0
Fri Feb 14 07:35:36.556 CEST
Node: 0/0/CPU0
----------------------------------------------------------------
RPF_STRICT usage is same on all NPs
NP: 0 struct 9: RPF_STRICT (maps to ucode stru = 15)
Used Entries: 0 Max Entries: 65536
-------------------------------------------------------------
IPV4_LEAF_P usage is same on all NPs
NP: 0 struct 23: IPV4_LEAF_P (maps to ucode stru = 54)
Used Entries: 807653 Max Entries: 1314000
-------------------------------------------------------------
IPV6_LEAF_P usage is same on all NPs
NP: 0 struct 24: IPV6_LEAF_P (maps to ucode stru = 55)
Used Entries: 21 Max Entries: 656500
-------------------------------------------------------------
Can someone help on this please?
We have another ASR9K with an older version of IOS-XR (4.2.1) which is actually configured with profile L3 only and have the same number of routes of the router version 5.3.4 but it never encountered any problem.
Thanks.
Regards,
Alex
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Recently i encountered a drop of all BGP sessions on ASR9K due to low memory.
All of the neighbors (around 40) status changed to OOM
RP/0/RSP0/CPU0:#sh bgp ipv4 unicast summary
Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 115785187 115785187 115785187 115785187 115785187 115785187
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
x.x.x.x 0 12345 4374395 186421 0 0 0 00:42:43 Idle (OOM)
The log stated:
RP/0/RSP0/CPU0:Jan 9 18:33:19.245 : wdsysmon[451]: %HA-HA_WD-4-TOP_MEMORY_USER_WARNING : 0: Process Name: bgp[1052], pid: 581965, Heap usage 810544 Kbytes, Virtual Shared memor
y usage: 56144 Kbytes.
RP/0/RSP0/CPU0:Jan 9 18:33:19.245 : wdsysmon[451]: %HA-HA_WD-4-TOP_MEMORY_USER_WARNING : 1: Process Name: mibd_entity[318], pid: 2167021, Heap usage 611188 Kbytes, Virtual Shar
ed memory usage: 26132 Kbytes.
RP/0/RSP0/CPU0:Jan 9 18:33:19.245 : wdsysmon[451]: %HA-HA_WD-4-TOP_MEMORY_USER_WARNING : 2: Process Name: ipv4_rib[1149], pid: 577855, Heap usage 312280 Kbytes, Virtual Shared
memory usage: 28516 Kbytes.
RP/0/RSP0/CPU0:Jan 9 18:33:19.245 : wdsysmon[451]: %HA-HA_WD-4-TOP_MEMORY_USER_WARNING : 3: Process Name: ipv6_rib[1150], pid: 577858, Heap usage 53604 Kbytes, Virtual Shared m
emory usage: 28584 Kbytes.
RP/0/RSP0/CPU0:Jan 9 18:33:19.245 : wdsysmon[451]: %HA-HA_WD-4-TOP_MEMORY_USER_WARNING : 4: Process Name: eth_server[60], pid: 53292, Heap usage 33420 Kbytes, Virtual Shared me
mory usage: 32568 Kbytes.
RP/0/RSP0/CPU0:Jan 9 18:33:20.251 : bgp[1052]: %ROUTING-BGP-5-LOWMEM_RESET : neighbor x.x.x.x reset due to SEVERE low memory condition - use 'clear bgp x.x.x.x' to
re-enable
RP/0/RSP0/CPU0:Jan 9 18:33:20.251 : bgp[1052]: %ROUTING-BGP-5-ADJCHANGE : neighbor x.x.x.x Down - Shutdown during SEVERE low memory condition (CEASE notification sent - o
ut of resource) (VRF: default) (AS: xxx)
RP/0/RSP1/CPU0:Jan 9 18:33:20.252 : bgp[1052]: %ROUTING-BGP-5-NBR_NSR_DISABLED_STANDBY : NSR disabled on neighbor x.x.x.x on standby RP due to Shutdown during SEVERE low
memory condition (VRF: default)
RP/0/RSP0/CPU0:Jan 9 18:33:34.226 : fib_mgr[221]: %OS-SHMWIN-2-ERROR_ENCOUNTERED : SHMWIN: Error encountered: System memory state is severe, please check the availability of th
e system memory
LC/0/2/CPU0:Jan 9 18:39:39.714 : fib_mgr[177]: %PLATFORM-PLAT_FIB-4-WARN_STR_PRM_OOR : The utilization of the object IPv4_LEAF_P drops below 83% on all NPs
LC/0/0/CPU0:Jan 9 18:45:10.210 : fib_mgr[177]: %PLATFORM-PLAT_FIB-4-WARN_STR_PRM_OOR : The utilization of the object IPv6_LEAF_P drops below 83% on all NPs
RP/0/RSP0/CPU0:(admin)#sh memory summary
Thu Jan 9 19:21:03.837 CEST
Physical Memory: 4096M total (565M available)
Application Memory : 3897M (565M available)
Image: 69M (bootram: 69M)
Reserved: 128M, IOMem: 2028M, flashfsys: 0
Total shared window: 191M
The IOS-XR installed is 5.3.4-SP10-1.0.0
While investigating we changed profile from L3 to L3XL inorder to be able to install more routes:
RP/0/RSP0/CPU0:#show cef platform resource summary loc 0/0/cpu0
Thu Jan 16 13:45:23.052 CEST
OBJECT USED MAX AVAILABLE
RPF_STRICT 0 65536 65536
IPv4_LEAF_P 780806 1314000 533194
IPv6_LEAF_P 389903 656500 266597
LEAF 826182 1460000 633818
TX_ADJ 826182 1460000 633818
NR_LDI 826182 1460000 633818
TE_NH_ADJ 0 32768 32768
RX_ADJ 28 32768 32740
R_LDI 13266 65536 52270
L2VPN_LDI 0 2048 2048
EXT_LSPA 0 0 0
IPv6_LL_LEAF_P 0 262144 262144
The problem is that we are unable to explain the value of 389903 concerning IPv6_LEAF_P. We tried dropping all IPv6 session but the value remained unchanged.
We decided to remove all BGP sessions:
RP/0/RSP0/CPU0:#sh cef platform resource summary location 0/0/CPU0
Fri Feb 14 07:25:27.917 CEST
OBJECT USED MAX AVAILABLE
RPF_STRICT 0 65536 65536
IPv4_LEAF_P 800 1314000 1313200
IPv6_LEAF_P 21 656500 656479 <—— Used=21. Value is correct
LEAF 40761 1460000 1419239
TX_ADJ 40761 1460000 1419239
NR_LDI 40761 1460000 1419239
TE_NH_ADJ 0 32768 32768
RX_ADJ 28 32768 32740
R_LDI 13265 65536 52271
L2VPN_LDI 0 2048 2048
EXT_LSPA 0 0 0
IPv6_LL_LEAF_P 0 262144 262144
RP/0/RSP0/CPU0:PA7#show cef platform resource location 0/0/CPU0
IPV6_LEAF_P usage is same on all NPs
NP: 0 struct 24: IPV6_LEAF_P (maps to ucode stru = 55)
Used Entries: 21 Max Entries: 656500
Adding one BGP IPv4 session:
RP/0/RSP0/CPU0:#sh cef platform resource summary location 0/0/CPU0
Fri Feb 14 07:34:57.943 CEST
OBJECT USED MAX AVAILABLE
RPF_STRICT 0 65536 65536
IPv4_LEAF_P 807653 1314000 506310
IPv6_LEAF_P 403345 656500 253155
LEAF 847658 1460000 612342
TX_ADJ 847658 1460000 612342
NR_LDI 847658 1460000 612342
TE_NH_ADJ 0 32768 32768
RX_ADJ 28 32768 32740
R_LDI 13266 65536 52270
L2VPN_LDI 0 2048 2048
EXT_LSPA 0 0 0
IPv6_LL_LEAF_P 0 262144 262144
But the following output states completely a different value:
RP/0/RSP0/CPU0:#show cef platform resource location 0/0/CPU0
Fri Feb 14 07:35:36.556 CEST
Node: 0/0/CPU0
----------------------------------------------------------------
RPF_STRICT usage is same on all NPs
NP: 0 struct 9: RPF_STRICT (maps to ucode stru = 15)
Used Entries: 0 Max Entries: 65536
-------------------------------------------------------------
IPV4_LEAF_P usage is same on all NPs
NP: 0 struct 23: IPV4_LEAF_P (maps to ucode stru = 54)
Used Entries: 807653 Max Entries: 1314000
-------------------------------------------------------------
IPV6_LEAF_P usage is same on all NPs
NP: 0 struct 24: IPV6_LEAF_P (maps to ucode stru = 55)
Used Entries: 21 Max Entries: 656500
-------------------------------------------------------------
Can someone help on this please?
We have another ASR9K with an older version of IOS-XR (4.2.1) which is actually configured with profile L3 only and have the same number of routes of the router version 5.3.4 but it never encountered any problem.
Thanks.
Regards,
Alex
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/