Mailing List Archive

Hello Vladimir,

What version IOS XR are you using?
We are doesn't have problems with FIB inconsistency. IOS XR 6.6.3.

#show processes memory detail location 0/RSP0/CPU0
Wed May 20 09:09:23.240 MSK
JID Text Data Stack Dynamic Dyn-Limit Shm-Tot Phy-Tot Process
------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- -------
1087 1M 10M 624K 818M 1658M 218M 829M bgp

#show memory summary location 0/0/CPU0
Wed May 20 09:10:34.206 MSK
node: node0_0_CPU0
--------------------------------------
Physical Memory: 8192M total
Application Memory : 7985M (4258M available)
Image: 78M (bootram: 78M)
Reserved: 128M, IOMem: 0, flashfsys: 0
Total shared window: 495M


On Tue, 19 May 2020 20:09:52 +0500
Vladimir Troitskiy <ruthenate@gmail.com> wrote:

> Hello everyone,
>
> ASR9001 has some memory usage limits:
> - 1658M for a BGP process on a RSP
> - 1536M for a shared memory window on a LC
> Those limits seems to be unconfigurable.
>
> Has anybody experienced any issues with these limits on high-loaded ASR9001
> boxes?
> We have a surprisingly high memory usage while the typical router setup is
> pretty lightweight - 4-5 full feeds (couple of upstreams and RRs). The only
> probably uncommon thing is we use "Internet in a VRF" approach.
>
> #show processes memory detail location 0/RSP0/CPU0
> > Tue May 19 19:39:12.592 Ural
> > JID Text Data Stack Dynamic Dyn-Limit Shm-Tot
> > Phy-Tot Process
> > ------ ---------- ---------- ---------- ---------- ---------- ----------
> > ---------- -------
> > 1054 1M 5M 516K 1485M 1658M 76M
> > 1491M bgp
> >
>
> #show memory summary location 0/0/CPU0
> >
> node: node0_0_CPU0
> > ------------------------------------------------------------------
> > Physical Memory: 8192M total
> > Application Memory : 7988M (3811M available)
> > Image: 75M (bootram: 75M)
> > Reserved: 128M, IOMem: 0, flashfsys: 0
> > Total shared window: 1327M
> >
>
> We have already had FIB inconsistency issues due to SHMWIN exhaustion
> despite the fact the total prefix amount was far from the platform limit
> (4M):
>
> > fib_mgr[184]: %OS-SHMWIN-3-ALLOC_ARENA_FAILED : SHMWIN: Failed to allocate
> > new arena from the server : 'SHMWIN_SVR' detected the 'fatal' condition 'VM
> > is exhausted or totally fragmented'
> > fib_mgr[184]: %ROUTING-FIB-3-ASSERT_RL : FIB internal inconsistency
> > detected
> > fib_mgr[184]: %ROUTING-FIB-3-PD_FAIL : FIB platform error: fib_leaf_insert
> > 5204 Cannot insert leaf
> >
>
> What are practical limits for BGP scaling on ASR9001 boxes? Could anyone
> share a memory usage stats?
> --
> Best regards,
> Vladimir Troitsky
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/


--
Alexandr Gurbo <gurbo@golas.ru>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Hello Alexandr,

Thank you for your input! We are using IOS XR 5.3.4 and 6.1.4 - no
significant difference in memory consumption between them.
How many peers/routes do you have on this box? Are those peers configured
in a GRT or in a VRF?

??, 20 ??? 2020 ?. ? 11:17, Alexandr Gurbo <gurbo@golas.ru>:

> Hello Vladimir,
>
> What version IOS XR are you using?
> We are doesn't have problems with FIB inconsistency. IOS XR 6.6.3.
>
> #show processes memory detail location 0/RSP0/CPU0
> Wed May 20 09:09:23.240 MSK
> JID Text Data Stack Dynamic Dyn-Limit Shm-Tot
> Phy-Tot Process
> ------ ---------- ---------- ---------- ---------- ---------- ----------
> ---------- -------
> 1087 1M 10M 624K 818M 1658M 218M
> 829M bgp
>
> #show memory summary location 0/0/CPU0
> Wed May 20 09:10:34.206 MSK
> node: node0_0_CPU0
> --------------------------------------
> Physical Memory: 8192M total
> Application Memory : 7985M (4258M available)
> Image: 78M (bootram: 78M)
> Reserved: 128M, IOMem: 0, flashfsys: 0
> Total shared window: 495M
>
> --
> Alexandr Gurbo <gurbo@golas.ru>
>
--
Best regards,
Vladimir Troitsky
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

All in GRT, 3 full tables, 2 big IX, couple private peers. No MPLS, only routing.

On Wed, 20 May 2020 11:44:05 +0500
Vladimir Troitskiy <ruthenate@gmail.com> wrote:

> Hello Alexandr,
>
> Thank you for your input! We are using IOS XR 5.3.4 and 6.1.4 - no
> significant difference in memory consumption between them.
> How many peers/routes do you have on this box? Are those peers configured
> in a GRT or in a VRF?
>
> ??, 20 ??? 2020 ?. ? 11:17, Alexandr Gurbo <gurbo@golas.ru>:
>
> > Hello Vladimir,
> >
> > What version IOS XR are you using?
> > We are doesn't have problems with FIB inconsistency. IOS XR 6.6.3.
> >
> > #show processes memory detail location 0/RSP0/CPU0
> > Wed May 20 09:09:23.240 MSK
> > JID Text Data Stack Dynamic Dyn-Limit Shm-Tot
> > Phy-Tot Process
> > ------ ---------- ---------- ---------- ---------- ---------- ----------
> > ---------- -------
> > 1087 1M 10M 624K 818M 1658M 218M
> > 829M bgp
> >
> > #show memory summary location 0/0/CPU0
> > Wed May 20 09:10:34.206 MSK
> > node: node0_0_CPU0
> > --------------------------------------
> > Physical Memory: 8192M total
> > Application Memory : 7985M (4258M available)
> > Image: 78M (bootram: 78M)
> > Reserved: 128M, IOMem: 0, flashfsys: 0
> > Total shared window: 495M
> >
> > --
> > Alexandr Gurbo <gurbo@golas.ru>
> >
> --
> Best regards,
> Vladimir Troitsky


--
Alexandr Gurbo <gurbo@golas.ru>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Slightly unrelated to this thread but also sort of related.

Did anyone else notice that this file appears in the ASR9001 IOS XR file list now?

asr9k-9000v-nV-x64-1.0.0.0-r702.x86_64.rpm

I was under the impression that ASR 9001 couldn't run x86_64 software and also... why is it an RPM rather than a tar?



-----Original Message-----
From: cisco-nsp <cisco-nsp-bounces@puck.nether.net> On Behalf Of Alexandr Gurbo
Sent: Wednesday, May 20, 2020 3:32 AM
To: Vladimir Troitskiy <ruthenate@gmail.com>
Cc: cisco-nsp@puck.nether.net
Subject: Re: [c-nsp] ASR9001 BGP scaling and memory shortage


All in GRT, 3 full tables, 2 big IX, couple private peers. No MPLS, only routing.

On Wed, 20 May 2020 11:44:05 +0500
Vladimir Troitskiy <ruthenate@gmail.com> wrote:

> Hello Alexandr,
>
> Thank you for your input! We are using IOS XR 5.3.4 and 6.1.4 - no
> significant difference in memory consumption between them.
> How many peers/routes do you have on this box? Are those peers
> configured in a GRT or in a VRF?
>
> ??, 20 ??? 2020 ?. ? 11:17, Alexandr Gurbo <gurbo@golas.ru>:
>
> > Hello Vladimir,
> >
> > What version IOS XR are you using?
> > We are doesn't have problems with FIB inconsistency. IOS XR 6.6.3.
> >
> > #show processes memory detail location 0/RSP0/CPU0 Wed May 20
> > 09:09:23.240 MSK
> > JID Text Data Stack Dynamic Dyn-Limit Shm-Tot
> > Phy-Tot Process
> > ------ ---------- ---------- ---------- ---------- ----------
> > ----------
> > ---------- -------
> > 1087 1M 10M 624K 818M 1658M 218M
> > 829M bgp
> >
> > #show memory summary location 0/0/CPU0 Wed May 20 09:10:34.206 MSK
> > node: node0_0_CPU0
> > --------------------------------------
> > Physical Memory: 8192M total
> > Application Memory : 7985M (4258M available)
> > Image: 78M (bootram: 78M)
> > Reserved: 128M, IOMem: 0, flashfsys: 0 Total shared window: 495M
> >
> > --
> > Alexandr Gurbo <gurbo@golas.ru>
> >
> --
> Best regards,
> Vladimir Troitsky


--
Alexandr Gurbo <gurbo@golas.ru>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

eXR is Linux based 64 bit vs classic XR which is the 32 bit qnx kernel.
Some releases have both

On Wednesday, May 20, 2020, Drew Weaver <drew.weaver@thenap.com> wrote:

> Slightly unrelated to this thread but also sort of related.
>
> Did anyone else notice that this file appears in the ASR9001 IOS XR file
> list now?
>
> asr9k-9000v-nV-x64-1.0.0.0-r702.x86_64.rpm
>
> I was under the impression that ASR 9001 couldn't run x86_64 software and
> also... why is it an RPM rather than a tar?
>
>
>
> -----Original Message-----
> From: cisco-nsp <cisco-nsp-bounces@puck.nether.net> On Behalf Of Alexandr
> Gurbo
> Sent: Wednesday, May 20, 2020 3:32 AM
> To: Vladimir Troitskiy <ruthenate@gmail.com>
> Cc: cisco-nsp@puck.nether.net
> Subject: Re: [c-nsp] ASR9001 BGP scaling and memory shortage
>
>
> All in GRT, 3 full tables, 2 big IX, couple private peers. No MPLS, only
> routing.
>
> On Wed, 20 May 2020 11:44:05 +0500
> Vladimir Troitskiy <ruthenate@gmail.com> wrote:
>
> > Hello Alexandr,
> >
> > Thank you for your input! We are using IOS XR 5.3.4 and 6.1.4 - no
> > significant difference in memory consumption between them.
> > How many peers/routes do you have on this box? Are those peers
> > configured in a GRT or in a VRF?
> >
> > ??, 20 ??? 2020 ?. ? 11:17, Alexandr Gurbo <gurbo@golas.ru>:
> >
> > > Hello Vladimir,
> > >
> > > What version IOS XR are you using?
> > > We are doesn't have problems with FIB inconsistency. IOS XR 6.6.3.
> > >
> > > #show processes memory detail location 0/RSP0/CPU0 Wed May 20
> > > 09:09:23.240 MSK
> > > JID Text Data Stack Dynamic Dyn-Limit Shm-Tot
> > > Phy-Tot Process
> > > ------ ---------- ---------- ---------- ---------- ----------
> > > ----------
> > > ---------- -------
> > > 1087 1M 10M 624K 818M 1658M
> 218M
> > > 829M bgp
> > >
> > > #show memory summary location 0/0/CPU0 Wed May 20 09:10:34.206 MSK
> > > node: node0_0_CPU0
> > > --------------------------------------
> > > Physical Memory: 8192M total
> > > Application Memory : 7985M (4258M available)
> > > Image: 78M (bootram: 78M)
> > > Reserved: 128M, IOMem: 0, flashfsys: 0 Total shared window: 495M
> > >
> > > --
> > > Alexandr Gurbo <gurbo@golas.ru>
> > >
> > --
> > Best regards,
> > Vladimir Troitsky
>
>
> --
> Alexandr Gurbo <gurbo@golas.ru>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Hello everyone,

Other list members have a significantly lower memory usage for a BGP
process and a shmwin on ASR9001 routers with more sessions/routes in GRT.

Saku Ytti has suggested me some useful notes which I would like to mention
as a summary for this thread:
- one could use 'hw-module profile scale l3xl' in admin mode to increase an
RLIMIT for a BGP process, even on Typhoon-based platforms (not only on
Trident-based ones as I thought);
- a shmwin shortage is probably caused by per-prefix label mode, the per-ce
mode will be much more scalable. We use the per-prefix mode because of BGP
PIC limitations, but maybe it's time to reconsider the feature-set used.

??, 19 ??? 2020 ?. ? 20:09, Vladimir Troitskiy <ruthenate@gmail.com>:

> Hello everyone,
>
> ASR9001 has some memory usage limits:
> - 1658M for a BGP process on a RSP
> - 1536M for a shared memory window on a LC
> Those limits seems to be unconfigurable.
>
> Has anybody experienced any issues with these limits on high-loaded
> ASR9001 boxes?
> We have a surprisingly high memory usage while the typical router setup is
> pretty lightweight - 4-5 full feeds (couple of upstreams and RRs). The only
> probably uncommon thing is we use "Internet in a VRF" approach.
>
> #show processes memory detail location 0/RSP0/CPU0
>> Tue May 19 19:39:12.592 Ural
>> JID Text Data Stack Dynamic Dyn-Limit Shm-Tot
>> Phy-Tot Process
>> ------ ---------- ---------- ---------- ---------- ---------- ----------
>> ---------- -------
>> 1054 1M 5M 516K 1485M 1658M 76M
>> 1491M bgp
>>
>
> #show memory summary location 0/0/CPU0
>>
> node: node0_0_CPU0
>> ------------------------------------------------------------------
>> Physical Memory: 8192M total
>> Application Memory : 7988M (3811M available)
>> Image: 75M (bootram: 75M)
>> Reserved: 128M, IOMem: 0, flashfsys: 0
>> Total shared window: 1327M
>>
>
> We have already had FIB inconsistency issues due to SHMWIN exhaustion
> despite the fact the total prefix amount was far from the platform limit
> (4M):
>
>> fib_mgr[184]: %OS-SHMWIN-3-ALLOC_ARENA_FAILED : SHMWIN: Failed to
>> allocate new arena from the server : 'SHMWIN_SVR' detected the 'fatal'
>> condition 'VM is exhausted or totally fragmented'
>> fib_mgr[184]: %ROUTING-FIB-3-ASSERT_RL : FIB internal inconsistency
>> detected
>> fib_mgr[184]: %ROUTING-FIB-3-PD_FAIL : FIB platform error:
>> fib_leaf_insert 5204 Cannot insert leaf
>>
>
> What are practical limits for BGP scaling on ASR9001 boxes? Could anyone
> share a memory usage stats?
> --
> Best regards,
> Vladimir Troitsky
>

--
Best regards,
Vladimir Troitsky
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/