Mailing List Archive

'show route summary'
'start shell pfe network fpcX'
'show jnh N pool summary'
'show jnh N pool usage'

On Wed, 4 May 2022 at 23:31, Nehul Patel <nehul.patel@gmail.com> wrote:
>
>
> Hi NANOG,
>
> We are seeing some strange behavior on our Juniper MX240 Chassis it is randomly dropping the routes to the certain destination IP address getting the following errors on the MX240 Chassis
>
> If Someone has seen these errors before please suggest how to resolve it
>
>
> May 4 12:42:00 cr01 newsyslog[44735]: logfile turned over due to size>1024K
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid)
> May 4 12:42:01 last message repeated 4 times
> May 4 12:42:01 fpc0 RT: IPv6:0 - 2600:40fc:1011::/48 (add rt entry into jtree failed)
> May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
> May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2600:40fc:1011::/48 nh 1048576
> May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
> May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2001:67c:20fc::/48 (No memory) on FE 0
> May 4 12:42:01 fpc0 RT: IPv6:0 - 2001:67c:20fc::/48 (add rt entry into jtree failed)
> May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
> May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2001:67c:20fc::/48 nh 1048576
> May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
> May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2606:2800:e004::/48 (No memory) on FE 0
> May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2a05:3181:ffff::/48 (No memory) on FE 0
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid)
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 2 (PREFIX DELETE) failed, err 5 (Invalid)
> May 4 12:42:02 fpc0 RT: Failed prefix add IPv4 - 79.120.22/24 (No memory) on FE 0
> May 4 12:42:02 fpc0 RT: IPv4:0 - 79.120.22/24 (add rt entry into jtree failed)
> May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
> May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv4,len 24 prefix 79.120.22/24 nh 1048583
> May 4 12:42:02 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
> May 4 12:42:02 fpc0 RT-HAL,rt_msg_handler,540: route process failed
>
> May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-pages Available:20 is less than LWM limit:1638, rsmon_syslog_limit()
> May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-dwords Available:1280 is less than LWM limit:104857, rsmon_syslog_limit()
> May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit()
> May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-pages Available:16 is less than LWM limit:1638, rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-dwords Available:1024 is less than LWM limit:104857, rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit()
> May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit()
> May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-pages Available:17 is less than LWM limit:1638, rsmon_syslog_limit()
> May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-dwords Available:1088 is less than LWM limit:104857, rsmon_syslog_limit()
> May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
> May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()
> May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
> May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()
>
> Any suggestions will be helpful
>
>
> Please do let me know if you have any questions.
>
>
>
>
> Regards and thanks,
> Nehul
>
>


--
++ytti

Actually is this DPCE? 'show jtree N summary'

On Wed, 4 May 2022 at 23:39, Saku Ytti <saku@ytti.fi> wrote:
>
> 'show route summary'
> 'start shell pfe network fpcX'
> 'show jnh N pool summary'
> 'show jnh N pool usage'
>
> On Wed, 4 May 2022 at 23:31, Nehul Patel <nehul.patel@gmail.com> wrote:
> >
> >
> > Hi NANOG,
> >
> > We are seeing some strange behavior on our Juniper MX240 Chassis it is randomly dropping the routes to the certain destination IP address getting the following errors on the MX240 Chassis
> >
> > If Someone has seen these errors before please suggest how to resolve it
> >
> >
> > May 4 12:42:00 cr01 newsyslog[44735]: logfile turned over due to size>1024K
> > May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
> > May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid)
> > May 4 12:42:01 last message repeated 4 times
> > May 4 12:42:01 fpc0 RT: IPv6:0 - 2600:40fc:1011::/48 (add rt entry into jtree failed)
> > May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
> > May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2600:40fc:1011::/48 nh 1048576
> > May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
> > May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2001:67c:20fc::/48 (No memory) on FE 0
> > May 4 12:42:01 fpc0 RT: IPv6:0 - 2001:67c:20fc::/48 (add rt entry into jtree failed)
> > May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
> > May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2001:67c:20fc::/48 nh 1048576
> > May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
> > May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2606:2800:e004::/48 (No memory) on FE 0
> > May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2a05:3181:ffff::/48 (No memory) on FE 0
> > May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid)
> > May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
> > May 4 12:42:01 /kernel: RT_PFE: RT msg op 2 (PREFIX DELETE) failed, err 5 (Invalid)
> > May 4 12:42:02 fpc0 RT: Failed prefix add IPv4 - 79.120.22/24 (No memory) on FE 0
> > May 4 12:42:02 fpc0 RT: IPv4:0 - 79.120.22/24 (add rt entry into jtree failed)
> > May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
> > May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv4,len 24 prefix 79.120.22/24 nh 1048583
> > May 4 12:42:02 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
> > May 4 12:42:02 fpc0 RT-HAL,rt_msg_handler,540: route process failed
> >
> > May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-pages Available:20 is less than LWM limit:1638, rsmon_syslog_limit()
> > May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-dwords Available:1280 is less than LWM limit:104857, rsmon_syslog_limit()
> > May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit()
> > May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit()
> > May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-pages Available:16 is less than LWM limit:1638, rsmon_syslog_limit()
> > May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-dwords Available:1024 is less than LWM limit:104857, rsmon_syslog_limit()
> > May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
> > May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()
> > May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit()
> > May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit()
> > May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-pages Available:17 is less than LWM limit:1638, rsmon_syslog_limit()
> > May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-dwords Available:1088 is less than LWM limit:104857, rsmon_syslog_limit()
> > May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
> > May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()
> > May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
> > May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()
> >
> > Any suggestions will be helpful
> >
> >
> > Please do let me know if you have any questions.
> >
> >
> >
> >
> > Regards and thanks,
> > Nehul
> >
> >
>
>
> --
> ++ytti



--
++ytti

What JUNOS version are you running?


Regards
Paschal Masha | Engineering
Skype ID: paschal.masha

----- Original Message -----
From: "Nehul Patel" <nehul.patel@gmail.com>
To: "nanog" <nanog@nanog.org>
Sent: Wednesday, May 4, 2022 10:56:14 PM
Subject: Strange behavior on the Juniper MX240


Hi NANOG,

We are seeing some strange behavior on our Juniper MX240 Chassis it is randomly dropping the routes to the certain destination IP address getting the following errors on the MX240 Chassis

If Someone has seen these errors before please suggest how to resolve it


May 4 12:42:00 cr01 newsyslog[44735]: logfile turned over due to size>1024K
May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid)
May 4 12:42:01 last message repeated 4 times
May 4 12:42:01 fpc0 RT: IPv6:0 - 2600:40fc:1011::/48 (add rt entry into jtree failed)
May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2600:40fc:1011::/48 nh 1048576
May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2001:67c:20fc::/48 (No memory) on FE 0
May 4 12:42:01 fpc0 RT: IPv6:0 - 2001:67c:20fc::/48 (add rt entry into jtree failed)
May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2001:67c:20fc::/48 nh 1048576
May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2606:2800:e004::/48 (No memory) on FE 0
May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2a05:3181:ffff::/48 (No memory) on FE 0
May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid)
May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
May 4 12:42:01 /kernel: RT_PFE: RT msg op 2 (PREFIX DELETE) failed, err 5 (Invalid)
May 4 12:42:02 fpc0 RT: Failed prefix add IPv4 - 79.120.22/24 (No memory) on FE 0
May 4 12:42:02 fpc0 RT: IPv4:0 - 79.120.22/24 (add rt entry into jtree failed)
May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv4,len 24 prefix 79.120.22/24 nh 1048583
May 4 12:42:02 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
May 4 12:42:02 fpc0 RT-HAL,rt_msg_handler,540: route process failed

May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-pages Available:20 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-dwords Available:1280 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-pages Available:16 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-dwords Available:1024 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-pages Available:17 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-dwords Available:1088 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()

Any suggestions will be helpful


Please do let me know if you have any questions.




Regards and thanks,
Nehul

'show chassis fpc' might also be useful (or, at least easier :-))

W


On Wed, May 04, 2022 at 4:43 PM, Saku Ytti <saku@ytti.fi> wrote:

> Actually is this DPCE? 'show jtree N summary'
>
> On Wed, 4 May 2022 at 23:39, Saku Ytti <saku@ytti.fi> wrote:
>
> 'show route summary'
> 'start shell pfe network fpcX'
> 'show jnh N pool summary'
> 'show jnh N pool usage'
>
> On Wed, 4 May 2022 at 23:31, Nehul Patel <nehul.patel@gmail.com> wrote:
>
> Hi NANOG,
>
> We are seeing some strange behavior on our Juniper MX240 Chassis it is
> randomly dropping the routes to the certain destination IP address getting
> the following errors on the MX240 Chassis
>
> If Someone has seen these errors before please suggest how to resolve it
>
> May 4 12:42:00 cr01 newsyslog[44735]: logfile turned over due to
> size>1024K May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD)
> failed, err 5 (Invalid) May 4 12:42:01 /kernel: RT_PFE: RT msg op 3
> (PREFIX CHANGE) failed, err 5 (Invalid) May 4 12:42:01 last message
> repeated 4 times
> May 4 12:42:01 fpc0 RT: IPv6:0 - 2600:40fc:1011::/48 (add rt entry into
> jtree failed) May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed May 4 12:42:01 fpc0
> RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix
> 2600:40fc:1011::/48 nh 1048576 May 4 12:42:01 fpc0
> RT-HAL,rt_msg_handler,540: route process failed May 4 12:42:01 fpc0 RT:
> Failed prefix add IPv6 - 2001:67c:20fc::/48 (No memory) on FE 0 May 4
> 12:42:01 fpc0 RT: IPv6:0 - 2001:67c:20fc::/48 (add rt entry into jtree
> failed) May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed May 4 12:42:01 fpc0
> RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix
> 2001:67c:20fc::/48 nh 1048576 May 4 12:42:01 fpc0
> RT-HAL,rt_msg_handler,540: route process failed May 4 12:42:01 fpc0 RT:
> Failed prefix add IPv6 - 2606:2800:e004::/48 (No memory) on FE 0 May 4
> 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2a05:3181:ffff::/48 (No memory)
> on FE 0 May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE)
> failed, err 5 (Invalid) May 4 12:42:01 /kernel: RT_PFE: RT msg op 1
> (PREFIX ADD) failed, err 5 (Invalid) May 4 12:42:01 /kernel: RT_PFE: RT
> msg op 2 (PREFIX DELETE) failed, err 5 (Invalid) May 4 12:42:02 fpc0 RT:
> Failed prefix add IPv4 - 79.120.22/24 (No memory) on FE 0 May 4 12:42:02
> fpc0 RT: IPv4:0 - 79.120.22/24 (add rt entry into jtree failed) May 4
> 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create
> failed May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto
> ipv4,len 24 prefix 79.120.22/24 nh 1048583 May 4 12:42:02 /kernel:
> RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid) May 4 12:42:02
> fpc0 RT-HAL,rt_msg_handler,540: route process failed
>
> May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0
> Type:free-pages Available:20 is less than LWM limit:1638,
> rsmon_syslog_limit() May 4 09:33:17 fpc0 RSMON: Resource Category:jtree
> Instance:jtree2-seg0 Type:free-dwords Available:1280 is less than LWM
> limit:104857, rsmon_syslog_limit() May 4 09:33:18 fpc0 RSMON: Resource
> Category:jtree Instance:jtree3-seg0 Type:free-pages Available:19 is less
> than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:18 fpc0 RSMON:
> Resource Category:jtree Instance:jtree3-seg0 Type:free-dwords
> Available:1216 is less than LWM limit:104857, rsmon_syslog_limit() May 4
> 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0
> Type:free-pages Available:16 is less than LWM limit:1638,
> rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON: Resource Category:jtree
> Instance:jtree0-seg0 Type:free-dwords Available:1024 is less than LWM
> limit:104857, rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON: Resource
> Category:jtree Instance:jtree1-seg0 Type:free-pages Available:15 is less
> than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:18 fpc1 RSMON:
> Resource Category:jtree Instance:jtree1-seg0 Type:free-dwords Available:960
> is less than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:18 fpc1
> RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-pages
> Available:19 is less than LWM limit:1638, rsmon_syslog_limit() May 4
> 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0
> Type:free-dwords Available:1216 is less than LWM limit:104857,
> rsmon_syslog_limit() May 4 09:33:19 fpc1 RSMON: Resource Category:jtree
> Instance:jtree3-seg0 Type:free-pages Available:17 is less than LWM
> limit:1638, rsmon_syslog_limit() May 4 09:33:19 fpc1 RSMON: Resource
> Category:jtree Instance:jtree3-seg0 Type:free-dwords Available:1088 is less
> than LWM limit:104857, rsmon_syslog_limit() May 4 09:33:19 fpc2 RSMON:
> Resource Category:jtree Instance:jtree0-seg0 Type:free-pages Available:15
> is less than LWM limit:1638, rsmon_syslog_limit() May 4 09:33:19 fpc2
> RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-dwords
> Available:960 is less than LWM limit:104857, rsmon_syslog_limit() May 4
> 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0
> Type:free-pages Available:15 is less than LWM limit:1638,
> rsmon_syslog_limit() May 4 09:33:19 fpc2 RSMON: Resource Category:jtree
> Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM
> limit:104857, rsmon_syslog_limit()
>
> Any suggestions will be helpful
>
> Please do let me know if you have any questions.
>
> Regards and thanks,
> Nehul
>
> --
> ++ytti
>
> --
> ++ytti
>

Thank you Saku and the warren Here is the requested output


show route summary

inet.0: 879635 destinations, 879649 routes (879634 active, 0 holddown, 1
hidden)
Direct: 9 routes, 8 active
Local: 8 routes, 8 active
OSPF: 928 routes, 925 active
BGP: 878686 routes, 878678 active
Static: 2 routes, 2 active
Aggregate: 15 routes, 12 active

inet.3: 718 destinations, 718 routes (718 active, 0 holddown, 0 hidden)
LDP: 718 routes, 718 active

Test_VRF.inet.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
Local: 1 routes, 1 active

mpls.0: 390 destinations, 390 routes (390 active, 0 holddown, 0 hidden)
MPLS: 3 routes, 3 active
LDP: 387 routes, 387 active

inet6.0: 143065 destinations, 286099 routes (143065 active, 0 holddown, 0
hidden)
Direct: 13 routes, 9 active
Local: 10 routes, 10 active
OSPF3: 16 routes, 15 active
BGP: 286060 routes, 143030 active
Static: 1 routes, 1 active

show chassis fpc
Temp CPU Utilization (%) Memory Utilization (%)
Slot State (C) Total Interrupt DRAM (MB) Heap Buffer
0 Online 31 11 0 1024 37 29
1 Online 31 11 0 1024 45 29
2 Online 30 4 0 1024 36 29

request pfe execute target fpc0 command "show jtree 0 memory extensive"
SENT: Ukern command: show jtree 0 memory extensive
GOT:
GOT: Jtree memory segment 0 (Context: 0x44817bb0)
GOT: -------------------------------------------
GOT: Memory Statistics:
GOT: 16777216 bytes total
GOT: 16715144 bytes used
GOT: 56880 bytes available (7168 bytes from free pages)
GOT: 3024 bytes wasted
GOT: 2168 bytes unusable
GOT: 32768 pages total
GOT: 32519 pages used (2568 pages used in page alloc)
GOT: 235 pages partially used
GOT: 14 pages free (max contiguous = 6)
GOT:
GOT: Partially Filled Pages (In bytes):-
GOT: Unit Avail Overhead
GOT: 8 26256 0
GOT: 16 14320 0
GOT: 24 8352 2040
GOT: 32 352 0
GOT: 48 432 128
GOT:
GOT: Free Page Lists(Pg Size = 512 bytes):-
GOT: Page Bucket Avail(Bytes)
GOT: 1-1 2560
GOT: 3-3 1536
GOT: 6-6 3072
GOT:
GOT: Fragmentation Index = 0.946, (largest free = 3072)
GOT: Counters:
GOT: 2643777 allocs (0 failed)
GOT: 0 releases(partial 0)
GOT: 1095040 frees
GOT: 0 holds
GOT: 7 pending frees(pending bytes 56)
GOT: 0 pending forced
GOT: 0 times free blocked
GOT: 0 sync writes
GOT: Error Counters:-
GOT: 0 bad params
GOT: 0 failed frees
GOT: 0 bad cookie
GOT:
GOT: Jtree memory segment 1 (Context: 0x448997f0)
GOT: -------------------------------------------
GOT: Memory Statistics:
GOT: 16777216 bytes total
GOT: 4589552 bytes used
GOT: 12185384 bytes available (12183552 bytes from free pages)
GOT: 2248 bytes wasted
GOT: 32 bytes unusable
GOT: 32768 pages total
GOT: 8967 pages used (8967 pages used in page alloc)
GOT: 5 pages partially used
GOT: 23796 pages free (max contiguous = 23793)
GOT:
GOT: Partially Filled Pages (In bytes):-
GOT: Unit Avail Overhead
GOT: 8 1416 0
GOT: 16 80 0
GOT: 48 336 32


request pfe execute target fpc1 command "show jtree 0 memory extensive"
SENT: Ukern command: show jtree 0 memory extensive
GOT:
GOT: Jtree memory segment 0 (Context: 0x447cc698)
GOT: -------------------------------------------
GOT: Memory Statistics:
GOT: 16777216 bytes total
GOT: 16715840 bytes used
GOT: 56184 bytes available (8192 bytes from free pages)
GOT: 3024 bytes wasted
GOT: 2168 bytes unusable
GOT: 32768 pages total
GOT: 32533 pages used (2568 pages used in page alloc)
GOT: 219 pages partially used
GOT: 16 pages free (max contiguous = 5)
GOT:
GOT: Partially Filled Pages (In bytes):-
GOT: Unit Avail Overhead
GOT: 8 25544 0
GOT: 16 13312 0
GOT: 24 8352 2040
GOT: 32 352 0
GOT: 48 432 128
GOT:
GOT: Free Page Lists(Pg Size = 512 bytes):-
GOT: Page Bucket Avail(Bytes)
GOT: 1-1 2048
GOT: 2-2 1024
GOT: 5-5 5120
GOT:
GOT: Fragmentation Index = 0.954, (largest free = 2560)
GOT: Counters:
GOT: 2645725 allocs (0 failed)
GOT: 2 releases(partial 0)
GOT: 1096891 frees
GOT: 0 holds
GOT: 0 pending frees(pending bytes 0)
GOT: 0 pending forced
GOT: 0 times free blocked
GOT: 0 sync writes
GOT: Error Counters:-
GOT: 0 bad params
GOT: 0 failed frees
GOT: 0 bad cookie
GOT:
GOT: Jtree memory segment 1 (Context: 0x4484e2d8)
GOT: -------------------------------------------
GOT: Memory Statistics:
GOT: 16777216 bytes total
GOT: 4589504 bytes used
GOT: 12185432 bytes available (12184576 bytes from free pages)
GOT: 2248 bytes wasted
GOT: 32 bytes unusable
GOT: 32768 pages total
GOT: 8967 pages used (8967 pages used in page alloc)
GOT: 3 pages partially used
GOT: 23798 pages free (max contiguous = 23798)
GOT:
GOT: Partially Filled Pages (In bytes):-
GOT: Unit Avail Overhead
GOT: 8 424 0
GOT: 16 96 0
GOT: 48 336 32
GOT:
GOT: Free Page Lists(Pg Size = 512 bytes):-
GOT: Page Bucket Avail(Bytes)
GOT: 27-32768 12184576
GOT:
GOT: Fragmentation Index = 0.000, (largest free = 12184576)
GOT: Counters:
GOT: 45 allocs (0 failed)
GOT: 0 releases(partial 0)
GOT: 0 frees
GOT: 0 holds

On Wed, May 4, 2022 at 1:39 PM Saku Ytti <saku@ytti.fi> wrote:

> 'show route summary'
> 'start shell pfe network fpcX'
> 'show jnh N pool summary'
> 'show jnh N pool usage'
>
> On Wed, 4 May 2022 at 23:31, Nehul Patel <nehul.patel@gmail.com> wrote:
> >
> >
> > Hi NANOG,
> >
> > We are seeing some strange behavior on our Juniper MX240 Chassis it is
> randomly dropping the routes to the certain destination IP address getting
> the following errors on the MX240 Chassis
> >
> > If Someone has seen these errors before please suggest how to resolve it
> >
> >
> > May 4 12:42:00 cr01 newsyslog[44735]: logfile turned over due to
> size>1024K
> > May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err
> 5 (Invalid)
> > May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed,
> err 5 (Invalid)
> > May 4 12:42:01 last message repeated 4 times
> > May 4 12:42:01 fpc0 RT: IPv6:0 - 2600:40fc:1011::/48 (add rt entry
> into jtree failed)
> > May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> > May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len
> 48 prefix 2600:40fc:1011::/48 nh 1048576
> > May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
> > May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2001:67c:20fc::/48
> (No memory) on FE 0
> > May 4 12:42:01 fpc0 RT: IPv6:0 - 2001:67c:20fc::/48 (add rt entry
> into jtree failed)
> > May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> > May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len
> 48 prefix 2001:67c:20fc::/48 nh 1048576
> > May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
> > May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2606:2800:e004::/48
> (No memory) on FE 0
> > May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2a05:3181:ffff::/48
> (No memory) on FE 0
> > May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed,
> err 5 (Invalid)
> > May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err
> 5 (Invalid)
> > May 4 12:42:01 /kernel: RT_PFE: RT msg op 2 (PREFIX DELETE) failed,
> err 5 (Invalid)
> > May 4 12:42:02 fpc0 RT: Failed prefix add IPv4 - 79.120.22/24 (No
> memory) on FE 0
> > May 4 12:42:02 fpc0 RT: IPv4:0 - 79.120.22/24 (add rt entry into
> jtree failed)
> > May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> > May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv4,len
> 24 prefix 79.120.22/24 nh 1048583
> > May 4 12:42:02 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err
> 5 (Invalid)
> > May 4 12:42:02 fpc0 RT-HAL,rt_msg_handler,540: route process failed
> >
> > May 4 09:33:17 fpc0 RSMON: Resource Category:jtree
> Instance:jtree2-seg0 Type:free-pages Available:20 is less than LWM
> limit:1638, rsmon_syslog_limit()
> > May 4 09:33:17 fpc0 RSMON: Resource Category:jtree
> Instance:jtree2-seg0 Type:free-dwords Available:1280 is less than LWM
> limit:104857, rsmon_syslog_limit()
> > May 4 09:33:18 fpc0 RSMON: Resource Category:jtree
> Instance:jtree3-seg0 Type:free-pages Available:19 is less than LWM
> limit:1638, rsmon_syslog_limit()
> > May 4 09:33:18 fpc0 RSMON: Resource Category:jtree
> Instance:jtree3-seg0 Type:free-dwords Available:1216 is less than LWM
> limit:104857, rsmon_syslog_limit()
> > May 4 09:33:18 fpc1 RSMON: Resource Category:jtree
> Instance:jtree0-seg0 Type:free-pages Available:16 is less than LWM
> limit:1638, rsmon_syslog_limit()
> > May 4 09:33:18 fpc1 RSMON: Resource Category:jtree
> Instance:jtree0-seg0 Type:free-dwords Available:1024 is less than LWM
> limit:104857, rsmon_syslog_limit()
> > May 4 09:33:18 fpc1 RSMON: Resource Category:jtree
> Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM
> limit:1638, rsmon_syslog_limit()
> > May 4 09:33:18 fpc1 RSMON: Resource Category:jtree
> Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM
> limit:104857, rsmon_syslog_limit()
> > May 4 09:33:18 fpc1 RSMON: Resource Category:jtree
> Instance:jtree2-seg0 Type:free-pages Available:19 is less than LWM
> limit:1638, rsmon_syslog_limit()
> > May 4 09:33:19 fpc1 RSMON: Resource Category:jtree
> Instance:jtree2-seg0 Type:free-dwords Available:1216 is less than LWM
> limit:104857, rsmon_syslog_limit()
> > May 4 09:33:19 fpc1 RSMON: Resource Category:jtree
> Instance:jtree3-seg0 Type:free-pages Available:17 is less than LWM
> limit:1638, rsmon_syslog_limit()
> > May 4 09:33:19 fpc1 RSMON: Resource Category:jtree
> Instance:jtree3-seg0 Type:free-dwords Available:1088 is less than LWM
> limit:104857, rsmon_syslog_limit()
> > May 4 09:33:19 fpc2 RSMON: Resource Category:jtree
> Instance:jtree0-seg0 Type:free-pages Available:15 is less than LWM
> limit:1638, rsmon_syslog_limit()
> > May 4 09:33:19 fpc2 RSMON: Resource Category:jtree
> Instance:jtree0-seg0 Type:free-dwords Available:960 is less than LWM
> limit:104857, rsmon_syslog_limit()
> > May 4 09:33:19 fpc2 RSMON: Resource Category:jtree
> Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM
> limit:1638, rsmon_syslog_limit()
> > May 4 09:33:19 fpc2 RSMON: Resource Category:jtree
> Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM
> limit:104857, rsmon_syslog_limit()
> >
> > Any suggestions will be helpful
> >
> >
> > Please do let me know if you have any questions.
> >
> >
> >
> >
> > Regards and thanks,
> > Nehul
> >
> >
>
>
> --
> ++ytti
>

JUNOS Version

JUNOS Base OS boot [10.4R9.2]
JUNOS Base OS Software Suite [10.4R9.2]
JUNOS Kernel Software Suite [10.4R9.2]
JUNOS Crypto Software Suite [10.4R9.2]
JUNOS Packet Forwarding Engine Support (M/T Common) [10.4R9.2]
JUNOS Packet Forwarding Engine Support (MX Common) [10.4R9.2]
JUNOS Online Documentation [10.4R9.2]
JUNOS Voice Services Container package [10.4R9.2]
JUNOS Border Gateway Function package [10.4R9.2]
JUNOS Services AACL Container package [10.4R9.2]
JUNOS Services LL-PDF Container package [10.4R9.2]
JUNOS Services PTSP Container package [10.4R9.2]
JUNOS Services Stateful Firewall [10.4R9.2]
JUNOS Services NAT [10.4R9.2]
JUNOS Services Application Level Gateways [10.4R9.2]



On Wed, May 4, 2022 at 1:49 PM Paschal Masha <paschal.masha@ke.wananchi.com>
wrote:

> What JUNOS version are you running?
>
>
> Regards
> Paschal Masha | Engineering
> Skype ID: paschal.masha
>
> ----- Original Message -----
> From: "Nehul Patel" <nehul.patel@gmail.com>
> To: "nanog" <nanog@nanog.org>
> Sent: Wednesday, May 4, 2022 10:56:14 PM
> Subject: Strange behavior on the Juniper MX240
>
>
> Hi NANOG,
>
> We are seeing some strange behavior on our Juniper MX240 Chassis it is
> randomly dropping the routes to the certain destination IP address getting
> the following errors on the MX240 Chassis
>
> If Someone has seen these errors before please suggest how to resolve it
>
>
> May 4 12:42:00 cr01 newsyslog[44735]: logfile turned over due to
> size>1024K
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5
> (Invalid)
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5
> (Invalid)
> May 4 12:42:01 last message repeated 4 times
> May 4 12:42:01 fpc0 RT: IPv6:0 - 2600:40fc:1011::/48 (add rt entry into
> jtree failed)
> May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48
> prefix 2600:40fc:1011::/48 nh 1048576
> May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
> May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2001:67c:20fc::/48 (No
> memory) on FE 0
> May 4 12:42:01 fpc0 RT: IPv6:0 - 2001:67c:20fc::/48 (add rt entry into
> jtree failed)
> May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48
> prefix 2001:67c:20fc::/48 nh 1048576
> May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
> May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2606:2800:e004::/48 (No
> memory) on FE 0
> May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2a05:3181:ffff::/48 (No
> memory) on FE 0
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5
> (Invalid)
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5
> (Invalid)
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 2 (PREFIX DELETE) failed, err 5
> (Invalid)
> May 4 12:42:02 fpc0 RT: Failed prefix add IPv4 - 79.120.22/24 (No memory)
> on FE 0
> May 4 12:42:02 fpc0 RT: IPv4:0 - 79.120.22/24 (add rt entry into jtree
> failed)
> May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv4,len 24
> prefix 79.120.22/24 nh 1048583
> May 4 12:42:02 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5
> (Invalid)
> May 4 12:42:02 fpc0 RT-HAL,rt_msg_handler,540: route process failed
>
> May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0
> Type:free-pages Available:20 is less than LWM limit:1638,
> rsmon_syslog_limit()
> May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0
> Type:free-dwords Available:1280 is less than LWM limit:104857,
> rsmon_syslog_limit()
> May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0
> Type:free-pages Available:19 is less than LWM limit:1638,
> rsmon_syslog_limit()
> May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0
> Type:free-dwords Available:1216 is less than LWM limit:104857,
> rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0
> Type:free-pages Available:16 is less than LWM limit:1638,
> rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0
> Type:free-dwords Available:1024 is less than LWM limit:104857,
> rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0
> Type:free-pages Available:15 is less than LWM limit:1638,
> rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0
> Type:free-dwords Available:960 is less than LWM limit:104857,
> rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0
> Type:free-pages Available:19 is less than LWM limit:1638,
> rsmon_syslog_limit()
> May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0
> Type:free-dwords Available:1216 is less than LWM limit:104857,
> rsmon_syslog_limit()
> May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0
> Type:free-pages Available:17 is less than LWM limit:1638,
> rsmon_syslog_limit()
> May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0
> Type:free-dwords Available:1088 is less than LWM limit:104857,
> rsmon_syslog_limit()
> May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0
> Type:free-pages Available:15 is less than LWM limit:1638,
> rsmon_syslog_limit()
> May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0
> Type:free-dwords Available:960 is less than LWM limit:104857,
> rsmon_syslog_limit()
> May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0
> Type:free-pages Available:15 is less than LWM limit:1638,
> rsmon_syslog_limit()
> May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0
> Type:free-dwords Available:960 is less than LWM limit:104857,
> rsmon_syslog_limit()
>
> Any suggestions will be helpful
>
>
> Please do let me know if you have any questions.
>
>
>
>
> Regards and thanks,
> Nehul
>
>
>
>

Dude, JunOS 10.4 end of support - 06/08/2014. You have an almost 8 years past end of Vendor support O/S still in production! No, just no.







From: NANOG <nanog-bounces+tony=wicks.co.nz@nanog.org> On Behalf Of Nehul Patel
Sent: Thursday, 5 May 2022 9:35 am
To: Paschal Masha <paschal.masha@ke.wananchi.com>
Cc: nanog <nanog@nanog.org>
Subject: Re: Strange behavior on the Juniper MX240



JUNOS Version



JUNOS Base OS boot [10.4R9.2]
JUNOS Base OS Software Suite [10.4R9.2]
JUNOS Kernel Software Suite [10.4R9.2]
JUNOS Crypto Software Suite [10.4R9.2]
JUNOS Packet Forwarding Engine Support (M/T Common) [10.4R9.2]
JUNOS Packet Forwarding Engine Support (MX Common) [10.4R9.2]
JUNOS Online Documentation [10.4R9.2]
JUNOS Voice Services Container package [10.4R9.2]
JUNOS Border Gateway Function package [10.4R9.2]
JUNOS Services AACL Container package [10.4R9.2]
JUNOS Services LL-PDF Container package [10.4R9.2]
JUNOS Services PTSP Container package [10.4R9.2]
JUNOS Services Stateful Firewall [10.4R9.2]
JUNOS Services NAT [10.4R9.2]
JUNOS Services Application Level Gateways [10.4R9.2]

----- On May 4, 2022, at 6:58 PM, Tony Wicks tony@wicks.co.nz wrote:

Hi,

> Dude, JunOS 10.4 end of support - 06/08/2014. You have an almost 8 years past
> end of Vendor support O/S still in production! No, just no.

Now I'm really interested in the uptime of that box...

Thanks,

Sabri

Your line cards (not RE's) are running out of route-storage memory.
As a short-term mitigation, you could try borrowing from segment 1,
normally dedicated to filters,

set chassis memory-enhanced route

but this option may not exist in the version of JunOS you're
running, which as already mentioned is very old.

If the command is accepted, and it lets you commit, you'll then
need to restart each of the FPC's, one at a time, by slot number,
which will take each out of service for a few minutes, so you
probably want to wait until a scheduled maintenance period, and
start with less-important FPC slots first:

request chassis fpc restart slot X



On Wed, May 04, 2022 at 02:34:45PM -0700, Nehul Patel wrote:
> Thank you Saku and the warren Here is the requested output
>
>
> show route summary
>
> inet.0: 879635 destinations, 879649 routes (879634 active, 0 holddown, 1
> hidden)
> Direct: 9 routes, 8 active
> Local: 8 routes, 8 active
> OSPF: 928 routes, 925 active
> BGP: 878686 routes, 878678 active
> Static: 2 routes, 2 active
> Aggregate: 15 routes, 12 active
>
> inet.3: 718 destinations, 718 routes (718 active, 0 holddown, 0 hidden)
> LDP: 718 routes, 718 active
>
> Test_VRF.inet.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
> Local: 1 routes, 1 active
>
> mpls.0: 390 destinations, 390 routes (390 active, 0 holddown, 0 hidden)
> MPLS: 3 routes, 3 active
> LDP: 387 routes, 387 active
>
> inet6.0: 143065 destinations, 286099 routes (143065 active, 0 holddown, 0
> hidden)
> Direct: 13 routes, 9 active
> Local: 10 routes, 10 active
> OSPF3: 16 routes, 15 active
> BGP: 286060 routes, 143030 active
> Static: 1 routes, 1 active
>
> show chassis fpc
> Temp CPU Utilization (%) Memory Utilization (%)
> Slot State (C) Total Interrupt DRAM (MB) Heap Buffer
> 0 Online 31 11 0 1024 37 29
> 1 Online 31 11 0 1024 45 29
> 2 Online 30 4 0 1024 36 29
>
> request pfe execute target fpc0 command "show jtree 0 memory extensive"
> SENT: Ukern command: show jtree 0 memory extensive
> GOT:
> GOT: Jtree memory segment 0 (Context: 0x44817bb0)
> GOT: -------------------------------------------
> GOT: Memory Statistics:
> GOT: 16777216 bytes total
> GOT: 16715144 bytes used
> GOT: 56880 bytes available (7168 bytes from free pages)
> GOT: 3024 bytes wasted
> GOT: 2168 bytes unusable
> GOT: 32768 pages total
> GOT: 32519 pages used (2568 pages used in page alloc)
> GOT: 235 pages partially used
> GOT: 14 pages free (max contiguous = 6)
> GOT:
> GOT: Partially Filled Pages (In bytes):-
> GOT: Unit Avail Overhead
> GOT: 8 26256 0
> GOT: 16 14320 0
> GOT: 24 8352 2040
> GOT: 32 352 0
> GOT: 48 432 128
> GOT:
> GOT: Free Page Lists(Pg Size = 512 bytes):-
> GOT: Page Bucket Avail(Bytes)
> GOT: 1-1 2560
> GOT: 3-3 1536
> GOT: 6-6 3072
> GOT:
> GOT: Fragmentation Index = 0.946, (largest free = 3072)
> GOT: Counters:
> GOT: 2643777 allocs (0 failed)
> GOT: 0 releases(partial 0)
> GOT: 1095040 frees
> GOT: 0 holds
> GOT: 7 pending frees(pending bytes 56)
> GOT: 0 pending forced
> GOT: 0 times free blocked
> GOT: 0 sync writes
> GOT: Error Counters:-
> GOT: 0 bad params
> GOT: 0 failed frees
> GOT: 0 bad cookie
> GOT:
> GOT: Jtree memory segment 1 (Context: 0x448997f0)
> GOT: -------------------------------------------
> GOT: Memory Statistics:
> GOT: 16777216 bytes total
> GOT: 4589552 bytes used
> GOT: 12185384 bytes available (12183552 bytes from free pages)
> GOT: 2248 bytes wasted
> GOT: 32 bytes unusable
> GOT: 32768 pages total
> GOT: 8967 pages used (8967 pages used in page alloc)
> GOT: 5 pages partially used
> GOT: 23796 pages free (max contiguous = 23793)
> GOT:
> GOT: Partially Filled Pages (In bytes):-
> GOT: Unit Avail Overhead
> GOT: 8 1416 0
> GOT: 16 80 0
> GOT: 48 336 32
>
>
> request pfe execute target fpc1 command "show jtree 0 memory extensive"
> SENT: Ukern command: show jtree 0 memory extensive
> GOT:
> GOT: Jtree memory segment 0 (Context: 0x447cc698)
> GOT: -------------------------------------------
> GOT: Memory Statistics:
> GOT: 16777216 bytes total
> GOT: 16715840 bytes used
> GOT: 56184 bytes available (8192 bytes from free pages)
> GOT: 3024 bytes wasted
> GOT: 2168 bytes unusable
> GOT: 32768 pages total
> GOT: 32533 pages used (2568 pages used in page alloc)
> GOT: 219 pages partially used
> GOT: 16 pages free (max contiguous = 5)
> GOT:
> GOT: Partially Filled Pages (In bytes):-
> GOT: Unit Avail Overhead
> GOT: 8 25544 0
> GOT: 16 13312 0
> GOT: 24 8352 2040
> GOT: 32 352 0
> GOT: 48 432 128
> GOT:
> GOT: Free Page Lists(Pg Size = 512 bytes):-
> GOT: Page Bucket Avail(Bytes)
> GOT: 1-1 2048
> GOT: 2-2 1024
> GOT: 5-5 5120
> GOT:
> GOT: Fragmentation Index = 0.954, (largest free = 2560)
> GOT: Counters:
> GOT: 2645725 allocs (0 failed)
> GOT: 2 releases(partial 0)
> GOT: 1096891 frees
> GOT: 0 holds
> GOT: 0 pending frees(pending bytes 0)
> GOT: 0 pending forced
> GOT: 0 times free blocked
> GOT: 0 sync writes
> GOT: Error Counters:-
> GOT: 0 bad params
> GOT: 0 failed frees
> GOT: 0 bad cookie
> GOT:
> GOT: Jtree memory segment 1 (Context: 0x4484e2d8)
> GOT: -------------------------------------------
> GOT: Memory Statistics:
> GOT: 16777216 bytes total
> GOT: 4589504 bytes used
> GOT: 12185432 bytes available (12184576 bytes from free pages)
> GOT: 2248 bytes wasted
> GOT: 32 bytes unusable
> GOT: 32768 pages total
> GOT: 8967 pages used (8967 pages used in page alloc)
> GOT: 3 pages partially used
> GOT: 23798 pages free (max contiguous = 23798)
> GOT:
> GOT: Partially Filled Pages (In bytes):-
> GOT: Unit Avail Overhead
> GOT: 8 424 0
> GOT: 16 96 0
> GOT: 48 336 32
> GOT:
> GOT: Free Page Lists(Pg Size = 512 bytes):-
> GOT: Page Bucket Avail(Bytes)
> GOT: 27-32768 12184576
> GOT:
> GOT: Fragmentation Index = 0.000, (largest free = 12184576)
> GOT: Counters:
> GOT: 45 allocs (0 failed)
> GOT: 0 releases(partial 0)
> GOT: 0 frees
> GOT: 0 holds

--
Jordan.

Ok, thank you all for the feedback we are going to start with the Junos OS
upgrade first on it but have to open the ticket with JTAC since currently
on the juniper support website they have the Junos 15.1 is available so not
sure we can directly jump from 10.4 to 15.1 maybe we have to do step by
step upgrade on it. Any other suggestions will be helpful as well

By the way, the uptime on the Juniper MX chassis was 1589 Days on it.










On Wed, May 4, 2022 at 7:11 PM Jordan <jnh@vt11.net> wrote:

> Your line cards (not RE's) are running out of route-storage memory.
> As a short-term mitigation, you could try borrowing from segment 1,
> normally dedicated to filters,
>
> set chassis memory-enhanced route
>
> but this option may not exist in the version of JunOS you're
> running, which as already mentioned is very old.
>
> If the command is accepted, and it lets you commit, you'll then
> need to restart each of the FPC's, one at a time, by slot number,
> which will take each out of service for a few minutes, so you
> probably want to wait until a scheduled maintenance period, and
> start with less-important FPC slots first:
>
> request chassis fpc restart slot X
>
>
>
> On Wed, May 04, 2022 at 02:34:45PM -0700, Nehul Patel wrote:
> > Thank you Saku and the warren Here is the requested output
> >
> >
> > show route summary
> >
> > inet.0: 879635 destinations, 879649 routes (879634 active, 0 holddown, 1
> > hidden)
> > Direct: 9 routes, 8 active
> > Local: 8 routes, 8 active
> > OSPF: 928 routes, 925 active
> > BGP: 878686 routes, 878678 active
> > Static: 2 routes, 2 active
> > Aggregate: 15 routes, 12 active
> >
> > inet.3: 718 destinations, 718 routes (718 active, 0 holddown, 0 hidden)
> > LDP: 718 routes, 718 active
> >
> > Test_VRF.inet.0: 1 destinations, 1 routes (1 active, 0 holddown, 0
> hidden)
> > Local: 1 routes, 1 active
> >
> > mpls.0: 390 destinations, 390 routes (390 active, 0 holddown, 0 hidden)
> > MPLS: 3 routes, 3 active
> > LDP: 387 routes, 387 active
> >
> > inet6.0: 143065 destinations, 286099 routes (143065 active, 0 holddown, 0
> > hidden)
> > Direct: 13 routes, 9 active
> > Local: 10 routes, 10 active
> > OSPF3: 16 routes, 15 active
> > BGP: 286060 routes, 143030 active
> > Static: 1 routes, 1 active
> >
> > show chassis fpc
> > Temp CPU Utilization (%) Memory Utilization
> (%)
> > Slot State (C) Total Interrupt DRAM (MB) Heap
> Buffer
> > 0 Online 31 11 0 1024 37
> 29
> > 1 Online 31 11 0 1024 45
> 29
> > 2 Online 30 4 0 1024 36
> 29
> >
> > request pfe execute target fpc0 command "show jtree 0 memory extensive"
> > SENT: Ukern command: show jtree 0 memory extensive
> > GOT:
> > GOT: Jtree memory segment 0 (Context: 0x44817bb0)
> > GOT: -------------------------------------------
> > GOT: Memory Statistics:
> > GOT: 16777216 bytes total
> > GOT: 16715144 bytes used
> > GOT: 56880 bytes available (7168 bytes from free pages)
> > GOT: 3024 bytes wasted
> > GOT: 2168 bytes unusable
> > GOT: 32768 pages total
> > GOT: 32519 pages used (2568 pages used in page alloc)
> > GOT: 235 pages partially used
> > GOT: 14 pages free (max contiguous = 6)
> > GOT:
> > GOT: Partially Filled Pages (In bytes):-
> > GOT: Unit Avail Overhead
> > GOT: 8 26256 0
> > GOT: 16 14320 0
> > GOT: 24 8352 2040
> > GOT: 32 352 0
> > GOT: 48 432 128
> > GOT:
> > GOT: Free Page Lists(Pg Size = 512 bytes):-
> > GOT: Page Bucket Avail(Bytes)
> > GOT: 1-1 2560
> > GOT: 3-3 1536
> > GOT: 6-6 3072
> > GOT:
> > GOT: Fragmentation Index = 0.946, (largest free = 3072)
> > GOT: Counters:
> > GOT: 2643777 allocs (0 failed)
> > GOT: 0 releases(partial 0)
> > GOT: 1095040 frees
> > GOT: 0 holds
> > GOT: 7 pending frees(pending bytes 56)
> > GOT: 0 pending forced
> > GOT: 0 times free blocked
> > GOT: 0 sync writes
> > GOT: Error Counters:-
> > GOT: 0 bad params
> > GOT: 0 failed frees
> > GOT: 0 bad cookie
> > GOT:
> > GOT: Jtree memory segment 1 (Context: 0x448997f0)
> > GOT: -------------------------------------------
> > GOT: Memory Statistics:
> > GOT: 16777216 bytes total
> > GOT: 4589552 bytes used
> > GOT: 12185384 bytes available (12183552 bytes from free pages)
> > GOT: 2248 bytes wasted
> > GOT: 32 bytes unusable
> > GOT: 32768 pages total
> > GOT: 8967 pages used (8967 pages used in page alloc)
> > GOT: 5 pages partially used
> > GOT: 23796 pages free (max contiguous = 23793)
> > GOT:
> > GOT: Partially Filled Pages (In bytes):-
> > GOT: Unit Avail Overhead
> > GOT: 8 1416 0
> > GOT: 16 80 0
> > GOT: 48 336 32
> >
> >
> > request pfe execute target fpc1 command "show jtree 0 memory extensive"
> > SENT: Ukern command: show jtree 0 memory extensive
> > GOT:
> > GOT: Jtree memory segment 0 (Context: 0x447cc698)
> > GOT: -------------------------------------------
> > GOT: Memory Statistics:
> > GOT: 16777216 bytes total
> > GOT: 16715840 bytes used
> > GOT: 56184 bytes available (8192 bytes from free pages)
> > GOT: 3024 bytes wasted
> > GOT: 2168 bytes unusable
> > GOT: 32768 pages total
> > GOT: 32533 pages used (2568 pages used in page alloc)
> > GOT: 219 pages partially used
> > GOT: 16 pages free (max contiguous = 5)
> > GOT:
> > GOT: Partially Filled Pages (In bytes):-
> > GOT: Unit Avail Overhead
> > GOT: 8 25544 0
> > GOT: 16 13312 0
> > GOT: 24 8352 2040
> > GOT: 32 352 0
> > GOT: 48 432 128
> > GOT:
> > GOT: Free Page Lists(Pg Size = 512 bytes):-
> > GOT: Page Bucket Avail(Bytes)
> > GOT: 1-1 2048
> > GOT: 2-2 1024
> > GOT: 5-5 5120
> > GOT:
> > GOT: Fragmentation Index = 0.954, (largest free = 2560)
> > GOT: Counters:
> > GOT: 2645725 allocs (0 failed)
> > GOT: 2 releases(partial 0)
> > GOT: 1096891 frees
> > GOT: 0 holds
> > GOT: 0 pending frees(pending bytes 0)
> > GOT: 0 pending forced
> > GOT: 0 times free blocked
> > GOT: 0 sync writes
> > GOT: Error Counters:-
> > GOT: 0 bad params
> > GOT: 0 failed frees
> > GOT: 0 bad cookie
> > GOT:
> > GOT: Jtree memory segment 1 (Context: 0x4484e2d8)
> > GOT: -------------------------------------------
> > GOT: Memory Statistics:
> > GOT: 16777216 bytes total
> > GOT: 4589504 bytes used
> > GOT: 12185432 bytes available (12184576 bytes from free pages)
> > GOT: 2248 bytes wasted
> > GOT: 32 bytes unusable
> > GOT: 32768 pages total
> > GOT: 8967 pages used (8967 pages used in page alloc)
> > GOT: 3 pages partially used
> > GOT: 23798 pages free (max contiguous = 23798)
> > GOT:
> > GOT: Partially Filled Pages (In bytes):-
> > GOT: Unit Avail Overhead
> > GOT: 8 424 0
> > GOT: 16 96 0
> > GOT: 48 336 32
> > GOT:
> > GOT: Free Page Lists(Pg Size = 512 bytes):-
> > GOT: Page Bucket Avail(Bytes)
> > GOT: 27-32768 12184576
> > GOT:
> > GOT: Fragmentation Index = 0.000, (largest free = 12184576)
> > GOT: Counters:
> > GOT: 45 allocs (0 failed)
> > GOT: 0 releases(partial 0)
> > GOT: 0 frees
> > GOT: 0 holds
>
> --
> Jordan.
>

Almost always direct upgrade works. If you ask TAC, they will likely
suggest a formal process and you'll be doing many upgrades, which
itself isn't actually something that is guaranteed to work (like in
WRL9 case, but that is vmhost RE, not yours).

And like Jordan said, you are out of resources but can extend them
with the command given, which should give you more run rate. You may
want to look in more detail how long you can keep running DPCE until
you're really out.


On Thu, 5 May 2022 at 06:08, Nehul Patel <nehul.patel@gmail.com> wrote:
>
> Ok, thank you all for the feedback we are going to start with the Junos OS upgrade first on it but have to open the ticket with JTAC since currently on the juniper support website they have the Junos 15.1 is available so not sure we can directly jump from 10.4 to 15.1 maybe we have to do step by step upgrade on it. Any other suggestions will be helpful as well
>
> By the way, the uptime on the Juniper MX chassis was 1589 Days on it.
>
>
>
>
>
>
>
>
>
>
> On Wed, May 4, 2022 at 7:11 PM Jordan <jnh@vt11.net> wrote:
>>
>> Your line cards (not RE's) are running out of route-storage memory.
>> As a short-term mitigation, you could try borrowing from segment 1,
>> normally dedicated to filters,
>>
>> set chassis memory-enhanced route
>>
>> but this option may not exist in the version of JunOS you're
>> running, which as already mentioned is very old.
>>
>> If the command is accepted, and it lets you commit, you'll then
>> need to restart each of the FPC's, one at a time, by slot number,
>> which will take each out of service for a few minutes, so you
>> probably want to wait until a scheduled maintenance period, and
>> start with less-important FPC slots first:
>>
>> request chassis fpc restart slot X
>>
>>
>>
>> On Wed, May 04, 2022 at 02:34:45PM -0700, Nehul Patel wrote:
>> > Thank you Saku and the warren Here is the requested output
>> >
>> >
>> > show route summary
>> >
>> > inet.0: 879635 destinations, 879649 routes (879634 active, 0 holddown, 1
>> > hidden)
>> > Direct: 9 routes, 8 active
>> > Local: 8 routes, 8 active
>> > OSPF: 928 routes, 925 active
>> > BGP: 878686 routes, 878678 active
>> > Static: 2 routes, 2 active
>> > Aggregate: 15 routes, 12 active
>> >
>> > inet.3: 718 destinations, 718 routes (718 active, 0 holddown, 0 hidden)
>> > LDP: 718 routes, 718 active
>> >
>> > Test_VRF.inet.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
>> > Local: 1 routes, 1 active
>> >
>> > mpls.0: 390 destinations, 390 routes (390 active, 0 holddown, 0 hidden)
>> > MPLS: 3 routes, 3 active
>> > LDP: 387 routes, 387 active
>> >
>> > inet6.0: 143065 destinations, 286099 routes (143065 active, 0 holddown, 0
>> > hidden)
>> > Direct: 13 routes, 9 active
>> > Local: 10 routes, 10 active
>> > OSPF3: 16 routes, 15 active
>> > BGP: 286060 routes, 143030 active
>> > Static: 1 routes, 1 active
>> >
>> > show chassis fpc
>> > Temp CPU Utilization (%) Memory Utilization (%)
>> > Slot State (C) Total Interrupt DRAM (MB) Heap Buffer
>> > 0 Online 31 11 0 1024 37 29
>> > 1 Online 31 11 0 1024 45 29
>> > 2 Online 30 4 0 1024 36 29
>> >
>> > request pfe execute target fpc0 command "show jtree 0 memory extensive"
>> > SENT: Ukern command: show jtree 0 memory extensive
>> > GOT:
>> > GOT: Jtree memory segment 0 (Context: 0x44817bb0)
>> > GOT: -------------------------------------------
>> > GOT: Memory Statistics:
>> > GOT: 16777216 bytes total
>> > GOT: 16715144 bytes used
>> > GOT: 56880 bytes available (7168 bytes from free pages)
>> > GOT: 3024 bytes wasted
>> > GOT: 2168 bytes unusable
>> > GOT: 32768 pages total
>> > GOT: 32519 pages used (2568 pages used in page alloc)
>> > GOT: 235 pages partially used
>> > GOT: 14 pages free (max contiguous = 6)
>> > GOT:
>> > GOT: Partially Filled Pages (In bytes):-
>> > GOT: Unit Avail Overhead
>> > GOT: 8 26256 0
>> > GOT: 16 14320 0
>> > GOT: 24 8352 2040
>> > GOT: 32 352 0
>> > GOT: 48 432 128
>> > GOT:
>> > GOT: Free Page Lists(Pg Size = 512 bytes):-
>> > GOT: Page Bucket Avail(Bytes)
>> > GOT: 1-1 2560
>> > GOT: 3-3 1536
>> > GOT: 6-6 3072
>> > GOT:
>> > GOT: Fragmentation Index = 0.946, (largest free = 3072)
>> > GOT: Counters:
>> > GOT: 2643777 allocs (0 failed)
>> > GOT: 0 releases(partial 0)
>> > GOT: 1095040 frees
>> > GOT: 0 holds
>> > GOT: 7 pending frees(pending bytes 56)
>> > GOT: 0 pending forced
>> > GOT: 0 times free blocked
>> > GOT: 0 sync writes
>> > GOT: Error Counters:-
>> > GOT: 0 bad params
>> > GOT: 0 failed frees
>> > GOT: 0 bad cookie
>> > GOT:
>> > GOT: Jtree memory segment 1 (Context: 0x448997f0)
>> > GOT: -------------------------------------------
>> > GOT: Memory Statistics:
>> > GOT: 16777216 bytes total
>> > GOT: 4589552 bytes used
>> > GOT: 12185384 bytes available (12183552 bytes from free pages)
>> > GOT: 2248 bytes wasted
>> > GOT: 32 bytes unusable
>> > GOT: 32768 pages total
>> > GOT: 8967 pages used (8967 pages used in page alloc)
>> > GOT: 5 pages partially used
>> > GOT: 23796 pages free (max contiguous = 23793)
>> > GOT:
>> > GOT: Partially Filled Pages (In bytes):-
>> > GOT: Unit Avail Overhead
>> > GOT: 8 1416 0
>> > GOT: 16 80 0
>> > GOT: 48 336 32
>> >
>> >
>> > request pfe execute target fpc1 command "show jtree 0 memory extensive"
>> > SENT: Ukern command: show jtree 0 memory extensive
>> > GOT:
>> > GOT: Jtree memory segment 0 (Context: 0x447cc698)
>> > GOT: -------------------------------------------
>> > GOT: Memory Statistics:
>> > GOT: 16777216 bytes total
>> > GOT: 16715840 bytes used
>> > GOT: 56184 bytes available (8192 bytes from free pages)
>> > GOT: 3024 bytes wasted
>> > GOT: 2168 bytes unusable
>> > GOT: 32768 pages total
>> > GOT: 32533 pages used (2568 pages used in page alloc)
>> > GOT: 219 pages partially used
>> > GOT: 16 pages free (max contiguous = 5)
>> > GOT:
>> > GOT: Partially Filled Pages (In bytes):-
>> > GOT: Unit Avail Overhead
>> > GOT: 8 25544 0
>> > GOT: 16 13312 0
>> > GOT: 24 8352 2040
>> > GOT: 32 352 0
>> > GOT: 48 432 128
>> > GOT:
>> > GOT: Free Page Lists(Pg Size = 512 bytes):-
>> > GOT: Page Bucket Avail(Bytes)
>> > GOT: 1-1 2048
>> > GOT: 2-2 1024
>> > GOT: 5-5 5120
>> > GOT:
>> > GOT: Fragmentation Index = 0.954, (largest free = 2560)
>> > GOT: Counters:
>> > GOT: 2645725 allocs (0 failed)
>> > GOT: 2 releases(partial 0)
>> > GOT: 1096891 frees
>> > GOT: 0 holds
>> > GOT: 0 pending frees(pending bytes 0)
>> > GOT: 0 pending forced
>> > GOT: 0 times free blocked
>> > GOT: 0 sync writes
>> > GOT: Error Counters:-
>> > GOT: 0 bad params
>> > GOT: 0 failed frees
>> > GOT: 0 bad cookie
>> > GOT:
>> > GOT: Jtree memory segment 1 (Context: 0x4484e2d8)
>> > GOT: -------------------------------------------
>> > GOT: Memory Statistics:
>> > GOT: 16777216 bytes total
>> > GOT: 4589504 bytes used
>> > GOT: 12185432 bytes available (12184576 bytes from free pages)
>> > GOT: 2248 bytes wasted
>> > GOT: 32 bytes unusable
>> > GOT: 32768 pages total
>> > GOT: 8967 pages used (8967 pages used in page alloc)
>> > GOT: 3 pages partially used
>> > GOT: 23798 pages free (max contiguous = 23798)
>> > GOT:
>> > GOT: Partially Filled Pages (In bytes):-
>> > GOT: Unit Avail Overhead
>> > GOT: 8 424 0
>> > GOT: 16 96 0
>> > GOT: 48 336 32
>> > GOT:
>> > GOT: Free Page Lists(Pg Size = 512 bytes):-
>> > GOT: Page Bucket Avail(Bytes)
>> > GOT: 27-32768 12184576
>> > GOT:
>> > GOT: Fragmentation Index = 0.000, (largest free = 12184576)
>> > GOT: Counters:
>> > GOT: 45 allocs (0 failed)
>> > GOT: 0 releases(partial 0)
>> > GOT: 0 frees
>> > GOT: 0 holds
>>
>> --
>> Jordan.



--
++ytti

On 5/4/22 21:56, Nehul Patel wrote:
>
> Hi NANOG,
>
> We are seeing some strange behavior on our Juniper MX240 Chassis it is
> randomly dropping the routes to the certain destination IP address
> getting the following errors on the MX240 Chassis
>
> If Someone has seen these errors before please suggest how to resolve it
>
>
> May  4 12:42:00 cr01 newsyslog[44735]: logfile turned over due to
> size>1024K
> May  4 12:42:01   /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed,
> err 5 (Invalid)
> May  4 12:42:01   /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed,
> err 5 (Invalid)
> May  4 12:42:01   last message repeated 4 times
> May  4 12:42:01   fpc0 RT: IPv6:0 - 2600:40fc:1011::/48 (add rt entry
> into jtree failed)
> May  4 12:42:01   fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> May  4 12:42:01   fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto
> ipv6,len 48 prefix 2600:40fc:1011::/48 nh 1048576
> May  4 12:42:01   fpc0 RT-HAL,rt_msg_handler,540: route process failed
> May  4 12:42:01   fpc0 RT: Failed prefix add IPv6 - 2001:67c:20fc::/48
> (No memory) on FE 0
> May  4 12:42:01   fpc0 RT: IPv6:0 - 2001:67c:20fc::/48 (add rt entry
> into jtree failed)
> May  4 12:42:01   fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> May  4 12:42:01   fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto
> ipv6,len 48 prefix 2001:67c:20fc::/48 nh 1048576
> May  4 12:42:01   fpc0 RT-HAL,rt_msg_handler,540: route process failed
> May  4 12:42:01   fpc0 RT: Failed prefix add IPv6 -
> 2606:2800:e004::/48 (No memory) on FE 0
> May  4 12:42:01   fpc0 RT: Failed prefix add IPv6 -
> 2a05:3181:ffff::/48 (No memory) on FE 0
> May  4 12:42:01   /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed,
> err 5 (Invalid)
> May  4 12:42:01   /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed,
> err 5 (Invalid)
> May  4 12:42:01   /kernel: RT_PFE: RT msg op 2 (PREFIX DELETE) failed,
> err 5 (Invalid)
> May  4 12:42:02   fpc0 RT: Failed prefix add IPv4 - 79.120.22/24 (No
> memory) on FE 0
> May  4 12:42:02   fpc0 RT: IPv4:0 - 79.120.22/24 (add rt entry into
> jtree failed)
> May  4 12:42:02   fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> May  4 12:42:02   fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto
> ipv4,len 24 prefix 79.120.22/24 nh 1048583
> May  4 12:42:02   /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed,
> err 5 (Invalid)
> May  4 12:42:02   fpc0 RT-HAL,rt_msg_handler,540: route process failed
>
> May  4 09:33:17   fpc0 RSMON: Resource Category:jtree
>  Instance:jtree2-seg0 Type:free-pages Available:20 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May  4 09:33:17   fpc0 RSMON: Resource Category:jtree
>  Instance:jtree2-seg0 Type:free-dwords Available:1280 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May  4 09:33:18   fpc0 RSMON: Resource Category:jtree
>  Instance:jtree3-seg0 Type:free-pages Available:19 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May  4 09:33:18   fpc0 RSMON: Resource Category:jtree
>  Instance:jtree3-seg0 Type:free-dwords Available:1216 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May  4 09:33:18   fpc1 RSMON: Resource Category:jtree
>  Instance:jtree0-seg0 Type:free-pages Available:16 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May  4 09:33:18   fpc1 RSMON: Resource Category:jtree
>  Instance:jtree0-seg0 Type:free-dwords Available:1024 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May  4 09:33:18   fpc1 RSMON: Resource Category:jtree
>  Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May  4 09:33:18   fpc1 RSMON: Resource Category:jtree
>  Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May  4 09:33:18   fpc1 RSMON: Resource Category:jtree
>  Instance:jtree2-seg0 Type:free-pages Available:19 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May  4 09:33:19   fpc1 RSMON: Resource Category:jtree
>  Instance:jtree2-seg0 Type:free-dwords Available:1216 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May  4 09:33:19   fpc1 RSMON: Resource Category:jtree
>  Instance:jtree3-seg0 Type:free-pages Available:17 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May  4 09:33:19   fpc1 RSMON: Resource Category:jtree
>  Instance:jtree3-seg0 Type:free-dwords Available:1088 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May  4 09:33:19   fpc2 RSMON: Resource Category:jtree
>  Instance:jtree0-seg0 Type:free-pages Available:15 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May  4 09:33:19   fpc2 RSMON: Resource Category:jtree
>  Instance:jtree0-seg0 Type:free-dwords Available:960 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May  4 09:33:19   fpc2 RSMON: Resource Category:jtree
>  Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May  4 09:33:19   fpc2 RSMON: Resource Category:jtree
>  Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM
> limit:104857, rsmon_syslog_limit()
>
> Any suggestions will be helpful

Looks like you are out of FIB slots.

Would recommend reducing the number of routes you need to send into FIB,
or upgrading to newer hardware that has more space.

Mark.

On 5/5/22 04:11, Jordan wrote:

> Your line cards (not RE's) are running out of route-storage memory.
> As a short-term mitigation, you could try borrowing from segment 1,
> normally dedicated to filters,
>
> set chassis memory-enhanced route
>
> but this option may not exist in the version of JunOS you're
> running, which as already mentioned is very old.

This feature was introduced in 10.4, so he should have it.

And yes, it's only supported for DPC's (I-chip).

Mark.

On 5/5/22 05:08, Nehul Patel wrote:

> Ok, thank you all for the feedback we are going to start with the
> Junos OS upgrade first on it but have to open the ticket with JTAC
> since currently on the juniper support website they have the Junos
> 15.1 is available so not sure we can directly jump from 10.4 to 15.1
> maybe we have to do step by step upgrade on it. Any other suggestions
> will be helpful as well
>
> By the way, the uptime on the Juniper MX chassis was 1589 Days on it.

Curious, what RE are you running?

If you have DPC's still, I'd assume something like the RE-S-1300 or
RE-S-2000, but not sure.

I ask because I'm not how late the older RE's can go.

Mark.

On 5/5/22 07:34, Saku Ytti wrote:

> And like Jordan said, you are out of resources but can extend them
> with the command given, which should give you more run rate. You may
> want to look in more detail how long you can keep running DPCE until
> you're really out.

Certainly an option, but this requires quite a bit of babysitting,
because as the DFZ oscillates, you can run into issues that send you
into circles, largely unaware about FIB issues, especially when just a
subset of routes are affected.

So yes, definitely an option, but the OP will need to watch the line
cards like a hawk, and be ultra sensitive to debugging regular issues
vs. FIB-related issues.

Mark.

Ok got it saku we will got with the direct upgrade of it

On Wed, May 4, 2022, 22:34 Saku Ytti <saku@ytti.fi> wrote:

> Almost always direct upgrade works. If you ask TAC, they will likely
> suggest a formal process and you'll be doing many upgrades, which
> itself isn't actually something that is guaranteed to work (like in
> WRL9 case, but that is vmhost RE, not yours).
>
> And like Jordan said, you are out of resources but can extend them
> with the command given, which should give you more run rate. You may
> want to look in more detail how long you can keep running DPCE until
> you're really out.
>
>
> On Thu, 5 May 2022 at 06:08, Nehul Patel <nehul.patel@gmail.com> wrote:
> >
> > Ok, thank you all for the feedback we are going to start with the Junos
> OS upgrade first on it but have to open the ticket with JTAC since
> currently on the juniper support website they have the Junos 15.1 is
> available so not sure we can directly jump from 10.4 to 15.1 maybe we have
> to do step by step upgrade on it. Any other suggestions will be helpful as
> well
> >
> > By the way, the uptime on the Juniper MX chassis was 1589 Days on it.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Wed, May 4, 2022 at 7:11 PM Jordan <jnh@vt11.net> wrote:
> >>
> >> Your line cards (not RE's) are running out of route-storage memory.
> >> As a short-term mitigation, you could try borrowing from segment 1,
> >> normally dedicated to filters,
> >>
> >> set chassis memory-enhanced route
> >>
> >> but this option may not exist in the version of JunOS you're
> >> running, which as already mentioned is very old.
> >>
> >> If the command is accepted, and it lets you commit, you'll then
> >> need to restart each of the FPC's, one at a time, by slot number,
> >> which will take each out of service for a few minutes, so you
> >> probably want to wait until a scheduled maintenance period, and
> >> start with less-important FPC slots first:
> >>
> >> request chassis fpc restart slot X
> >>
> >>
> >>
> >> On Wed, May 04, 2022 at 02:34:45PM -0700, Nehul Patel wrote:
> >> > Thank you Saku and the warren Here is the requested output
> >> >
> >> >
> >> > show route summary
> >> >
> >> > inet.0: 879635 destinations, 879649 routes (879634 active, 0
> holddown, 1
> >> > hidden)
> >> > Direct: 9 routes, 8 active
> >> > Local: 8 routes, 8 active
> >> > OSPF: 928 routes, 925 active
> >> > BGP: 878686 routes, 878678 active
> >> > Static: 2 routes, 2 active
> >> > Aggregate: 15 routes, 12 active
> >> >
> >> > inet.3: 718 destinations, 718 routes (718 active, 0 holddown, 0
> hidden)
> >> > LDP: 718 routes, 718 active
> >> >
> >> > Test_VRF.inet.0: 1 destinations, 1 routes (1 active, 0 holddown, 0
> hidden)
> >> > Local: 1 routes, 1 active
> >> >
> >> > mpls.0: 390 destinations, 390 routes (390 active, 0 holddown, 0
> hidden)
> >> > MPLS: 3 routes, 3 active
> >> > LDP: 387 routes, 387 active
> >> >
> >> > inet6.0: 143065 destinations, 286099 routes (143065 active, 0
> holddown, 0
> >> > hidden)
> >> > Direct: 13 routes, 9 active
> >> > Local: 10 routes, 10 active
> >> > OSPF3: 16 routes, 15 active
> >> > BGP: 286060 routes, 143030 active
> >> > Static: 1 routes, 1 active
> >> >
> >> > show chassis fpc
> >> > Temp CPU Utilization (%) Memory
> Utilization (%)
> >> > Slot State (C) Total Interrupt DRAM (MB) Heap
> Buffer
> >> > 0 Online 31 11 0 1024 37
> 29
> >> > 1 Online 31 11 0 1024 45
> 29
> >> > 2 Online 30 4 0 1024 36
> 29
> >> >
> >> > request pfe execute target fpc0 command "show jtree 0 memory
> extensive"
> >> > SENT: Ukern command: show jtree 0 memory extensive
> >> > GOT:
> >> > GOT: Jtree memory segment 0 (Context: 0x44817bb0)
> >> > GOT: -------------------------------------------
> >> > GOT: Memory Statistics:
> >> > GOT: 16777216 bytes total
> >> > GOT: 16715144 bytes used
> >> > GOT: 56880 bytes available (7168 bytes from free pages)
> >> > GOT: 3024 bytes wasted
> >> > GOT: 2168 bytes unusable
> >> > GOT: 32768 pages total
> >> > GOT: 32519 pages used (2568 pages used in page alloc)
> >> > GOT: 235 pages partially used
> >> > GOT: 14 pages free (max contiguous = 6)
> >> > GOT:
> >> > GOT: Partially Filled Pages (In bytes):-
> >> > GOT: Unit Avail Overhead
> >> > GOT: 8 26256 0
> >> > GOT: 16 14320 0
> >> > GOT: 24 8352 2040
> >> > GOT: 32 352 0
> >> > GOT: 48 432 128
> >> > GOT:
> >> > GOT: Free Page Lists(Pg Size = 512 bytes):-
> >> > GOT: Page Bucket Avail(Bytes)
> >> > GOT: 1-1 2560
> >> > GOT: 3-3 1536
> >> > GOT: 6-6 3072
> >> > GOT:
> >> > GOT: Fragmentation Index = 0.946, (largest free = 3072)
> >> > GOT: Counters:
> >> > GOT: 2643777 allocs (0 failed)
> >> > GOT: 0 releases(partial 0)
> >> > GOT: 1095040 frees
> >> > GOT: 0 holds
> >> > GOT: 7 pending frees(pending bytes 56)
> >> > GOT: 0 pending forced
> >> > GOT: 0 times free blocked
> >> > GOT: 0 sync writes
> >> > GOT: Error Counters:-
> >> > GOT: 0 bad params
> >> > GOT: 0 failed frees
> >> > GOT: 0 bad cookie
> >> > GOT:
> >> > GOT: Jtree memory segment 1 (Context: 0x448997f0)
> >> > GOT: -------------------------------------------
> >> > GOT: Memory Statistics:
> >> > GOT: 16777216 bytes total
> >> > GOT: 4589552 bytes used
> >> > GOT: 12185384 bytes available (12183552 bytes from free pages)
> >> > GOT: 2248 bytes wasted
> >> > GOT: 32 bytes unusable
> >> > GOT: 32768 pages total
> >> > GOT: 8967 pages used (8967 pages used in page alloc)
> >> > GOT: 5 pages partially used
> >> > GOT: 23796 pages free (max contiguous = 23793)
> >> > GOT:
> >> > GOT: Partially Filled Pages (In bytes):-
> >> > GOT: Unit Avail Overhead
> >> > GOT: 8 1416 0
> >> > GOT: 16 80 0
> >> > GOT: 48 336 32
> >> >
> >> >
> >> > request pfe execute target fpc1 command "show jtree 0 memory
> extensive"
> >> > SENT: Ukern command: show jtree 0 memory extensive
> >> > GOT:
> >> > GOT: Jtree memory segment 0 (Context: 0x447cc698)
> >> > GOT: -------------------------------------------
> >> > GOT: Memory Statistics:
> >> > GOT: 16777216 bytes total
> >> > GOT: 16715840 bytes used
> >> > GOT: 56184 bytes available (8192 bytes from free pages)
> >> > GOT: 3024 bytes wasted
> >> > GOT: 2168 bytes unusable
> >> > GOT: 32768 pages total
> >> > GOT: 32533 pages used (2568 pages used in page alloc)
> >> > GOT: 219 pages partially used
> >> > GOT: 16 pages free (max contiguous = 5)
> >> > GOT:
> >> > GOT: Partially Filled Pages (In bytes):-
> >> > GOT: Unit Avail Overhead
> >> > GOT: 8 25544 0
> >> > GOT: 16 13312 0
> >> > GOT: 24 8352 2040
> >> > GOT: 32 352 0
> >> > GOT: 48 432 128
> >> > GOT:
> >> > GOT: Free Page Lists(Pg Size = 512 bytes):-
> >> > GOT: Page Bucket Avail(Bytes)
> >> > GOT: 1-1 2048
> >> > GOT: 2-2 1024
> >> > GOT: 5-5 5120
> >> > GOT:
> >> > GOT: Fragmentation Index = 0.954, (largest free = 2560)
> >> > GOT: Counters:
> >> > GOT: 2645725 allocs (0 failed)
> >> > GOT: 2 releases(partial 0)
> >> > GOT: 1096891 frees
> >> > GOT: 0 holds
> >> > GOT: 0 pending frees(pending bytes 0)
> >> > GOT: 0 pending forced
> >> > GOT: 0 times free blocked
> >> > GOT: 0 sync writes
> >> > GOT: Error Counters:-
> >> > GOT: 0 bad params
> >> > GOT: 0 failed frees
> >> > GOT: 0 bad cookie
> >> > GOT:
> >> > GOT: Jtree memory segment 1 (Context: 0x4484e2d8)
> >> > GOT: -------------------------------------------
> >> > GOT: Memory Statistics:
> >> > GOT: 16777216 bytes total
> >> > GOT: 4589504 bytes used
> >> > GOT: 12185432 bytes available (12184576 bytes from free pages)
> >> > GOT: 2248 bytes wasted
> >> > GOT: 32 bytes unusable
> >> > GOT: 32768 pages total
> >> > GOT: 8967 pages used (8967 pages used in page alloc)
> >> > GOT: 3 pages partially used
> >> > GOT: 23798 pages free (max contiguous = 23798)
> >> > GOT:
> >> > GOT: Partially Filled Pages (In bytes):-
> >> > GOT: Unit Avail Overhead
> >> > GOT: 8 424 0
> >> > GOT: 16 96 0
> >> > GOT: 48 336 32
> >> > GOT:
> >> > GOT: Free Page Lists(Pg Size = 512 bytes):-
> >> > GOT: Page Bucket Avail(Bytes)
> >> > GOT: 27-32768 12184576
> >> > GOT:
> >> > GOT: Fragmentation Index = 0.000, (largest free = 12184576)
> >> > GOT: Counters:
> >> > GOT: 45 allocs (0 failed)
> >> > GOT: 0 releases(partial 0)
> >> > GOT: 0 frees
> >> > GOT: 0 holds
> >>
> >> --
> >> Jordan.
>
>
>
> --
> ++ytti
>

Friend of mine had this issue recently on an MX chassis running DPC's and RE-2000's.

The extend memory command others have mentioned worked for him.

His instance drove us crazy for a bit. The device would learn a route, show that it was installed (show routes) but traffic to said prefix would bounce net unreachable. We even pushed a static just for S&G's and that still didn't resolve it. It was a single prefix that a customer had reported.

Some things to consider, as others have mentioned.


1. IPv6 routes share the same space. And use more per-route. You can extend the life of this box (probably considerably) by dropping full tables for IPv6. Perhaps taking just a default (Same goes for v4).
2. It seems from your previous output that you're taking ~1 full v4 table. And 2x v6 tables. Do you really need a full table if you're only taking 1 v4 table? Consider switching to a default only? In my Colleagues case, he was taking 2 full tables of v4 and v6 until he hit the same issue.
3. While you're RE's could use a nice upgrade too. Your linecards are actually the problem here. If you move to anything > DPC you get the trio chipset with much more FIB space (2 Million routes I believe?). I'd consider new RE's and new line cards for this box. Which might also mean new switch fabric controllers.... Basically, we'd be talking a full overhaul sans the power supplies and chassis.
4. Consider taking a default + full routes. Then filtering > /24 (if you even have anything < /24 learned now) (/48 on IPv6).

Start with the memory command first and see where that gets you. But keep a watchful eye out for this to happen again (as the DFZ grows). Eventually your only option will be to filter routes and rely on a default or upgrade.
________________________________
From: NANOG <nanog-bounces+nick=141networks.com@nanog.org> on behalf of Nehul Patel <nehul.patel@gmail.com>
Sent: Wednesday, May 4, 2022 3:56 PM
To: nanog@nanog.org <nanog@nanog.org>
Subject: Strange behavior on the Juniper MX240


Hi NANOG,

We are seeing some strange behavior on our Juniper MX240 Chassis it is randomly dropping the routes to the certain destination IP address getting the following errors on the MX240 Chassis

If Someone has seen these errors before please suggest how to resolve it


May 4 12:42:00 cr01 newsyslog[44735]: logfile turned over due to size>1024K
May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid)
May 4 12:42:01 last message repeated 4 times
May 4 12:42:01 fpc0 RT: IPv6:0 - 2600:40fc:1011::/48 (add rt entry into jtree failed)
May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2600:40fc:1011::/48 nh 1048576
May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2001:67c:20fc::/48 (No memory) on FE 0
May 4 12:42:01 fpc0 RT: IPv6:0 - 2001:67c:20fc::/48 (add rt entry into jtree failed)
May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2001:67c:20fc::/48 nh 1048576
May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2606:2800:e004::/48 (No memory) on FE 0
May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2a05:3181:ffff::/48 (No memory) on FE 0
May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid)
May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
May 4 12:42:01 /kernel: RT_PFE: RT msg op 2 (PREFIX DELETE) failed, err 5 (Invalid)
May 4 12:42:02 fpc0 RT: Failed prefix add IPv4 - 79.120.22/24 (No memory) on FE 0
May 4 12:42:02 fpc0 RT: IPv4:0 - 79.120.22/24 (add rt entry into jtree failed)
May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv4,len 24 prefix 79.120.22/24 nh 1048583
May 4 12:42:02 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
May 4 12:42:02 fpc0 RT-HAL,rt_msg_handler,540: route process failed

May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-pages Available:20 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-dwords Available:1280 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-pages Available:16 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-dwords Available:1024 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-pages Available:17 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-dwords Available:1088 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()

Any suggestions will be helpful


Please do let me know if you have any questions.




Regards and thanks,
Nehul

Hi Nick,

Thank you for the feedback on it. Would you please let me know which Juno
OS version he had installed on the MX Chassis that works with the extended
memory command of it?

On Thu, May 5, 2022 at 12:50 PM Nick Olsen <Nick@141networks.com> wrote:

> Friend of mine had this issue recently on an MX chassis running DPC's and
> RE-2000's.
>
> The extend memory command others have mentioned worked for him.
>
> His instance drove us crazy for a bit. The device would learn a route,
> show that it was installed (show routes) but traffic to said prefix would
> bounce net unreachable. We even pushed a static just for S&G's and that
> still didn't resolve it. It was a single prefix that a customer had
> reported.
>
> Some things to consider, as others have mentioned.
>
>
> 1. IPv6 routes share the same space. And use more per-route. You can
> extend the life of this box (probably considerably) by dropping full tables
> for IPv6. Perhaps taking just a default (Same goes for v4).
> 2. It seems from your previous output that you're taking ~1 full v4
> table. And 2x v6 tables. Do you really need a full table if you're only
> taking 1 v4 table? Consider switching to a default only? In my Colleagues
> case, he was taking 2 full tables of v4 and v6 until he hit the same issue.
> 3. While you're RE's could use a nice upgrade too. Your linecards are
> actually the problem here. If you move to anything > DPC you get the trio
> chipset with much more FIB space (2 Million routes I believe?). I'd
> consider new RE's and new line cards for this box. Which might also mean
> new switch fabric controllers.... Basically, we'd be talking a full
> overhaul sans the power supplies and chassis.
> 4. Consider taking a default + full routes. Then filtering > /24 (if
> you even have anything < /24 learned now) (/48 on IPv6).
>
> Start with the memory command first and see where that gets you. But keep
> a watchful eye out for this to happen again (as the DFZ grows). Eventually
> your only option will be to filter routes and rely on a default or upgrade.
> ------------------------------
> *From:* NANOG <nanog-bounces+nick=141networks.com@nanog.org> on behalf of
> Nehul Patel <nehul.patel@gmail.com>
> *Sent:* Wednesday, May 4, 2022 3:56 PM
> *To:* nanog@nanog.org <nanog@nanog.org>
> *Subject:* Strange behavior on the Juniper MX240
>
>
> Hi NANOG,
>
> We are seeing some strange behavior on our Juniper MX240 Chassis it is
> randomly dropping the routes to the certain destination IP address getting
> the following errors on the MX240 Chassis
>
> If Someone has seen these errors before please suggest how to resolve it
>
>
> May 4 12:42:00 cr01 newsyslog[44735]: logfile turned over due to
> size>1024K
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5
> (Invalid)
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err
> 5 (Invalid)
> May 4 12:42:01 last message repeated 4 times
> May 4 12:42:01 fpc0 RT: IPv6:0 - 2600:40fc:1011::/48 (add rt entry into
> jtree failed)
> May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len
> 48 prefix 2600:40fc:1011::/48 nh 1048576
> May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
> May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2001:67c:20fc::/48 (No
> memory) on FE 0
> May 4 12:42:01 fpc0 RT: IPv6:0 - 2001:67c:20fc::/48 (add rt entry into
> jtree failed)
> May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len
> 48 prefix 2001:67c:20fc::/48 nh 1048576
> May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
> May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2606:2800:e004::/48
> (No memory) on FE 0
> May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2a05:3181:ffff::/48
> (No memory) on FE 0
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err
> 5 (Invalid)
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5
> (Invalid)
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 2 (PREFIX DELETE) failed, err
> 5 (Invalid)
> May 4 12:42:02 fpc0 RT: Failed prefix add IPv4 - 79.120.22/24 (No
> memory) on FE 0
> May 4 12:42:02 fpc0 RT: IPv4:0 - 79.120.22/24 (add rt entry into jtree
> failed)
> May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv4,len
> 24 prefix 79.120.22/24 nh 1048583
> May 4 12:42:02 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5
> (Invalid)
> May 4 12:42:02 fpc0 RT-HAL,rt_msg_handler,540: route process failed
>
> May 4 09:33:17 fpc0 RSMON: Resource Category:jtree
> Instance:jtree2-seg0 Type:free-pages Available:20 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May 4 09:33:17 fpc0 RSMON: Resource Category:jtree
> Instance:jtree2-seg0 Type:free-dwords Available:1280 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May 4 09:33:18 fpc0 RSMON: Resource Category:jtree
> Instance:jtree3-seg0 Type:free-pages Available:19 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May 4 09:33:18 fpc0 RSMON: Resource Category:jtree
> Instance:jtree3-seg0 Type:free-dwords Available:1216 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree
> Instance:jtree0-seg0 Type:free-pages Available:16 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree
> Instance:jtree0-seg0 Type:free-dwords Available:1024 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree
> Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree
> Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree
> Instance:jtree2-seg0 Type:free-pages Available:19 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May 4 09:33:19 fpc1 RSMON: Resource Category:jtree
> Instance:jtree2-seg0 Type:free-dwords Available:1216 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May 4 09:33:19 fpc1 RSMON: Resource Category:jtree
> Instance:jtree3-seg0 Type:free-pages Available:17 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May 4 09:33:19 fpc1 RSMON: Resource Category:jtree
> Instance:jtree3-seg0 Type:free-dwords Available:1088 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May 4 09:33:19 fpc2 RSMON: Resource Category:jtree
> Instance:jtree0-seg0 Type:free-pages Available:15 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May 4 09:33:19 fpc2 RSMON: Resource Category:jtree
> Instance:jtree0-seg0 Type:free-dwords Available:960 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May 4 09:33:19 fpc2 RSMON: Resource Category:jtree
> Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May 4 09:33:19 fpc2 RSMON: Resource Category:jtree
> Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM
> limit:104857, rsmon_syslog_limit()
>
> Any suggestions will be helpful
>
>
> Please do let me know if you have any questions.
>
>
>
>
> Regards and thanks,
> Nehul
>
>
>

Nehul,

He was running the 15 code train. I think 15.1R6.7. But don't take that as fact. I just know it was 15 for sure.
________________________________
From: Nehul Patel <nehul.patel@gmail.com>
Sent: Thursday, May 5, 2022 6:40 PM
To: Nick Olsen <Nick@141networks.com>
Cc: nanog@nanog.org <nanog@nanog.org>
Subject: Re: Strange behavior on the Juniper MX240

Hi Nick,

Thank you for the feedback on it. Would you please let me know which Juno OS version he had installed on the MX Chassis that works with the extended memory command of it?

On Thu, May 5, 2022 at 12:50 PM Nick Olsen <Nick@141networks.com<mailto:Nick@141networks.com>> wrote:
Friend of mine had this issue recently on an MX chassis running DPC's and RE-2000's.

The extend memory command others have mentioned worked for him.

His instance drove us crazy for a bit. The device would learn a route, show that it was installed (show routes) but traffic to said prefix would bounce net unreachable. We even pushed a static just for S&G's and that still didn't resolve it. It was a single prefix that a customer had reported.

Some things to consider, as others have mentioned.


1. IPv6 routes share the same space. And use more per-route. You can extend the life of this box (probably considerably) by dropping full tables for IPv6. Perhaps taking just a default (Same goes for v4).
2. It seems from your previous output that you're taking ~1 full v4 table. And 2x v6 tables. Do you really need a full table if you're only taking 1 v4 table? Consider switching to a default only? In my Colleagues case, he was taking 2 full tables of v4 and v6 until he hit the same issue.
3. While you're RE's could use a nice upgrade too. Your linecards are actually the problem here. If you move to anything > DPC you get the trio chipset with much more FIB space (2 Million routes I believe?). I'd consider new RE's and new line cards for this box. Which might also mean new switch fabric controllers.... Basically, we'd be talking a full overhaul sans the power supplies and chassis.
4. Consider taking a default + full routes. Then filtering > /24 (if you even have anything < /24 learned now) (/48 on IPv6).

Start with the memory command first and see where that gets you. But keep a watchful eye out for this to happen again (as the DFZ grows). Eventually your only option will be to filter routes and rely on a default or upgrade.
________________________________
From: NANOG <nanog-bounces+nick=141networks.com@nanog.org<mailto:141networks.com@nanog.org>> on behalf of Nehul Patel <nehul.patel@gmail.com<mailto:nehul.patel@gmail.com>>
Sent: Wednesday, May 4, 2022 3:56 PM
To: nanog@nanog.org<mailto:nanog@nanog.org> <nanog@nanog.org<mailto:nanog@nanog.org>>
Subject: Strange behavior on the Juniper MX240


Hi NANOG,

We are seeing some strange behavior on our Juniper MX240 Chassis it is randomly dropping the routes to the certain destination IP address getting the following errors on the MX240 Chassis

If Someone has seen these errors before please suggest how to resolve it


May 4 12:42:00 cr01 newsyslog[44735]: logfile turned over due to size>1024K
May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid)
May 4 12:42:01 last message repeated 4 times
May 4 12:42:01 fpc0 RT: IPv6:0 - 2600:40fc:1011::/48 (add rt entry into jtree failed)
May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2600:40fc:1011::/48 nh 1048576
May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2001:67c:20fc::/48 (No memory) on FE 0
May 4 12:42:01 fpc0 RT: IPv6:0 - 2001:67c:20fc::/48 (add rt entry into jtree failed)
May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len 48 prefix 2001:67c:20fc::/48 nh 1048576
May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2606:2800:e004::/48 (No memory) on FE 0
May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2a05:3181:ffff::/48 (No memory) on FE 0
May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err 5 (Invalid)
May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
May 4 12:42:01 /kernel: RT_PFE: RT msg op 2 (PREFIX DELETE) failed, err 5 (Invalid)
May 4 12:42:02 fpc0 RT: Failed prefix add IPv4 - 79.120.22/24 (No memory) on FE 0
May 4 12:42:02 fpc0 RT: IPv4:0 - 79.120.22/24 (add rt entry into jtree failed)
May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2028: rt_halp_vectors->rt_create failed
May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv4,len 24 prefix 79.120.22/24 nh 1048583
May 4 12:42:02 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5 (Invalid)
May 4 12:42:02 fpc0 RT-HAL,rt_msg_handler,540: route process failed

May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-pages Available:20 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:17 fpc0 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-dwords Available:1280 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:18 fpc0 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-pages Available:16 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-dwords Available:1024 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:18 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-pages Available:19 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree2-seg0 Type:free-dwords Available:1216 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-pages Available:17 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:19 fpc1 RSMON: Resource Category:jtree Instance:jtree3-seg0 Type:free-dwords Available:1088 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree0-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()
May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM limit:1638, rsmon_syslog_limit()
May 4 09:33:19 fpc2 RSMON: Resource Category:jtree Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM limit:104857, rsmon_syslog_limit()

Any suggestions will be helpful


Please do let me know if you have any questions.




Regards and thanks,
Nehul

ok got it thank you Nick

On Thu, May 5, 2022 at 3:43 PM Nick Olsen <Nick@141networks.com> wrote:

> Nehul,
>
> He was running the 15 code train. I think 15.1R6.7. But don't take that as
> fact. I just know it was 15 for sure.
> ------------------------------
> *From:* Nehul Patel <nehul.patel@gmail.com>
> *Sent:* Thursday, May 5, 2022 6:40 PM
> *To:* Nick Olsen <Nick@141networks.com>
> *Cc:* nanog@nanog.org <nanog@nanog.org>
> *Subject:* Re: Strange behavior on the Juniper MX240
>
> Hi Nick,
>
> Thank you for the feedback on it. Would you please let me know which Juno
> OS version he had installed on the MX Chassis that works with the extended
> memory command of it?
>
> On Thu, May 5, 2022 at 12:50 PM Nick Olsen <Nick@141networks.com> wrote:
>
> Friend of mine had this issue recently on an MX chassis running DPC's and
> RE-2000's.
>
> The extend memory command others have mentioned worked for him.
>
> His instance drove us crazy for a bit. The device would learn a route,
> show that it was installed (show routes) but traffic to said prefix would
> bounce net unreachable. We even pushed a static just for S&G's and that
> still didn't resolve it. It was a single prefix that a customer had
> reported.
>
> Some things to consider, as others have mentioned.
>
>
> 1. IPv6 routes share the same space. And use more per-route. You can
> extend the life of this box (probably considerably) by dropping full tables
> for IPv6. Perhaps taking just a default (Same goes for v4).
> 2. It seems from your previous output that you're taking ~1 full v4
> table. And 2x v6 tables. Do you really need a full table if you're only
> taking 1 v4 table? Consider switching to a default only? In my Colleagues
> case, he was taking 2 full tables of v4 and v6 until he hit the same issue.
> 3. While you're RE's could use a nice upgrade too. Your linecards are
> actually the problem here. If you move to anything > DPC you get the trio
> chipset with much more FIB space (2 Million routes I believe?). I'd
> consider new RE's and new line cards for this box. Which might also mean
> new switch fabric controllers.... Basically, we'd be talking a full
> overhaul sans the power supplies and chassis.
> 4. Consider taking a default + full routes. Then filtering > /24 (if
> you even have anything < /24 learned now) (/48 on IPv6).
>
> Start with the memory command first and see where that gets you. But keep
> a watchful eye out for this to happen again (as the DFZ grows). Eventually
> your only option will be to filter routes and rely on a default or upgrade.
> ------------------------------
> *From:* NANOG <nanog-bounces+nick=141networks.com@nanog.org> on behalf of
> Nehul Patel <nehul.patel@gmail.com>
> *Sent:* Wednesday, May 4, 2022 3:56 PM
> *To:* nanog@nanog.org <nanog@nanog.org>
> *Subject:* Strange behavior on the Juniper MX240
>
>
> Hi NANOG,
>
> We are seeing some strange behavior on our Juniper MX240 Chassis it is
> randomly dropping the routes to the certain destination IP address getting
> the following errors on the MX240 Chassis
>
> If Someone has seen these errors before please suggest how to resolve it
>
>
> May 4 12:42:00 cr01 newsyslog[44735]: logfile turned over due to
> size>1024K
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5
> (Invalid)
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err
> 5 (Invalid)
> May 4 12:42:01 last message repeated 4 times
> May 4 12:42:01 fpc0 RT: IPv6:0 - 2600:40fc:1011::/48 (add rt entry into
> jtree failed)
> May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len
> 48 prefix 2600:40fc:1011::/48 nh 1048576
> May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
> May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2001:67c:20fc::/48 (No
> memory) on FE 0
> May 4 12:42:01 fpc0 RT: IPv6:0 - 2001:67c:20fc::/48 (add rt entry into
> jtree failed)
> May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> May 4 12:42:01 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len
> 48 prefix 2001:67c:20fc::/48 nh 1048576
> May 4 12:42:01 fpc0 RT-HAL,rt_msg_handler,540: route process failed
> May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2606:2800:e004::/48
> (No memory) on FE 0
> May 4 12:42:01 fpc0 RT: Failed prefix add IPv6 - 2a05:3181:ffff::/48
> (No memory) on FE 0
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed, err
> 5 (Invalid)
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5
> (Invalid)
> May 4 12:42:01 /kernel: RT_PFE: RT msg op 2 (PREFIX DELETE) failed, err
> 5 (Invalid)
> May 4 12:42:02 fpc0 RT: Failed prefix add IPv4 - 79.120.22/24 (No
> memory) on FE 0
> May 4 12:42:02 fpc0 RT: IPv4:0 - 79.120.22/24 (add rt entry into jtree
> failed)
> May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> May 4 12:42:02 fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv4,len
> 24 prefix 79.120.22/24 nh 1048583
> May 4 12:42:02 /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err 5
> (Invalid)
> May 4 12:42:02 fpc0 RT-HAL,rt_msg_handler,540: route process failed
>
> May 4 09:33:17 fpc0 RSMON: Resource Category:jtree
> Instance:jtree2-seg0 Type:free-pages Available:20 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May 4 09:33:17 fpc0 RSMON: Resource Category:jtree
> Instance:jtree2-seg0 Type:free-dwords Available:1280 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May 4 09:33:18 fpc0 RSMON: Resource Category:jtree
> Instance:jtree3-seg0 Type:free-pages Available:19 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May 4 09:33:18 fpc0 RSMON: Resource Category:jtree
> Instance:jtree3-seg0 Type:free-dwords Available:1216 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree
> Instance:jtree0-seg0 Type:free-pages Available:16 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree
> Instance:jtree0-seg0 Type:free-dwords Available:1024 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree
> Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree
> Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May 4 09:33:18 fpc1 RSMON: Resource Category:jtree
> Instance:jtree2-seg0 Type:free-pages Available:19 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May 4 09:33:19 fpc1 RSMON: Resource Category:jtree
> Instance:jtree2-seg0 Type:free-dwords Available:1216 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May 4 09:33:19 fpc1 RSMON: Resource Category:jtree
> Instance:jtree3-seg0 Type:free-pages Available:17 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May 4 09:33:19 fpc1 RSMON: Resource Category:jtree
> Instance:jtree3-seg0 Type:free-dwords Available:1088 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May 4 09:33:19 fpc2 RSMON: Resource Category:jtree
> Instance:jtree0-seg0 Type:free-pages Available:15 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May 4 09:33:19 fpc2 RSMON: Resource Category:jtree
> Instance:jtree0-seg0 Type:free-dwords Available:960 is less than LWM
> limit:104857, rsmon_syslog_limit()
> May 4 09:33:19 fpc2 RSMON: Resource Category:jtree
> Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM
> limit:1638, rsmon_syslog_limit()
> May 4 09:33:19 fpc2 RSMON: Resource Category:jtree
> Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM
> limit:104857, rsmon_syslog_limit()
>
> Any suggestions will be helpful
>
>
> Please do let me know if you have any questions.
>
>
>
>
> Regards and thanks,
> Nehul
>
>
>

On 5/5/22 21:50, Nick Olsen wrote:

> His instance drove us crazy for a bit. The device would learn a route,
> show that it was installed (show routes) but traffic to said prefix
> would bounce net unreachable. We even pushed a static just for S&G's
> and that still didn't resolve it. It was a single prefix that a
> customer had reported.

> Start with the memory command first and see where that gets you. But
> keep a watchful eye out for this to happen again (as the DFZ grows).
> Eventually your only option will be to filter routes and rely on a
> default or upgrade.

These are the reasons why I was saying that while there may be some
commands to move FIB allocations around, it's a lot of admin. because
the DFZ is very dynamic, and FIB programming issues due to lack of slots
that affect different prefixes in different ways can have you chasing
your tail for weeks before you figure things out.

I think doing this should be a short-term solution as you make plans to
get newer hardware. As a long-term strategy, it will tax day-to-day
operations.

Mark.

On Fri, 6 May 2022 at 10:59, Mark Tinka <mark@tinka.africa> wrote:

> These are the reasons why I was saying that while there may be some commands to move FIB allocations around, it's a lot of admin. because the DFZ is very dynamic, and FIB programming issues due to lack of slots that affect different prefixes in different ways can have you chasing your tail for weeks before you figure things out.
>
> I think doing this should be a short-term solution as you make plans to get newer hardware. As a long-term strategy, it will tax day-to-day operations.

This seems like a strange position. The device has 16MB+16MB jtree
segments. The first is IP, the second is filters (Broadly).

OP has 16MB of first used.
OP has <5MB of second used.

What if the platform had originally shipped with a different balance
between filters and IP, and OP would have never hit this problem?

It is easy to see in many scenarios filter growth is negligible toi 0,
IP growth is not. OP would technically have 70% FIB growth left, so
DFZ of about 1.7M, which puts him in the year >2030 (potentially far
beyond, but at least that).

I view the recarving as fixing poorly dimensioned memory use. And had
it shipped with more sensible carving this discussion didn't exist,
and no one would suggest they are in any sort of tactical situation.
Saying there is a problem is logical fallacy, what if your platform
shipped carving of 1 prefix, and rest for filters, and you could do
50M+50M by config toggle. By your logic, this would be a tactical
temporary fix. No, we need to understand what we are doing, what is
the problem, what the solution is, we cannot categorically say this is
a tactical fix.




--
++ytti

On 5/6/22 10:09, Saku Ytti wrote:

> This seems like a strange position. The device has 16MB+16MB jtree
> segments. The first is IP, the second is filters (Broadly).
>
> OP has 16MB of first used.
> OP has <5MB of second used.
>
> What if the platform had originally shipped with a different balance
> between filters and IP, and OP would have never hit this problem?
>
> It is easy to see in many scenarios filter growth is negligible toi 0,
> IP growth is not. OP would technically have 70% FIB growth left, so
> DFZ of about 1.7M, which puts him in the year >2030 (potentially far
> beyond, but at least that).
>
> I view the recarving as fixing poorly dimensioned memory use. And had
> it shipped with more sensible carving this discussion didn't exist,
> and no one would suggest they are in any sort of tactical situation.
> Saying there is a problem is logical fallacy, what if your platform
> shipped carving of 1 prefix, and rest for filters, and you could do
> 50M+50M by config toggle. By your logic, this would be a tactical
> temporary fix. No, we need to understand what we are doing, what is
> the problem, what the solution is, we cannot categorically say this is
> a tactical fix.

My response is to be taken in the context of running a (large) network,
and not the view of a single box.

We have run into issues with platforms that have shipped with FIB's in
favour of IPv4 and less for IPv6 and MPLS labels. Shifted around, you
could give up whatever is left for IPv6 and ACL's to give more to IPv4,
but you then end up losing native IPv6 scalability. And, of course,
whatever other permutation you may think of that leaves you in a
babysitting scenario for the protocol(s) assigned to peasantry.

When considered against the backdrop of a (large) network, one has to
also consider the FIB requirements for the IGP, MPLS label space, e.t.c.
And not to mention that IPv6 will require more FIB space than IPv4, both
for the IGP and BGP.

I'd love to say people's ACL's are simple, but who knows what folk
populate into every RADIUS PPPoE session that they think filters are a
solution for?

So yes, it is important to understand the limitations (or capabilities)
of your specific platform, but also look at the overall picture of your
entire backbone, and get a full understanding of what re-juggling FIB
memory may mean in the short and long term; of course, bearing in mind
that for some operators, short-term could also be 10 years or more.

So all I'm saying is if there is a hack like this to help you delay
moving to newer hardware, go for it. But know your hardware in the
global context of your network, which will require a lot more attention
to avoid getting caught out when you least expect it. I'd be remiss if I
suggested that "implement, move on and forget" is a normal way to treat
this hack.

Mark.

On Fri, 6 May 2022 at 12:17, Mark Tinka <mark@tinka.africa> wrote:

> We have run into issues with platforms that have shipped with FIB's in favour of IPv4 and less for IPv6 and MPLS labels. Shifted around, you could give up whatever is left for IPv6 and ACL's to give more to IPv4, but you then end up losing native IPv6 scalability. And, of course, whatever other permutation you may think of that leaves you in a babysitting scenario for the protocol(s) assigned to peasantry.

You are always here. You always need to understand your scale and how
much resources you have available, what is possible and what is not.
Existence or non-existence of configuration toggle to affect this
doesn't make that situation worse or better.
It is not implied to be a tactical short-term fix, it depends and you
must understand what you are doing to determine your position in any
case.

--
++ytti