Mailing List Archive

On 11/5/2016 5:13 AM, Arnaud Launay wrote:
> Hello,
>
> Since shutdowning one router (on three) to add more SFP ports, we
> see like 200 lines like this:
>
> 2016/11/05 10:50:37 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=564170, pid=0
> 2016/11/05 10:50:37 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=564171, pid=0
> 2016/11/05 10:50:37 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=564172, pid=0
> 2016/11/05 10:50:37 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=564173, pid=0
> 2016/11/05 10:50:37 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=564174, pid=0
>
>
> At boot, and then a few dozen more each day, not necessarily
> grouped. Does anyone has an idea on where to look at ? We found
> very few references on google, mostly on very old releases, but
> we use a recent one...
>
> Linux 4.4.26
> quagga-1.0.20160315-r4.ebuild on Gentoo (it includes the RA,
> ospfd dangling and bgpd logging patches)
>
> Strangely enough, this issue doesn't appear on the other routers.
> Should we try the very last quagga ?
Arnaud,

What are the other routers? Gentoo as well? same Quagga version?

Without testing to see what exactly is going wrong I can only
speculate. Could it be that the Gentoo box doesn't have support for
network namespaces? and that is causing issues with Quagga trying to
send vrf fields in RTM msgs? What do yo see if you do "ip netns list" on
the router where you see this issue? is the output any different on
other routers?

--Jafar



_______________________________________________
Quagga-users mailing list
Quagga-users@lists.quagga.net
https://lists.quagga.net/mailman/listinfo/quagga-users

Hello,

Le Sat, Nov 05, 2016 at 11:50:56PM -0500, Jafar Al-Gharaibeh a écrit:
> > Since shutdowning one router (on three) to add more SFP ports, we
> > see like 200 lines like this:
> > 2016/11/05 10:50:37 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=564170, pid=0
> > 2016/11/05 10:50:37 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=564171, pid=0
> > 2016/11/05 10:50:37 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=564172, pid=0
> > 2016/11/05 10:50:37 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=564173, pid=0
> > 2016/11/05 10:50:37 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=564174, pid=0
> >
> >
> > At boot, and then a few dozen more each day, not necessarily
> > grouped. Does anyone has an idea on where to look at ? We found
> > very few references on google, mostly on very old releases, but
> > we use a recent one...
> >
> > Linux 4.4.26
> > quagga-1.0.20160315-r4.ebuild on Gentoo (it includes the RA,
> > ospfd dangling and bgpd logging patches)
> >
> > Strangely enough, this issue doesn't appear on the other routers.
> > Should we try the very last quagga ?
>
> What are the other routers? Gentoo as well? same Quagga version?

Yes. Same Gentoo, same quagga, same kernel config, same sysctl.

> Without testing to see what exactly is going wrong I can
> only speculate. Could it be that the Gentoo box doesn't have
> support for network namespaces? and that is causing issues
> with Quagga trying to send vrf fields in RTM msgs? What do
> yo see if you do "ip netns list" on the router where you see
> this issue? is the output any different on other routers?

It's the same on all three, output on the first one:

qua1 ~ # ip netns list
qua1 ~ #

I don't think they support network NS. It can be added, though.
Would it help ? And why until now was that never needed ?

Arnaud.

Arnaud,

I looked at the code in zebra/rt_netlink.c:netlink_parse_info()
where we parse netlink packets.
The error log you see takes place at line 372 in the same file.

In summary, you are receiving and error packet from the kernel in
response to a previous RTM_NEWROUTE msg.
To see what is going on you need to enable zebra-kernel debugging by
adding the following to zebra.conf:

debug zebra kernel

right after where you specify the log file would do. Using the sequence
number you can then match the error msg
sequence number with that of the offending RTM_NEWROUTE msg. Please
post back the result and then we
can continue from there.

Regards,
Jafar


On 11/6/2016 2:17 PM, Arnaud Launay wrote:
> Hello,
>
> Le Sat, Nov 05, 2016 at 11:50:56PM -0500, Jafar Al-Gharaibeh a écrit:
>>> Since shutdowning one router (on three) to add more SFP ports, we
>>> see like 200 lines like this:
>>> 2016/11/05 10:50:37 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=564170, pid=0
>>> 2016/11/05 10:50:37 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=564171, pid=0
>>> 2016/11/05 10:50:37 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=564172, pid=0
>>> 2016/11/05 10:50:37 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=564173, pid=0
>>> 2016/11/05 10:50:37 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=564174, pid=0
>>>
>>>
>>> At boot, and then a few dozen more each day, not necessarily
>>> grouped. Does anyone has an idea on where to look at ? We found
>>> very few references on google, mostly on very old releases, but
>>> we use a recent one...
>>>
>>> Linux 4.4.26
>>> quagga-1.0.20160315-r4.ebuild on Gentoo (it includes the RA,
>>> ospfd dangling and bgpd logging patches)
>>>
>>> Strangely enough, this issue doesn't appear on the other routers.
>>> Should we try the very last quagga ?
>> What are the other routers? Gentoo as well? same Quagga version?
> Yes. Same Gentoo, same quagga, same kernel config, same sysctl.
>
>> Without testing to see what exactly is going wrong I can
>> only speculate. Could it be that the Gentoo box doesn't have
>> support for network namespaces? and that is causing issues
>> with Quagga trying to send vrf fields in RTM msgs? What do
>> yo see if you do "ip netns list" on the router where you see
>> this issue? is the output any different on other routers?
> It's the same on all three, output on the first one:
>
> qua1 ~ # ip netns list
> qua1 ~ #
>
> I don't think they support network NS. It can be added, though.
> Would it help ? And why until now was that never needed ?
>
> Arnaud.
>
>
> _______________________________________________
> Quagga-users mailing list
> Quagga-users@lists.quagga.net
> https://lists.quagga.net/mailman/listinfo/quagga-users

Le Mon, Nov 07, 2016 at 09:59:43AM -0600, Jafar Al-Gharaibeh a écrit:

with:

> debug zebra kernel
> debug zebra rib
> right after where you specify the log file would do. Using the sequence
> number you can then match the error msg
> sequence number with that of the offending RTM_NEWROUTE msg. Please
> post back the result and then we
> can continue from there.

So I did that (+rib), the problem appeared again this afternoon,
I have just one more line:

2016/11/09 16:34:40 ZEBRA: netlink_talk: netlink-cmd (vrf 0) type RTM_NEWROUTE(24), seq=2153396
2016/11/09 16:34:40 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=2153396, pid=0

Plus a bunch of rib_ stuff, but which has no seq version number -
and I'm unsure if it's related or not as everything is dated on
the same second...

Arnaud.

_______________________________________________
Quagga-users mailing list
Quagga-users@lists.quagga.net
https://lists.quagga.net/mailman/listinfo/quagga-users

On 11/9/2016 11:58 AM, Arnaud Launay wrote:
> 2016/11/09 16:34:40 ZEBRA: netlink_talk: netlink-cmd (vrf 0) type RTM_NEWROUTE(24), seq=2153396
> 2016/11/09 16:34:40 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=2153396, pid=0

These two lines confirm that we are sending a command to the kernel, and
it is not happy about the supplied argument. The first line is where we
actually talk to the kernel, and the error msg in the second line has a
sequence number that matched the first. The next step is to look into
what is wrong with the command we are sending.

The only place where we send RTM_NEWROUTE commands is in rt_netlink.c
in function: netlink_route_multipath(). From a quick code inspection, I
expect to see more log messages before these. Can you please post
earlier logs?

BTW, do you have only ipv4 addresses? or a mix if ipv4 and ipv6?

Do get little bit more info, can you also replace the last line in the
function above:

return netlink_talk (&req.n, &zvrf->netlink_cmd, zvrf);

with this:

/* Talk to netlink socket. */
{
int rc = netlink_talk (&req.n, &zvrf->netlink_cmd, zvrf);
if (rc = -1) // something went wrong!
zlog_debug ("netlink_route_multipath(): nexthop_num(%d) discard(%d)
", nexthop_num, discard);
return rc;
}

You are welcome to print more information there like the contents of
most rtm/nlmsg fields.

Regards,
Jafar

_______________________________________________
Quagga-users mailing list
Quagga-users@lists.quagga.net
https://lists.quagga.net/mailman/listinfo/quagga-users

Hello,

Le Wed, Nov 09, 2016 at 03:15:40PM -0600, Jafar Al-Gharaibeh a écrit:
> > 2016/11/09 16:34:40 ZEBRA: netlink_talk: netlink-cmd (vrf 0) type RTM_NEWROUTE(24), seq=2153396
> > 2016/11/09 16:34:40 ZEBRA: netlink-cmd (vrf 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=2153396, pid=0

We went offlist not to pollute here, so a little followup:

the kernel was NOT configured with CONFIG_IP_ROUTE_MULTIPATH ,
hence the newroute error. Since then I recompiled the kernel to
include the CONFIG_IP_ROUTE_MULTIPATH:

Networking support > Networking options > TCP/IP networking > IP:
advanced router > IP: equal cost multipath

And the problem just disappeared. Jafar is looking into the
possibility of having a more explicit message from quagga,
indicating to activate the kernel feature, if it's possible to
derive this possibility from the error codes.

Thanks a lot Jafar !

Arnaud.