Mailing List Archive: Linux and netlink

Linux and netlink

Apr 6, 2004, 4:52 PM

Post #1 of 6 (1151 views)

I tried to play with idea to move my routers configs to the zebra
daemon. Would require some custom stuff to handle vlans etc. but
nothing too hard. Started with ip addresses, but ...

Moved ip addresses of 70 vlan interfaces to the zebra.conf and no go.
Zebra daemon isn't able to start and machine hangs. All I see is a
lot of "recvmsg overrun" messages. After some googling I tried to
play with two ideas.

1) Increasing buffer size with SO_RCVBUF. I got it working with huge
buffers, but I wouldn't rely on it in production system. There is no
way I can quarantee that I don't hit limit again if number of vlan
interfaces is increased.

2) Disabling netlink at all. But in this case I loose features. No
proto zebra flag, no equalize flag etc. Also, it seems that it's
missing some privs raise/lower bits (I will look into it later, but
it's not high priority).

So, there is no go at the moment. I still have to configure my routers
in two places - interfaces/addresses in init scripts and routing in
quagga.

But all this netlink stuff sounds quite dangerous for me and should be
fixed somehow IMHO. According to linux network developers correct
solution would be detect that something was lost and resync:

http://oss.sgi.com/projects/netdev/archive/2001-01/msg00215.html
(whole thread is quite interesting reading)

Ideas?

--
Hasso Tepper
Elion Enterprises Ltd.
WAN administrator

Re: Linux and netlink [ In reply to ]

paul at clubi

Apr 7, 2004, 8:44 AM

Post #2 of 6 (1132 views)

Permalink

On Wed, 7 Apr 2004, Hasso Tepper wrote:

> 1) Increasing buffer size with SO_RCVBUF. I got it working with
> huge buffers, but I wouldn't rely on it in production system. There
> is no way I can quarantee that I don't hit limit again if number of
> vlan interfaces is increased.

> But all this netlink stuff sounds quite dangerous for me and should be
> fixed somehow IMHO. According to linux network developers correct
> solution would be detect that something was lost and resync:
>
> http://oss.sgi.com/projects/netdev/archive/2001-01/msg00215.html
> (whole thread is quite interesting reading)

Yep.

And one of the things I'd like to do soonish is get workqueues nice
enough to merge in (tisnt at moment), then one of the things that can
make use of it is rt_netlink to keep proper track of and also
coalesce netlink requests.

> Ideas?

Lets bump the buffer up for 0.96.5.

regards,
--
Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A
warning: do not ever send email to spam@dishone.st
Fortune:
It is the nature of extreme self-lovers, as they will set an house on fire,
and it were but to roast their eggs.
-- Francis Bacon

Re: Linux and netlink [ In reply to ]

hasso at estpak

Apr 8, 2004, 12:57 AM

Post #3 of 6 (1128 views)

Permalink

Paul Jakma wrote:
> On Wed, 7 Apr 2004, Hasso Tepper wrote:
> > 1) Increasing buffer size with SO_RCVBUF. I got it working with
> > huge buffers, but I wouldn't rely on it in production system.
> > There is no way I can quarantee that I don't hit limit again if
> > number of vlan interfaces is increased.
> >
> > But all this netlink stuff sounds quite dangerous for me and
> > should be fixed somehow IMHO. According to linux network
> > developers correct solution would be detect that something was
> > lost and resync:
> >
> > http://oss.sgi.com/projects/netdev/archive/2001-01/msg00215.html
> > (whole thread is quite interesting reading)
>
> Yep.
>
> And one of the things I'd like to do soonish is get workqueues nice
> enough to merge in (tisnt at moment), then one of the things that
> can make use of it is rt_netlink to keep proper track of and also
> coalesce netlink requests.
>
> > Ideas?
>
> Lets bump the buffer up for 0.96.5.

It's not so easy. From man 7 socket:

SO_RCVBUF
Sets or gets the maximum socket receive buffer in bytes. The
default value is set by the rmem_default sysctl and the maximum
allowed value is set by the rmem_max sysctl.

hasso@test3:/proc/sys$ cat /proc/sys/net/core/rmem_default
65535
hasso@test3:/proc/sys$ cat /proc/sys/net/core/rmem_max
65535

I got my router working with 524280. Btw, it depends what features you
have enabled in router. Enabled multicast and ipv6 make things worse
of course (more routes are added when interface is upped).

It wasn't so bad with older zebra, btw and I have feeling that one
patch (http://marc.theaimsgroup.com/?l=zebra&m=102379207214720&w=2)
made things worse.

--
Hasso Tepper
Elion Enterprises Ltd.
WAN administrator

Re: Linux and netlink [ In reply to ]

paul at clubi

Apr 8, 2004, 2:03 AM

Post #4 of 6 (1131 views)

Permalink

On Thu, 8 Apr 2004, Hasso Tepper wrote:

> It's not so easy. From man 7 socket:
>
> SO_RCVBUF
> Sets or gets the maximum socket receive buffer in bytes. The
> default value is set by the rmem_default sysctl and the maximum
> allowed value is set by the rmem_max sysctl.
>
> hasso@test3:/proc/sys$ cat /proc/sys/net/core/rmem_default
> 65535
> hasso@test3:/proc/sys$ cat /proc/sys/net/core/rmem_max
> 65535

ah dear, yes.

> I got my router working with 524280. Btw, it depends what features
> you have enabled in router. Enabled multicast and ipv6 make things
> worse of course (more routes are added when interface is upped).

yes.

> It wasn't so bad with older zebra, btw and I have feeling that one
> patch (http://marc.theaimsgroup.com/?l=zebra&m=102379207214720&w=2)
> made things worse.

Quite possibly.

Could we then just document the rmem_max sysctl and make the rcvbuf
configurable? (we cant twiddle the sysctl because the kernel doesnt
honour CAP_NET_ADMIN for net related sysctl's) (and grr #2, iirc
zebra reads the routing table before it parses the config file).

regards,
--
Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A
warning: do not ever send email to spam@dishone.st
Fortune:
You plan things that you do not even attempt because of your extreme caution.

Re: Linux and netlink [ In reply to ]

paul at clubi

Apr 8, 2004, 4:24 AM

Post #5 of 6 (1134 views)

Permalink

On Thu, 8 Apr 2004, Hasso Tepper wrote:

> Maybe documenting rmem_max and rmem_default in enough?

just rmem_max, then some way to configure the RCVBUF (which will
probably need to be compile time i think). That'll have to do till we
can do netlink 'properly', or we get initial information via some
other means, eg AF_ROUTE. Thing is, will doing netlink 'properly'
have any effect on this problem?

regards,
--
Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A
warning: do not ever send email to spam@dishone.st
Fortune:
Life would be so much easier if we could just look at the source code.
-- Dave Olson

Re: Linux and netlink [ In reply to ]

hasso at estpak

Apr 8, 2004, 4:25 AM

Post #6 of 6 (1131 views)

Permalink

Paul Jakma wrote:
> On Thu, 8 Apr 2004, Hasso Tepper wrote:
> > It's not so easy. From man 7 socket:
> >
> > SO_RCVBUF
> > Sets or gets the maximum socket receive buffer in bytes. The
> > default value is set by the rmem_default sysctl and the maximum
> > allowed value is set by the rmem_max sysctl.
> >
> > hasso@test3:/proc/sys$ cat /proc/sys/net/core/rmem_default
> > 65535
> > hasso@test3:/proc/sys$ cat /proc/sys/net/core/rmem_max
> > 65535
>
> ah dear, yes.
>
> > I got my router working with 524280. Btw, it depends what
> > features you have enabled in router. Enabled multicast and ipv6
> > make things worse of course (more routes are added when interface
> > is upped).
>
> yes.
>
> > It wasn't so bad with older zebra, btw and I have feeling that
> > one patch
> > (http://marc.theaimsgroup.com/?l=zebra&m=102379207214720&w=2)
> > made things worse.
>
> Quite possibly.
>
> Could we then just document the rmem_max sysctl and make the rcvbuf
> configurable? (we cant twiddle the sysctl because the kernel doesnt
> honour CAP_NET_ADMIN for net related sysctl's) (and grr #2, iirc
> zebra reads the routing table before it parses the config file).

Maybe documenting rmem_max and rmem_default in enough? In case of
incereased rmem_default every created socket will have this receive
buffer size though.

--
Hasso Tepper
Elion Enterprises Ltd.
WAN administrator