Mailing List Archive

1 2  View All
Re: kernel routing socket handling problems [ In reply to ]
>
> I have just committed the changes, so I think this leaves only the
> open issue of data structure consistency in the face of unplumbing
> (and likely cardbus/etc. interface removal on other systems too).
> I'm glad my rtm message length check exposed this problem so we could
> fix it cleanly!
>

yup. though, if we are on a cleaning spree, a more appropriate structure
would be

struct
{

union
{
struct rt_msghdr rtm;
struct if_msghdr ifm;
struct ifa_msghdr ifa;
#ifdef RTM_IFANNOUNCE
struct if_announcemsghdr ifan;
#endif /* RTM_IFANNOUNCE */
} bufhdr;
struct sockaddr_storage addr[RTAX_MAX];
} buf;

but that's another project.

--Sowmini
Re: kernel routing socket handling problems [ In reply to ]
On Thu, 8 Jan 2004, Greg Troxel wrote:

> humans deal with, I agree that names are primary. But here we are
> talking about zebra/kernel communication on the routing socket, and
> the conventions there are not necessarily the same, or determined
> by the human/zebra interface.

Indeed.

> Well, I'm still afraid that there is underyling complexity being
> hidden by this, and I don't want to paper over it without fully
> understanding it.

Ok, yes.

> The whole notion that Solaris has about having a
> new ifindex for a 'reused' name to comply with SNMP makes a lot of
> sense, and has implications about whether we have a 'new interface'
> v.s. a new version of the same interface.

We need to know:

- when can an index be invalidated?

I'm guessing unplumb/delete is the most obvious case, in which case
we should set index to -1 - the interface is gone. Only message we
could get about this interface would be a NEWLINK type message, which
gives us the index again (as well as name).

It is indeed important to get this right, so that if we do
cross-check name lookups with resulting retrieved index, and vice
versa that we can tell when things have gone wrong. However, if the
OS is sane, this shouldnt be /too/ difficult.

> I think on all systems if the index matches there is truly a match.

It should be so, yes.

Presuming we are always notified of events which could change an
index / cause an index to a candidate for change (so we can
invalidate it).

> The issue is that a name can appear with a new index, and perhaps
> that should cause the old interface structure to be deleted.

If the platform is such that we should have known the index was
susceptable to change, then this should never occur. The name lookup
should result in an interface with index set to -1, and we reuse the
interface.

However, is there any case where an interface can still exist in the
system (eg by name) yet not have a stable index? On Linux, TTBOMK,
invalid index implies the link is gone - ie we received an
RTM_DELLINK.

Why would we ever receive a message from the kernel about a
non-existent interface? The only possibility, at least on linux,
(TTBOMK, gilad?) is RTM_NEWLINK - which gives us index and name.
(which is what we key from). For netlink we dont even care about the
index, we always trust the kernel and update ifindex with what the
kernel has given us.

What is the situation on kernel_socket systems?

> Really kernels which do this should issue a routing socket message
> that the name/ifindex association has been ended - one of the
> design goals of the routing socket is that a program can listen and
> track the state of the kernel's interface/routing activities, and
> that's just what zebra does. So if there is trouble I think it
> will turn out to be because zebra's database and the kernel's are
> getting out of sync.

Yes.

> Sowmini: you might ask the Solaris people responsible for
> plumb/unplumb the mechanism a program like zebra is supposed to use
> to become aware that an interface has become unplumbed, so that the
> program may delete its state that associates the (old) ifindex and
> the name.
>
> Perhaps, since there is apparently no routing socket message, we
> should check on creating an interface if there is another interface
> structure with the same name. It should have state down (as an
> assert), and probably should be deleted when this happens, since
> I'd argue this situation is a latent detection of a previous
> unplumb.

I dont know the intricacies of BSD route sockets, but why not key off
the name as much as possible? If you have the name, use that to
lookup the interface. If the interface index doesnt match the index
in the message just received then: if message is one which is
'allowed' to update the index, update the zebra interface index,
otherwise panic().

this presumes:

- kernel socket messages for interface events usually do include the
interface name (its in the sockaddr_dl data right? always?)

- we know exactly which messages fall in the "index is allowed to
change" class.

regards,
--
Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A
warning: do not ever send email to spam@dishone.st
Fortune:
A mathematician is a device for turning coffee into theorems.
-- P. Erdos
Re: kernel routing socket handling problems [ In reply to ]
> > I think on all systems if the index matches there is truly a match.
>
> It should be so, yes.

Is this really true for linux? If I have 2 intfs, eth0 (index 1)
and eth1 (index 2), and then eth0 "goes away" (whatever that means
on linux) and I rename eth1 to be eth0, unless zebra has been
notified and has carefully kept up with this dance, it is
now going to find that the new eth0 has index 1.

> What is the situation on kernel_socket systems?

The traditional kernel_socket system is BSD has no equivalent
of unplumb. However, I think RTM_IFANNOUNCE was added at some point
in BSD to deal with this.

> I dont know the intricacies of BSD route sockets, but why not key off
> the name as much as possible? If you have the name, use that to

No problem in theory. Just takes a little longer because we
are talking strcmp() versus integer comparison.

> - kernel socket messages for interface events usually do include the
> interface name (its in the sockaddr_dl data right? always?)

the sockaddr_dl thing is a solaris peculiarity. However the RTM_IFANNOUNCE
message announces both the name and the index of the arriving/departing
interface.

and no, solaris doesn't currently support the equivalent of RTM_IFANNOUNCE.

--Sowmini
Re: kernel routing socket handling problems [ In reply to ]
You won't get messages about nonexistent interfaces. The problem is
that you can have state in zebra for an interface (by index) which no
longer exists (unplumbed).

this presumes:

- kernel socket messages for interface events usually do include the
interface name (its in the sockaddr_dl data right? always?)

But this is not true. Because the ifindex is considered the primary
handle on the routing socket, some things, like if_msghdr, sometimes
send up/down notifications without a sockaddr_dl. (Solaris might
always include a sockaddr_dl, but the interface definition, such as it
is, doesn't require it.)

- we know exactly which messages fall in the "index is allowed to
change" class.

The whole 'change' concept arguably postdates the interface design, so
this is scary. But really when the kernel breaks the ifindex/name
binding, this needs to be communicated to the routing socket listeners
so the user-space state can stay consistent. Anything else I can
think of is arguably a bit of a bandaid to work around this.

Example bandaid

when an interface with a previously unknown ifindex appears,
search by name to find an existing ifindex/name binding. If that
ifindex is different (must be, since we said it was new), then
execute the (not yet written perhaps) if_disappeared procedure to
clear all state and set ifindex to -1, matching the state of
'interface was in config file but kernel hasn't told us about it'.
Then proceed.

Really if_disappeared should be called on unplumb.
Re: kernel routing socket handling problems [ In reply to ]
On Thu, 8 Jan 2004 sowmini.varadhan@sun.com wrote:

> Is this really true for linux?

yep.

> If I have 2 intfs, eth0 (index 1) and eth1 (index 2), and then eth0
> "goes away" (whatever that means on linux)

That'd be RTM_DELLINK.

> and I rename eth1 to be eth0, unless zebra has been notified and
> has carefully kept up with this dance, it is now going to find that
> the new eth0 has index 1.

Hmm... renaming. Good point. Will have to test this.

> The traditional kernel_socket system is BSD has no equivalent of
> unplumb. However, I think RTM_IFANNOUNCE was added at some point in
> BSD to deal with this.

Ok.

> No problem in theory. Just takes a little longer because we are
> talking strcmp() versus integer comparison.

But we're doing a list walk anyway in either case. I'm going to guess
the list walk is the majority cost - its the memory fetches and
attendant cache misses which will eat the cycles, not the cost of
comparing 4ish words against one word. (yeah, you have to fetch 4
words, not one - but those 4 words of interface name will most likely
be contiguous in memory and prefetched by modern CPUs anyway.
hopefully). Ie i'd like to see a benchmark to be convinced strcmp()
would be a significant extra cost.

Further, if we always base our lookups on index numbers, then we have
to deal with 'list walk misses' for index numbers and go walk the
list again for the name anyway.

Unless of course we know when index is unlikely to be valid and
lookup by name instead (and update the index).

> the sockaddr_dl thing is a solaris peculiarity.

Sure? Its embedded in Stevens' examples on BSD AF_ROUTE and sysctl().
It appears to be a general BSD interface description structure.
(unless you refer to a peculiarity of sockaddr_dl on solaris, rather
than it being peculiar to solaris).

> However the RTM_IFANNOUNCE message announces both the name and the
> index of the arriving/departing interface.

Ah, good, so that shouldnt be too difficult to handle?

> and no, solaris doesn't currently support the equivalent of
> RTM_IFANNOUNCE.

That would be a problem then.

> --Sowmini

regards,
--
Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A
warning: do not ever send email to spam@dishone.st
Fortune:
Have you reconsidered a computer career?
Re: kernel routing socket handling problems [ In reply to ]
Interfaces do not come and go frequently. I don't think efficiency is
a big deal here. I'm worried about correctness, particularly as the
world gets more complicated.
Re: kernel routing socket handling problems [ In reply to ]
> > No problem in theory. Just takes a little longer because we are
> > talking strcmp() versus integer comparison.
>
> But we're doing a list walk anyway in either case. I'm going to guess
> the list walk is the majority cost - its the memory fetches and

agreed. and in practice, this is not a frequently executed operation,
anyway, so this cost is not phenomenal. I was just mentioning the
reason people prefer (in general) to compare indices than to do
a strcmp. Also the ifindex lends itself nicely to hashing etc.

> > the sockaddr_dl thing is a solaris peculiarity.
>
> Sure? Its embedded in Stevens' examples on BSD AF_ROUTE and sysctl().

I meant the feature about sending the sockaddr_dl with an RTM_IFINFO.
Evidently netbsd doesn't do this (though I was under the impression
that it does. shrug.).

--Sowmini
Re: kernel routing socket handling problems [ In reply to ]
On Thu, 8 Jan 2004, Greg Troxel wrote:

> You won't get messages about nonexistent interfaces.

yes :)

> The problem is that you can have state in zebra for an interface
> (by index) which no longer exists (unplumbed).

right.

So you need to either be able to realise this as it occurs (because
the kernel announced the unplumbing) and invalidate the index or
be able to realise it later (because you get a 'plumbing this
interface' message) and update the index.

In the netlink case, we appear to get RTM_NEWLINK messages whenever
interface comes up or changes. (at least this is what rt_netlink.c
expects) and we update the index from the kernel's message each time.

> But this is not true. Because the ifindex is considered the
> primary handle on the routing socket, some things, like if_msghdr,
> sometimes send up/down notifications without a sockaddr_dl.
> (Solaris might always include a sockaddr_dl, but the interface
> definition, such as it is, doesn't require it.)

But do you get a notification?

> Example bandaid
>
> when an interface with a previously unknown ifindex appears,
> search by name to find an existing ifindex/name binding. If that
> ifindex is different (must be, since we said it was new), then
> execute the (not yet written perhaps) if_disappeared procedure to
> clear all state and set ifindex to -1, matching the state of
> 'interface was in config file but kernel hasn't told us about it'.
> Then proceed.
>
> Really if_disappeared should be called on unplumb.

regards,
--
Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A
warning: do not ever send email to spam@dishone.st
Fortune:
A man is known by the company he organizes.
-- Ambrose Bierce
Re: kernel routing socket handling problems [ In reply to ]
On Thu, 8 Jan 2004 sowmini.varadhan@Sun.COM wrote:

> a strcmp. Also the ifindex lends itself nicely to hashing etc.

as does the interface name surely? :)

> --Sowmini

regards,
--
Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A
warning: do not ever send email to spam@dishone.st
Fortune:
Suggest you just sit there and wait till life gets easier.
Re: kernel routing socket handling problems [ In reply to ]
On Thu, Jan 08, 2004 at 10:46:40AM -0500, Greg Troxel wrote:
<cut>
> I have just committed the changes, so I think this leaves only the
> open issue of data structure consistency in the face of unplumbing
> (and likely cardbus/etc. interface removal on other systems too).
<cut>

I suppose this accounts for ppp, tunnel etc. interfaces on Linux too. So
probably it have something to do with:

http://mail.gnu.org/archive/html/bug-zebra/2002-02/msg00013.html
http://lists.quagga.net/pipermail/quagga-dev/2003-December/000576.html

And while speaking about kernel routing socket please see this message:

http://lists.shmoo.com/pipermail/hostap/2004-January/005110.html

Best regards,

--
Theodor Milkov Head Network Administrator
Davidov Net Phone: +359 (2) 730158
Re: kernel routing socket handling problems [ In reply to ]
Thanks for the pointers.

http://mail.gnu.org/archive/html/bug-zebra/2002-02/msg00013.html

This is a bug about semantic processing of up/down events, rather than
a problem in handling the routing socket. The kernel_socket.c code
just calls an if_down routine. (Not saying it shouldn't be fixed,
just that it is a separate issue.) up/down is much milder than
removal.

Is this still a problem with quagga?

http://lists.quagga.net/pipermail/quagga-dev/2003-December/000576.html

This is about what to do when an interface goes away. The patch looks
sane at first glance (I didn't study it carefully), but I think here
we should also set the ifindex to -1 so that when the named interface
is resurrected it will match this structure even if the ifindex is
different. I suppose the invariant of 'interface structure for
interface which does not exist in the kernel must have ifindex == -1,
and structures for which the interface does exist in the kernel must
have the same ifindex as the kernel' should be documented in the code.

http://lists.shmoo.com/pipermail/hostap/2004-January/005110.html

That's about netlink, not 'the routing socket', but certainly if Linux
emits new netlink messages that aren't of interest then zebra should
gracefully ignore them. That said, there is merit to logging unknown
messages, since then we learn something when they pop up and can then
choose whether we wish to ignore them or not.

A patch to ignore the wireless status netlink messages should be very
easy to generate (complete with a comment about why we don't care
about these messages, following HACKING from CVS :-).
Re: kernel routing socket handling problems [ In reply to ]
Hello!

Greg Troxel's patch fixes the crash of zebra daemon when initializing
interfaces on FreeBSD systems. Big thanks to Greg!

Regards,
Boris

1 2  View All