Mailing List Archive

Chuck Ebbert wrote:
> I was testing a 2.6.20 kernel and got a soft
> lockup on shutdown:
>
> _raw_write_lock+0x5a
> nf_ct_iterate_cleanup+0x3e
> kill_l3proto+0x0
> nf_conntrack_l3proto_unregister+0x85
> nf_conntrack_l3proto_ipv4_fini+0x1e
> sys_delete_module+0x18a
> remove_vma+0x45
> do_munmap+0x196
> syscall_call+0x7

Happened three times in a row, trying to remove
netfilter modules.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Chuck Ebbert wrote:
> Chuck Ebbert wrote:
>
>>I was testing a 2.6.20 kernel and got a soft
>>lockup on shutdown:
>>
>>_raw_write_lock+0x5a
>>nf_ct_iterate_cleanup+0x3e
>>kill_l3proto+0x0
>>nf_conntrack_l3proto_unregister+0x85
>>nf_conntrack_l3proto_ipv4_fini+0x1e
>>sys_delete_module+0x18a
>>remove_vma+0x45
>>do_munmap+0x196
>>syscall_call+0x7
>
>
> Happened three times in a row, trying to remove
> netfilter modules.

Does it happen when you unload only nf_conntrack_ipv4, or do
you need to unload other modules previously? Does it really
lock up?

Please also send your config.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Patrick McHardy wrote:
> Chuck Ebbert wrote:
>> Chuck Ebbert wrote:
>>
>>> I was testing a 2.6.20 kernel and got a soft
>>> lockup on shutdown:
>>>
>>> _raw_write_lock+0x5a
>>> nf_ct_iterate_cleanup+0x3e
>>> kill_l3proto+0x0
>>> nf_conntrack_l3proto_unregister+0x85
>>> nf_conntrack_l3proto_ipv4_fini+0x1e
>>> sys_delete_module+0x18a
>>> remove_vma+0x45
>>> do_munmap+0x196
>>> syscall_call+0x7
>>
>> Happened three times in a row, trying to remove
>> netfilter modules.
>
> Does it happen when you unload only nf_conntrack_ipv4, or do
> you need to unload other modules previously? Does it really
> lock up?
>

It is the standard Fedora script "iptables" and it recursively
unloads all referring modules, then the base ones: ip_tables and
ip_conntrack. I can't tell which module it stopped on because
that computer is at home today.

And yes, it really locks up until it's caught by the softlockup
watchdog. Has only happened on a uniprocessor i686 system so far,
but happens every time on shutdown.

> Please also send your config.

Standard Fedora 6 i686 config, updated for 2.6.20, not released yet.
I'll attach it.

Chuck Ebbert wrote:
> It is the standard Fedora script "iptables" and it recursively
> unloads all referring modules, then the base ones: ip_tables and
> ip_conntrack. I can't tell which module it stopped on because
> that computer is at home today.
>
> And yes, it really locks up until it's caught by the softlockup
> watchdog. Has only happened on a uniprocessor i686 system so far,
> but happens every time on shutdown.


Can you try this patch please?

On Sun, 2007-02-25 at 18:31 +0100, Patrick McHardy wrote:

> [NETFILTER]: conntrack: fix {nf,ip}_ct_iterate_cleanup endless loops
>
> {nf,ip}_ct_iterate_cleanup iterate over the unconfirmed list for cleaning
> up conntrack entries, which is wrong for multiple reasons:
>
> - unconfirmed entries can not be killed manually, which means we might
> iterate forever without making forward progress.
>
> This can happen in combination with the conntrack event cache, which
> holds a reference to the conntrack entry, which is only released when
> the packet makes it all the way through the stack or a different
> packet is handled.
>
> - taking references to an unconfirmed entry and using it outside the
> locked section doesn't work, the list entries are not refcounted and
> another CPU might already be waiting to destroy the entry
>
> Split ip_ct_iterate_cleanup in ip_ct_iterate, which iterates over both
> confirmed and unconfirmed entries, but doesn't attempt to kill them,
> and ip_ct_cleanup, which makes sure no unconfirmed entries exist by
> calling synchronize_net() prior to walking the conntrack hash.

What about this case:

1. Conntrack entry is created and placed on the unconfirmed list
2. The event cache bumps the refcount of the conntrack entry
3. module removal of ip_conntrack unregisters all hooks
4. packet is dropped by an iptables rule
5. packet is freed but we still have a refcount on the conntrack entry

Now there's no way to get that refcount to decrease as that only happens
when the event cache receives another packet or the current packet makes
it through the stack as you wrote above. And neither of this will happen
since we unregistered the hooks providing the packets and dropped the
packet.

I ran into this case a while ago during stresstesting and rewrote the
event cache to not increase the refcount, but this has the drawback that
events caused by dropped packets won't be reported. This may not be a
good thing...

Old patch can be found here:
http://performance.netfilter.org/patches/nf_conntrack_ecache-fix

--
/Martin

Martin Josefsson wrote:
> What about this case:
>
> 1. Conntrack entry is created and placed on the unconfirmed list
> 2. The event cache bumps the refcount of the conntrack entry
> 3. module removal of ip_conntrack unregisters all hooks
> 4. packet is dropped by an iptables rule
> 5. packet is freed but we still have a refcount on the conntrack entry
>
> Now there's no way to get that refcount to decrease as that only happens
> when the event cache receives another packet or the current packet makes
> it through the stack as you wrote above. And neither of this will happen
> since we unregistered the hooks providing the packets and dropped the
> packet.

The event cache is flushed on conntrack module unload after the hooks
have been unregistered, which should release all references.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Patrick McHardy wrote:
> Chuck Ebbert wrote:
>
> Can you try this patch please?
> ------------------------------------------------------------------------
>
> [NETFILTER]: conntrack: fix {nf,ip}_ct_iterate_cleanup endless loops
>

> include/linux/netfilter_ipv4/ip_conntrack.h | 5 ++--
> include/net/netfilter/nf_conntrack.h | 5 +++-
> include/net/netfilter/nf_nat_rule.h | 3 ++
> net/ipv4/netfilter/ip_conntrack_core.c | 30 ++++++++++++++++++------
> net/ipv4/netfilter/ip_conntrack_standalone.c | 5 ++--
> net/ipv4/netfilter/ip_nat_core.c | 5 ++--
> net/ipv4/netfilter/ipt_MASQUERADE.c | 4 ++-
> net/ipv4/netfilter/nf_nat_core.c | 7 ++----
> net/netfilter/nf_conntrack_core.c | 33 ++++++++++++++++++++------
> net/netfilter/nf_conntrack_proto.c | 4 ++-
> 10 files changed, 69 insertions(+), 32 deletions(-)
>

Changes to nf_nat_core.c, ip_nat_core.c and nf_conntrack_proto.c
do not apply, and nf_conntrack_core.c changes are already there.

This is vanilla 2.6.20; looks like there have been a bunch
of changes since then. I tried adding all of the RCU patches
but even they won't apply.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Chuck Ebbert wrote:
> Changes to nf_nat_core.c, ip_nat_core.c and nf_conntrack_proto.c
> do not apply, and nf_conntrack_core.c changes are already there.
>
> This is vanilla 2.6.20; looks like there have been a bunch
> of changes since then. I tried adding all of the RCU patches
> but even they won't apply.

It seems to conflict with the net/ whitespace cleanup we had since
then. This patch is against current stable-2.6.20, it applies
cleanly to 2.6.20 as well.

Patrick McHardy wrote:
> This patch is against current stable-2.6.20, it applies
> cleanly to 2.6.20 as well.
>

Everything works OK, but I get:

BUG: warning: (!list_empty(&unconfirmed)) at
net/netfilter/nf_conntrack_core.c:1068/nf_ct_cleanup()

<> nf_ct_cleanup+0x66/0x122 [nf_conntrack]
<> kill_l4proto+0x0 [nf_conntrack]
<> nf_conntrack_l4proto_unregister+0x7d/0x82 [nf_conntrack]
<> nf_conntrack_l3proto_ipv4_fini+0x3c/0x46 [nf_conntrack_ipv4]
<> sys_delete_module+0x18a/0x1b1
...

netconsole doesn't seem to work as a module for me, so hand
copied from screen.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Chuck Ebbert wrote:
> Patrick McHardy wrote:
>
>>This patch is against current stable-2.6.20, it applies
>>cleanly to 2.6.20 as well.
>
>
> Everything works OK, but I get:
>
> BUG: warning: (!list_empty(&unconfirmed)) at
> net/netfilter/nf_conntrack_core.c:1068/nf_ct_cleanup()
>
> <> nf_ct_cleanup+0x66/0x122 [nf_conntrack]
> <> kill_l4proto+0x0 [nf_conntrack]
> <> nf_conntrack_l4proto_unregister+0x7d/0x82 [nf_conntrack]
> <> nf_conntrack_l3proto_ipv4_fini+0x3c/0x46 [nf_conntrack_ipv4]
> <> sys_delete_module+0x18a/0x1b1


Thanks, the previous approach doesn't seem to work properly without
unpleasant event cache hacks. This patch takes a simpler approach
and keeps the unconfirmed list iteration, but makes sure to make
forward progress.

Patrick McHardy wrote:
> Thanks, the previous approach doesn't seem to work properly without
> unpleasant event cache hacks. This patch takes a simpler approach
> and keeps the unconfirmed list iteration, but makes sure to make
> forward progress.
>
>
>
> ------------------------------------------------------------------------
>
> [NETFILTER]: conntrack: fix {nf,ip}_ct_iterate_cleanup endless loops
>
> Fix {nf,ip}_ct_iterate_cleanup unconfirmed list handling:
>

Works great: survived three reboots without lockup or warning messages.
And it's a nice simple patch, too...

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/