Mailing List Archive

Jordan Russell wrote:
> Hi,
>
> My machine is functioning as a NAT box. It has two NICs:
> - eth0, connected to the LAN, IP address 192.168.0.1
> - eth1, connected to the Internet, IP address 123.23.23.23
>
> In the OUTPUT chain, I accept packets sent out eth0 with a destination
> address of 192.168.0.x. Any packets sent out other interfaces with a
> destination address of 192.168.0.x are logged and dropped:
>
> -A OUTPUT -d 192.168.0.0/24 -o eth0 -j ACCEPT
> -A OUTPUT -d 192.168.0.0/24 -j LOG '[outdrop] '
> -A OUTPUT -d 192.168.0.0/24 -j DROP
>
> In kernel 2.6.19 and earlier, the LOG & DROP rules never matched
> anything, just as expected.
>
> With 2.6.20.12 and 2.6.21.5, however, they occasionally catch "ICMP
> TYPE=3 CODE=3" packets going out eth1. Example:
>
> [outdrop] IN= OUT=eth1 SRC=123.23.23.23 DST=192.168.0.4 LEN=68 TOS=0x00
> PREC=0xC0 TTL=64 ID=61136 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.4
> DST=123.23.23.23 LEN=40 TOS=0x00 PREC=0x20 TTL=53 ID=16088 PROTO=TCP
> SPT=1229 DPT=44851 WINDOW=0 RES=0x00 ACK RST FIN URGP=0 ]
>

That's a port unreachable message, seemingly coming from the outside, in
response to a RST coming from the inside. That in itself is not very
usual, but it can happen.

This packet going out the wrong interface seems like a pretty serious
bug. I advice you to repost on the netfilter-devel list, there are many
more people there that are really into this. However, this may be a bug
in the network code, not the netfilter code. Or something else completely.


HTH,
M4

On Wed, 2007-06-27 at 00:22 +0200, Martijn Lievaart wrote:
> Jordan Russell wrote:
> > Hi,
> >
> > My machine is functioning as a NAT box. It has two NICs:
> > - eth0, connected to the LAN, IP address 192.168.0.1
> > - eth1, connected to the Internet, IP address 123.23.23.23
> >
> > In the OUTPUT chain, I accept packets sent out eth0 with a destination
> > address of 192.168.0.x. Any packets sent out other interfaces with a
> > destination address of 192.168.0.x are logged and dropped:
> >
> > -A OUTPUT -d 192.168.0.0/24 -o eth0 -j ACCEPT
> > -A OUTPUT -d 192.168.0.0/24 -j LOG '[outdrop] '
> > -A OUTPUT -d 192.168.0.0/24 -j DROP
> >
> > In kernel 2.6.19 and earlier, the LOG & DROP rules never matched
> > anything, just as expected.
> >
> > With 2.6.20.12 and 2.6.21.5, however, they occasionally catch "ICMP
> > TYPE=3 CODE=3" packets going out eth1. Example:
> >
> > [outdrop] IN= OUT=eth1 SRC=123.23.23.23 DST=192.168.0.4 LEN=68 TOS=0x00
> > PREC=0xC0 TTL=64 ID=61136 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.4
> > DST=123.23.23.23 LEN=40 TOS=0x00 PREC=0x20 TTL=53 ID=16088 PROTO=TCP
> > SPT=1229 DPT=44851 WINDOW=0 RES=0x00 ACK RST FIN URGP=0 ]
> >
>
> That's a port unreachable message, seemingly coming from the outside, in
> response to a RST coming from the inside. That in itself is not very
> usual, but it can happen.
>
> This packet going out the wrong interface seems like a pretty serious
> bug. I advice you to repost on the netfilter-devel list, there are many
> more people there that are really into this. However, this may be a bug
> in the network code, not the netfilter code. Or something else completely.
>
>
> HTH,
> M4
>
Doesn't seem like a bug ...

The port unreachable ICMP replay is coming from your machine in response
to a packet coming in eth1 trying to get to 192.168.0.x ...
You have just denied the echo replies by youe DROP rule.

--
--------------------------------------------------
Raymond Leach (spoons@rchq.co.za)
RCHQ Hobbies cc Web: http://www.rchq.co.za/
Tel: +27 82 575 6975 Fax: +27 86 652 2773
"No matter where you go, there you are ..."
--------------------------------------------------

Ray Leach wrote:
> The port unreachable ICMP replay is coming from your machine in response
> to a packet coming in eth1 trying to get to 192.168.0.x ...

I don't think that's correct. Take a look at the square-bracketed part
of the log entry, which is (as I understand it) the packet that caused
the ICMP response to be generated. 192.168.0.4 was trying to send a TCP
packet *out*. The kernel then (for reasons unclear) tried to send an
ICMP message back to 192.168.0.4, but sent it out the wrong interface,
eth1, thus tripping my rule.

--
Jordan Russell

Ray Leach wrote:
> On Wed, 2007-06-27 at 00:22 +0200, Martijn Lievaart wrote:
>
>> Jordan Russell wrote:
>>
>>> Hi,
>>>
>>> My machine is functioning as a NAT box. It has two NICs:
>>> - eth0, connected to the LAN, IP address 192.168.0.1
>>> - eth1, connected to the Internet, IP address 123.23.23.23
>>>
>>>

(snip)

>>> [outdrop] IN= OUT=eth1 SRC=123.23.23.23 DST=192.168.0.4 LEN=68 TOS=0x00
>>> PREC=0xC0 TTL=64 ID=61136 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.4
>>> DST=123.23.23.23 LEN=40 TOS=0x00 PREC=0x20 TTL=53 ID=16088 PROTO=TCP
>>> SPT=1229 DPT=44851 WINDOW=0 RES=0x00 ACK RST FIN URGP=0 ]
>>>
>>>
>> That's a port unreachable message, seemingly coming from the outside, in
>> response to a RST coming from the inside. That in itself is not very
>> usual, but it can happen.
>>
>> This packet going out the wrong interface seems like a pretty serious
>> bug. I advice you to repost on the netfilter-devel list, there are many
>> more people there that are really into this. However, this may be a bug
>> in the network code, not the netfilter code. Or something else completely.
>>
>>
>> HTH,
>> M4
>>
>>
> Doesn't seem like a bug ...
>
> The port unreachable ICMP replay is coming from your machine in response
> to a packet coming in eth1 trying to get to 192.168.0.x ...
> You have just denied the echo replies by youe DROP rule.
>

The source of the packet is 123.23.23.23, it comes from the Internet.
The destination is local. Why do you think this packet is generated locally?

M4

Martijn Lievaart wrote:
> The source of the packet is 123.23.23.23, it comes from the Internet.
> The destination is local. Why do you think this packet is generated locally?

123.23.23.23 is the address of the machine's eth1 (Internet) interface.

Since this only happens when I boot into certain kernel versions (2.6.20
and 2.6.21), it seems unlikely that someone on the Internet is sending
me ICMP packets with a spoofed source address.

--
Jordan Russell

Jordan Russell wrote:
> Martijn Lievaart wrote:
>
>> The source of the packet is 123.23.23.23, it comes from the Internet.
>> The destination is local. Why do you think this packet is generated locally?
>>
>
> 123.23.23.23 is the address of the machine's eth1 (Internet) interface.
>
> Since this only happens when I boot into certain kernel versions (2.6.20
> and 2.6.21), it seems unlikely that someone on the Internet is sending
> me ICMP packets with a spoofed source address.
>

My bad. But still seems like a bug somewhere.

M4

From: Jordan Russell <jr-list-2007@quo.to>
Date: Sat, 16 Jun 2007 11:43:06 -0500

> Hi,
>
> My machine is functioning as a NAT box. It has two NICs:
> - eth0, connected to the LAN, IP address 192.168.0.1
> - eth1, connected to the Internet, IP address 123.23.23.23
>
> In the OUTPUT chain, I accept packets sent out eth0 with a destination
> address of 192.168.0.x. Any packets sent out other interfaces with a
> destination address of 192.168.0.x are logged and dropped:
>
> -A OUTPUT -d 192.168.0.0/24 -o eth0 -j ACCEPT
> -A OUTPUT -d 192.168.0.0/24 -j LOG '[outdrop] '
> -A OUTPUT -d 192.168.0.0/24 -j DROP
>
> In kernel 2.6.19 and earlier, the LOG & DROP rules never matched
> anything, just as expected.
>
> With 2.6.20.12 and 2.6.21.5, however, they occasionally catch "ICMP
> TYPE=3 CODE=3" packets going out eth1. Example:
>
> [outdrop] IN= OUT=eth1 SRC=123.23.23.23 DST=192.168.0.4 LEN=68 TOS=0x00
> PREC=0xC0 TTL=64 ID=61136 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.4
> DST=123.23.23.23 LEN=40 TOS=0x00 PREC=0x20 TTL=53 ID=16088 PROTO=TCP
> SPT=1229 DPT=44851 WINDOW=0 RES=0x00 ACK RST FIN URGP=0 ]
>
> What the logged packets have in common:
>
> 1. The second SRC= is always a machine on the LAN (192.168.0.x).
> (Note: The machines aren't doing anything 'weird', just normal web
> browsing.)
> 2. The second DST= is always the address of the Internet interface
> (123.23.23.23). (This doesn't seem right as no machines on the LAN
> ever connect to that address directly?)

What is real TCP packet on wire ? Is it really from 192.168.0.4 to
123.23.23.23 ? If there is bug in kernel, we cannot believe
output of LOG because NAT usually mangles addresses and ports in ICMP body.

> 3. DPT is always within the range of ports I have allocated for NAT use
> (I use "-p tcp --to-ports 41800-59999" in my MASQUERADE rule).
>
> Any ideas as to what's causing packets destined for the LAN to be sent
> out the Internet interface, and only in 2.6.20+? Does the
> "DST=123.23.23.23" indicate a failure to perform address translation?
> (Could a conntrack entry be expiring prematurely?)

What is your kernel configration and other nat rules ? And please try to
'echo 1 > /proc/sys/net/nf_conntrack_log_invalid' and see whether any
packet is logged or not.

-- Yasuyuki Kozakai

[re-sending without the attachment]

Yasuyuki KOZAKAI wrote:
> What is real TCP packet on wire ? Is it really from 192.168.0.4 to
> 123.23.23.23 ? If there is bug in kernel, we cannot believe
> output of LOG because NAT usually mangles addresses and ports in ICMP body.

For this logged packet:

Jul 4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39698 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]

the real packet on eth1 according to tcpdump seems to be:

14:54:33.931831 IP (tos 0x20, ttl 239, id 39262, offset 0, flags [none],
proto: TCP (6), length: 40) 70.243.226.250.1703 > 123.23.23.23.25000: R,
cksum 0xacb6 (correct), 4070626809:4070626809(0) win 64172

> What is your kernel configration

Full kernel config here: http://76.187.96.234/config-2.6.21.5
All settings under /proc/sys/net/netfilter are default.

> and other nat rules ?

I can see the logged packets with just the following rules:

*nat
:PREROUTING ACCEPT [32:2910]
:POSTROUTING ACCEPT [29:2330]
:OUTPUT ACCEPT [2:152]
-A PREROUTING -i eth1 -p tcp -m tcp --dport 25000 -j DNAT
--to-destination 192.168.0.133
-A PREROUTING -i eth1 -p udp -m udp --dport 25000 -j DNAT
--to-destination 192.168.0.133
-A POSTROUTING -o eth1 -j MASQUERADE
COMMIT
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -j ACCEPT
-A INPUT -i eth1 -m state --state ESTABLISHED -j ACCEPT
-A INPUT -i eth1 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -j DROP
-A FORWARD -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -d 192.168.0.0/255.255.255.0 -o eth0 -j ACCEPT
-A OUTPUT -d 192.168.0.0/255.255.255.0 -j LOG --log-prefix "[packet out
wrong interface] "
-A OUTPUT -o eth1 -j ACCEPT
-A OUTPUT -j DROP
COMMIT

And I've found that the problem occurs with DNAT connections too.

Steps to reproduce:

1. On the server, iptables-restore the above rules.
2. On 192.168.0.133, install BitTorrent client (e.g. Azureus) with TCP
and UDP ports set to 25000.
3. Download
<http://torrent.fedoraproject.org/torrents/Fedora-7-i386.torrent> using
the BitTorrent client.

Within 5-20 minutes, packets begin matching the LOG rule:

Jul 4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39698 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]
Jul 4 14:58:04 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=92 TOS=0x00 PREC=0xC0 TTL=64
ID=32353 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
LEN=64 TOS=0x00 PREC=0x20 TTL=38 ID=47850 DF PROTO=TCP SPT=25000
DPT=25000 WINDOW=63 RES=0x00 ACK URGP=0 ]
Jul 4 15:01:06 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39699 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39688 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]
Jul 4 15:09:18 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39700 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=40226 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]
Jul 4 15:11:10 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=71 TOS=0x00 PREC=0xC0 TTL=64
ID=21127 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
LEN=43 TOS=0x00 PREC=0x20 TTL=17 ID=0 PROTO=TCP SPT=25000 DPT=25000
WINDOW=0 RES=0x00 RST URGP=0 ]

> And please try to
> 'echo 1 > /proc/sys/net/nf_conntrack_log_invalid' and see whether any
> packet is logged or not.

When I do that, some "bad HW ICMP checksum" messages are logged, e.g.:

Jul 4 14:58:39 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
SRC=80.133.170.211 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=234
ID=22079 PROTO=ICMP TYPE=3 CODE=1 [.SRC=123.23.23.23 DST=80.133.170.211
LEN=91 TOS=0x00 PREC=0x00 TTL=114 ID=25502 PROTO=UDP SPT=25000 DPT=21519
LEN=71 ]

but the times don't coincide with the "[packet out wrong interface]" LOG
messages.

--
Jordan Russell

Hi,

Cc: netfilter-devel (He said kernel <= 2.6.19 is OK, but problem on 2.6.20.5)

From: Jordan Russell <jr-list-2007@quo.to>
Date: Wed, 04 Jul 2007 16:49:34 -0500

> Yasuyuki KOZAKAI wrote:
> > What is real TCP packet on wire ? Is it really from 192.168.0.4 to
> > 123.23.23.23 ? If there is bug in kernel, we cannot believe
> > output of LOG because NAT usually mangles addresses and ports in ICMP body.
>
> For this logged packet:
>
> Jul 4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> ID=39698 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=64172 RES=0x00 RST URGP=0 ]
>
> the real packet on eth1 according to tcpdump seems to be:
>
> 14:54:33.931831 IP (tos 0x20, ttl 239, id 39262, offset 0, flags [none],
> proto: TCP (6), length: 40) 70.243.226.250.1703 > 123.23.23.23.25000: R,
> cksum 0xacb6 (correct), 4070626809:4070626809(0) win 64172

Thanks, I want to see dump of real ICMP packet. 'cksum' of ICMP packet is
marked 'correct' ?

> > What is your kernel configration
>
> Full kernel config attached.
> All settings under /proc/sys/net/netfilter are default.

Sorry for ambiguous, I asked .config for building kernel.
Anyway I've got that you are using nf_conntrack.


> > and other nat rules ?
>
> I can see the logged packets with just the following rules:
>
> *nat
> :PREROUTING ACCEPT [32:2910]
> :POSTROUTING ACCEPT [29:2330]
> :OUTPUT ACCEPT [2:152]
> -A PREROUTING -i eth1 -p tcp -m tcp --dport 25000 -j DNAT
> --to-destination 192.168.0.133
> -A PREROUTING -i eth1 -p udp -m udp --dport 25000 -j DNAT
> --to-destination 192.168.0.133
> -A POSTROUTING -o eth1 -j MASQUERADE
> COMMIT
> *filter
> :INPUT DROP [0:0]
> :FORWARD DROP [0:0]
> :OUTPUT DROP [0:0]
> -A INPUT -i lo -j ACCEPT
> -A INPUT -i eth0 -j ACCEPT
> -A INPUT -i eth1 -m state --state ESTABLISHED -j ACCEPT
> -A INPUT -i eth1 -j REJECT --reject-with icmp-port-unreachable
> -A INPUT -j DROP
> -A FORWARD -j ACCEPT
> -A OUTPUT -o lo -j ACCEPT
> -A OUTPUT -d 192.168.0.0/255.255.255.0 -o eth0 -j ACCEPT
> -A OUTPUT -d 192.168.0.0/255.255.255.0 -j LOG --log-prefix "[packet out
> wrong interface] "
> -A OUTPUT -o eth1 -j ACCEPT
> -A OUTPUT -j DROP
> COMMIT
>
> And I've found that the problem occurs with DNAT connections too.
>
> Steps to reproduce:
>
> 1. On the server, iptables-restore the above rules.
> 2. On 192.168.0.133, install BitTorrent client (e.g. Azureus) with TCP
> and UDP ports set to 25000.
> 3. Download
> <http://torrent.fedoraproject.org/torrents/Fedora-7-i386.torrent> using
> the BitTorrent client.
>
> Within 5-20 minutes, packets begin matching the LOG rule:
>
> Jul 4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> ID=39698 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=64172 RES=0x00 RST URGP=0 ]
> Jul 4 14:58:04 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=92 TOS=0x00 PREC=0xC0 TTL=64
> ID=32353 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
> LEN=64 TOS=0x00 PREC=0x20 TTL=38 ID=47850 DF PROTO=TCP SPT=25000
> DPT=25000 WINDOW=63 RES=0x00 ACK URGP=0 ]
> Jul 4 15:01:06 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> ID=39699 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39688 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=64172 RES=0x00 RST URGP=0 ]
> Jul 4 15:09:18 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> ID=39700 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=40226 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=64172 RES=0x00 RST URGP=0 ]
> Jul 4 15:11:10 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=71 TOS=0x00 PREC=0xC0 TTL=64
> ID=21127 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
> LEN=43 TOS=0x00 PREC=0x20 TTL=17 ID=0 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=0 RES=0x00 RST URGP=0 ]
>
> > And please try to
> > 'echo 1 > /proc/sys/net/nf_conntrack_log_invalid' and see whether any
> > packet is logged or not.
>
> When I do that, some "bad HW ICMP checksum" messages are logged, e.g.:

OK, then might be kernel bug in netfilter, bug in IP stack, or broken
hardware, of cause, if the checksum of ICMP packet is really correct.

workaround fix is disable hardware checksum offload if you use it.

developpers, anyone has idea ? I have not deeply checked codes yet,
but I suspect following.

commit 43bc0ca7eadc024e9e5b935fa5e0892df4fec9eb
Author: Al Viro <viro@zeniv.linux.org.uk>
Date: Tue Nov 14 21:43:23 2006 -0800

[NET]: netfilter checksum annotations

Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: David S. Miller <davem@davemloft.net>


> Jul 4 14:58:39 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
> SRC=80.133.170.211 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=234
> ID=22079 PROTO=ICMP TYPE=3 CODE=1 [.SRC=123.23.23.23 DST=80.133.170.211
> LEN=91 TOS=0x00 PREC=0x00 TTL=114 ID=25502 PROTO=UDP SPT=25000 DPT=21519
> LEN=71 ]

This is ICMP error for UDP pakcet. ICMP packets TYPE=3 and CODE=3 were
logged ?


> but the times don't coincide with the "[packet out wrong interface]" LOG
> messages.

This seems other problem, but indeed strange..

-- Yasuyuki Kozakai

From: Yasuyuki KOZAKAI <yasuyuki.kozakai@toshiba.co.jp>
Date: Thu, 05 Jul 2007 10:11:55 +0900 (JST)

> Cc: netfilter-devel (He said kernel <= 2.6.19 is OK, but problem on
> 2.6.20.5)

Sorry, not 2.6.20.5, but 2.6.20.12 and 2.6.21.5.

-- Yasuyuki Kozakai

Yasuyuki KOZAKAI wrote:
>> Jul 4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
>> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
>> ID=39698 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
>> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
>> WINDOW=64172 RES=0x00 RST URGP=0 ]
>>
>> the real packet on eth1 according to tcpdump seems to be:
>>
>> 14:54:33.931831 IP (tos 0x20, ttl 239, id 39262, offset 0, flags [none],
>> proto: TCP (6), length: 40) 70.243.226.250.1703 > 123.23.23.23.25000: R,
>> cksum 0xacb6 (correct), 4070626809:4070626809(0) win 64172
>
> Thanks, I want to see dump of real ICMP packet. 'cksum' of ICMP packet is
> marked 'correct' ?

The logged ICMP packet doesn't seem to show up in the tcpdump output.
When I grep for the ID 39698 there are no matches at 14:54:xx. (??)

BTW: does the LOG output indicate that netfilter translated the source
address of 70.243.226.250 to 192.168.0.133? If so, shouldn't it have
instead translated the *destination* address of 123.23.23.23 (=eth1) to
192.168.0.133? Could this be why the ICMP packet was generated in the
first place?

> workaround fix is disable hardware checksum offload if you use it.

eth1 is running off a 10-year-old 3Com 3C905 adapter, which doesn't
appear to support hardware checksums. dmesg says:

0000:01:0c.0: scatter/gather disabled. h/w checksums disabled

>> Jul 4 14:58:39 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
>> SRC=80.133.170.211 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=234
>> ID=22079 PROTO=ICMP TYPE=3 CODE=1 [.SRC=123.23.23.23 DST=80.133.170.211
>> LEN=91 TOS=0x00 PREC=0x00 TTL=114 ID=25502 PROTO=UDP SPT=25000 DPT=21519
>> LEN=71 ]
>
> This is ICMP error for UDP pakcet. ICMP packets TYPE=3 and CODE=3 were
> logged ?

Yes, there are TYPE=3 CODE=3 too. Here's a log snippet showing the "bad
HW ICMP checksum" messages together with the messages from my LOG rule:

...
Jul 4 14:53:57 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
SRC=80.203.45.12 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=102
ID=23775 PROTO=ICMP TYPE=3 CODE=3 [.SRC=123.23.23.23 DST=80.203.45.12
LEN=91 TOS=0x00 PREC=0x00 TTL=115 ID=41422 PROTO=UDP SPT=25000 DPT=21227
LEN=71 ]
Jul 4 14:53:57 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
SRC=80.203.45.12 DST=123.23.23.23 LEN=98 TOS=0x00 PREC=0x20 TTL=102
ID=23783 PROTO=ICMP TYPE=3 CODE=3 [.SRC=123.23.23.23 DST=80.203.45.12
LEN=70 TOS=0x00 PREC=0x00 TTL=115 ID=41504 PROTO=UDP SPT=25000 DPT=21227
LEN=50 ]
Jul 4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39698 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]
Jul 4 14:58:04 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=92 TOS=0x00 PREC=0xC0 TTL=64
ID=32353 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
LEN=64 TOS=0x00 PREC=0x20 TTL=38 ID=47850 DF PROTO=TCP SPT=25000
DPT=25000 WINDOW=63 RES=0x00 ACK URGP=0 ]
Jul 4 14:58:39 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
SRC=80.133.170.211 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=234
ID=22079 PROTO=ICMP TYPE=3 CODE=1 [.SRC=123.23.23.23 DST=80.133.170.211
LEN=91 TOS=0x00 PREC=0x00 TTL=114 ID=25502 PROTO=UDP SPT=25000 DPT=21519
LEN=71 ]
Jul 4 15:01:06 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39699 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39688 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]
Jul 4 15:09:18 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39700 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=40226 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]
Jul 4 15:11:10 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=71 TOS=0x00 PREC=0xC0 TTL=64
ID=21127 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
LEN=43 TOS=0x00 PREC=0x20 TTL=17 ID=0 PROTO=TCP SPT=25000 DPT=25000
WINDOW=0 RES=0x00 RST URGP=0 ]
...

--
Jordan Russell

From: Jordan Russell <jr-list-2007@quo.to>
Date: Thu, 05 Jul 2007 00:51:05 -0500

> Yasuyuki KOZAKAI wrote:
> >> Jul 4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
> >> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> >> ID=39698 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
> >> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
> >> WINDOW=64172 RES=0x00 RST URGP=0 ]
> >>
> >> the real packet on eth1 according to tcpdump seems to be:
> >>
> >> 14:54:33.931831 IP (tos 0x20, ttl 239, id 39262, offset 0, flags [none],
> >> proto: TCP (6), length: 40) 70.243.226.250.1703 > 123.23.23.23.25000: R,
> >> cksum 0xacb6 (correct), 4070626809:4070626809(0) win 64172
> >
> > Thanks, I want to see dump of real ICMP packet. 'cksum' of ICMP packet is
> > marked 'correct' ?
>
> The logged ICMP packet doesn't seem to show up in the tcpdump output.
> When I grep for the ID 39698 there are no matches at 14:54:xx. (??)

Indeed it seems to be generated by local node and dropped.


> BTW: does the LOG output indicate that netfilter translated the source
> address of 70.243.226.250 to 192.168.0.133? If so, shouldn't it have
> instead translated the *destination* address of 123.23.23.23 (=eth1) to
> 192.168.0.133? Could this be why the ICMP packet was generated in the
> first place?

Hmmm, REJECT in your rule might generate it, but I'm not sure.


> > workaround fix is disable hardware checksum offload if you use it.
>
> eth1 is running off a 10-year-old 3Com 3C905 adapter, which doesn't
> appear to support hardware checksums. dmesg says:
>
> 0000:01:0c.0: scatter/gather disabled. h/w checksums disabled

Then it's purely kernel bug, I think.


> >> Jul 4 14:58:39 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
> >> SRC=80.133.170.211 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=234
> >> ID=22079 PROTO=ICMP TYPE=3 CODE=1 [.SRC=123.23.23.23 DST=80.133.170.211
> >> LEN=91 TOS=0x00 PREC=0x00 TTL=114 ID=25502 PROTO=UDP SPT=25000 DPT=21519
> >> LEN=71 ]
> >
> > This is ICMP error for UDP pakcet. ICMP packets TYPE=3 and CODE=3 were
> > logged ?
>
> Yes, there are TYPE=3 CODE=3 too. Here's a log snippet showing the "bad
> HW ICMP checksum" messages together with the messages from my LOG rule:
>
> ...
> Jul 4 14:53:57 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
> SRC=80.203.45.12 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=102
> ID=23775 PROTO=ICMP TYPE=3 CODE=3 [.SRC=123.23.23.23 DST=80.203.45.12
> LEN=91 TOS=0x00 PREC=0x00 TTL=115 ID=41422 PROTO=UDP SPT=25000 DPT=21227
> LEN=71 ]
> Jul 4 14:53:57 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
> SRC=80.203.45.12 DST=123.23.23.23 LEN=98 TOS=0x00 PREC=0x20 TTL=102
> ID=23783 PROTO=ICMP TYPE=3 CODE=3 [.SRC=123.23.23.23 DST=80.203.45.12
> LEN=70 TOS=0x00 PREC=0x00 TTL=115 ID=41504 PROTO=UDP SPT=25000 DPT=21227
> LEN=50 ]
> Jul 4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> ID=39698 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=64172 RES=0x00 RST URGP=0 ]
> Jul 4 14:58:04 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=92 TOS=0x00 PREC=0xC0 TTL=64
> ID=32353 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
> LEN=64 TOS=0x00 PREC=0x20 TTL=38 ID=47850 DF PROTO=TCP SPT=25000
> DPT=25000 WINDOW=63 RES=0x00 ACK URGP=0 ]
> Jul 4 14:58:39 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
> SRC=80.133.170.211 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=234
> ID=22079 PROTO=ICMP TYPE=3 CODE=1 [.SRC=123.23.23.23 DST=80.133.170.211
> LEN=91 TOS=0x00 PREC=0x00 TTL=114 ID=25502 PROTO=UDP SPT=25000 DPT=21519
> LEN=71 ]
> Jul 4 15:01:06 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> ID=39699 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39688 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=64172 RES=0x00 RST URGP=0 ]
> Jul 4 15:09:18 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> ID=39700 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=40226 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=64172 RES=0x00 RST URGP=0 ]
> Jul 4 15:11:10 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=71 TOS=0x00 PREC=0xC0 TTL=64
> ID=21127 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
> LEN=43 TOS=0x00 PREC=0x20 TTL=17 ID=0 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=0 RES=0x00 RST URGP=0 ]
> ...

The issue of UDP and TCP seems to be different.

-- Yasuyuki Kozakai

Yasuyuki KOZAKAI wrote:
> From: Jordan Russell <jr-list-2007@quo.to>
> Date: Thu, 05 Jul 2007 00:51:05 -0500
>
>
>>Yasuyuki KOZAKAI wrote:
>>
>>>>Jul 4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
>>>>SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
>>>>ID=39698 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
>>>>LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
>>>>WINDOW=64172 RES=0x00 RST URGP=0 ]
>>>>
>>BTW: does the LOG output indicate that netfilter translated the source
>>address of 70.243.226.250 to 192.168.0.133? If so, shouldn't it have
>>instead translated the *destination* address of 123.23.23.23 (=eth1) to
>>192.168.0.133? Could this be why the ICMP packet was generated in the
>>first place?

> Hmmm, REJECT in your rule might generate it, but I'm not sure.


Its pretty certain the REJECT target, it defauls to port unreachable
and the network stack doesn't generate port unreachables for TCP.
Jordan, please post your ruleset.

>>>workaround fix is disable hardware checksum offload if you use it.
>>
>>eth1 is running off a 10-year-old 3Com 3C905 adapter, which doesn't
>>appear to support hardware checksums. dmesg says:
>>
>> 0000:01:0c.0: scatter/gather disabled. h/w checksums disabled


I can't find this message in the kernel tree. Which driver are you
using?

On Thu, 5 Jul 2007, Patrick McHardy wrote:
<CUT>
>>>> workaround fix is disable hardware checksum offload if you use it.
>>>
>>> eth1 is running off a 10-year-old 3Com 3C905 adapter, which doesn't
>>> appear to support hardware checksums. dmesg says:
>>>
>>> 0000:01:0c.0: scatter/gather disabled. h/w checksums disabled
>
>
> I can't find this message in the kernel tree. Which driver are you
> using?

3c59x.c: printk(KERN_INFO "%s: scatter/gather %sabled. h/w checksums %sabled\n",

Best reragds,

Krzysztof Olêdzki

Patrick McHardy wrote:
> Its pretty certain the REJECT target, it defauls to port unreachable
> and the network stack doesn't generate port unreachables for TCP.
> Jordan, please post your ruleset.

Yes, I have a REJECT rule for non-ESTABLISHED incoming packets on eth1.
Please see this post for rules sufficient to reproducing the issue:

http://lists.netfilter.org/pipermail/netfilter/2007-July/069182.html

>>> 0000:01:0c.0: scatter/gather disabled. h/w checksums disabled
>
>
> I can't find this message in the kernel tree. Which driver are you
> using?

It's 3c59x.

--
Jordan Russell

From: Patrick McHardy <kaber@trash.net>
Date: Thu, 05 Jul 2007 14:21:52 +0200

> Yasuyuki KOZAKAI wrote:
> > From: Jordan Russell <jr-list-2007@quo.to>
> > Date: Thu, 05 Jul 2007 00:51:05 -0500
> >
> >
> >>Yasuyuki KOZAKAI wrote:
> >>
> >>>>Jul 4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
> >>>>SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> >>>>ID=39698 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
> >>>>LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
> >>>>WINDOW=64172 RES=0x00 RST URGP=0 ]
> >>>>
> >>BTW: does the LOG output indicate that netfilter translated the source
> >>address of 70.243.226.250 to 192.168.0.133? If so, shouldn't it have
> >>instead translated the *destination* address of 123.23.23.23 (=eth1) to
> >>192.168.0.133? Could this be why the ICMP packet was generated in the
> >>first place?
>
> > Hmmm, REJECT in your rule might generate it, but I'm not sure.
>
>
> Its pretty certain the REJECT target, it defauls to port unreachable
> and the network stack doesn't generate port unreachables for TCP.
> Jordan, please post your ruleset.

He has already posted it. REJECT is in INPUT chain. This means TCP packet
was not mangled. TCP packet might be handled as error, but there seemed to
be no log "nf_ct_tcp: ...".

Jordan, is there the message "nf_conntrack: table full, dropping packet"
in your log ? I've heard that BitTorrent creates huge connections.

> > *nat
> > :PREROUTING ACCEPT [32:2910]
> > :POSTROUTING ACCEPT [29:2330]
> > :OUTPUT ACCEPT [2:152]
> > -A PREROUTING -i eth1 -p tcp -m tcp --dport 25000 -j DNAT
> > --to-destination 192.168.0.133
> > -A PREROUTING -i eth1 -p udp -m udp --dport 25000 -j DNAT
> > --to-destination 192.168.0.133
> > -A POSTROUTING -o eth1 -j MASQUERADE
> > COMMIT
> > *filter
> > :INPUT DROP [0:0]
> > :FORWARD DROP [0:0]
> > :OUTPUT DROP [0:0]
> > -A INPUT -i lo -j ACCEPT
> > -A INPUT -i eth0 -j ACCEPT
> > -A INPUT -i eth1 -m state --state ESTABLISHED -j ACCEPT
> > -A INPUT -i eth1 -j REJECT --reject-with icmp-port-unreachable
> > -A INPUT -j DROP
> > -A FORWARD -j ACCEPT
> > -A OUTPUT -o lo -j ACCEPT
> > -A OUTPUT -d 192.168.0.0/255.255.255.0 -o eth0 -j ACCEPT
> > -A OUTPUT -d 192.168.0.0/255.255.255.0 -j LOG --log-prefix "[packet out
> > wrong interface] "
> > -A OUTPUT -o eth1 -j ACCEPT
> > -A OUTPUT -j DROP
> > COMMIT

-- Yasuyuki Kozakai

Yasuyuki KOZAKAI wrote:
> Jordan, is there the message "nf_conntrack: table full, dropping packet"
> in your log ? I've heard that BitTorrent creates huge connections.

No; there are no messages from the kernel other than "nf_ct_icmp: bad HW
ICMP checksum" and "[packet out wrong interface]" (my LOG rule).

Note that my LOG rule isn't only triggered when using BitTorrent. It's
hit during normal web browsing from the client machines too (i.e.
outbound port 80 through my MASQUERADE rule), just not nearly as often.

--
Jordan Russell

Jordan Russell wrote:
> BTW: does the LOG output indicate that netfilter translated the source
> address of 70.243.226.250 to 192.168.0.133? If so, shouldn't it have
> instead translated the *destination* address of 123.23.23.23 (=eth1) to
> 192.168.0.133? Could this be why the ICMP packet was generated in the
> first place?

To clarify my question:

If tcpdump on eth1 reports:

70.243.226.250.1703 > 123.23.23.23.25000

while my LOG rule reports for the same packet:

... [SRC=192.168.0.133 DST=123.23.23.23 ... SPT=25000 DPT=25000

isn't this saying that netfilter translated the *source* address of the
packet?

Since port 25000 is covered by a DNAT rule:

-A PREROUTING -i eth1 -p tcp -m tcp --dport 25000 -j DNAT
--to-destination 192.168.0.133

shouldn't it have set the *destination* address of the packet to
192.168.0.133, while leaving the source address unchanged?

So: It appears as though netfilter is (in rare cases) translating the
source address of packets when it should be translating the destination
address. Or am I misinterpreting the log output?

--
Jordan Russell

Hi, Jordan,

I'll sort out observations from your report and try to explain my idea in
other mail after finishing some tests.


From: Jordan Russell <jr-list-2007@quo.to>
Date: Fri, 06 Jul 2007 12:42:24 -0500

> Jordan Russell wrote:
> > BTW: does the LOG output indicate that netfilter translated the source
> > address of 70.243.226.250 to 192.168.0.133? If so, shouldn't it have
> > instead translated the *destination* address of 123.23.23.23 (=eth1) to
> > 192.168.0.133? Could this be why the ICMP packet was generated in the
> > first place?
>
> To clarify my question:
>
> If tcpdump on eth1 reports:
>
> 70.243.226.250.1703 > 123.23.23.23.25000
>
> while my LOG rule reports for the same packet:
>
> ... [SRC=192.168.0.133 DST=123.23.23.23 ... SPT=25000 DPT=25000
>
> isn't this saying that netfilter translated the *source* address of the
> packet?

Yes, but from this log, we can say only that source address of TCP packet
was mangled, OR the source address of IP header in ICMP body was mangled.

Inserting LOG rule just before REJECT rule will clarify that.


> Since port 25000 is covered by a DNAT rule:
>
> -A PREROUTING -i eth1 -p tcp -m tcp --dport 25000 -j DNAT
> --to-destination 192.168.0.133
>
> shouldn't it have set the *destination* address of the packet to
> 192.168.0.133, while leaving the source address unchanged?

If TCP packet is valid, yes. If not, no address is mangled.

> So: It appears as though netfilter is (in rare cases) translating the
> source address of packets when it should be translating the destination
> address. Or am I misinterpreting the log output?

The current NAT code does that, but I suspect that it is due to bug.

-- Yasuyuki Kozakai

From: Yasuyuki KOZAKAI <yasuyuki.kozakai@toshiba.co.jp>
Date: Sat, 07 Jul 2007 15:27:06 +0900 (JST)

> Hi, Jordan,
>
> I'll sort out observations from your report and try to explain my idea in
> other mail after finishing some tests.

OK, I try to explain what the current netfilter does in Jordan's situation.
I also attach the patch to fix this problem. Jordan, can you try it ?

Patrick, please consider to apply it. Even if following my idea is not
correct, clearly it fixes a bug. I'm wondering why I missed '!'
when I copied & pasted ip_conntrack_proto_icmp.c...


- This situation occurred on 2.6.20.12 and 2.6.21.5 at least.
- your rules are following

*nat
:PREROUTING ACCEPT [32:2910]
:POSTROUTING ACCEPT [29:2330]
:OUTPUT ACCEPT [2:152]
-A PREROUTING -i eth1 -p tcp -m tcp --dport 25000 -j DNAT
--to-destination 192.168.0.133
-A PREROUTING -i eth1 -p udp -m udp --dport 25000 -j DNAT
--to-destination 192.168.0.133
-A POSTROUTING -o eth1 -j MASQUERADE
COMMIT
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -j ACCEPT
-A INPUT -i eth1 -m state --state ESTABLISHED -j ACCEPT
-A INPUT -i eth1 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -j DROP
-A FORWARD -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -d 192.168.0.0/255.255.255.0 -o eth0 -j ACCEPT
-A OUTPUT -d 192.168.0.0/255.255.255.0 -j LOG --log-prefix "[packet out wrong interface] "
-A OUTPUT -o eth1 -j ACCEPT
-A OUTPUT -j DROP
COMMIT

- ICMP error is generated by this router but destination address of it is strange.

Jul 4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39698 PROTO=ICMP TYPE=3 CODE=3 [.SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]

- original TCP reset is following

14:54:33.931831 IP (tos 0x20, ttl 239, id 39262, offset 0, flags [none],
proto: TCP (6), length: 40) 70.243.226.250.1703 > 123.23.23.23.25000: R,
cksum 0xacb6 (correct), 4070626809:4070626809(0) win 64172

- no error message is printed by nf_ct_tcp, no 'table is full' error.


My idea is following.

- This TCP reset is not initial packet of TCP connection. If it is initial packet,
no address in ICMP packet should be mangled. Jordan, if you see
/proc/net/netfilter/nf_conntrack, you will find the entry matched the TCP packet.

- TCP packet was marked as error packet. Because '--state ESTABLISHED'
didn't match the packet. No conntrack entry wasn't assigned to the
packet. Usually, error log by nf_conntrack_tcp should be generated in
such case, but no message is generated in some cases. I don't know why
this TCP reset was handled as error.

- Then ICMP error generated at this router was not assigned to any conntrack entry.

- nf_conntarck_icmp.c assigns the ICMP error to the conntrack which matches
the TCP reset. But IP_CT_IS_REPLY didn't set to *ctinfo. This is bug.

h = nf_conntrack_find_get(&innertuple, NULL);
if (!h) {
/* Locally generated ICMPs will match inverted if they
haven't been SNAT'ed yet */
/* FIXME: NAT code has to handle half-done double NAT --RR */
if (hooknum == NF_IP_LOCAL_OUT)
h = nf_conntrack_find_get(&origtuple, NULL);

if (!h) {
DEBUGP("icmp_error_message: no match\n");
return -NF_ACCEPT;
}

/* Reverse direction from that found */
if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
*ctinfo += IP_CT_IS_REPLY;
} else {
if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
*ctinfo += IP_CT_IS_REPLY;


- As a result, nf_nat_packet mistook the address in ICMP packet which
should be mangled.

On Sat, 7 Jul 2007, Yasuyuki KOZAKAI wrote:

> OK, I try to explain what the current netfilter does in Jordan's situation.
> I also attach the patch to fix this problem. Jordan, can you try it ?
>
> Patrick, please consider to apply it. Even if following my idea is not
> correct, clearly it fixes a bug. I'm wondering why I missed '!'
> when I copied & pasted ip_conntrack_proto_icmp.c...


I'm currently travelling, I'll look into it tomorrow. Just one
question below ..

> My idea is following.
>
> - This TCP reset is not initial packet of TCP connection. If it is initial packet,
> no address in ICMP packet should be mangled. Jordan, if you see
> /proc/net/netfilter/nf_conntrack, you will find the entry matched the TCP packet.
>
> - TCP packet was marked as error packet. Because '--state ESTABLISHED'
> didn't match the packet. No conntrack entry wasn't assigned to the
> packet. Usually, error log by nf_conntrack_tcp should be generated in
> such case, but no message is generated in some cases. I don't know why
> this TCP reset was handled as error.
>
> - Then ICMP error generated at this router was not assigned to any conntrack entry.
>
> - nf_conntarck_icmp.c assigns the ICMP error to the conntrack which matches
> the TCP reset. But IP_CT_IS_REPLY didn't set to *ctinfo. This is bug.
>
> h = nf_conntrack_find_get(&innertuple, NULL);
> if (!h) {
> /* Locally generated ICMPs will match inverted if they
> haven't been SNAT'ed yet */
> /* FIXME: NAT code has to handle half-done double NAT --RR */
> if (hooknum == NF_IP_LOCAL_OUT)
> h = nf_conntrack_find_get(&origtuple, NULL);


The local ICMP tracking is basically useless nowadays since we always
manually attach the conntrack reference from the original packet
(exactly because of the half-done double NAT FIXME quoted above).
But this is an interesting case, the connection tracking code itself
thought the RST was invalid, but ICMP tracking will still associate
the ICMP containing the RST with the original connection. I'm wondering
whether it should really do that. In case it shouldn't, just removing
all locally generated ICMP special-casing should also fix the bug,
right?

From: Patrick McHardy <kaber@trash.net>
Date: Sat, 7 Jul 2007 17:34:49 +0200 (CEST)

> > My idea is following.
> >
> > - This TCP reset is not initial packet of TCP connection. If it is initial packet,
> > no address in ICMP packet should be mangled. Jordan, if you see
> > /proc/net/netfilter/nf_conntrack, you will find the entry matched the TCP packet.
> >
> > - TCP packet was marked as error packet. Because '--state ESTABLISHED'
> > didn't match the packet. No conntrack entry wasn't assigned to the
> > packet. Usually, error log by nf_conntrack_tcp should be generated in
> > such case, but no message is generated in some cases. I don't know why
> > this TCP reset was handled as error.
> >
> > - Then ICMP error generated at this router was not assigned to any conntrack entry.
> >
> > - nf_conntarck_icmp.c assigns the ICMP error to the conntrack which matches
> > the TCP reset. But IP_CT_IS_REPLY didn't set to *ctinfo. This is bug.
> >
> > h = nf_conntrack_find_get(&innertuple, NULL);
> > if (!h) {
> > /* Locally generated ICMPs will match inverted if they
> > haven't been SNAT'ed yet */
> > /* FIXME: NAT code has to handle half-done double NAT --RR */
> > if (hooknum == NF_IP_LOCAL_OUT)
> > h = nf_conntrack_find_get(&origtuple, NULL);
>
>
> The local ICMP tracking is basically useless nowadays since we always
> manually attach the conntrack reference from the original packet
> (exactly because of the half-done double NAT FIXME quoted above).
> But this is an interesting case, the connection tracking code itself
> thought the RST was invalid, but ICMP tracking will still associate
> the ICMP containing the RST with the original connection. I'm wondering
> whether it should really do that. In case it shouldn't, just removing
> all locally generated ICMP special-casing should also fix the bug,
> right?

At first I thought so. But I didn't come up with any bad situation caused
by returning ICMP error to such invalid packets.

-- Yasuyuki Kozakai

From: Yasuyuki KOZAKAI <yasuyuki.kozakai@toshiba.co.jp>
Date: Sun, 08 Jul 2007 02:28:54 +0900 (JST)

> From: Patrick McHardy <kaber@trash.net>
> Date: Sat, 7 Jul 2007 17:34:49 +0200 (CEST)
> >
> > The local ICMP tracking is basically useless nowadays since we always
> > manually attach the conntrack reference from the original packet
> > (exactly because of the half-done double NAT FIXME quoted above).
> > But this is an interesting case, the connection tracking code itself
> > thought the RST was invalid, but ICMP tracking will still associate
> > the ICMP containing the RST with the original connection. I'm wondering
> > whether it should really do that. In case it shouldn't, just removing
> > all locally generated ICMP special-casing should also fix the bug,
> > right?
>
> At first I thought so. But I didn't come up with any bad situation caused
> by returning ICMP error to such invalid packets.

Can kernel correctly return ICMP error without conntrack in this case ?
If so, I agree. (maybe yes, I'll check it after waking up).

-- Yasuyuki Kozakai

Yasuyuki KOZAKAI wrote:
> OK, I try to explain what the current netfilter does in Jordan's situation.
> I also attach the patch to fix this problem. Jordan, can you try it ?

Excellent. The patch does appear to fix the problem. I repeated the
BitTorrent test, and no packets matched my LOG rule.

I'll keep watching the logs and report back if it happens again.

Thanks for looking into this!

--
Jordan Russell