Mailing List Archive

xen, fc4, bridging, iptables and conntrack problem
Hi,

I'm testing out Xen on FC4. I'm using bridging for networking, as
well as iptables to firewall, configured with the standard Fedora
'system-config-security-level' tool. However I have really strange
problem with conntrack not seeming to catch outbound connections.
This prevents outbound connections working from dom0. Connections
from domU's however /do/ work.

The problem appears to boil down to the following:

Chain INPUT (policy ACCEPT 210K packets, 18M bytes)
pkts bytes target prot opt in out source destination
111K 8778K RH-Firewall-1-INPUT all -- xen-br+ any anywhere anywhere
0 0 RH-Firewall-1-INPUT all -- vif+ any anywhere anywhere
1 73 RH-Firewall-1-INPUT all -- eth0 any anywhere anywhere

Chain FORWARD (policy ACCEPT 2812K packets, 311M bytes)
pkts bytes target prot opt in out source destination
<empty>

Chain RH-Firewall-1-INPUT (3 references)
pkts bytes target prot opt in out source destination
33 2485 ACCEPT all -- lo any anywhere anywhere
253 16338 ACCEPT icmp -- any any anywhere anywhere icmp any
0 0 ACCEPT ipv6-crypt-- any any anywhere anywhere
0 0 ACCEPT ipv6-auth-- any any anywhere anywhere
68483 6004K ACCEPT all -- any any anywhere anywhere state RELATED,ESTABLISHED
<snip remaining standard RH-Firewall rules to allow in certain ports>


The FORWARD chain is empty and policy ACCEPT, which maybe explains
why domU's work.

The INPUT side of stuff though seems to not work because the
RELATED,ESTABLISHED conntrack rule doesn't match. And this would
appear to be because the original /outgoing/ packets are never caught
by connection track and entered into its state.

If I tcpdump xen-br0, I can see packets leave, and I can even see the
remote SYN|ACK come in, which is very strange (and not inline with my
only hypothesis so far, a conntrack problem):

# tcpdump -i xen-br0 port 25
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on xen-br0, link-type EN10MB (Ethernet), capture size 96 bytes
18:48:54.138909 IP domain0.38261 > remote.smtp:
S 710403207:710403207(0) win 5840 <mss 1460,sackOK,timestamp 181127121 0,nop,wscale 2>
18:48:54.271062 IP remote.smtp > domain0.38261:
S 746149051:746149051(0) ack 710403208 win 5792 <mss 1460,sackOK,timestamp 1332954470 181127121,nop,wscale 0>
18:48:57.138797 IP domain0.38261 > remote.smtp:
S 710403207:710403207(0) win 5840 <mss 1460,sackOK,timestamp 181127421 0,nop,wscale 2>
18:48:57.270302 IP remote.smtp > domain0.38261:
S 749148214:749148214(0) ack 710403208 win 5792 <mss 1460,sackOK,timestamp 1332954770 181127421,nop,wscale 0>

Has anyone seen this problem before?

Is it specific to bridging (but it affects local packets though), to
Xen somehow, to FC4?

regards,
--
Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A
Fortune:
That's always the way when you discover something new; everyone thinks
you're crazy.
-- Evelyn E. Smith

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
re: xen, fc4, bridging, iptables and conntrack problem [ In reply to ]
Hi Paul, I have Fedora Core 4 and I am having exactly the same problem
as you. I will provide some detail below.
Out of two installs this happened both times.
You are right, this is a conntrack failure but I don't know if it's on
the iptables or xen side, although everything works fine until xend
starts-creates the bridge and bingo! conntrack stops working. Bit of a
showstopper really.

Here is some of my info:-
Problem:-
New install of fedora core 4 with xen kernel running. Iptables rules that under
the regular kernel work fine stop working when in bridge mode under xen in dom0.
This stops the conntrack system working on the xen host machine and i can't then
log in via ssh.
It seems that the conntrack system is failing to match already accepted
connections. The initial packet seems to get accepted by the INPUT rule, then
the reply packet slips past the ESTABLISHED,RELATED rule and gets logged then
dropped by the default policy.

This is the packet that gets logged:-
xen kernel: OUTPUT IN= OUT=xen-br0 PHYSOUT=eth0 SRC=192.168.0.45
DST=192.168.0.39 LEN=48 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=22
DPT=1152 WINDOW=5840 RES=0x00 ACK SYN URGP=0

This happens whether i start a guest os up or not.
This was reproduced on another machine at work with a Fedora Core 4 install.

xen host machine address:192.168.0.45
ssh client address:192.168.0.39

rules:-
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
state RELATED,ESTABLISHED
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0
LOG flags 0 level 4 prefix `FORWARD '

Chain INPUT (policy DROP 54 packets, 7483 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
304 21532 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
state RELATED,ESTABLISHED
1 48 ACCEPT tcp -- * * 192.168.0.39 192.168.0.45
tcp spts:1024:65535 dpt:22 state NEW
54 7483 LOG all -- * * 0.0.0.0/0 0.0.0.0/0
LOG flags 0 level 4 prefix `INPUT '

Chain OUTPUT (policy DROP 8 packets, 384 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * lo 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
state RELATED,ESTABLISHED
0 0 ACCEPT udp -- * * 192.168.0.45 192.168.0.19
udp spts:1024:65535 dpt:53
0 0 ACCEPT tcp -- * * 192.168.0.45 0.0.0.0/0
tcp spts:1024:65535 dpt:80
0 0 ACCEPT icmp -- * * 192.168.0.45 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0
LOG flags 0 level 4 prefix `OUTPUT '

interfaces:-
eth0 Link encap:Ethernet HWaddr 00:08:43:EE:50:CE
inet addr:192.168.0.45 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:24684 errors:0 dropped:0 overruns:0 frame:0
TX packets:4406 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1992235 (1.8 MiB) TX bytes:631910 (617.0 KiB)
Base address:0xecc0 Memory:ff8e0000-ff900000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

xen-br0 Link encap:Ethernet HWaddr 00:08:43:EE:50:CE
inet addr:192.168.0.45 Bcast:192.168.0.255 Mask:255.255.255.255
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:24682 errors:0 dropped:0 overruns:0 frame:0
TX packets:4451 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1538495 (1.4 MiB) TX bytes:618890 (604.3 KiB)

routes:-
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 xen-br0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 xen-br0
0.0.0.0 192.168.0.250 0.0.0.0 UG 0 0 0 xen-br0

operating system:-
Fedora Core 4

kernel version:-
2.6.11-1.1369_FC4xen0

iptables version:-
iptables v1.3.0

xen version:-
xen-2-20050522

network driver:-
e1000

Had everything working under fedora core 3 before with iptables and 5 virtual
machines conntracking beautifully

There's nothing obvious, all the iptables modules are loaded and work
fine until the bridge goes up. No error messages associated with the
bridge creation either.
Will try to dig further.
Hope somebody has some ideas as I am running out of them!


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
re: xen, fc4, bridging, iptables and conntrack problem [ In reply to ]
Hi,

On Sat, 25 Jun 2005, Jon Howse wrote:

> Hi Paul,

> I have Fedora Core 4 and I am having exactly the same problem as
> you.

Aha, so it's not just me. Time to raise a bug with fedora.

> I will provide some detail below. Out of two installs this happened
> both times. You are right, this is a conntrack failure

Seems to be.

> but I don't know if it's on the iptables or xen side, although
> everything works fine until xend starts-creates the bridge and
> bingo! conntrack stops working.

Yep.

> Bit of a showstopper really.

Definitely.

> machine and i can't then log in via ssh. It seems that the
> conntrack system is failing to match already accepted connections.

See above. For me, all dom0 initiated connections fail to appear in
conntrack state (but strangely the remote replies still get seen by
tcpdump on xen-br0). domU's work fine though, as FORWARD is
unrestricted.

> The initial packet seems to get accepted by the INPUT rule, then
> the reply packet slips past the ESTABLISHED,RELATED rule and gets
> logged then dropped by the default policy.

Ah.

> This happens whether i start a guest os up or not. This was
> reproduced on another machine at work with a Fedora Core 4 install.

> There's nothing obvious, all the iptables modules are loaded and
> work fine until the bridge goes up. No error messages associated
> with the bridge creation either. Will try to dig further.

I created a bug for Fedora. See
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=161792 and
please add your comments to it.

regards,
--
Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A
Fortune:
Ask not what's inside your head, but what your head's inside of.
-- J.J. Gibson

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
re: xen, fc4, bridging, iptables and conntrack problem [ In reply to ]
Hi Paul, I too posted to bugzilla at redhat since we last spoke. Should
I delete my bug and add to yours?

I also posted to the xen bugzilla system...bug number 82.

Will check back soon...

Jonny



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Re: xen, fc4, bridging, iptables and conntrack problem [ In reply to ]
Paul Jakma wrote:


> On Sat, 25 Jun 2005, Jon Howse wrote:
>
>> Hi Paul,
>
>> I have Fedora Core 4 and I am having exactly the same problem as you.
>
> Aha, so it's not just me. Time to raise a bug with fedora.

I can confirm the problem here.

[snip]
>> machine and i can't then log in via ssh. It seems that the conntrack
>> system is failing to match already accepted connections.
>
> See above. For me, all dom0 initiated connections fail to appear in
> conntrack state (but strangely the remote replies still get seen by
> tcpdump on xen-br0). domU's work fine though, as FORWARD is unrestricted.
>
>> The initial packet seems to get accepted by the INPUT rule, then the
>> reply packet slips past the ESTABLISHED,RELATED rule and gets logged then
>> dropped by the default policy.

[snip]
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=161792 and please add
> your comments to it.


The snapshot for -unstable used for the latest FC4 package is quite old: *
Tue Apr 26 2005 Rik van Riel <...> 2-20050424
- upgrade to last night's snapshot

So perhaps this is already fixed in xen-unstable. Or it was just an artefact
of code changes, similar to the problem that xm restore does not work
correctly in that snapshot.

Rik said he would upgrade to a new snapshot for rawhide rather soon. Not
sure when that will be, though.

Can anyone not using FC4 confirm problems with iptables and conntrack in the
latest -unstable?

Best Regards,
Michael Paesold


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
re: xen, fc4, bridging, iptables and conntrack problem [ In reply to ]
Hi Paul, I subscribe to the netfilter mailing list and I saw this in a
posting by someone:-

<POSTING>
Has anyone else seen this? A working bridge running Fedora Core 3 fails
after an upgrade to the one of the latest Fedora kernels. What I've
found is:

kernel-2.6.11-1.14_FC3 and earlier work fine

kernel-2.6.11-1.27_FC3 and kernel-2.6.11-1.35_FC3 fail!

The problem is with netfilter not with bridging. With iptables
shutdown the bridge works fine but even with very simple iptables
rules any network connections to/from or through the bridge fail.

(Fedora kernels are patched so it's difficult to say which standard
kernel version they correspond to but it looks like some new kernel
patch has caused or will cause problems with bridge-nf).
<POSTING>

I know it's fedora core 3...but it may be connected.
It's not great detail but indicates a trend that seems to fit the
evidence that we have. No network connectivity...ESTABLISHED or RELATED
reply packets not getting out/in?

These kernels are relatively recent for 3 as I have a fc3 workstation at
work and those revisions ring a bell. I'll check back with you after I
find out at work. The last fc3 working xen environment I had working was
before those revisions...hmmmm.

Any help?

Jonny



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users