Mailing List Archive

Hi Andrew!

andrew mathes wrote:
> i'm confused what the "mac" paramater on vif's in the startup
> configuration file for a domain does.
>
> i have this in my configuration file:
> vif = [ 'mac=aa:00:00:1a:%d:11' % (vmid), 'bridge=xen-br0']
> vif = [ 'mac=aa:00:00:1a:%d:12' % (vmid), 'bridge=xen-br1']
>
> and while my REAL question is why this doesn't bind the second vif to
> bridge1,

The first time I used xen, I thought all this would happen
automagically. Butwhen I figured out (like you) that this is not true,
so I started to configure everything outside of xen's startup scripts in
advance of creating domains:

1. Create all the bridge devices that are mentioned in xenU
configuration files; since xen-br0 is already there, in your case only
xen-br1 has to be created:

brctl addbr xen-br1
brctl stp xen-br1 off
brctl setfd xen-br1 0
ip link set xen-br1 up

I disable spanning tree since normally it causes more problems than it
solves and it unnecessarily delays the initial forwarding. When the
domUs boot, their eth0, eth1 etc. are added to the bridges as specified
by your xen config file and get the MAC addresses as specified there
with the mac=... parameter.

2. If you want to do networking between xen0 and xenU, configure the
bridge device with an IP address, i. e.

ifconfig xen-br1 10.1.2.3 netmask 255.255.255.0 broadcast 10.2.3.255

the bridge xen-br1 has the MAC address fe:ff:ff:ff:ff:ff. This is ok and
poses no problem, because this is a valid (local scope individual) MAC
address.

3. If you want to do external networking, you have to add one of dom0's
physical networking devices to the bridge, i. e.

brctl addif xen-br1 eth1

The bridge will get the MAC address of the physical NIC eth1. And as
above, dom0 communicates via xen-br1, not via eth1, so leave eth1
unconfigured.

You can also use VLAN devices instead of physical NICs here, they are
equivalent to physical NICs in this context (at least as long
/proc/sys/net/bridge/bridge-nf-filter-vlan-tagged is 1).

4. Since VLAN devs carry their own MAC address (by default the same as
their physical base device), you could add a VLAN to a bridge to change
the bridges MAC address, i.e. if you would like to give xen-br1 the MAC
address 00:00:01:02:03, you could use

ip link set eth0 up
vconfig add eth0 815
ip link set addr eth0.815 00:00:00:01:02:03
brctl addif xen-br1 eth0.815

Of course you could also change the MAC of a physical NIC instead of
using a VLAN device but I prefer using VLANs.


> my first concern is that what's happening is the second line's
> MAC is being given to eth0 on my VM, and eth1 is getting something
> fairly random looking.

My xenU eth's MAC addresses are exactly what I specify in the
configuration files. I have no idea where this random MAC stuff that you
experience comes from.

> is this mac the mac of the bridge being binded to, the mac of the new
> vif in the domain?

The MAC of xenU's eth0, eth1 etc. The MAC of vifX.Y in dom0 always shows
fe:ff:ff:ff:ff.

> And if it's the mac of the if in the domain, why
> would these two lines not assign those two macs to eth0/1 respectively?

That's what they do at least for me. Although I don't use %d etc., I
don't think that this makes any difference (I use scripts to generate
all xen configuration files, virtual disks, startup configs inside those
virtual disks, the bridges etc.).


> I'm sure someone out there has bound multiple ethernet cards to
> different interfaces ... seems like a very common server setup, any
> enlightenment from xen guru's out there?

I'm not a guru, but maybe the above can help a bit.


> mostly curious about the "vif" paramater in your domain startup
> configuration as well as if you used the default "vif-bridge" script.

Sorry, I'm doing everything outside the xen autoconfiguration magic, so
no vif-bridge script.

Mark.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sun, May 01, 2005 at 09:45:22PM -0700, andrew mathes wrote:
> i have this in my configuration file:
> vif = [ 'mac=aa:00:00:1a:%d:11' % (vmid), 'bridge=xen-br0']
> vif = [ 'mac=aa:00:00:1a:%d:12' % (vmid), 'bridge=xen-br1']

the configuration files are actually python scripts, so the
2nd line there is replacing the information from the first. This is
what you actually need:

vif = [ 'mac=aa:00:00:1a:%d:11, bridge=xen-br0' % (vmid),
'mac=aa:00:00:1a:%d:12, bridge=xen-br1' % (vmid)]


J

--
Jody Belka
knew (at) pimb (dot) org

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

everyone has been very helpful so far, thank you all, hopefully i will
become knowledgeable enough to help out others.

However ... if I use the below, it DOES set up two vif's with the correct
addresses, however, they can't reach/be reached by the internet. (Only
eth3 (of dom0)->xen-br1 is external available, so it seems to me they
aren't binding to br1, but to br0.

My reason for thinking this is 1) the no external availability and 2) in
the dom0 ifconfig the vif.1 has the same MAC as br0, leading me to believe
it's bound to that MAC.

Mark wrote:
"The MAC of xenU's eth0, eth1 etc. The MAC of vifX.Y in dom0 always shows
fe:ff:ff:ff:ff."

is this true? It seems vifX.1 (bound to br1) should have MAC of br1,
vifX.0 bound to br0 should have MAC of br0. Maybe this is where I'm going
wrong.

Everything worked fine when i was using eth0/br0 as my external interface
... so i feel like the bridge binding is my only problem ...

ugh, sorry I'm having so many problems ...

On Mon, 2 May 2005, Jody Belka wrote:

> On Sun, May 01, 2005 at 09:45:22PM -0700, andrew mathes wrote:
>> i have this in my configuration file:
>> vif = [ 'mac=aa:00:00:1a:%d:11' % (vmid), 'bridge=xen-br0']
>> vif = [ 'mac=aa:00:00:1a:%d:12' % (vmid), 'bridge=xen-br1']
>
> the configuration files are actually python scripts, so the
> 2nd line there is replacing the information from the first. This is
> what you actually need:
>
> vif = [ 'mac=aa:00:00:1a:%d:11, bridge=xen-br0' % (vmid),
> 'mac=aa:00:00:1a:%d:12, bridge=xen-br1' % (vmid)]
>
>
> J
>
> --
> Jody Belka
> knew (at) pimb (dot) org
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

ok, so, ... brctl show shows this:

xen-br0 8000.001143fd7101 no eth0
vif10.0
vif11.0
vif12.0
vif9.0
xen-br1 8000.001143fd7102 no eth3
vif10.1
vif11.1
vif12.1
vif9.1

so obviously the interfaces are bound to the correct bridges, i just can't
reach any of the interfaces through the second bridge (eth0 isn't plugged
in ...)

I know eth0 worked when it was plugged in ... so maybe there's a different
issue here? i know eth3 works, i ssh in to it and it can get out ...
don't know where to go from here ...

any suggestions?

On Mon, 2 May 2005, andrew mathes wrote:

> everyone has been very helpful so far, thank you all, hopefully i will become
> knowledgeable enough to help out others.
>
> However ... if I use the below, it DOES set up two vif's with the correct
> addresses, however, they can't reach/be reached by the internet. (Only eth3
> (of dom0)->xen-br1 is external available, so it seems to me they aren't
> binding to br1, but to br0.
>
> My reason for thinking this is 1) the no external availability and 2) in the
> dom0 ifconfig the vif.1 has the same MAC as br0, leading me to believe it's
> bound to that MAC.
>
> Mark wrote:
> "The MAC of xenU's eth0, eth1 etc. The MAC of vifX.Y in dom0 always shows
> fe:ff:ff:ff:ff."
>
> is this true? It seems vifX.1 (bound to br1) should have MAC of br1, vifX.0
> bound to br0 should have MAC of br0. Maybe this is where I'm going wrong.
>
> Everything worked fine when i was using eth0/br0 as my external interface ...
> so i feel like the bridge binding is my only problem ...
>
> ugh, sorry I'm having so many problems ...
>
> On Mon, 2 May 2005, Jody Belka wrote:
>
>> On Sun, May 01, 2005 at 09:45:22PM -0700, andrew mathes wrote:
>> > i have this in my configuration file:
>> > vif = [ 'mac=aa:00:00:1a:%d:11' % (vmid), 'bridge=xen-br0']
>> > vif = [ 'mac=aa:00:00:1a:%d:12' % (vmid), 'bridge=xen-br1']
>>
>> the configuration files are actually python scripts, so the
>> 2nd line there is replacing the information from the first. This is
>> what you actually need:
>>
>> vif = [ 'mac=aa:00:00:1a:%d:11, bridge=xen-br0' % (vmid),
>> 'mac=aa:00:00:1a:%d:12, bridge=xen-br1' % (vmid)]
>>
>>
>> J
>>
>> --
>> Jody Belka
>> knew (at) pimb (dot) org
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xensource.com
>> http://lists.xensource.com/xen-users
>>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Andrew!

andrew mathes wrote:
> ok, so, ... brctl show shows this:
>
> xen-br0 8000.001143fd7101 no eth0
> vif10.0
> vif11.0
> vif12.0
> vif9.0
> xen-br1 8000.001143fd7102 no eth3
> vif10.1
> vif11.1
> vif12.1
> vif9.1
>
> so obviously the interfaces are bound to the correct bridges, i just
> can't reach any of the interfaces through the second bridge (eth0 isn't
> plugged in ...)

This looks good. So since it worked for me, check if

- all interfaces are up including (the unconfigured) eth3 in domain0
- eth1 of domain10 (domain11, domain12) and the external host connected
to eth3 of domain0 are configured with the same IP subnet
- make shure that spoof protection makes no problems, so better disable
it by setting /proc/sys/net/conf/*/rp_filter to 0
- the bridge is in forwarding state? With dmesg you should see something
like

xen-br1: port 1(vif10.1) entering learning state
xen-br1: topology change detected, propagating
xen-br1: port 1(vif10.1) entering forwarding state

and so on for the other vifs

For debugging you should use something simpler than ssh. I always use
"ping" together with "tcpdump". I. e. in domain0 use "tcpdump -eni eth3
icmp or arp") to check how far the address resolution and the ping
packets go and if answers are coming in. Try this on all hosts.
Sometimes a broadcast ping, i. e. "ping -b 10.11.12.255" if your network
is 10.11.12.0/24, or "ping -I eth0 -b 255.255.255.255" if eth0 is on the
subet you wish to check, is more robust in the presence of misconfigured
IP addresses.

Mark.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

hm, tcpdump from external -> eth3 of dom0 looks fine

tcpdump from exteran -> eth1 of domU = no reply

tcpdump from eth3 -> ext = fine

tcpdump from eth3->domU ... the ping says it's ok, but tcpdump shows
nothing

likewise domU->eth3 ... says it's pinging, but tcpdump shows nothing

so is what's happening here is both sides pinging the bridge, but nothing
is going through? i have this in dmesg:

device vif1.0 entered promiscuous mode
xen-br0: port 1(vif1.0) entering learning state
xen-br0: topology change detected, propagating
xen-br0: port 1(vif1.0) entering forwarding state
device vif1.1 entered promiscuous mode
xen-br1: port 2(vif1.1) entering learning state
xen-br1: topology change detected, propagating
xen-br1: port 2(vif1.1) entering forwarding state

so that looks good ...

i'm not supposed to make the domU gateway the same as the domO nic or
anything am i?

perhaps that dump information will clarify the problem ...

i hope =)

On Mon, 2 May 2005, Mark Doll wrote:

> Hi Andrew!
>
> andrew mathes wrote:
>> ok, so, ... brctl show shows this:
>>
>> xen-br0 8000.001143fd7101 no eth0
>> vif10.0
>> vif11.0
>> vif12.0
>> vif9.0
>> xen-br1 8000.001143fd7102 no eth3
>> vif10.1
>> vif11.1
>> vif12.1
>> vif9.1
>>
>> so obviously the interfaces are bound to the correct bridges, i just
>> can't reach any of the interfaces through the second bridge (eth0 isn't
>> plugged in ...)
>
> This looks good. So since it worked for me, check if
>
> - all interfaces are up including (the unconfigured) eth3 in domain0
> - eth1 of domain10 (domain11, domain12) and the external host connected
> to eth3 of domain0 are configured with the same IP subnet
> - make shure that spoof protection makes no problems, so better disable
> it by setting /proc/sys/net/conf/*/rp_filter to 0
> - the bridge is in forwarding state? With dmesg you should see something
> like
>
> xen-br1: port 1(vif10.1) entering learning state
> xen-br1: topology change detected, propagating
> xen-br1: port 1(vif10.1) entering forwarding state
>
> and so on for the other vifs
>
> For debugging you should use something simpler than ssh. I always use
> "ping" together with "tcpdump". I. e. in domain0 use "tcpdump -eni eth3
> icmp or arp") to check how far the address resolution and the ping
> packets go and if answers are coming in. Try this on all hosts.
> Sometimes a broadcast ping, i. e. "ping -b 10.11.12.255" if your network
> is 10.11.12.0/24, or "ping -I eth0 -b 255.255.255.255" if eth0 is on the
> subet you wish to check, is more robust in the presence of misconfigured
> IP addresses.
>
> Mark.
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

andrew mathes wrote:

> hm, tcpdump from external -> eth3 of dom0 looks fine
>
> tcpdump from exteran -> eth1 of domU = no reply
>
> tcpdump from eth3 -> ext = fine
>
> tcpdump from eth3->domU ... the ping says it's ok, but tcpdump shows
> nothing
>
> likewise domU->eth3 ... says it's pinging, but tcpdump shows nothing
>
> so is what's happening here is both sides pinging the bridge, but
> nothing is going through? i have this in dmesg:
>
> device vif1.0 entered promiscuous mode
> xen-br0: port 1(vif1.0) entering learning state
> xen-br0: topology change detected, propagating
> xen-br0: port 1(vif1.0) entering forwarding state
> device vif1.1 entered promiscuous mode
> xen-br1: port 2(vif1.1) entering learning state
> xen-br1: topology change detected, propagating
> xen-br1: port 2(vif1.1) entering forwarding state
>
> so that looks good ...
>
> i'm not supposed to make the domU gateway the same as the domO nic or
> anything am i?
>

Sorry to come so late to this thread - but did you
share your routes (ip route show, or netstat -rn) output,
also your arp -a, ifconfig -a?

If you have two real interfaces on the same subnet,
you may need to set arp_filter on.

thanks,
Nivedita


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Andrew!

andrew mathes wrote:
> hm, tcpdump from external -> eth3 of dom0 looks fine
>
> tcpdump from exteran -> eth1 of domU = no reply
>
> tcpdump from eth3 -> ext = fine
>
> tcpdump from eth3->domU ... the ping says it's ok, but tcpdump shows
> nothing
>
> likewise domU->eth3 ... says it's pinging, but tcpdump shows nothing
>

Run tcpump on xen-br1 instead of eth3 and you should see the ICMP packets.


> so is what's happening here is both sides pinging the bridge, but
> nothing is going through? i have this in dmesg:
>
> device vif1.0 entered promiscuous mode
> xen-br0: port 1(vif1.0) entering learning state
> xen-br0: topology change detected, propagating
> xen-br0: port 1(vif1.0) entering forwarding state
> device vif1.1 entered promiscuous mode
> xen-br1: port 2(vif1.1) entering learning state
> xen-br1: topology change detected, propagating
> xen-br1: port 2(vif1.1) entering forwarding state
>
> so that looks good ...

ACK.


> i'm not supposed to make the domU gateway the same as the domO nic or
> anything am i?

dom0 isn't a gateway (forwarding is disabled). Bridging runs entirely on
layer 2 (Ethernet), not layer 3 (IP).

>
> perhaps that dump information will clarify the problem ...
>
> i hope =)

Not really. I have ne idea where the mistake in your config is. :-(

Mark.

all i can think of now is it's a routing issue- here is my route -n on
dom0:

10.255.0.0 10.254.1.254 255.255.255.0 UG 0 0 0 eth0
10.254.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 xen-br1
10.254.0.0 10.254.1.254 255.255.0.0 UG 0 0 0 eth0
0.0.0.0 10.1.1.254 0.0.0.0 UG 0 0 0 xen-br1

and from domU:

Destination Gateway Genmask Flags Metric Ref Use
Iface
10.255.0.129 10.254.1.254 255.255.255.255 UGH 0 0 0 eth0
10.255.0.0 10.254.1.254 255.255.255.0 UG 0 0 0 eth0
10.254.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.254.0.0 10.254.1.254 255.255.0.0 UG 0 0 0 eth0
0.0.0.0 10.1.1.254 0.0.0.0 UG 0 0 0 eth1

so, it looks to me like both things are routing correctly ... but I can't
get anywhere through the 10.1.1. subnet ... (eth1/xen-br1 (tied to dom0
eth3 which is 10.1.1.53))

arp -a (dom0):
sudo arp -a
? (10.1.1.243) at 00:11:43:FD:71:8C [ether] on xen-br1
? (10.254.1.254) at <incomplete> on eth0

no arp -a output on domU

ifconfig output looks good, as well as brctl, everything looks bound
correctly, although i'm still confused why vifX.1 while bound to xen-br1
doesn't get it's mac address ... seems odd to me

well, that's all the output i can do, and if there's nothing wrong in
these routes ... i'm stumped, one network card for me i guess =/

On Mon, 2 May 2005, Nivedita Singhvi wrote:

> andrew mathes wrote:
>
>> hm, tcpdump from external -> eth3 of dom0 looks fine
>>
>> tcpdump from exteran -> eth1 of domU = no reply
>>
>> tcpdump from eth3 -> ext = fine
>>
>> tcpdump from eth3->domU ... the ping says it's ok, but tcpdump shows
>> nothing
>>
>> likewise domU->eth3 ... says it's pinging, but tcpdump shows nothing
>>
>> so is what's happening here is both sides pinging the bridge, but nothing
>> is going through? i have this in dmesg:
>>
>> device vif1.0 entered promiscuous mode
>> xen-br0: port 1(vif1.0) entering learning state
>> xen-br0: topology change detected, propagating
>> xen-br0: port 1(vif1.0) entering forwarding state
>> device vif1.1 entered promiscuous mode
>> xen-br1: port 2(vif1.1) entering learning state
>> xen-br1: topology change detected, propagating
>> xen-br1: port 2(vif1.1) entering forwarding state
>>
>> so that looks good ...
>>
>> i'm not supposed to make the domU gateway the same as the domO nic or
>> anything am i?
>>
>
> Sorry to come so late to this thread - but did you
> share your routes (ip route show, or netstat -rn) output,
> also your arp -a, ifconfig -a?
>
> If you have two real interfaces on the same subnet,
> you may need to set arp_filter on.
>
> thanks,
> Nivedita
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

andrew mathes schrieb:
> all i can think of now is it's a routing issue- here is my route -n on
> dom0:
>
> 10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0
> xen-br1
> 0.0.0.0 10.1.1.254 0.0.0.0 UG 0 0 0
> xen-br1
>
> and from domU:
>
> Destination Gateway Genmask Flags Metric Ref Use
> Iface
> 10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0
> eth1
> 0.0.0.0 10.1.1.254 0.0.0.0 UG 0 0 0
> eth1

OK, network 10.1.1.0/24 with router 10.1.1.254 via eth3 (xen-br3). Both
dom0 and domU are simple hosts and don't route. Fine.


> ifconfig output looks good, as well as brctl, everything looks bound
> correctly, although i'm still confused why vifX.1 while bound to xen-br1
> doesn't get it's mac address ... seems odd to me

My vifX.Y always have fe:ff:ff:ff:ff:ff. Do you mean, while vifX.1 does
not get the configured MAC your vifX.0 does?


> well, that's all the output i can do, and if there's nothing wrong in
> these routes ... i'm stumped, one network card for me i guess =/

One last idea. I don't remember: Have you tried xen-br0 on eth3 and it
worked? If not, are your eth0 and eth3 NICs different hardware
models/use different device drivers? Maybe it's a NIC driver or hardware
problem. Maybe setting the NIC in promicious mode might help "ip l s
eth3 promisc on" or "ifconfig eth3 promisc". Maybe the driver is
incompatible with the bridging code.

Mark.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Might be worth checking all the MAC addresses (the VIFS at both "ends" (the
dom0 vif devices and the domU eth devices). Might as well check the "eth"
devices in dom0.

Can you post those MAC addresses, plus your domain configs where they are set.
If you get clashes in the MAC addresses, all sorts of weird behaviour can
happen!

Cheers,
Mark

On Wednesday 04 May 2005 09:49, Mark Doll wrote:
> andrew mathes schrieb:
> > all i can think of now is it's a routing issue- here is my route -n on
> > dom0:
> >
> > 10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0
> > xen-br1
> > 0.0.0.0 10.1.1.254 0.0.0.0 UG 0 0 0
> > xen-br1
> >
> > and from domU:
> >
> > Destination Gateway Genmask Flags Metric Ref Use
> > Iface
> > 10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0
> > eth1
> > 0.0.0.0 10.1.1.254 0.0.0.0 UG 0 0 0
> > eth1
>
> OK, network 10.1.1.0/24 with router 10.1.1.254 via eth3 (xen-br3). Both
> dom0 and domU are simple hosts and don't route. Fine.
>
> > ifconfig output looks good, as well as brctl, everything looks bound
> > correctly, although i'm still confused why vifX.1 while bound to xen-br1
> > doesn't get it's mac address ... seems odd to me
>
> My vifX.Y always have fe:ff:ff:ff:ff:ff. Do you mean, while vifX.1 does
> not get the configured MAC your vifX.0 does?
>
> > well, that's all the output i can do, and if there's nothing wrong in
> > these routes ... i'm stumped, one network card for me i guess =/
>
> One last idea. I don't remember: Have you tried xen-br0 on eth3 and it
> worked? If not, are your eth0 and eth3 NICs different hardware
> models/use different device drivers? Maybe it's a NIC driver or hardware
> problem. Maybe setting the NIC in promicious mode might help "ip l s
> eth3 promisc on" or "ifconfig eth3 promisc". Maybe the driver is
> incompatible with the bridging code.
>
> Mark.
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users