Mailing List Archive

Am Freitag, den 15.04.2005, 09:20 +0200 schrieb Roland Paterson-Jones:
> Hi
>
> I had a brief look at the routing scripts in /etc/xen/scripts.
> Essentially the main script turns on ip forwarding in dom-0, and the
> dom-U vif script seems to configure a 169.254.1.0 address for each vif
> (auto-configure address, I think), then enable proxy ARP on the vif.
>
> Some questions: How do remote machines pick up routing information for
> the dom-U's? Do I have to run a routing protocol in dom-0 (maybe with
> zebra) so that remote machines can 'see' the dom-U's?

Depends on network configuration: If you use bridging or proxy-arp or
NAT that's not necessary. If not, routes can be configured statically
into remote machines or dynamically via routing protocals like RIP or
OSPF.

This is not a Xen-specific question, look around for networking howtos.


> Could someone maybe explain the details of the ifconfig <vif>
> 169.254.1.0 ...

Nils, thanks for the great response.

Some more details: we have a network of dom-0's that will host a number
of dom-U's.

We need dynamic addressing, so proxy-arp sounds the simplest for us
(simpler than full-on routing, that is).

Bridging is not so nice cos it exposes ethernet to the (untrusted) dom-U's.

We can't use NAT cos we want the dom-U's to be externally addressable.

Something I'm still unclear on - we don't want to reserve dom-U
addresses for each dom-0 (it'll be wasteful), so we want dom-U to use
DHCP. But then we've got to do DHCP relaying in dom-0, I think, and
capture the dom-U IP address, unless there's a better way.

Another thing that's confusing me is that I expect there should be a
left-hand (dom-0) and right-hand (dom-U) address for each of the vif's
in routing mode, but I see only the one address in the scripts.

I hope this makes sense - as you might have noticed I'm approaching this
from first principles. I'm sure I'll get there in the end :(

Regards
Roland


Nils Toedtmann wrote:

>Am Freitag, den 15.04.2005, 09:20 +0200 schrieb Roland Paterson-Jones:
>
>
>>Hi
>>
>>I had a brief look at the routing scripts in /etc/xen/scripts.
>>Essentially the main script turns on ip forwarding in dom-0, and the
>>dom-U vif script seems to configure a 169.254.1.0 address for each vif
>>(auto-configure address, I think), then enable proxy ARP on the vif.
>>
>>Some questions: How do remote machines pick up routing information for
>>the dom-U's? Do I have to run a routing protocol in dom-0 (maybe with
>>zebra) so that remote machines can 'see' the dom-U's?
>>
>>
>
>Depends on network configuration: If you use bridging or proxy-arp or
>NAT that's not necessary. If not, routes can be configured statically
>into remote machines or dynamically via routing protocals like RIP or
>OSPF.
>
>This is not a Xen-specific question, look around for networking howtos.
>
>
>
>
>>Could someone maybe explain the details of the ifconfig <vif>
>>169.254.1.0 ...
>>
>>
>
>>From RFC 3330 <http://www.faqs.org/rfcs/rfc3330.html>:
>
> 169.254.0.0/16 - This is the "link local" block. It is allocated for
> communication between hosts on a single link. Hosts obtain these
> addresses by auto-configuration, such as when a DHCP server may not
> be found.
>
>You may use random IPs in this range as a poor backup alternative to
>dhcp. MS Windows and many devices like printers use such IPs if they
>cannot find a dhcp server.
>
>
>
>
>>and what the proxy ARP stuff does?
>>
>>
>
>It kinda "pseudo-bridging". For example if your domU and your dom0 shall
>use ips within the same IP prefix (say 192.168.1.0/24), and another
>physical host is acting as default gateway (lets say dom0=192.168.1.2,
>domU=192.168.1.3, gw=192.168.1.1), there are (at least) four
>alternatives:
>
>* DNAT all domU-services on dom0
>
> iptables -t nat -A PREROUTING -j DNAT -d 192.168.1.2 \
> --dport 80 --to-destination 192.168.1.3
>
>* hostroutes for domU
>
> gw# ip route add 192.168.1.3/32 via 192.168.1.2
> dom0# ip route add 192.168.1.3/32 dev vif1.0
>
>* bridging
>
>* proxy-arp: When gw tries to send an IP packet to domU it thinks domU
> is link-local, so it tries to resolve 192.168.1.3 to a MAC address by
> ARP-request. But that ARP-request can never reach domU (it's not
> bridged). Now
>
> ip route add 192.168.1.3/32 dev vif1.0
> sysctl -w net.ipv4.conf.eth0.proxy_arp=1
>
> (or has it to be "net.ipv4.conf.vif1.0.proxy_arp"?) tells dom0 to
> reply to that ARP-request with dom0's MAC-address on behalf of domU.
>
> A better way to do proxyarp are static arp entries:
> <http://www.tldp.org/HOWTO/Proxy-ARP-Subnet/index.html>
>
>
>So the xen-script 169.254.0.0/16 ips plus proxyarp on vif* is probably
>for automagical inter-domU-communication.
>
>
>/nils.
>
>
>
>


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Am Freitag, den 15.04.2005, 14:59 +0200 schrieb Roland Paterson-Jones:
> Nils, thanks for the great response.
>
> Some more details: we have a network of dom-0's that will host a number
> of dom-U's.
>
> We need dynamic addressing, so proxy-arp sounds the simplest for us
> (simpler than full-on routing, that is).

Maybe not: at least the dom0 needs hostroutes for the domUs it is
hosting onto the vif* devices. Otherwise it won't proxyarp.


> Bridging is not so nice cos it exposes ethernet to the (untrusted) dom-U's.

Not really. In _all_ cases you want to do some [ip|arp|eb]tables stuff
to filter traffic. Nowadays with CONFIG_BRIDGE_NETFILTER filtering and
routing/bridging is almost independant. Whatever topology you take you
can encapsulate the domUs.


> We can't use NAT cos we want the dom-U's to be externally addressable.

Would still be possible with DNAT (for example any dom0 hosting n domUs
needs n+1 IPs, n of them DNATed), but that would be _really_ _ugly_! You
are right not to do that ;)


> Something I'm still unclear on - we don't want to reserve dom-U
> addresses for each dom-0 (it'll be wasteful), so we want dom-U to use
> DHCP. But then we've got to do DHCP relaying in dom-0, I think, and
> capture the dom-U IP address, unless there's a better way.

dhcrelay(8) is your friend, but indeed dom0 must somehow be informed
about the combination IP-adress/virtual-interface (and if you want
filtering: MAC) to set the hostroute. And that hostroute has to be
deleted when the domU shuts down ... so:

(a) turn on a RIP/OSPF routing daemon on all domUs and all dom0s
(b) or: patch dhcrelay to set the hostroute (but on shutdown?)
(c) or: dom0 catches dhcp traffic on vif* devices and parses it

(a) is overkill and insecure, (b) & (c) are evel & ugly!


> Another thing that's confusing me is that I expect there should be a
> left-hand (dom-0) and right-hand (dom-U) address for each of the vif's
> in routing mode, but I see only the one address in the scripts.

That's ok. There are again two possibilities (at least):

* the vif* are up, the corrosponding domU-eth0 have IPs in the same
subnet, dom0 has hostroutes for those domU-IPs onto the according
vif*. In this case, it does not matter if the vif* have an IP at all.
dom0 has proxy-arp activated.

* each dom0-vif/domU-eth0 pair is configured in a different transport-
subnet, eg vif1.0=192.168.1.1/30, eth0(dom1)=192.168.1.2/30,
vif2.0=192.168.1.5/30, eth0(dom2)=192.168.1.6/30

I don't know what the xend routing scripts actually do. You may fire up
two domUs and send me the output of "/sbin/ip addr" and "/sbin/ip route"
then i'll tell.


> I hope this makes sense - as you might have noticed I'm approaching this
> from first principles. I'm sure I'll get there in the end :(

To sum it up: You are trying to set up a more complex scenario, several
dom0s hosting different numbers of domU. But you do not want to reserve
IP prefixes to dom0s. My strong recommendation: BRIDGING + fixed MAC/IP
pairs + ebtables filtering those pairs. You may use dhcp if you do not
want to configure each domU. The MAC/IP pairing could be algorithmical
like IP=o1.o2.o3.o4 ==> MAC=FE:00:o1:o2:o3:o4 (this time in hex).

Everthing else (like proxyarp + dhcrelay) will get too fast too complex.
For example you could do dhcp & real dynamic routing, but then every
domU has to run a routing daemon announcing its IP, so a malicious domU
could redirect all traffic to itself. Or you have to do some evil hack
to pass routing information from dhcp to the dom0s.

Just my 2c. /nils.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nils Toedtmann wrote:

>>Bridging is not so nice cos it exposes ethernet to the (untrusted) dom-U's.
>>
>>
>
>Not really. In _all_ cases you want to do some [ip|arp|eb]tables stuff
>to filter traffic. Nowadays with CONFIG_BRIDGE_NETFILTER filtering and
>routing/bridging is almost independant. Whatever topology you take you
>can encapsulate the domUs.
>
>
Can we ensure that dom-U is not sending ethernet packets with fake
destination mac addresses if we're using bridging?

How do we prevent a dom-U filling up our LAN with bogus ethernet addresses?

I guess we want to restrict the dom-U to IP packets with IP/MAC pairs
that match previous ARP results. Can ebtables in dom-0 filter this
accurately?

>To sum it up: You are trying to set up a more complex scenario, several
>dom0s hosting different numbers of domU. But you do not want to reserve
>IP prefixes to dom0s. My strong recommendation: BRIDGING + fixed MAC/IP
>pairs + ebtables filtering those pairs. You may use dhcp if you do not
>want to configure each domU. The MAC/IP pairing could be algorithmical
>like IP=o1.o2.o3.o4 ==> MAC=FE:00:o1:o2:o3:o4 (this time in hex).
>
>
Actually I have played with such an algorithmic MAC/IP pairing in a
prototype. But then the aim was to specify the MAC address in the Xen
config for the dom-U, let the dom-U use DHCP, and ensure that the DHCP
mapped the MAC to the corresponding IP address, all in order that I knew
the IP address of the dom-U up front (but let the dom-U use DHCP rather
than static for more flexibility etc.).

Bridging is definitely easier to manage than routing. However, given
that I'm paranoid about untrusted dom-U's, how can we prevent dom-U's
from abusing the ethernet network?

Also, there will be more ARP'ing with bridging, since all the dom-U's
will ARP independently (can we short-circuit ARP responses in dom-0?).

Thanks again for your detailed help.
Roland

>
>


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Can we ensure that dom-U is not sending ethernet packets with
> fake destination mac addresses if we're using bridging?

Sure. Just add the apprioriate netfilter or ebtables rules to
'vif-bridge'.

> How do we prevent a dom-U filling up our LAN with bogus
> ethernet addresses?

There's an example of a netfilter rule to prevent spoofing of bogus src
IP addrs.

> I guess we want to restrict the dom-U to IP packets with
> IP/MAC pairs that match previous ARP results. Can ebtables in
> dom-0 filter this accurately?

Sure. If you don't know all the rules at domain creation time you'll
probably need to cook up your own little daemon to add rules/

> Also, there will be more ARP'ing with bridging, since all the
> dom-U's will ARP independently (can we short-circuit ARP
> responses in dom-0?).

Why would you want to? It's hardly high bandwidth.

Ian

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ian Pratt wrote:

>>I guess we want to restrict the dom-U to IP packets with
>>IP/MAC pairs that match previous ARP results. Can ebtables in
>>dom-0 filter this accurately?
>>
>>
>
>Sure. If you don't know all the rules at domain creation time you'll
>probably need to cook up your own little daemon to add rules/
>
>
I think I might be able to achieve what I want with ebtables by brouting
all outgoing traffic. So dom-0 is a router for outgoing traffic but a
bridge for incoming traffic. I think I just have to enable
ip_forwarding, but otherwise use the xen 'bridging' scripts.

>>Also, there will be more ARP'ing with bridging, since all the
>>dom-U's will ARP independently (can we short-circuit ARP
>>responses in dom-0?).
>>
>>
>
>Why would you want to? It's hardly high bandwidth.
>
>
Well, ARP is broadcast and across all bridged networks. What if the
dom-U did an ARP-bomb attack, for example. I don't know really. I guess
you could rate limit ARP's with ebtables.

Anyway, if we're brouting outbound traffic, then we can use --arpreply
<bogus-address> to short-circuit outbound ARP requests. They're no use
anyway, if we're brouting all outbound traffic.

Does this all sound plausible or maybe even sensible?

Thanks for your help
Roland


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Am Sonntag, den 17.04.2005, 18:56 +0200 schrieb Roland Paterson-Jones:
> Ian Pratt wrote:
> > > I guess we want to restrict the dom-U to IP packets with
> > > IP/MAC pairs that match previous ARP results. Can ebtables in
> > > dom-0 filter this accurately?
> >
> > Sure. If you don't know all the rules at domain creation time you'll
> > probably need to cook up your own little daemon to add rules/
> >
> >
> I think I might be able to achieve what I want with ebtables by brouting
> all outgoing traffic.

What is "brouting"? There's an ebtables chain with that name, but i
never heard this term (yet) as a name for a network topology ...?

> So dom-0 is a router for outgoing traffic but a
> bridge for incoming traffic.

Ah! Is that standard terminology?

> I think I just have to enable
> ip_forwarding, but otherwise use the xen 'bridging' scripts.

What advantage you gain over proper bridging?

> > > Also, there will be more ARP'ing with bridging, since all the
> > > dom-U's will ARP independently (can we short-circuit ARP
> > > responses in dom-0?).
> >
> > Why would you want to? It's hardly high bandwidth.
>
> Well, ARP is broadcast and across all bridged networks. What if the
> dom-U did an ARP-bomb attack, for example. I don't know really. I guess
> you could rate limit ARP's with ebtables.

That's not ARP nor bridging/routing specific. A malicious domU could do
MAC/IP-broad/multi/unicast attact if you do not filter and ratelimit
it's traffic properly. It's always the same principal problem, whatever
topology you use. With routing for example it can still try to SYN-
flood, ping-of-death, overload net with massive UDP fragment
traffic, ...


> Anyway, if we're brouting outbound traffic, then we can use --arpreply
> <bogus-address> to short-circuit outbound ARP requests. They're no use
> anyway, if we're brouting all outbound traffic.
>
> Does this all sound plausible or maybe even sensible?

Hmmm ... some general guidelines:

* Making things more complicated than necessary reduces security

* Using network topology not fully understood reduces security (sorry,
could not resist ;-)

* Do not try to filter traffic by using a special network topology.
Use filters for it.


At domU creation time, dom0 knows it's dedicated MAC, and (according to
your own rules) the according IP of that domU. As Ian wrote: extend the
vif-bridge (which now knows the IP/MAC/VIF combination) using

* ebtables to enforce the correct MAC (--> no MAC spoofing, no STP
attacks)

* arptables to enforce the correct IP/MAC pairing (--> no ARP spoofing)

* iptables to enforce the correct IP (--> no IP spoofing)

* ebtables to force ethernet frames to contain IP or ARP only (no bogus
IPv6/MPLS/PPPoE stuff on the wire)

* ebtables to limit linklayer broadcasts to ARP and IP broadcasts

* iptables to limit IP broadcasts to the necessary ones (for example
dhcp)

* iptables to limit unicast IP traffic to the services the domU is
allowed to offer/use.

* some ratelimiting (particulary on TCP/SYN) such that the domU cannot
DoS somebody via ARP/IP/whatever


This is just an example, but quiet paranoid. You would have to do most
of it with (b)routing, too (even with routing you have to protect dom0
from MAC/ARP-spoofing!).

Just my 2ct, /nils.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nils Toedtmann wrote:

>Am Sonntag, den 17.04.2005, 18:56 +0200 schrieb Roland Paterson-Jones:
>
>
>>I think I might be able to achieve what I want with ebtables by brouting
>>all outgoing traffic.
>>
>>
>
>What is "brouting"? There's an ebtables chain with that name, but i
>never heard this term (yet) as a name for a network topology ...?
>
>
I would call it a hack rather than a network topology. The only
advantage is that dom-0 doesn't have to know the dom-U IP addresses, but
can still exert firm control over traffic from dom-U's.

>>So dom-0 is a router for outgoing traffic but a
>>bridge for incoming traffic.
>>
>>
>
>Ah! Is that standard terminology?
>
>
I doubt it ;)

>What advantage you gain over proper bridging?
>
>
I'm assuming iptables doesn't see bridged ethernet traffic(!?) So using
ebtables' brouting forces the outbound IP traffic through IP routing
letting iptables take a look.

>At domU creation time, dom0 knows it's dedicated MAC, and (according to
>your own rules) the according IP of that domU. As Ian wrote: extend the
>vif-bridge (which now knows the IP/MAC/VIF combination) using
>
>
The MAC -> IP mapping is a pain with DHCP, cos dhcpd scripting doesn't
extend to mangling the hardware address into the resulting (fixed) IP
address. In the prototype, I had a hard-coded rule for each MAC -> IP.
This is not very scalable!

However, another way to do it is to use iptables to QUEUE DHCP responses
to a custom ipq app which pulls out the IP address and does the same. In
other words, to sniff the DHCP allocations in dom-0.

And, yes, I think you DO need to know the IP address to do effective
firewalling in dom-0. Previously, I was hoping to avoid dom-0 knowing
the IP address at all by using bridging.

>* iptables to enforce the correct IP (--> no IP spoofing)
>
>
Does iptables get to see ethernet-bridged traffic? I thought ethernet
traffic snuck through under the iptables radar since it doesn't
(shouldn't?) touch the IP stack.

Thanks again for the frank discussion
Roland




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Am Montag, den 18.04.2005, 16:02 +0200 schrieb Roland Paterson-Jones:
>
> Nils Toedtmann wrote:
>
> >Am Sonntag, den 17.04.2005, 18:56 +0200 schrieb Roland Paterson-Jones:
> I'm assuming iptables doesn't see bridged ethernet traffic(!?)
[...]
> >* iptables to enforce the correct IP (--> no IP spoofing)
> Does iptables get to see ethernet-bridged traffic? I thought ethernet
> traffic snuck through under the iptables radar since it doesn't
> (shouldn't?) touch the IP stack.

That depends. In the old days of linux-2.4 you needed the br-nf-patch
from the ebtables site to make bridged ip packets visible to iptable.
But as i already said there is now "CONFIG_BRIDGE_NETFILTER" in
linux-2.6: if you compile a kernel with "CONFIG_BRIDGE_NETFILTER=y" (as
all distributers i know do and as xen does in it's default dom0 config)
then iptables sees every forwarded frame which has ethertype 0x0800
(IPv4). If you want it more detailed, see this netfilter/ebtables flow
chart:

<http://l7-filter.sourceforge.net/PacketFlow.png>

So: everything you want to filter you can filter with bridging.

/nils.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nils Toedtmann wrote:

>Am Montag, den 18.04.2005, 16:02 +0200 schrieb Roland Paterson-Jones:
>
>
>>Does iptables get to see ethernet-bridged traffic? I thought ethernet
>>traffic snuck through under the iptables radar since it doesn't
>>(shouldn't?) touch the IP stack.
>>
>>
>
>That depends. In the old days of linux-2.4 you needed the br-nf-patch
>from the ebtables site to make bridged ip packets visible to iptable.
>But as i already said there is now "CONFIG_BRIDGE_NETFILTER" in
>linux-2.6: if you compile a kernel with "CONFIG_BRIDGE_NETFILTER=y" (as
>all distributers i know do and as xen does in it's default dom0 config)
>then iptables sees every forwarded frame which has ethertype 0x0800
>(IPv4). If you want it more detailed, see this netfilter/ebtables flow
>chart:
>
> <http://l7-filter.sourceforge.net/PacketFlow.png>
>
>So: everything you want to filter you can filter with bridging.
>
>
Cool. That makes it much easier. I must admit I missed the significance
of your CONFIG_BRIDGE_NETFILTER comment last time.

Now really, the only ugly part is discovering/forcing the dom-U IP
addresses.

Thanks again
Roland


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Am Montag, den 18.04.2005, 16:30 +0200 schrieb Roland Paterson-Jones:
> Now really, the only ugly part is discovering/forcing the dom-U IP
> addresses.

If you do not want to patch dhcpd, you may do funny other things:

* generate huge dhcpd.conf with bash/perl scripts (ok, that's not so
funny)

* supply all domUs with a network initscript which greps network
information from kernel command line /proc/cmdline (for example
"eth0=192.168.1.2/24 gw=192.168.1.1". Put not only mac address
into /etc/xen/domUconf, but also 'extra+=" eth0=..."'

* poor man's dhcp: encode all networking information (ip/mask/gw) within
the MAC (for networks <= /21 possible) and supply all domUs with
a network initscript that decodes the MAC. For example 192.168.1.2/24
with gw 192.168.1.1 could be encoded as

0x80:1:192:168:1:2

(decimal octets have to be converted to hex): the first four bits are
the prefix-complemneent "32-24=8", the next tree bits are void, the
eighth bit has to be zero (to be a valid unicast MAC), the second MAC-
octet is the last IP-octet of the gw-ip (.1), the last four MAC-octets
are the ip.

Evil, eh?

/nils.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users