Mailing List Archive

How to setup a multi vlan connection to an (Ubuntu) XEN-server !!??
I try to configure a Xen server based on the latest Ubuntu server. The server will be connected to a network trunk using vlan's.

?

I hoped this was relatively simple, however it is not. When installing ubuntu-22.04.1-live-server-amd64.iso, I did not manage to setup the network, so I did start digging into the problem. I did open lots of config files searching for the network configuration ........ a crime ..... and discovered that there are tons of (conflicting) tools and related ways to do that.

?

?

After many hours I discovered that part of the network config was stored in the file "/etc/network/00-installer-config.yaml" and that file was generated by an application called 'subiquity'.

By the way subiquity'. is using vconfig is which seems to be deprecated and not ip(route2).

I could not find a decent specification of the yami file. I noted that AFTER changing the config an "sudo netplan apply" is needed.

?

I also noted that the classic method via "/etc/network/interfaces" does not work anymore and

that "systemd.network - Network configuration" is not used. All files in "/etc/systemd" are at their defaults.

My feeling is that systend is the most modern way to config the network (I do not have any experience), but that that does not match the way Ubuntu is setup and that all those network setup methods are conflicting with each other.



?

And that is only Ubuntu itself. After that there is the interaction between the XEN-hypervisor network and Ubuntu. So some will understand that I have a headache.



?

What I would like to archive is:

- to manage the XEN-server from a (home compiled) XEN Orchestra instance

- that XEN Orchestra is in first instance running in a windows10 hosted VM and later on perhaps on the XEN-server itself

- the server wil host some vm's most of them connected to one of the vlan's provided by the trunk

- some vm's will share the same vlan

- each vlan will (or course) have >> its own gatgeway and related its own routing table <<

- I will also use a second nic for managment purposes (dom0 / ubuntu itself)

- I am also using IPV6

?

?

So it is clear that I need some advice and examples. Of course a decent gui would help very much ….

?

Louis
Re: How to setup a multi vlan connection to an (Ubuntu) XEN-server !!?? [ In reply to ]
Hi,



> Am 05.10.2022 um 09:59 schrieb louis.forums@xs4all.nl <lsrbreda@xs4all.nl>:
>
> I try to configure a Xen server based on the latest Ubuntu server. The server will be connected to a network trunk using vlan's.
>
> I hoped this was relatively simple, however it is not. When installing ubuntu-22.04.1-live-server-amd64.iso, I did not manage to setup the network, so I did start digging into the problem. I did open lots of config files searching for the network configuration ........ a crime ..... and discovered that there are tons of (conflicting) tools and related ways to do that.
>

it has become rather stupid with network configuration. I don’t know what the point of it is, but at least I’ve found ways to use it well.

I don’t use all things you do, but generally, I’ve found the combination of „netplan“ using its /etc/netplan/blah.yaml files
with the „systemd-networkd“ renderer to be reliable to a good level, including the important edge cases when something goes wrong.

The alternative you probably need to look into is to rely more on openvswitch for all the vlan related tasks.

what i’d recommend you to try:
have one dedicated host mgmt interface (with a 01-mgmt-lan.yaml for example) and a second set of interfaces that will be driven by openvswitch (55-ovs-vmnet.yaml)

Greetings,
Flo
Re: How to setup a multi vlan connection to an (Ubuntu) XEN-server !!?? [ In reply to ]
louis.forums@xs4all.nl <lsrbreda@xs4all.nl> wrote:

> I try to configure a Xen server based on the latest Ubuntu server. The server will be connected to a network trunk using vlan's.

Should be easy enough …

> I hoped this was relatively simple, however it is not. When installing ubuntu-22.04.1-live-server-amd64.iso, I did not manage to setup the network, so I did start digging into the problem. I did open lots of config files searching for the network configuration ........ a crime ..... and discovered that there are tons of (conflicting) tools and related ways to do that.
>
>
> After many hours I discovered that part of the network config was stored in the file "/etc/network/00-installer-config.yaml" and that file was generated by an application called 'subiquity'.
> By the way subiquity'. is using vconfig is which seems to be deprecated and not ip(route2).
> I could not find a decent specification of the yami file. I noted that AFTER changing the config an "sudo netplan apply" is needed.
>
> I also noted that the classic method via "/etc/network/interfaces" does not work anymore and
> that "systemd.network - Network configuration" is not used. All files in "/etc/systemd" are at their defaults.
> My feeling is that systend is the most modern way to config the network (I do not have any experience), but that that does not match the way Ubuntu is setup and that all those network setup methods are conflicting with each other.

My personal approach would be to track down and nuke from orbit all supposedly “smart” network management tools that are doing nothing but get in the way. IMO, none of these sort of tools have a place on a server with a static network config.

Once you get down to being able to configure the network via /etc/network/interfaces then it becomes fairly easy to do.

As suggested, these days it may be worth looking into openvswitch - but when I was doing this, it wasn’t quite mature enough. So I just used the built in tools.

The way I did it was :
Rename my physical interfaces to meaningful names - this means that if you change hardware in any way, the only place you need to change anything is wherever the interface name is configured (used to be /etc/udev/rules.d/70-persistent-net.rules).
In the absence of openvswitch, I split out each VLAN to a separate bridge, so something along the lines of : bridge “ethnic”, with a member pethtrunk.10; bridge “ethext” with member pethtrunk.11; and so on.

> iface ethint init static
> bridge_ports pethtrunk.10
> iface ethext init static
> bridge_ports pethtrunk.11

Then you can attach a guest to one or more of the bridges - with each VLAN appearing as a separate virtual interface.

I had a mix of trunked and physically separate LANs, so as well a pethtrunk, I had a pethext, pethback, and so on - just adjust the bridge declarations accordingly.

This obviously only scales so far, so I suspect these days going down the openvswitch route may make more sense - and present a trunk interface to the guest. But you will still need to track down and terminate with extreme prejudice all those tools getting in the way.


Just my 2d worth, Simon
RE: How to setup a multi vlan connection to an (Ubuntu) XEN-server !!?? [ In reply to ]
My concerns with that are:
- Ubuntu seems to have deliberately chosen not to use the /etc/network/interfaces method. That method seems to be "end of life"
- I am not sure if it is a good idea to use a second config method next to the method canonical.com has chosen
- system seems to be the most modern tool and also the tool which structure is more logical to me than other methods
( I admit I have no real/practical experience with all of the possible options)

So I am not convinced about that direction yet, apart from the fact that I do not know how to define the intended setup (including vlans and multiple routing tables etc) using the "/etc/network/interfaces" method.

Louis
-----Original Message-----
From: Xen-users <xen-users-bounces@lists.xenproject.org> On Behalf Of Simon
Sent: Wednesday, October 5, 2022 7:45 PM
To: xen-users@lists.xenproject.org
Subject: Re: How to setup a multi vlan connection to an (Ubuntu) XEN-server !!??

louis.forums@xs4all.nl <lsrbreda@xs4all.nl> wrote:

> I try to configure a Xen server based on the latest Ubuntu server. The server will be connected to a network trunk using vlan's.

Should be easy enough …

> I hoped this was relatively simple, however it is not. When installing ubuntu-22.04.1-live-server-amd64.iso, I did not manage to setup the network, so I did start digging into the problem. I did open lots of config files searching for the network configuration ........ a crime ..... and discovered that there are tons of (conflicting) tools and related ways to do that.
>
>
> After many hours I discovered that part of the network config was stored in the file "/etc/network/00-installer-config.yaml" and that file was generated by an application called 'subiquity'.
> By the way subiquity'. is using vconfig is which seems to be deprecated and not ip(route2).
> I could not find a decent specification of the yami file. I noted that AFTER changing the config an "sudo netplan apply" is needed.
>
> I also noted that the classic method via "/etc/network/interfaces"
> does not work anymore and that "systemd.network - Network configuration" is not used. All files in "/etc/systemd" are at their defaults.
> My feeling is that systend is the most modern way to config the network (I do not have any experience), but that that does not match the way Ubuntu is setup and that all those network setup methods are conflicting with each other.

My personal approach would be to track down and nuke from orbit all supposedly “smart” network management tools that are doing nothing but get in the way. IMO, none of these sort of tools have a place on a server with a static network config.

Once you get down to being able to configure the network via /etc/network/interfaces then it becomes fairly easy to do.

As suggested, these days it may be worth looking into openvswitch - but when I was doing this, it wasn’t quite mature enough. So I just used the built in tools.

The way I did it was :
Rename my physical interfaces to meaningful names - this means that if you change hardware in any way, the only place you need to change anything is wherever the interface name is configured (used to be /etc/udev/rules.d/70-persistent-net.rules).
In the absence of openvswitch, I split out each VLAN to a separate bridge, so something along the lines of : bridge “ethnic”, with a member pethtrunk.10; bridge “ethext” with member pethtrunk.11; and so on.

> iface ethint init static
> bridge_ports pethtrunk.10
> iface ethext init static
> bridge_ports pethtrunk.11

Then you can attach a guest to one or more of the bridges - with each VLAN appearing as a separate virtual interface.

I had a mix of trunked and physically separate LANs, so as well a pethtrunk, I had a pethext, pethback, and so on - just adjust the bridge declarations accordingly.

This obviously only scales so far, so I suspect these days going down the openvswitch route may make more sense - and present a trunk interface to the guest. But you will still need to track down and terminate with extreme prejudice all those tools getting in the way.


Just my 2d worth, Simon
Re: How to setup a multi vlan connection to an (Ubuntu) XEN-server !!?? [ In reply to ]
On 2022-10-05 00:59, louis.forums@xs4all.nl wrote:
> I try to configure a Xen server based on the latest Ubuntu server. The
> server will be connected to a network trunk using vlan's.

Ubuntu network configuration has moved to using netplan. netplan is
really
a wrapper around other network tools on the system and intended to make
the
configuration of systemd network or netmanager networking simpler. Took
me
a while to get a trunked configuration to work, but once you get it is
fairly
straight forward. Here is an example:

# file: /etc/netplan/config-trunk.yaml
network:
version: 2
renderer: networkd
ethernets:
eno2:
mtu: 9000
dhcp4: no
dhcp6: no
bridges:
br179:
macaddress: 66:c5:7f:bf:f6:00
mtu: 9000
dhcp4: no
dhcp6: no
interfaces: [ vlan179 ]
br180:
macaddress: 66:c5:7f:bf:f6:10
mtu: 9000
dhcp4: no
dhcp6: no
interfaces: [ vlan180 ]
vlans:
vlan179:
mtu: 9000
id: 179
link: eno2
dhcp4: no
dhcp6: no
vlan180:
mtu: 9000
id: 180
link: eno2
dhcp4: no
dhcp6: no

This assumes that you have the netplan.io package installed. Be
careful about the MAC addresses you use. Since you make them up
make sure you don't duplicate anything on the local network. You
will note in my example that the second NIC is used for trunking.
This is just because we use FAI for building systems and PXE booting
is not supported on trunked ports. The first NIC is a simple access
port.

Once you get the bridged network up then the network definition in the
xen configuration need to reference the VLAN. I use xen-create-image to
build guests which just means I add --vlan=N to the command. An
example from a xen cfg file is:

vif = [ 'ip=10.0.0.18, mac=00:16:3E:1B:D1:EA, vlan=180' ]


Bill

--
"What can be asserted without evidence can also be dismissed without
evidence."
Christopher Hitchens
RE: How to setup a multi vlan connection to an (Ubuntu) XEN-server !!?? [ In reply to ]
Yep,

If canonical is using netplan. IMHO all the network definitions for the server should be configured that way.

Your example is in the direction I would like to go, however a lot of things still to be solved. To mention a few:
- I would like a proper specification of the yaml file format. Is that available somewhere??
- you are using macaddresses. Why?
- you do not assign ip-addresses. Why?
- you do not specify vlan related gateways and routing tables .... ??
- you are talking about a simple nic and a trunk. There seems only a trunk definition in the example ?
- is het necessary to add a bridge if there is only one vm using a certain vlan

So lot of questions, never the less it supports my idea that this is the way to go


Louis

-----Original Message-----
From: Xen-users <xen-users-bounces@lists.xenproject.org> On Behalf Of Bill MacAllister
Sent: Thursday, October 6, 2022 12:12 AM
To: louis.forums@xs4all.nl <lsrbreda@xs4all.nl>
Cc: xen-users@lists.xenproject.org
Subject: Re: How to setup a multi vlan connection to an (Ubuntu) XEN-server !!??

On 2022-10-05 00:59, louis.forums@xs4all.nl wrote:
> I try to configure a Xen server based on the latest Ubuntu server. The
> server will be connected to a network trunk using vlan's.

Ubuntu network configuration has moved to using netplan. netplan is really a wrapper around other network tools on the system and intended to make the configuration of systemd network or netmanager networking simpler. Took me a while to get a trunked configuration to work, but once you get it is fairly straight forward. Here is an example:

# file: /etc/netplan/config-trunk.yaml
network:
version: 2
renderer: networkd
ethernets:
eno2:
mtu: 9000
dhcp4: no
dhcp6: no
bridges:
br179:
macaddress: 66:c5:7f:bf:f6:00
mtu: 9000
dhcp4: no
dhcp6: no
interfaces: [ vlan179 ]
br180:
macaddress: 66:c5:7f:bf:f6:10
mtu: 9000
dhcp4: no
dhcp6: no
interfaces: [ vlan180 ]
vlans:
vlan179:
mtu: 9000
id: 179
link: eno2
dhcp4: no
dhcp6: no
vlan180:
mtu: 9000
id: 180
link: eno2
dhcp4: no
dhcp6: no

This assumes that you have the netplan.io package installed. Be careful about the MAC addresses you use. Since you make them up make sure you don't duplicate anything on the local network. You will note in my example that the second NIC is used for trunking.
This is just because we use FAI for building systems and PXE booting is not supported on trunked ports. The first NIC is a simple access port.

Once you get the bridged network up then the network definition in the xen configuration need to reference the VLAN. I use xen-create-image to build guests which just means I add --vlan=N to the command. An example from a xen cfg file is:

vif = [ 'ip=10.0.0.18, mac=00:16:3E:1B:D1:EA, vlan=180' ]


Bill

--
"What can be asserted without evidence can also be dismissed without evidence."
Christopher Hitchens
RE: How to setup a multi vlan connection to an (Ubuntu) XEN-server !!?? [ In reply to ]
I found two webpages which helps to understand how to configure the ^ /etc/netplan/config-trunk.yaml^ file
# https://linuxconfig.org/netplan-network-configuration-tutorial-for-beginners
# https://netplan.io/examples

Not that this makes life simple, but it helps ...

Louis

-----Original Message-----
From: Xen-users <xen-users-bounces@lists.xenproject.org> On Behalf Of louis.forums@xs4all.nl
Sent: Thursday, October 6, 2022 4:30 PM
To: 'Bill MacAllister' <bill@ca-zephyr.org>
Cc: xen-users@lists.xenproject.org
Subject: RE: How to setup a multi vlan connection to an (Ubuntu) XEN-server !!??

Yep,

If canonical is using netplan. IMHO all the network definitions for the server should be configured that way.

Your example is in the direction I would like to go, however a lot of things still to be solved. To mention a few:
- I would like a proper specification of the yaml file format. Is that available somewhere??
- you are using macaddresses. Why?
- you do not assign ip-addresses. Why?
- you do not specify vlan related gateways and routing tables .... ??
- you are talking about a simple nic and a trunk. There seems only a trunk definition in the example ?
- is het necessary to add a bridge if there is only one vm using a certain vlan

So lot of questions, never the less it supports my idea that this is the way to go


Louis

-----Original Message-----
From: Xen-users <xen-users-bounces@lists.xenproject.org> On Behalf Of Bill MacAllister
Sent: Thursday, October 6, 2022 12:12 AM
To: louis.forums@xs4all.nl <lsrbreda@xs4all.nl>
Cc: xen-users@lists.xenproject.org
Subject: Re: How to setup a multi vlan connection to an (Ubuntu) XEN-server !!??

On 2022-10-05 00:59, louis.forums@xs4all.nl wrote:
> I try to configure a Xen server based on the latest Ubuntu server. The
> server will be connected to a network trunk using vlan's.

Ubuntu network configuration has moved to using netplan. netplan is really a wrapper around other network tools on the system and intended to make the configuration of systemd network or netmanager networking simpler. Took me a while to get a trunked configuration to work, but once you get it is fairly straight forward. Here is an example:

# file: /etc/netplan/config-trunk.yaml
network:
version: 2
renderer: networkd
ethernets:
eno2:
mtu: 9000
dhcp4: no
dhcp6: no
bridges:
br179:
macaddress: 66:c5:7f:bf:f6:00
mtu: 9000
dhcp4: no
dhcp6: no
interfaces: [ vlan179 ]
br180:
macaddress: 66:c5:7f:bf:f6:10
mtu: 9000
dhcp4: no
dhcp6: no
interfaces: [ vlan180 ]
vlans:
vlan179:
mtu: 9000
id: 179
link: eno2
dhcp4: no
dhcp6: no
vlan180:
mtu: 9000
id: 180
link: eno2
dhcp4: no
dhcp6: no

This assumes that you have the netplan.io package installed. Be careful about the MAC addresses you use. Since you make them up make sure you don't duplicate anything on the local network. You will note in my example that the second NIC is used for trunking.
This is just because we use FAI for building systems and PXE booting is not supported on trunked ports. The first NIC is a simple access port.

Once you get the bridged network up then the network definition in the xen configuration need to reference the VLAN. I use xen-create-image to build guests which just means I add --vlan=N to the command. An example from a xen cfg file is:

vif = [ 'ip=10.0.0.18, mac=00:16:3E:1B:D1:EA, vlan=180' ]


Bill

--
"What can be asserted without evidence can also be dismissed without evidence."
Christopher Hitchens
RE: How to setup a multi vlan connection to an (Ubuntu) XEN-server !!?? [ In reply to ]
At the end, after a lot of research and testing I did manage to get my system working.

?

## my requirement ##

# traffic arriving at a particular interface, fysical or vlan should

# also be answered via *that* interface

?

## Helpful links ##

# https://cloudinit.readthedocs.io/en/latest/topics/network-config-format-v2.html

# https://linuxconfig.org/netplan-network-configuration-tutorial-for-beginners

# https://netplan.io/examples

# https://netplan.io/reference

# https://manpages.ubuntu.com/manpages/jammy/man5/netplan.5.html helped me

?

## Issues left ##

# 1) So I did not define a server wide default gateway ....

# .... however that is not true if enp5s0 becomes active ....

# .... I simply do not know how to assign a routing and routing-policy to

# an dhcp provided address

# 2) The trunk is a container for vlans, it does not have an address ...

# .... however I had to give it a dummy address to make this file work

?

The attached a jaml config file gives being an example for a server having three interfaces. One of the interfaces is the vlan trunk. In the attached file one vlan, but of course it is no problem to add other vlans or to add IPV6 (I assume).

?

Louis



?

?
Re: How to setup a multi vlan connection to an (Ubuntu) XEN-server !!?? [ In reply to ]
Sorry, a bit of rambling ...


louis.forums@xs4all.nl <lsrbreda@xs4all.nl> wrote:

> - you are using macaddresses. Why?

MAC addresses are used by bridges for a number of functions.
For STP, in the absence of specific priority settings, the MAC addresses will determine the network topology.
I would assume that if you want to attach an IP address to the bridge and use it as an interface from the host then it will need a MAC address.
There are probably other reasons I’m not aware of - quite likely the network stack needs it even if the host isn’t using it as an interface.

> - you do not assign ip-addresses. Why?

The host does not need an IP address in any plan/bridge it isn’t using for it’s own networking.

> - you do not specify vlan related gateways and routing tables .... ??

Again, not needed if the host is not actively using that interface for its own traffic

> - is het necessary to add a bridge if there is only one vm using a certain vlan

I believe so. In any case, it’s the simplest way to do it. The host can create the trunk interface, VLANs, and bridges at boot time, then they are ready for the the guests to attach to as and when required. Note that the guest side of the networking is not configured until the guest starts some time after the host network config is set up. Also, the guest side will be deleted if/when the guest is shut down.


> ## Issues left ##
> # 1) So I did not define a server wide default gateway ....
> # .... however that is not true if enp5s0 becomes active ....
> # .... I simply do not know how to assign a routing and routing-policy to
> # an dhcp provided address

I notice you have default routes configured on multiple interfaces. Normally this is incorrect - you’d only want a single default route, and potentially other specific routes via specific interfaces. Is this the reason for policy based routing (see below) ?
Are you forced to use DHCP for this ? If not, I would suggest static config would be preferable.

> # 2) The trunk is a container for vlans, it does not have an address ...
> # .... however I had to give it a dummy address to make this file work

So that’s a problem with the configuration tool because - as you point out - it absolutely does not need an IP address **unless** you wish to use it directly as a host interface. I would stick to using an address from the RFC1918 ranges BTW.


> The attached a jaml config file gives being an example for a server having three interfaces. One of the interfaces is the vlan trunk. In the attached file one vlan, but of course it is no problem to add other vlans or to add IPV6 (I assume).

I am curious why you are applying policy based routing - either there's some detail (probably not important to the discussion) you’ve omitted, or possibly it’s not needed. In many years, I’ve only needed policy based (source address) routing in one network - and that was a campus network where I had to handle 2 providers, separate IP ranges, and phased migration, hence needing to route via a specific interface according the IP address used by a connected client.
Ah, I see https://netplan.io/examples gives something like your config (key difference - example has different routing table numbers), but from the explanation I cannot fathom out what it’s saying ! I’m used to working with systems that handle what I think the page is saying “out of the box”, but since I don’t understand what problem is being fixed, I can’t see if this is relevant or not.

While on routing ...
The “on-link” statements are not needed as your router addresses are within the same subnets as the interface addresses. on-link is specifically for the case where this is not so - where normally the system would ignore the route as the router address is notionally unreachable. I think all you need is :
> routes:
> - to: default
> via: n.n.n.n
(https://netplan.io/examples)


BTW - my personal opinion is that you should really look into whether you can give the interfaces “sensible” names. Which enp3sf1 is not the worst of the naming schemes, you will really curse it the first time you change the host networking. As it is, you will need to find every instance of the old interface name in every config file its used in - it won’t just be in this file. And if you want to move a guest to a different host, you’ll need to change its config to suit the host’s interface names.
I’ve always done this via dev rules - and while some consider it heresy to use the MAC address as an interface identifier, it’s literally one line per interface and only one place to update if anything changes (replacement NIC, add/remove a card that renumbers other cards, move to new hardware, whatever).

Ah, I see from https://cloudinit.readthedocs.io/en/latest/topics/network-config-format-v2.html that it’s possible to do it with a match statement in the config file.
> In the generic case, they can be selected by match: rules on desired properties, such as name/name pattern, MAC address, driver, or device paths.


https://netplan.io/examples gives a worked example :
> ethernets:
> mainif:
> match:
> macaddress: "de:ad:be:ef:ca:fe"
> set-name: mainif

And mainif is used elsewhere in the config.


I also note that your config does not define any bridges.


Simon
RE: How to setup a multi vlan connection to an (Ubuntu) XEN-server !!?? [ In reply to ]
Simon, here some explanation and reaction to your comments

Lets start with the fact that I have to deal with two aspects:
1) Interfacing the host with the network
2) interfacing the host with the xen-hypervisor
The first problem to solve was how to interface the host with the network.

The host is connected to the network via vlan's (or untagged interface). Those vlans are there for security reasons. And traffic should NEVER be allowed to pass from vlan to another one in an endpoint like a server. Unless of course the server is used as firewall or managed switch. That is the reason that I absolutely require that traffic which enters via vlan-x is also answered via vlan-x.

As a consequence things like:
- a server wide default gateway is in principle a no go (I can live with it if it is assured by other means that traffic cannot leave a vlan)
- priority settings and routes OK but only withing the scope of a particular vlan (I do not use that here)
- as far as I know I need routes for incoming traffic and routing-policy for outgoing traffic. If I do not use both e.g. the answers to incoming traffic will be send through the host wide default gateway, where I want the answers send back via the same vlan as the incoming traffic arrived.

Related to MAC-addresses:
- of course I know the mac-addresses of the interfaces, however I do not see any reason to use them here on IP-level, ......... apart from one reason. My original intention was to change the interface names via "set-name", but apart that the syntax is bizarre (first that name and then the definition (strange), it did not work when using match name <interface name> ...... and it is clear that match on mac does work (ihmo it is a bug)

Related to IP-addresses:
- I partly agree. I completely agree that the IP-address of a VM should be defined in the VM. However, the network is also there in favor of the host and DOM0. And at least the host (ubuntu) does need an IP.
- for testing the vlans it is handy to have an ip assigned as well

The trunk does not need an address, we agree and also on the fact that I should use an RFC1918 ranges address

Related to routing in conjunction to DHCP
- The first NIC is used to setup the system and for emergency purposes. The NIC is connected to a normal PC-lan and it is most logical to have the IP assigned via DNCP. However in this case traffic arriving via this interfaces should be answered via this interface as well. So ideally I would define the routes and routing-policy for this interface as well ...
- however that seems to be impossible since the address is not static, unless I can do something with the "on-link" keyword.
- note that I tried that and was not successful (perhaps I did something wrong ??)

Related to naming:
- I completely agree that it is better to give interfaces, but also e.g. vlans sensible names. Not sure that is possible for .e.g. vlans

Policy based routing
- You could use that to send specific destinations to a particular interface, apart if that is a server or a network switch function, I do not need that here
- the reason I use it is that as far as I know is to force the host/vm answers back into the vlan from which the answer arrived

Related to Bridges
- I do not know yet I need them. That is because I do not yet know yet how to connect one or more vm's, or dom0, to the host its network as defined in the actual stage
- perhaps I can simply connect one or more vm's or dom0 to an interface or vlan, however perhaps I need to use a bridge for that. My feeling is that a bridge is the equivalent of an unmanged switch ... only required if you want to connect multiple (logical) devices (vm's !?) to an interface / vlan.


Louis


-----Original Message-----
From: Xen-users <xen-users-bounces@lists.xenproject.org> On Behalf Of Simon
Sent: Saturday, October 8, 2022 10:17 PM
To: xen-users@lists.xenproject.org
Subject: Re: How to setup a multi vlan connection to an (Ubuntu) XEN-server !!??

Sorry, a bit of rambling ...


louis.forums@xs4all.nl <lsrbreda@xs4all.nl> wrote:

> - you are using macaddresses. Why?

MAC addresses are used by bridges for a number of functions.
For STP, in the absence of specific priority settings, the MAC addresses will determine the network topology.
I would assume that if you want to attach an IP address to the bridge and use it as an interface from the host then it will need a MAC address.
There are probably other reasons I’m not aware of - quite likely the network stack needs it even if the host isn’t using it as an interface.

> - you do not assign ip-addresses. Why?

The host does not need an IP address in any plan/bridge it isn’t using for it’s own networking.

> - you do not specify vlan related gateways and routing tables .... ??

Again, not needed if the host is not actively using that interface for its own traffic

> - is het necessary to add a bridge if there is only one vm using a
> certain vlan

I believe so. In any case, it’s the simplest way to do it. The host can create the trunk interface, VLANs, and bridges at boot time, then they are ready for the the guests to attach to as and when required. Note that the guest side of the networking is not configured until the guest starts some time after the host network config is set up. Also, the guest side will be deleted if/when the guest is shut down.


> ## Issues left ##
> # 1) So I did not define a server wide default gateway ....
> # .... however that is not true if enp5s0 becomes active ....
> # .... I simply do not know how to assign a routing and routing-policy
> to # an dhcp provided address

I notice you have default routes configured on multiple interfaces. Normally this is incorrect - you’d only want a single default route, and potentially other specific routes via specific interfaces. Is this the reason for policy based routing (see below) ?
Are you forced to use DHCP for this ? If not, I would suggest static config would be preferable.

> # 2) The trunk is a container for vlans, it does not have an address ...
> # .... however I had to give it a dummy address to make this file work

So that’s a problem with the configuration tool because - as you point out - it absolutely does not need an IP address **unless** you wish to use it directly as a host interface. I would stick to using an address from the RFC1918 ranges BTW.


> The attached a jaml config file gives being an example for a server having three interfaces. One of the interfaces is the vlan trunk. In the attached file one vlan, but of course it is no problem to add other vlans or to add IPV6 (I assume).

I am curious why you are applying policy based routing - either there's some detail (probably not important to the discussion) you’ve omitted, or possibly it’s not needed. In many years, I’ve only needed policy based (source address) routing in one network - and that was a campus network where I had to handle 2 providers, separate IP ranges, and phased migration, hence needing to route via a specific interface according the IP address used by a connected client.
Ah, I see https://netplan.io/examples gives something like your config (key difference - example has different routing table numbers), but from the explanation I cannot fathom out what it’s saying ! I’m used to working with systems that handle what I think the page is saying “out of the box”, but since I don’t understand what problem is being fixed, I can’t see if this is relevant or not.

While on routing ...
The “on-link” statements are not needed as your router addresses are within the same subnets as the interface addresses. on-link is specifically for the case where this is not so - where normally the system would ignore the route as the router address is notionally unreachable. I think all you need is :
> routes:
> - to: default
> via: n.n.n.n
(https://netplan.io/examples)


BTW - my personal opinion is that you should really look into whether you can give the interfaces “sensible” names. Which enp3sf1 is not the worst of the naming schemes, you will really curse it the first time you change the host networking. As it is, you will need to find every instance of the old interface name in every config file its used in - it won’t just be in this file. And if you want to move a guest to a different host, you’ll need to change its config to suit the host’s interface names.
I’ve always done this via dev rules - and while some consider it heresy to use the MAC address as an interface identifier, it’s literally one line per interface and only one place to update if anything changes (replacement NIC, add/remove a card that renumbers other cards, move to new hardware, whatever).

Ah, I see from https://cloudinit.readthedocs.io/en/latest/topics/network-config-format-v2.html that it’s possible to do it with a match statement in the config file.
> In the generic case, they can be selected by match: rules on desired properties, such as name/name pattern, MAC address, driver, or device paths.


https://netplan.io/examples gives a worked example :
> ethernets:
> mainif:
> match:
> macaddress: "de:ad:be:ef:ca:fe"
> set-name: mainif

And mainif is used elsewhere in the config.


I also note that your config does not define any bridges.


Simon
Re: How to setup a multi vlan connection to an (Ubuntu) XEN-server !!?? [ In reply to ]
louis.forums@xs4all.nl <lsrbreda@xs4all.nl> wrote:

> Lets start with the fact that I have to deal with two aspects:
> 1) Interfacing the host with the network
> 2) interfacing the host with the xen-hypervisor
> The first problem to solve was how to interface the host with the network.
>
> The host is connected to the network via vlan's (or untagged interface). Those vlans are there for security reasons. And traffic should NEVER be allowed to pass from vlan to another one in an endpoint like a server. Unless of course the server is used as firewall or managed switch. That is the reason that I absolutely require that traffic which enters via vlan-x is also answered via vlan-x.

OK, the detail that wasn’t previously stated. In that case, yes you’ll be needing policy routing.
One thing worth considering is whether the host needs addresses in multiple VLANs, or whether you could reduce host access to just one address on one network. Just trying to think of ways you could simplify things - though I realise that there are probably reasons for having it as it is.
It’s certainly a valid (in terms of what the OS needs) config for the host to have no addresses on some or all of the networks used by guests.


> Related to MAC-addresses:
> - of course I know the mac-addresses of the interfaces, however I do not see any reason to use them here on IP-level, ......... apart from one reason. My original intention was to change the interface names via "set-name", but apart that the syntax is bizarre (first that name and then the definition (strange), it did not work when using match name <interface name> ...... and it is clear that match on mac does work (ihmo it is a bug)

I agree that “set-name” does seem redundant - but the docs do suggest that it’s needed. It’s not a huge task anyway, you’re only building this config once.
Matching by MAC address is, IMO, by far the most reliable way to do things. Unless you change the hardware (i.e. replace a NIC) then it’s not going to change - and if you do replace a NIC then you know that there’s only one - precisely ONE - place you need to change the MAC address to keep your network config the same.


> Related to naming:
> - I completely agree that it is better to give interfaces, but also e.g. vlans sensible names. Not sure that is possible for .e.g. vlans

It is - just use a meaningful name for the VLAN definition instead of (e.g.) “enp3s0f1.100”.

Based on what I recall of reading the docs, something like this should work :
# Define the trunk (2e 10G)
trunk1:
match:
macaddress: “aa:bb:cc:dd:ee:ff"
set-name: trunk1 # requires match (on mac !!??)


# Define vlan100 (vm-managment)
vlans:
vlan100:
id: 100
link: trunk1
addresses:
- 192.168.100.10/24
...


> Related to Bridges
> - I do not know yet I need them. That is because I do not yet know yet how to connect one or more vm's, or dom0, to the host its network as defined in the actual stage
> - perhaps I can simply connect one or more vm's or dom0 to an interface or vlan, however perhaps I need to use a bridge for that. My feeling is that a bridge is the equivalent of an unmanged switch ... only required if you want to connect multiple (logical) devices (vm's !?) to an interface / vlan.

Well you are connecting multiple devices - the host and at least one guest - to an interface.
I’ve always left the physical interface un-numbered and put an IP address on the bridge. I do know there was a reason for that, but I don’t recall what it was - possibly networking between host and guest doesn’t work properly if the host spits the packet out of the physical interface rather than into the bridge.

You are correct to liken the bridge to an un-managed switch - that is exactly what it is. In the world of separate boxes, you’d have a switch per VLAN, physically plug in network cables, then power up the device. In the virtualised world, when you start a guest, Xen plugs in virtual network cable(s) between the guest and switch(es), then starts the guest. With no guest running, it’s valid to have a bridge with only the physical port connected - analogous to having a real switch with only the uplink connected, waiting for you to plug devices into it.

TBH, I don’t know if you can do what you want without bridges - hopefully someone else can help with that. What I do know is that using bridges is trivially easy - create bridge, connect physical interface (or VLAN interface), connect guest(s) as required.

So putting the above together, I’d probably have a config along the lines of :

# Define the trunk (2e 10G)
trunk1:
match:
macaddress: “aa:bb:cc:dd:ee:ff"
set-name: trunk1 # requires match (on mac !!??)

# Define vlan100 (vm-managment)
vlans:
pvlan100:
id: 100
link: trunk1

# Bridge for VLAN 100
bridges:
vlan100:
interfaces:
- pvlan100
addresses:
- 192.168.100.10/24
...

That, barring typos and me not understanding the docs properly, should create you a bridge called “vlan100” attached to VLAN 100 on the interface with MAC “aa:bb:cc:dd:ee:ff”, and with address 192.168.100.10/24 (plus the routing stuff etc).


I think that’s me out of ideas now. As I may have mentioned, I’ve not done this sort of advanced networking for a few years now (changed jobs) and no longer have access to any of the configs to look at.


Simon