Mailing List Archive

Packets not passing from domU to dom0
Hi all!



I have spent hours debugging network issues between domU and dom0 - without success. I hope you have some ideas what could be the cause of the problem, or you can give me some hints how to debug this issue.



Situation: dom0 and PV domU are Ubuntu22.04 with XEN

4.16.0-1~ubuntu2.1(I had the same issues with Ubuntu 20.04 and XEN

4.11.3+24-g14b62ab3e5-1ubuntu2.3). The dom0 was setup by my provider

where I rent this baremetal server. We have 20+ other servers with other providers

with more or less identical setup and without issues.





dom0 vif1.0 <---------------> eth0 domU

192.168.0.1/24 192.168.0.2/24



No nftables, iptables, ebtables.



Packets pass from dom0 to domU, but almost not from domU to dom0.

"Almost not" means, that once in a while a packet pass through from the

domU to the dom0.



For example, dom0 pings 102.168.0.2. I use tcpdump in dom0 with "-i

vif1.0" and in domU with "-i eth0".



In the dom0 I see the ARP request for 192.168.0.2.

In the domU I see the ARP request for 192.168.0.2 arriving and I see the

ARP response back.

But the ARP response does not arrive in the dom0 (I do not see it in

tcpdump).



Once in a while the ARP response passes through to the domU. Then the

dom0 send ICMP ping request, which arrives in the domU, and the domU

responds with an ICMP response which does not arrive in the domU. Again,

once in a while a ping response arrives in the domU.



RX/TX statistics in the domU show plenty of packets coming in and sent

out.

RX/TX statistics in the dom0 show plenty of packets sent out, but only a

few packets coming in. For example TX=5000 packets, RX=20 packets.

No "dropped" or "error" indications.



As said before I have no idea where or why the packets do not pass

through to the dom0. I have plenty of identical setups on other servers.

I have enabled log_martians, checked verious sysctl setttings.

Everything looks fine. Also no indications in the kernel logs.





Questions: First, I would like to find out where the packets get dropped/filtered -

in the vif eth0 of the domU, in the vif vif1.0 of the dom0, or somewhere

inbetween?



Who forwards packets between the two vifs - the hypervisor or the dom0?



Can I somehow debug if a packet enters the vif but get silenty dropped?



Thanks

Klaus


--
Klaus Darilion, Head of Operations
nic.at GmbH, Jakob-Haringer-Stra?e 8/V
5020 Salzburg, Austria
Re: Packets not passing from domU to dom0 [ In reply to ]
On 1/24/2023 6:50 AM, Klaus Darilion wrote:
>
> Hi all!
>
>  
>
> I have spent hours debugging network issues between domU and dom0 - without success. I hope you have some ideas what could be the cause of the problem, or you can give me some hints how to debug this issue.
>
>  
>
> Situation: dom0 and PV domU are Ubuntu22.04 with XEN
>
> 4.16.0-1~ubuntu2.1(I had the same issues with Ubuntu 20.04 and XEN
>
> 4.11.3+24-g14b62ab3e5-1ubuntu2.3). The dom0 was setup by my provider
>
> where I rent this baremetal server. We have 20+ other servers with other providers
>
> with more or less identical setup and without issues.
>
>  
>
>  
>
> dom0  vif1.0 <---------------> eth0 domU
>
>   192.168.0.1/24            192.168.0.2/24
>
>  
>
> No nftables, iptables, ebtables.
>
>  
>
> Packets pass from dom0 to domU, but almost not from domU to dom0.
>
> "Almost not" means, that once in a while a packet pass through from the
>
> domU to the dom0.
>
>  
>
> For example, dom0 pings 102.168.0.2. I use tcpdump in dom0 with "-i
>
> vif1.0" and in domU with "-i eth0".
>
>  
>
> In the dom0 I see the ARP request for 192.168.0.2.
>
> In the domU I see the ARP request for 192.168.0.2 arriving and I see the
>
> ARP response back.
>
> But the ARP response does not arrive in the dom0 (I do not see it in
>
> tcpdump).
>
>  
>
> Once in a while the ARP response passes through to the domU. Then the
>
> dom0 send ICMP ping request, which arrives in the domU, and the domU
>
> responds with an ICMP response which does not arrive in the domU. Again,
>
> once in a while a ping response arrives in the domU.
>
>  
>
> RX/TX statistics in the domU show plenty of packets coming in and sent
>
> out.
>
> RX/TX statistics in the dom0 show plenty of packets sent out, but only a
>
> few packets coming in. For example TX=5000 packets, RX=20 packets.
>
> No "dropped" or "error" indications.
>
>  
>
> As said before I have no idea where or why the packets do not pass
>
> through to the dom0. I have plenty of identical setups on other servers.
>
> I have enabled log_martians, checked verious sysctl setttings.
>
> Everything looks fine. Also no indications in the kernel logs.
>
>  
>
>  
>
> Questions: First, I would like to find out where the packets get dropped/filtered -
>
> in the vif eth0 of the domU, in the vif vif1.0 of the dom0, or somewhere
>
> inbetween?
>

IIUC it is a direct connection between the vif in the dom0 and the
eth0 in the domU. But networking scripts that run in dom0 shipped
by Xen can introduce some iptables rules that can filter things. See,
for example:

http://xenbits.xen.org/gitweb/?p=xen.git;a=tree;f=tools/hotplug/Linux;h=1e5e4f9af283c68626fc518559f593eee810b629;hb=HEAD

In there is a bunch of hotplug scripts for Linux that run in dom0,
including the networking scripts (vif-common, vif-bridge, vif-route,
etc). Some of them, such as vif-common.sh, do setup some iptables
rules, among other things. These scripts don't always work in every
networking setup and sometimes they need to be tweaked on a
particular server in a particular networking setup. Your provider must
not have setup those vif-* scripts correctly for the network setup of
your provider. You should probably report the problem to the provider
and simply tell them if they can't fix it you will just use another provider.

>  
>
> Who forwards packets between the two vifs - the hypervisor or the dom0?
>
>  
>
> Can I somehow debug if a packet enters the vif but get silenty dropped?
>
>  
>
> Thanks
>
> Klaus
>
>  
>
>  
>
> --
>
> Klaus Darilion, Head of Operations
>
> nic.at GmbH, Jakob-Haringer-Straße 8/V
>
> 5020 Salzburg, Austria
>
>  
>
Re: Packets not passing from domU to dom0 [ In reply to ]
Hi!

> I have spent hours debugging network issues between domU and dom0 - without
> success. I hope you have some ideas what could be the cause of the problem,
> or you can give me some hints how to debug this issue.
We had a very similar issue with an incomplete patch for XSA-423. This
is a fix in the linux kernel and V1 of the advisory lacked the
initialization a variable[1][2]
Maybe the kernel lacks the second part of the patch?

-- Adi

[1] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?
id=7dfa764e0223a324366a2a1fc056d4d9d4e95491
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1026035
AW: Packets not passing from domU to dom0 [ In reply to ]
> -----Ursprüngliche Nachricht-----
> Von: Xen-users <xen-users-bounces@lists.xenproject.org> Im Auftrag von
> Adi Kriegisch
> Gesendet: Mittwoch, 25. Jänner 2023 11:10
> An: xen-users@lists.xenproject.org
> Cc: Klaus Darilion <klaus.darilion@nic.at>
> Betreff: Re: Packets not passing from domU to dom0
>
> Hi!
>
> > I have spent hours debugging network issues between domU and dom0 -
> without
> > success. I hope you have some ideas what could be the cause of the
> problem,
> > or you can give me some hints how to debug this issue.
> We had a very similar issue with an incomplete patch for XSA-423. This
> is a fix in the linux kernel and V1 of the advisory lacked the
> initialization a variable[1][2]
> Maybe the kernel lacks the second part of the patch?
>
> -- Adi
>
> [1]
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/
> ?
> id=7dfa764e0223a324366a2a1fc056d4d9d4e95491
> [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1026035

Thanks for the hint.

Meanwhile we fixed it by using an old Ubuntu 22.04 kernel in the dom0. So we now upgrade the kernel minor version by minor version to find out when the problem appeared first.

regards
Klaus
AW: Packets not passing from domU to dom0 [ In reply to ]
> Hi!
>
> > I have spent hours debugging network issues between domU and dom0 -
> without
> > success. I hope you have some ideas what could be the cause of the
> problem,
> > or you can give me some hints how to debug this issue.
> We had a very similar issue with an incomplete patch for XSA-423. This
> is a fix in the linux kernel and V1 of the advisory lacked the
> initialization a variable[1][2]
> Maybe the kernel lacks the second part of the patch?

It seems that Ubuntu has not backported the second part too:
http://changelogs.ubuntu.com/changelogs/pool/main/l/linux/linux_5.15.0-58.64/changelog only mentions the first part.

I guess this is due the fact that the commit message is named "fix build warning" and did nit received much attention.

I will open an issue with Ubuntu.

thanks
Klaus
AW: Packets not passing from domU to dom0 [ In reply to ]
> I will open an issue with Ubuntu.

For the records: there is already a bug report for this issue. I have added the details about the problem.
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2002889

regards
Klaus