Mailing List Archive: "swiotlb buffer is full" in DomU with PCI passthrough is caused by low memory in Dom0

"swiotlb buffer is full" in DomU with PCI passthrough is caused by low memory in Dom0

Apr 25, 2023, 1:30 PM

Post #1 of 3 (148 views)

Hi,

this message is meant mainly to document an issue and the solution to
help others in a similar situation.

I came across this situation when installing Xen on a small PC (PC
Engines APU4D4) with 4 GB of RAM and passing through a PCI device (a
Compex WLE900VX wifi card) to a OpenWRT DomU. In the beginning, I wanted
to use as little memory as possible for the Debian Dom0, therefore I set
its memory to 512 MB by using "dom0_mem=512M,max:512M". When trying to
pass through the PCI card to the DomU, the DomU could see it, but it did
not work. Dmesg output in the OpenWRT DomU showed several instances of
the following line:

"ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112 bytes), total
2048 (slots), used 2034 (slots)"

Everything else in DomU seemed to work. Searching the web for this error
message did not give me an indication for the cause. I also could not
find any cause in the Dom0 logs. Therefore, I could not find the reason
for this issue for quite some time.

I then randomly tried out to enlarge Dom0 memory to 768 MB, and the
error message in the DomU disappeared during the next boot. The PCI
device was working.

I can't rule out that there actually is a log entry in Dom0 for this
situation, and my Xen and/or linux knowledge was not good enough to let
me find it. If this is the case, I'd appreciate some pointers.

Best regards,

Paul

Re: "swiotlb buffer is full" in DomU with PCI passthrough is caused by low memory in Dom0 [ In reply to ]

slack at rabbit

Apr 28, 2023, 8:28 AM

Post #2 of 3 (147 views)

Permalink

On 25 Apr 2023 22:30, Paul Leiber wrote:
> Hi,

Hi,

> I can't rule out that there actually is a log entry in Dom0 for this
> situation, and my Xen and/or linux knowledge was not good enough to let
> me find it. If this is the case, I'd appreciate some pointers.

Then why not posting your logs ?

Did you add "loglvl=all" and/or "guest_loglvl=all" to Xen cmdline ?
What logs did you read: "dmesg", "xl dmesg" or logs in /var/log/xen ?

From the (few) posts I've read, this problems happens on
non-virtualized systems, so maybe it's not related to Xen at all.
Maybe it's a driver problem.
(Search query used: "swiotlb buffer is full").

Also I found pointers related to the *domU* kernel command line option
"swiotlb", like using "swiotlb=1" or "swiotlb=65536"

PS: I'm not an expert, just a Xen user ;)

Re: "swiotlb buffer is full" in DomU with PCI passthrough is caused by low memory in Dom0 [ In reply to ]

paul at onlineschubla

Apr 28, 2023, 9:05 AM

Post #3 of 3 (147 views)

Permalink

Am 28.04.2023 um 17:28 schrieb zithro:
> On 25 Apr 2023 22:30, Paul Leiber wrote:
>> Hi,
>
> Hi,
>
>> I can't rule out that there actually is a log entry in Dom0 for this
>> situation, and my Xen and/or linux knowledge was not good enough to
>> let me find it. If this is the case, I'd appreciate some pointers.
>
> Then why not posting your logs ?

I didn't find anything that seemed relevant, that's why I didn't post
any logs. Also, I am not sure which logs are relevant (see below).

As this is from a "production" machine (the family internet connection
gateway at home), I need to literally beg for time windows where I can
do the testing, hence I am hesitant to promise any logs soon.

That put aside: Which logs would be relevant? The ones below, or
more/others?

> Did you add "loglvl=all" and/or "guest_loglvl=all" to Xen cmdline ?

Yes.

> What logs did you read: "dmesg", "xl dmesg" or logs in /var/log/xen ?

I looked at the logs in /var/log/xen and dmesg, don't remember if a also
looked at "xl dmesg".

> From the (few) posts I've read, this problems happens on
> non-virtualized systems, so maybe it's not related to Xen at all.
> Maybe it's a driver problem.
> (Search query used: "swiotlb buffer is full").

I also noticed that most reports of swiotlib issues are for
non-virtualized systems. But: If my reported issue was not Xen related,
why would changing the amount of Dom0 memory influence DomU behavior?

> Also I found pointers related to the *domU* kernel command line option
> "swiotlb", like using "swiotlb=1" or "swiotlb=65536"

Interesting, thank you for the hint.

> PS: I'm not an expert, just a Xen user ;)

Same, although I still consider myself a Xen beginner. :-)