Mailing List Archive: Revisiting Slow Apt (actually disk write)

Revisiting Slow Apt (actually disk write)

Jul 26, 2021, 3:19 PM

Post #1 of 3 (257 views)

Sorry for the long message, but wanted to give a lack-of-progress report

Did a lot of troubleshooting but it hasn’t gotten me very far:

System: SuperMicro
        Xeon E3-1230 V3 @ 3.3 Ghz
       12 GB RAM
       Physical Disk: 4 x 2 TB ST2000NM0033 - Enterprise

Dom0 OS: Debian Buster 10.10, kernel 4.19.0-16-amd64
                 Xen 4.11

DomUs: All run debian buster as well. they are using Disk.Img for their
swap and root drives. All have at least 1 GB of RAM allocated. My test
VM has 2GB.

When I ran debian wheezy, I was not having this problem.

Dom0 is fast in disk read and writes.

Problem:   DomU's Seem to be very slow disk writes. For example, if I
just to an "apt update" this is my output:

Get:1 http://security.debian.org buster/updates InRelease [65.4 kB]
Get:2 http://ftp.us.debian.org/debian buster InRelease [122 kB]
Get:3 http://security.debian.org buster/updates/main Sources [195 kB]
Get:4 http://security.debian.org buster/updates/main amd64 Packages [297 kB]
Get:5 http://ftp.us.debian.org/debian buster/non-free Sources [85.7 kB]
Get:6 http://ftp.us.debian.org/debian buster/main Sources [7836 kB]
Get:7 http://ftp.us.debian.org/debian buster/contrib Sources [42.5 kB]
Get:8 http://ftp.us.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:9 http://ftp.us.debian.org/debian buster/main Translation-en [5968 kB]
Get:10 http://ftp.us.debian.org/debian buster/contrib amd64 Packages
[50.1 kB]
Get:11 http://ftp.us.debian.org/debian buster/contrib Translation-en
[44.2 kB]
Get:12 http://ftp.us.debian.org/debian buster/non-free amd64 Packages
[87.7 kB]
Get:13 http://ftp.us.debian.org/debian buster/non-free Translation-en
[88.9 kB]
Fetched 22.7 MB in 52min 3s (7277 B/s)
Reading package lists... Done

Notice that it says "52min 3s", this is something that Dom0 takes about
3 seconds to do to the same servers. Meaning, it is not a network issue.

What I've tried:
- I first went through the I/O tweaks from the xen wiki pages. Zero effect

- Moved all the DomUs off to another server except one (tester).

- Formatted and reinstalled entire OS from scratch on Dom0.

- Tried using LVM disk instead of disk images.   Slight improvement,
but still extremely slow.

- Tried using physical disk, dedicating one drive entirely to the test
DomU, mounting with "phys" in the config.   This was also a slight
improvement, but still very slow disk writes.

"top" on the DomU shows WA very high while trying to get the update.
Drops to <1 when idle. All other "top" values are <1.

I thought I solved it by going back to an earlier kernel,
4.19.0-14-amd64, and I tried doing that again and it had zero effect
this time.

Is it possible that there is a kernel flag that is turned off that would
effect VM disk writes?   Reading seems to be normal speed.   Writing to
a network drive from the DomU is very fast as well.

Flag Info from /proc/cpu on Dom0

flags        : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush
acpi mmx fxsr sse sse2 ss ht syscall nx

lm constant_tsc rep_good nopl nonstop_tsc cpuid pni pclmulqdq monitor
est ssse3 cx16 sse4_1 sse4_2

popcnt xsave avx f16c rdrand hypervisor lahf_lm cpuid_fault ssbd ibrs
ibpb stibp fsgsbase erms xsaveopt md_clear

Re: Revisiting Slow Apt (actually disk write) [ In reply to ]

chrismyers81 at gmail

Jul 27, 2021, 10:42 AM

Post #2 of 3 (257 views)

Permalink

Out of curiosity is it all disk activity or just apt-related? Eg. how does

dd if=/dev/zero of=blah bs=1M count=512

look?

I'm running a similar-ish config as you (asrock "J" motherboards and
SSD-backed LVM storage instead of regular hard drives) and thus far am not
experiencing that specific issue.

I have had weird stuff happen with other motherboards however, in one case
where activity on any domu (including dom0) incurred a large "st"eal,
unless the VM was pinned to one specific dedicated CPU. I never did figure
that one out; even went as far as turning off all ACPI stuff just in case
it was interfering.

On Mon, Jul 26, 2021 at 5:19 PM TheBearAK <thebearak@gmail.com> wrote:

> Sorry for the long message, but wanted to give a lack-of-progress report
>
> Did a lot of troubleshooting but it hasn’t gotten me very far:
>
>
> System: SuperMicro
> Xeon E3-1230 V3 @ 3.3 Ghz
> 12 GB RAM
> Physical Disk: 4 x 2 TB ST2000NM0033 - Enterprise
>
> Dom0 OS: Debian Buster 10.10, kernel 4.19.0-16-amd64
> Xen 4.11
>
> DomUs: All run debian buster as well. they are using Disk.Img for their
> swap and root drives. All have at least 1 GB of RAM allocated. My test
> VM has 2GB.
>
> When I ran debian wheezy, I was not having this problem.
>
> Dom0 is fast in disk read and writes.
>
> Problem: DomU's Seem to be very slow disk writes. For example, if I
> just to an "apt update" this is my output:
>
> Get:1 http://security.debian.org buster/updates InRelease [65.4 kB]
> Get:2 http://ftp.us.debian.org/debian buster InRelease [122 kB]
> Get:3 http://security.debian.org buster/updates/main Sources [195 kB]
> Get:4 http://security.debian.org buster/updates/main amd64 Packages [297
> kB]
> Get:5 http://ftp.us.debian.org/debian buster/non-free Sources [85.7 kB]
> Get:6 http://ftp.us.debian.org/debian buster/main Sources [7836 kB]
> Get:7 http://ftp.us.debian.org/debian buster/contrib Sources [42.5 kB]
> Get:8 http://ftp.us.debian.org/debian buster/main amd64 Packages [7907 kB]
> Get:9 http://ftp.us.debian.org/debian buster/main Translation-en [5968 kB]
> Get:10 http://ftp.us.debian.org/debian buster/contrib amd64 Packages
> [50.1 kB]
> Get:11 http://ftp.us.debian.org/debian buster/contrib Translation-en
> [44.2 kB]
> Get:12 http://ftp.us.debian.org/debian buster/non-free amd64 Packages
> [87.7 kB]
> Get:13 http://ftp.us.debian.org/debian buster/non-free Translation-en
> [88.9 kB]
> Fetched 22.7 MB in 52min 3s (7277 B/s)
> Reading package lists... Done
>
> Notice that it says "52min 3s", this is something that Dom0 takes about
> 3 seconds to do to the same servers. Meaning, it is not a network issue.
>
> What I've tried:
> - I first went through the I/O tweaks from the xen wiki pages. Zero
> effect
>
> - Moved all the DomUs off to another server except one (tester).
>
> - Formatted and reinstalled entire OS from scratch on Dom0.
>
> - Tried using LVM disk instead of disk images. Slight improvement,
> but still extremely slow.
>
> - Tried using physical disk, dedicating one drive entirely to the test
> DomU, mounting with "phys" in the config. This was also a slight
> improvement, but still very slow disk writes.
>
>
> "top" on the DomU shows WA very high while trying to get the update.
> Drops to <1 when idle. All other "top" values are <1.
>
> I thought I solved it by going back to an earlier kernel,
> 4.19.0-14-amd64, and I tried doing that again and it had zero effect
> this time.
>
>
> Is it possible that there is a kernel flag that is turned off that would
> effect VM disk writes? Reading seems to be normal speed. Writing to
> a network drive from the DomU is very fast as well.
>
>
> Flag Info from /proc/cpu on Dom0
>
> flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush
> acpi mmx fxsr sse sse2 ss ht syscall nx
>
> lm constant_tsc rep_good nopl nonstop_tsc cpuid pni pclmulqdq monitor
> est ssse3 cx16 sse4_1 sse4_2
>
> popcnt xsave avx f16c rdrand hypervisor lahf_lm cpuid_fault ssbd ibrs
> ibpb stibp fsgsbase erms xsaveopt md_clear
>
>
>

--
People use duct tape to fix everything....God used nails.

http://www.myerscountry.net

Re: Revisiting Slow Apt (actually disk write) [ In reply to ]

thebearak at gmail

Jul 27, 2021, 9:32 PM

Post #3 of 3 (257 views)

Permalink

All disk write activity.   Read is within normal.

On 7/27/21 9:42 AM, Chris Myers wrote:
> Out of curiosity is it all disk activity or just apt-related? Eg. how
> does
>
> dd if=/dev/zero of=blah bs=1M count=512
>
> look?
>
> I'm running a similar-ish config as you (asrock "J" motherboards and
> SSD-backed LVM storage instead of regular hard drives) and thus far am
> not experiencing that specific issue.
>
> I have had weird stuff happen with other motherboards however, in one
> case where activity on any domu (including dom0) incurred a large
> "st"eal, unless the VM was pinned to one specific dedicated CPU. I
> never did figure that one out; even went as far as turning off all
> ACPI stuff just in case it was interfering.
>
> On Mon, Jul 26, 2021 at 5:19 PM TheBearAK <thebearak@gmail.com
> <mailto:thebearak@gmail.com>> wrote:
>
> Sorry for the long message, but wanted to give a lack-of-progress
> report
>
> Did a lot of troubleshooting but it hasn’t gotten me very far:
>
>
> System: SuperMicro
>          Xeon E3-1230 V3 @ 3.3 Ghz
>         12 GB RAM
>         Physical Disk: 4 x 2 TB ST2000NM0033 - Enterprise
>
> Dom0 OS: Debian Buster 10.10, kernel 4.19.0-16-amd64
>                   Xen 4.11
>
> DomUs: All run debian buster as well. they are using Disk.Img for
> their
> swap and root drives. All have at least 1 GB of RAM allocated.
> My test
> VM has 2GB.
>
> When I ran debian wheezy, I was not having this problem.
>
> Dom0 is fast in disk read and writes.
>
> Problem:   DomU's Seem to be very slow disk writes. For example,
> if I
> just to an "apt update" this is my output:
>
> Get:1 http://security.debian.org <http://security.debian.org>
> buster/updates InRelease [65.4 kB]
> Get:2 http://ftp.us.debian.org/debian
> <http://ftp.us.debian.org/debian> buster InRelease [122 kB]
> Get:3 http://security.debian.org <http://security.debian.org>
> buster/updates/main Sources [195 kB]
> Get:4 http://security.debian.org <http://security.debian.org>
> buster/updates/main amd64 Packages [297 kB]
> Get:5 http://ftp.us.debian.org/debian
> <http://ftp.us.debian.org/debian> buster/non-free Sources [85.7 kB]
> Get:6 http://ftp.us.debian.org/debian
> <http://ftp.us.debian.org/debian> buster/main Sources [7836 kB]
> Get:7 http://ftp.us.debian.org/debian
> <http://ftp.us.debian.org/debian> buster/contrib Sources [42.5 kB]
> Get:8 http://ftp.us.debian.org/debian
> <http://ftp.us.debian.org/debian> buster/main amd64 Packages [7907 kB]
> Get:9 http://ftp.us.debian.org/debian
> <http://ftp.us.debian.org/debian> buster/main Translation-en [5968 kB]
> Get:10 http://ftp.us.debian.org/debian
> <http://ftp.us.debian.org/debian> buster/contrib amd64 Packages
> [50.1 kB]
> Get:11 http://ftp.us.debian.org/debian
> <http://ftp.us.debian.org/debian> buster/contrib Translation-en
> [44.2 kB]
> Get:12 http://ftp.us.debian.org/debian
> <http://ftp.us.debian.org/debian> buster/non-free amd64 Packages
> [87.7 kB]
> Get:13 http://ftp.us.debian.org/debian
> <http://ftp.us.debian.org/debian> buster/non-free Translation-en
> [88.9 kB]
> Fetched 22.7 MB in 52min 3s (7277 B/s)
> Reading package lists... Done
>
> Notice that it says "52min 3s", this is something that Dom0 takes
> about
> 3 seconds to do to the same servers. Meaning, it is not a network
> issue.
>
> What I've tried:
>   - I first went through the I/O tweaks from the xen wiki pages.
> Zero effect
>
>   - Moved all the DomUs off to another server except one (tester).
>
>   - Formatted and reinstalled entire OS from scratch on Dom0.
>
>   - Tried using LVM disk instead of disk images.   Slight
> improvement,
> but still extremely slow.
>
>   - Tried using physical disk, dedicating one drive entirely to
> the test
> DomU, mounting with "phys" in the config.   This was also a slight
> improvement, but still very slow disk writes.
>
>
> "top" on the DomU shows WA very high while trying to get the update.
> Drops to <1 when idle. All other "top" values are <1.
>
> I thought I solved it by going back to an earlier kernel,
> 4.19.0-14-amd64, and I tried doing that again and it had zero effect
> this time.
>
>
> Is it possible that there is a kernel flag that is turned off that
> would
> effect VM disk writes?   Reading seems to be normal speed. Writing to
> a network drive from the DomU is very fast as well.
>
>
> Flag Info from /proc/cpu on Dom0
>
> flags        : fpu de tsc msr pae mce cx8 apic sep mca cmov pat
> clflush
> acpi mmx fxsr sse sse2 ss ht syscall nx
>
> lm constant_tsc rep_good nopl nonstop_tsc cpuid pni pclmulqdq monitor
> est ssse3 cx16 sse4_1 sse4_2
>
> popcnt xsave avx f16c rdrand hypervisor lahf_lm cpuid_fault ssbd ibrs
> ibpb stibp fsgsbase erms xsaveopt md_clear
>
>
>
>
> --
> People use duct tape to fix everything....God used nails.
>
> http://www.myerscountry.net <http://www.myerscountry.net>