Mailing List Archive: VMs randomly become to read-only mode

VMs randomly become to read-only mode

Jul 21, 2020, 8:12 AM

Post #1 of 3 (342 views)

Hi, time to time we're facing with the weird situations when VMs are
coming to readonly mode:

Eg. /dev/vda is allowed to read but not for write:

(initramfs) dd if=/dev/vda of=/tmp/1 bs=1 count=1
1+0 records in
1+0 records out
(initramfs) dd if=/tmp/1 if=/dev/vda bs=1 count=1
[ 1335.967438] print_req_error: I/O error, dev vda, sector 0
[ 1335.969843] Buffer I/O error on dev vda, logical block 0, lost
async page write
1+0 records in
1+0 records out
(initramfs)

Today about 8 VMs in different clusters become to this state, so I had
a time to experiment with them:

- dmesg messages are clean
- drbd state is seems ok:

one-vm-604-disk-2 role:Primary
disk:UpToDate
m1c10 role:Secondary
peer-disk:UpToDate
m1c7 role:Secondary
peer-disk:Diskless

- from the host side the disk is accessible and writable
- if VM has many disks all of them will become readonly
- simple reboot does not solves the problem, poweroff then resume does.
- virsh domfsthaw returns:

error: Unable to thaw filesystems
error: Guest agent is not responding: QEMU guest agent is not connected

Has anyone the similar cases?

Best Regards,
Andrei Kvapil
_______________________________________________
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user

Re: VMs randomly become to read-only mode [ In reply to ]

kvapss at gmail

Jul 21, 2020, 8:50 AM

Post #2 of 3 (342 views)

Permalink

I was wrong about few facts:

- if VM has many disks all of them will become readonly
I just found the VM with the same symptoms but only one disk were in
read-only mode

- virsh domfsthaw returns error
for this vm it is working correctly, but no change and disk is still
not writable afterwards

I can also append that usually this is happening after the snapshot
operations on drive (we are using linstor with lvmthin backend)

drbd version:

version: 9.0.23-1 (api:2/proto:86-116)
GIT-hash: d16bfab7a4033024fed2d99d3b179aa6bb6eb300 build by
@runner-cpuncbtu-project-69-concurrent-15tjkf, 2020-06-09 23:46:40
Transports (api:16): tcp (9.0.23-1)

Best Regards,
Andrei Kvapil

On Tue, Jul 21, 2020 at 5:12 PM kvaps <kvapss@gmail.com> wrote:
>
> Hi, time to time we're facing with the weird situations when VMs are
> coming to readonly mode:
>
> Eg. /dev/vda is allowed to read but not for write:
>
> (initramfs) dd if=/dev/vda of=/tmp/1 bs=1 count=1
> 1+0 records in
> 1+0 records out
> (initramfs) dd if=/tmp/1 if=/dev/vda bs=1 count=1
> [ 1335.967438] print_req_error: I/O error, dev vda, sector 0
> [ 1335.969843] Buffer I/O error on dev vda, logical block 0, lost
> async page write
> 1+0 records in
> 1+0 records out
> (initramfs)
>
> Today about 8 VMs in different clusters become to this state, so I had
> a time to experiment with them:
>
> - dmesg messages are clean
> - drbd state is seems ok:
>
> one-vm-604-disk-2 role:Primary
> disk:UpToDate
> m1c10 role:Secondary
> peer-disk:UpToDate
> m1c7 role:Secondary
> peer-disk:Diskless
>
> - from the host side the disk is accessible and writable
> - if VM has many disks all of them will become readonly
> - simple reboot does not solves the problem, poweroff then resume does.
> - virsh domfsthaw returns:
>
> error: Unable to thaw filesystems
> error: Guest agent is not responding: QEMU guest agent is not connected
>
> Has anyone the similar cases?
>
> Best Regards,
> Andrei Kvapil
_______________________________________________
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user

Re: VMs randomly become to read-only mode [ In reply to ]

roland.kammerer at linbit

Jul 22, 2020, 1:06 AM

Post #3 of 3 (338 views)

Permalink

On Tue, Jul 21, 2020 at 05:12:37PM +0200, kvaps wrote:
> Hi, time to time we're facing with the weird situations when VMs are
> coming to readonly mode:
>
> - dmesg messages are clean

Can you double check the dmesg? It sounds a bit like they might have
lost quorum. Should be easily grep-able.

Best, rck
_______________________________________________
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user