Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. DRBD>
  3. Users
Proxmox:Linstore, new third node, all volumes diskless, why?
volker.jaenisch at inqbus
Dec 2, 2020, 12:08 PM
Post #1 of 4 (450 views)
Permalink
Dear DRBD Users!

I have setup a proxmox 6.2 cluster linstore 1.9, linstore-proxmox with
two nodes Hydra3/4 two months ago. Works like a charm.

Now I added a third node Hydra1 (proxmox 6.3, linstore 1.9,
linstore-proxmox).

[.This project is a Proxmox 4->6 migration. I migrated all VMs from
Hydra1/2 (Proxmox4) by hand to our new Hydra3/4 (Proxmox6) servers and
now reintegrate Hydra1/2]

Also no problem. But all the volumes are now and remain in diskless
state on Hydra1?

DRBDTOP v0.2.3-4-g5fa4b14 (kernel: 9.0.25; utils: 9.15.1; host: hydra1)
- Details for vm-118-disk-1

Resource: vm-118-disk-1: (Overall danger score:
0)                                                                                          

? Local
Disc(*Secondary*):                                                                                                                                    

?  volume 0 (/dev/drbd1023): *Diskless*(detached from backing
disk)                                                                 
?                                                                                                                                                                         

? Connection to hydra3(*Primary*): Connected(connected to
hydra3)                                                             
?  volume
0:                                                                                                                                                        

?   UpToDate(normal disk
state)                                                                                                                          

?                                                                                                                                                                          

? Connection to hydra4(Secondary): Connected(connected to
hydra4)                                                              
?  volume
0:                                                                                                                                                         

?   UpToDate(normal disk state)         

The 3-Node cluster is fully operational, creation of VMs and live
migration between all three node works perfectly.

So when I migrate e.g. VM118 from Hydra3 to the new Hydra1 the VMs
switches really fine. But in the end the DRBD-Device is still diskless
on Hydra1

?Resource: vm-118-disk-1: (Overall danger score:
0)                                                                                                                                         

? Local
Disc(*Primary*):                                                                                                                                                                      

?  volume 0 (/dev/drbd1023): *Diskless*(detached from backing
disk)                                                                                                                           

?                                                                                                                                                                                           

? Connection to hydra3(*Secondary*): Connected(connected to
hydra3)                                                                                                                           

?  volume
0:                                                                                                                                                                                

?   UpToDate(normal disk
state)                                                                                                                                                             

?                                                                                                                                                                                           

? Connection to hydra4(Secondary): Connected(connected to
hydra4)                                                                                                                           

?  volume
0:                                                                                                                                                                                

?   UpToDate(normal disk state)

Stupid question: How do I get the volumes also DRBD-replicated onto the
third node? I thought that this would be done in the moment when I start
a migration.

/etc/pve/storage.cfg

drbd: drbdstorage
    content images, rootdir
    controller 10.1.3.18
    resourcegroup ResGrpDRBD

root@hydra1:~# linstor rg list
????????????????????????????????????????????????????????????????????
? ResourceGroup ? SelectFilter              ? VlmNrs ? Description ?
????????????????????????????????????????????????????????????????????
? DfltRscGrp    ? PlaceCount: 2             ?        ?             ?
????????????????????????????????????????????????????????????????????
? ResGrpDRBD    ? PlaceCount: 2             ? 0      ?             ?
?               ? StoragePool(s): pool_drbd ?        ?             ?
????????????????????????????????????????????????????????????????????

root@hydra1:~# linstor volume-definition list
?????????????????????????????????????????????????????????????????????
? ResourceName  ? VolumeNr ? VolumeMinor ? Size     ? Gross ? State ?
?????????????????????????????????????????????????????????????????????
? vm-100-disk-1 ? 0        ? 1000        ? 50 GiB   ?       ? ok    ?
? vm-100-disk-2 ? 0        ? 1001        ? 900 GiB  ?       ? ok    ?
? vm-101-disk-1 ? 0        ? 1002        ? 100 GiB  ?       ? ok    ?
? vm-102-disk-1 ? 0        ? 1003        ? 16 GiB   ?       ? ok    ?
? vm-103-disk-1 ? 0        ? 1004        ? 32 GiB   ?       ? ok    ?
? vm-104-disk-1 ? 0        ? 1005        ? 32 GiB   ?       ? ok    ?
? vm-105-disk-1 ? 0        ? 1006        ? 50 GiB   ?       ? ok    ?
? vm-105-disk-2 ? 0        ? 1007        ? 100 GiB  ?       ? ok    ?
? vm-106-disk-1 ? 0        ? 1008        ? 50 GiB   ?       ? ok    ?
? vm-107-disk-1 ? 0        ? 1009        ? 32 GiB   ?       ? ok    ?
? vm-107-disk-2 ? 0        ? 1010        ? 80 GiB   ?       ? ok    ?
? vm-108-disk-1 ? 0        ? 1011        ? 32 GiB   ?       ? ok    ?
? vm-109-disk-1 ? 0        ? 1012        ? 100 GiB  ?       ? ok    ?
? vm-110-disk-1 ? 0        ? 1013        ? 100 GiB  ?       ? ok    ?
? vm-110-disk-2 ? 0        ? 1014        ? 1.46 TiB ?       ? ok    ?
? vm-111-disk-1 ? 0        ? 1015        ? 50 GiB   ?       ? ok    ?
? vm-112-disk-1 ? 0        ? 1016        ? 32 GiB   ?       ? ok    ?
? vm-113-disk-1 ? 0        ? 1017        ? 50 GiB   ?       ? ok    ?
? vm-114-disk-1 ? 0        ? 1018        ? 50 GiB   ?       ? ok    ?
? vm-115-disk-1 ? 0        ? 1019        ? 22 GiB   ?       ? ok    ?
? vm-115-disk-2 ? 0        ? 1020        ? 100 GiB  ?       ? ok    ?
? vm-116-disk-1 ? 0        ? 1021        ? 32 GiB   ?       ? ok    ?
? vm-117-disk-1 ? 0        ? 1022        ? 32 GiB   ?       ? ok    ?
? vm-118-disk-1 ? 0        ? 1023        ? 32 GiB   ?       ? ok    ?
?????????????????????????????????????????????????????????????????????

root@hydra1:~# linstor resource-definition list
????????????????????????????????????????????????
? ResourceName  ? Port ? ResourceGroup ? State ?
????????????????????????????????????????????????
? vm-100-disk-1 ? 7000 ? ResGrpDRBD    ? ok    ?
? vm-100-disk-2 ? 7001 ? ResGrpDRBD    ? ok    ?
? vm-101-disk-1 ? 7002 ? ResGrpDRBD    ? ok    ?
? vm-102-disk-1 ? 7003 ? ResGrpDRBD    ? ok    ?
? vm-103-disk-1 ? 7004 ? ResGrpDRBD    ? ok    ?
? vm-104-disk-1 ? 7005 ? ResGrpDRBD    ? ok    ?
? vm-105-disk-1 ? 7006 ? ResGrpDRBD    ? ok    ?
? vm-105-disk-2 ? 7007 ? ResGrpDRBD    ? ok    ?
? vm-106-disk-1 ? 7008 ? ResGrpDRBD    ? ok    ?
? vm-107-disk-1 ? 7009 ? ResGrpDRBD    ? ok    ?
? vm-107-disk-2 ? 7010 ? ResGrpDRBD    ? ok    ?
? vm-108-disk-1 ? 7011 ? ResGrpDRBD    ? ok    ?
? vm-109-disk-1 ? 7012 ? ResGrpDRBD    ? ok    ?
? vm-110-disk-1 ? 7013 ? ResGrpDRBD    ? ok    ?
? vm-110-disk-2 ? 7014 ? ResGrpDRBD    ? ok    ?
? vm-111-disk-1 ? 7015 ? ResGrpDRBD    ? ok    ?
? vm-112-disk-1 ? 7016 ? ResGrpDRBD    ? ok    ?
? vm-113-disk-1 ? 7017 ? ResGrpDRBD    ? ok    ?
? vm-114-disk-1 ? 7018 ? ResGrpDRBD    ? ok    ?
? vm-115-disk-1 ? 7019 ? ResGrpDRBD    ? ok    ?
? vm-115-disk-2 ? 7020 ? ResGrpDRBD    ? ok    ?
? vm-116-disk-1 ? 7021 ? ResGrpDRBD    ? ok    ?
? vm-117-disk-1 ? 7022 ? ResGrpDRBD    ? ok    ?
? vm-118-disk-1 ? 7023 ? ResGrpDRBD    ? ok    ?
????????????????????????????????????????????????

root@hydra1:~# linstor resource list
???????????????????????????????????????????????????????????????????????????????????
? ResourceName  ? Node   ? Port ? Usage  ? Conns ?    State ?
CreatedOn           ?
???????????????????????????????????????????????????????????????????????????????????
? vm-100-disk-1 ? hydra3 ? 7000 ? Unused ? Ok    ? UpToDate ? 2020-11-04
18:53:37 ?
? vm-100-disk-1 ? hydra4 ? 7000 ? InUse  ? Ok    ? UpToDate ? 2020-11-04
18:53:37 ?
? vm-100-disk-2 ? hydra3 ? 7001 ? Unused ? Ok    ? UpToDate ? 2020-11-04
19:49:55 ?
? vm-100-disk-2 ? hydra4 ? 7001 ? InUse  ? Ok    ? UpToDate ? 2020-11-04
19:49:55 ?
? vm-101-disk-1 ? hydra3 ? 7002 ? Unused ? Ok    ? UpToDate ? 2020-11-04
20:00:09 ?
? vm-101-disk-1 ? hydra4 ? 7002 ? InUse  ? Ok    ? UpToDate ? 2020-11-04
20:00:08 ?
? vm-102-disk-1 ? hydra3 ? 7003 ? Unused ? Ok    ? UpToDate ? 2020-11-04
20:04:28 ?
? vm-102-disk-1 ? hydra4 ? 7003 ? Unused ? Ok    ? UpToDate ? 2020-11-04
20:04:27 ?
? vm-103-disk-1 ? hydra3 ? 7004 ? Unused ? Ok    ? UpToDate ? 2020-11-20
17:54:11 ?
? vm-103-disk-1 ? hydra4 ? 7004 ? InUse  ? Ok    ? UpToDate ? 2020-11-20
17:54:11 ?
? vm-104-disk-1 ? hydra3 ? 7005 ? Unused ? Ok    ? UpToDate ? 2020-11-20
21:06:05 ?
? vm-104-disk-1 ? hydra4 ? 7005 ? InUse  ? Ok    ? UpToDate ? 2020-11-20
21:06:05 ?
? vm-105-disk-1 ? hydra3 ? 7006 ? Unused ? Ok    ? UpToDate ? 2020-11-20
21:09:24 ?
? vm-105-disk-1 ? hydra4 ? 7006 ? InUse  ? Ok    ? UpToDate ? 2020-11-20
21:09:22 ?
? vm-105-disk-2 ? hydra3 ? 7007 ? Unused ? Ok    ? UpToDate ? 2020-11-20
21:09:58 ?
? vm-105-disk-2 ? hydra4 ? 7007 ? InUse  ? Ok    ? UpToDate ? 2020-11-20
21:09:56 ?
? vm-106-disk-1 ? hydra3 ? 7008 ? Unused ? Ok    ? UpToDate ? 2020-11-20
22:12:26 ?
? vm-106-disk-1 ? hydra4 ? 7008 ? InUse  ? Ok    ? UpToDate ? 2020-11-20
22:12:26 ?
? vm-107-disk-1 ? hydra3 ? 7009 ? Unused ? Ok    ? UpToDate ? 2020-11-20
22:52:51 ?
? vm-107-disk-1 ? hydra4 ? 7009 ? InUse  ? Ok    ? UpToDate ? 2020-11-20
22:52:51 ?
? vm-107-disk-2 ? hydra3 ? 7010 ? Unused ? Ok    ? UpToDate ? 2020-11-20
22:53:27 ?
? vm-107-disk-2 ? hydra4 ? 7010 ? InUse  ? Ok    ? UpToDate ? 2020-11-20
22:53:27 ?
? vm-108-disk-1 ? hydra3 ? 7011 ? Unused ? Ok    ? UpToDate ? 2020-11-24
08:18:52 ?
? vm-108-disk-1 ? hydra4 ? 7011 ? InUse  ? Ok    ? UpToDate ? 2020-11-24
08:18:52 ?
? vm-109-disk-1 ? hydra3 ? 7012 ? Unused ? Ok    ? UpToDate ? 2020-11-24
08:20:33 ?
? vm-109-disk-1 ? hydra4 ? 7012 ? InUse  ? Ok    ? UpToDate ? 2020-11-24
08:20:34 ?
? vm-110-disk-1 ? hydra3 ? 7013 ? InUse  ? Ok    ? UpToDate ? 2020-11-24
11:17:06 ?
? vm-110-disk-1 ? hydra4 ? 7013 ? Unused ? Ok    ? UpToDate ? 2020-11-24
11:17:06 ?
? vm-110-disk-2 ? hydra3 ? 7014 ? InUse  ? Ok    ? UpToDate ? 2020-11-24
11:17:44 ?
? vm-110-disk-2 ? hydra4 ? 7014 ? Unused ? Ok    ? UpToDate ? 2020-11-24
11:17:52 ?
? vm-111-disk-1 ? hydra3 ? 7015 ? Unused ? Ok    ? UpToDate ? 2020-11-24
13:14:20 ?
? vm-111-disk-1 ? hydra4 ? 7015 ? InUse  ? Ok    ? UpToDate ? 2020-11-24
13:14:22 ?
? vm-112-disk-1 ? hydra3 ? 7016 ? Unused ? Ok    ? UpToDate ? 2020-11-24
15:46:42 ?
? vm-112-disk-1 ? hydra4 ? 7016 ? InUse  ? Ok    ? UpToDate ? 2020-11-24
15:46:44 ?
? vm-113-disk-1 ? hydra3 ? 7017 ? InUse  ? Ok    ? UpToDate ? 2020-11-26
11:18:34 ?
? vm-113-disk-1 ? hydra4 ? 7017 ? Unused ? Ok    ? UpToDate ? 2020-11-26
11:18:34 ?
? vm-114-disk-1 ? hydra3 ? 7018 ? InUse  ? Ok    ? UpToDate ? 2020-11-26
22:47:51 ?
? vm-114-disk-1 ? hydra4 ? 7018 ? Unused ? Ok    ? UpToDate ? 2020-11-26
22:47:51 ?
? vm-115-disk-1 ? hydra3 ? 7019 ? InUse  ? Ok    ? UpToDate ? 2020-11-27
12:47:43 ?
? vm-115-disk-1 ? hydra4 ? 7019 ? Unused ? Ok    ? UpToDate ? 2020-11-27
12:47:42 ?
? vm-115-disk-2 ? hydra3 ? 7020 ? InUse  ? Ok    ? UpToDate ? 2020-11-27
12:58:01 ?
? vm-115-disk-2 ? hydra4 ? 7020 ? Unused ? Ok    ? UpToDate ? 2020-11-27
12:58:01 ?
? vm-116-disk-1 ? hydra3 ? 7021 ? InUse  ? Ok    ? UpToDate ? 2020-12-02
18:42:30 ?
? vm-116-disk-1 ? hydra4 ? 7021 ? Unused ? Ok    ? UpToDate ? 2020-12-02
18:42:30 ?
? vm-117-disk-1 ? hydra3 ? 7022 ? Unused ? Ok    ? UpToDate ? 2020-12-02
19:31:31 ?
? vm-117-disk-1 ? hydra4 ? 7022 ? Unused ? Ok    ? UpToDate ? 2020-12-02
19:31:31 ?
? vm-118-disk-1 ? hydra1 ? 7023 ? InUse  ? Ok    ? Diskless ? 2020-12-02
19:40:00 ?
? vm-118-disk-1 ? hydra3 ? 7023 ? Unused ? Ok    ? UpToDate ? 2020-12-02
19:40:04 ?
? vm-118-disk-1 ? hydra4 ? 7023 ? Unused ? Ok    ? UpToDate ? 2020-12-02
19:40:02 ?
???????????????????????????????????????????????????????????????????????????????????

root@hydra1:~# linstor storage-pool list
?????????????????????????????????????????????????????????????????????????????????????????????????????????????
? StoragePool          ? Node   ? Driver   ? PoolName ? FreeCapacity ?
TotalCapacity ? CanSnapshots ? State ?
?????????????????????????????????????????????????????????????????????????????????????????????????????????????
? DfltDisklessStorPool ? hydra1 ? DISKLESS ?          ?             
?               ? False        ? Ok    ?
? DfltDisklessStorPool ? hydra3 ? DISKLESS ?          ?             
?               ? False        ? Ok    ?
? DfltDisklessStorPool ? hydra4 ? DISKLESS ?          ?             
?               ? False        ? Ok    ?
? pool_drbd            ? hydra1 ? LVM      ? vg_drbd  ?     7.28 TiB
?      7.28 TiB ? False        ? Ok    ?
? pool_drbd            ? hydra3 ? LVM      ? vg_drbd  ?    14.70 TiB
?     18.19 TiB ? False        ? Ok    ?
? pool_drbd            ? hydra4 ? LVM      ? vg_drbd  ?    14.70 TiB
?     18.19 TiB ? False        ? Ok    ?
?????????????????????????????????????????????????????????????????????????????????????????????????????????????

Any pointers appreciated

Volker


--
=========================================================
inqbus Scientific Computing Dr. Volker Jaenisch
Hungerbichlweg 3 +49 (8860) 9222 7 92
86977 Burggen https://inqbus.de
=========================================================
Re: Proxmox:Linstore, new third node, all volumes diskless, why? [ In reply to ]
gabor.hernadi at linbit
Dec 2, 2020, 10:32 PM
Post #2 of 4 (450 views)
Permalink
Hi,

> But all the volumes are now and remain in diskless state on Hydra1?

when you added the third node, the command `node create Hydra1` triggered
Linstor's auto-tiebreaker feature. DRBD can use quorum[1] (useful if your
cluster gets split), but needs 3 peers for that (so there can be a majority
of still connected nodes, which is kinda difficult with only 2 peers). That
is the reason why Linstor automatically deploys all resources where you
already had 2 diskful replicas to the third as diskless peer. You can also
check the .res files to see that quorum is now active on all your
DRBD-resources.

A Linstor-managed tiebreaking resource is by default hidden/filtered in the
`resource list`. You can still show them with `resource list --all`.

> But in the end the DRBD-Device is still diskless on Hydra1

That will also stay this way until you tell Linstor to do something
differently.

> How do I get the volumes also DRBD-replicated onto the third node?

Usually Linstor complains if you try to create a resource while the
resource is already deployed on the given node ("resource already exists"
or something like that).
However, Linstor's tiebreaker is different as that one allows you to either
`resource create Hydra1 $resource -s ...` or do a `resource toggle-disk
Hydra1 $resource -s ...`, whereas for the special case of Linstor
tiebreakers, the first method will internally be mapped to the second one :)

For me everything looks normal / as expected.

Best regards,
Gabor


[1]
https://www.linbit.com/drbd-user-guide/drbd-guide-9_0-en/#s-feature-quorum
Re: Proxmox:Linstore, new third node, all volumes diskless, why? [ In reply to ]
rene.peinthor at linbit
Dec 2, 2020, 10:47 PM
Post #3 of 4 (450 views)
Permalink
If you want diskful replicas on all nodes, you also need to change your
place-count on your used resource group.
Otherwise it will always only be diskful on 2 nodes (which ones are decided
by linstor).

Cheers,
Rene

On Thu, Dec 3, 2020 at 7:33 AM Gábor Hernádi <gabor.hernadi@linbit.com>
wrote:

> Hi,
>
> > But all the volumes are now and remain in diskless state on Hydra1?
>
> when you added the third node, the command `node create Hydra1` triggered
> Linstor's auto-tiebreaker feature. DRBD can use quorum[1] (useful if your
> cluster gets split), but needs 3 peers for that (so there can be a majority
> of still connected nodes, which is kinda difficult with only 2 peers). That
> is the reason why Linstor automatically deploys all resources where you
> already had 2 diskful replicas to the third as diskless peer. You can also
> check the .res files to see that quorum is now active on all your
> DRBD-resources.
>
> A Linstor-managed tiebreaking resource is by default hidden/filtered in
> the `resource list`. You can still show them with `resource list --all`.
>
> > But in the end the DRBD-Device is still diskless on Hydra1
>
> That will also stay this way until you tell Linstor to do something
> differently.
>
> > How do I get the volumes also DRBD-replicated onto the third node?
>
> Usually Linstor complains if you try to create a resource while the
> resource is already deployed on the given node ("resource already exists"
> or something like that).
> However, Linstor's tiebreaker is different as that one allows you to
> either `resource create Hydra1 $resource -s ...` or do a `resource
> toggle-disk Hydra1 $resource -s ...`, whereas for the special case of
> Linstor tiebreakers, the first method will internally be mapped to the
> second one :)
>
> For me everything looks normal / as expected.
>
> Best regards,
> Gabor
>
>
> [1]
> https://www.linbit.com/drbd-user-guide/drbd-guide-9_0-en/#s-feature-quorum
> _______________________________________________
> Star us on GITHUB: https://github.com/LINBIT
> drbd-user mailing list
> drbd-user@lists.linbit.com
> https://lists.linbit.com/mailman/listinfo/drbd-user
>
Re: Proxmox:Linstore, new third node, all volumes diskless, why? [ In reply to ]
volker.jaenisch at inqbus
Dec 6, 2020, 12:52 PM
Post #4 of 4 (444 views)
Permalink
Dear Gábor!

Thank you so much!

This was exactly what I was looking for:

linstor resource toggle-disk hydra1 vm-118-disk-1 -s pool_drbd

Works like a charm.

Cheers,

Volker


On 03.12.20 07:32, Gábor Hernádi wrote:
> Hi,
>
> > But all the volumes are now and remain in diskless state on Hydra1?
>
> when you added the third node, the command `node create Hydra1`
> triggered Linstor's auto-tiebreaker feature. DRBD can use quorum[1]
> (useful if your cluster gets split), but needs 3 peers for that (so
> there can be a majority of still connected nodes, which is kinda
> difficult with only 2 peers). That is the reason why Linstor
> automatically deploys all resources where you already had 2 diskful
> replicas to the third as diskless peer. You can also check the .res
> files to see that quorum is now active on all your DRBD-resources.
>
> A Linstor-managed tiebreaking resource is by default hidden/filtered
> in the `resource list`. You can still show them with `resource list
> --all`.
>
> > But in the end the DRBD-Device is still diskless on Hydra1
>
> That will also stay this way until you tell Linstor to do something
> differently.
>
> > How do I get the volumes also DRBD-replicated onto the third node?
>
> Usually Linstor complains if you try to create a resource while the
> resource is already deployed on the given node ("resource already
> exists" or something like that).
> However, Linstor's tiebreaker is different as that one allows you to
> either `resource create Hydra1 $resource -s ...` or do a `resource
> toggle-disk Hydra1 $resource -s ...`, whereas for the special case of
> Linstor tiebreakers, the first method will internally be mapped to the
> second one :)
>
> For me everything looks normal / as expected.
>
> Best regards,
> Gabor
>
>
> [1]
> https://www.linbit.com/drbd-user-guide/drbd-guide-9_0-en/#s-feature-quorum
> <https://www.linbit.com/drbd-user-guide/drbd-guide-9_0-en/#s-feature-quorum>

--
=========================================================
inqbus Scientific Computing Dr. Volker Jaenisch
Hungerbichlweg 3 +49 (8860) 9222 7 92
86977 Burggen https://inqbus.de
=========================================================

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal