Mailing List Archive

UpToDate and out-of-sync inconsistence
Hi all,

On drbd-9.0.23 (CentOS 8.2), we are facing an inconsistence of node status.
That is, although the state of both DRBD device is UpToDate, there exists
some out-of-sync value.

#drbdsetup status --verbose --statistics
r008 node-id:0 role:Primary suspended:no
write-ordering:flush
volume:0 minor:8 disk:UpToDate quorum:yes
size:109757124 read:3065 written:105640 al-writes:27 bm-writes:0
upper-pending:0 lower-pending:0 al-suspended:no blocked:no
fs100 node-id:1 connection:Connected role:Secondary congested:no
ap-in-flight:0 rs-in-flight:0
volume:0 replication:Established peer-disk:UpToDate
resync-suspended:no
received:0 sent:105028 out-of-sync:12 pending:0 unacked:0

I see this issue on several systems running drbd-9.0, and I made a stable
reproducer (attached for reference). This repeats umount/mount of DRBD disk
on primary node, and disconnect/connect on secondary node at the same time.

When I run this reproducer on drbd-8.9 for comparison, I never see the
issue.
On the other hand, I see this not only drbd-9.0.23, but also the latest 9.1.

Am I doing something wrong or do we have a bug in out-of-sync calculation
and state transition to UpToDate?
Any feedback would be greatly appreciated.
Re: UpToDate and out-of-sync inconsistence [ In reply to ]
Hi all,

I am continuing to investigate this.
As a result, I found one fact.

This problem only occurs when the state transitions from "OutDated" to
"UpToDate".
And it never occurs when the state transitions from "Inconsistent" to
"UpTodate".

Probably,I presume that the state transition from "OutDated" to "UpToDate"
is not exclusive in the state transition and asynchronous calculation.

Is this a specification? Or is it a bug?
If it's a bug, where in the source code should I add exclusive processing?

I'm worried because I don't know how to fix it.
I would be grateful if anyone could tell me.

Best regards,
shuhei
Re: UpToDate and out-of-sync inconsistence [ In reply to ]
Hi all,

I have confirmed that this problem can be reproduced with the latest
drbd-9.1.16.

On primary and secondary DRBD devices
Both states are UpToDate, but there are values that are out of sync.

Any feedback would be greatly appreciated.


Best regards,
shuhei

On Fri, Mar 18, 2022 at 10:01 AM Shuhei Tsutsumi <stsutsumi@mvista.com>
wrote:

> Hi all,
>
> I am continuing to investigate this.
> As a result, I found one fact.
>
> This problem only occurs when the state transitions from "OutDated" to
> "UpToDate".
> And it never occurs when the state transitions from "Inconsistent" to
> "UpTodate".
>
> Probably,I presume that the state transition from "OutDated" to "UpToDate"
> is not exclusive in the state transition and asynchronous calculation.
>
> Is this a specification? Or is it a bug?
> If it's a bug, where in the source code should I add exclusive processing?
>
> I'm worried because I don't know how to fix it.
> I would be grateful if anyone could tell me.
>
> Best regards,
> shuhei
>
>
Re: UpToDate and out-of-sync inconsistence [ In reply to ]
Hi folks,

I would be highly interested in a explanation (and a possible workaround
or fix?) for this inconsistency, too. How comes that drbd doesn't ring all
alarm bells it has access to in such a situation? Is this considered as
"not important"?


Regards

Harri
_______________________________________________
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user
Re: UpToDate and out-of-sync inconsistence [ In reply to ]
Hi all,

There are facts found in additional investigations.

Even in the same UpToDate state
This issue occurs if the transition source is Outdated.
However, if the transition source is Inconsistent, this problem does not
occur at all.

What is the difference between the two?

Regards
Shuhei

On Fri, Mar 25, 2022 at 8:47 PM Harald Dunkel <harald.dunkel@aixigo.com>
wrote:

>
> Hi folks,
>
> I would be highly interested in a explanation (and a possible workaround
> or fix?) for this inconsistency, too. How comes that drbd doesn't ring all
> alarm bells it has access to in such a situation? Is this considered as
> "not important"?
>
>
> Regards
>
> Harri
> _______________________________________________
> Star us on GITHUB: https://github.com/LINBIT
> drbd-user mailing list
> drbd-user@lists.linbit.com
> https://lists.linbit.com/mailman/listinfo/drbd-user
>
Re: UpToDate and out-of-sync inconsistence [ In reply to ]
Hi, all

We received the following responses from community developers.
- States such as "UpToDate" are calculated based on the UUID comparison
results.
- "out-of-sync" is calculated according to the bitmap settings.
- Because the calculation method is different between the two
Even in the UpTpDate state, it occurs when out-oy-sync is not 0.

However, the following two points are doubtful.

1.This does not occur with DRBD 8.x

Is this a specification designed after DRBD 9.x?
Or is it the same specification as DRBD 8.x?

2.If you execute verify in this case, you can see that there are
unsynchronized blocks.

When this phenomenon occurs, if you execute the "drbdadm verify" command,
You can see that asynchronous blocks remain.
However, it seems strange that the disk status is UpToDate.

Can anyone give us an opinion on the above two points?

Regards
Shuhei

On Thu, Mar 31, 2022 at 9:46 AM Shuhei Tsutsumi <stsutsumi@mvista.com>
wrote:

> Hi all,
>
> There are facts found in additional investigations.
>
> Even in the same UpToDate state
> This issue occurs if the transition source is Outdated.
> However, if the transition source is Inconsistent, this problem does not
> occur at all.
>
> What is the difference between the two?
>
> Regards
> Shuhei
>
> On Fri, Mar 25, 2022 at 8:47 PM Harald Dunkel <harald.dunkel@aixigo.com>
> wrote:
>
>>
>> Hi folks,
>>
>> I would be highly interested in a explanation (and a possible workaround
>> or fix?) for this inconsistency, too. How comes that drbd doesn't ring all
>> alarm bells it has access to in such a situation? Is this considered as
>> "not important"?
>>
>>
>> Regards
>>
>> Harri
>> _______________________________________________
>> Star us on GITHUB: https://github.com/LINBIT
>> drbd-user mailing list
>> drbd-user@lists.linbit.com
>> https://lists.linbit.com/mailman/listinfo/drbd-user
>>
>