Mailing List Archive: Suicide fencing and watchdog questions

Suicide fencing and watchdog questions

Nov 25, 2014, 3:37 AM

Post #1 of 6 (2623 views)

Hi,

Is there any information how watchdog integration is intended to work?
What are currently-evaluated use-cases for that?
It seems to be forcibly disabled id SBD is not detected...

Also, is there any way to make node (in one-node cluster ;) ) to suicide
if it detects fencing is required? Technically, that can be done with
IPMI 'power cycle' or 'power reset' commands - but node (and thus the
"whole" cluster) will not know about fencing is succeeded, because if it
received the answer, then fencing failed. But node will be hard reboot
and thus cleaned up otherwise.

Best,
Vladislav

_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: Suicide fencing and watchdog questions [ In reply to ]

andrew at beekhof

Nov 26, 2014, 4:43 PM

Post #2 of 6 (2559 views)

Permalink

> On 25 Nov 2014, at 10:37 pm, Vladislav Bogdanov <bubble@hoster-ok.com> wrote:
>
> Hi,
>
> Is there any information how watchdog integration is intended to work?
> What are currently-evaluated use-cases for that?
> It seems to be forcibly disabled id SBD is not detected...

Are you referring to no-quorum-policy=suicide?

>
> Also, is there any way to make node (in one-node cluster ;) ) to suicide
> if it detects fencing is required? Technically, that can be done with
> IPMI 'power cycle' or 'power reset' commands - but node (and thus the
> "whole" cluster) will not know about fencing is succeeded, because if it
> received the answer, then fencing failed. But node will be hard reboot
> and thus cleaned up otherwise.
>
> Best,
> Vladislav
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: Suicide fencing and watchdog questions [ In reply to ]

bubble at hoster-ok

Nov 26, 2014, 9:24 PM

Post #3 of 6 (2556 views)

Permalink

27.11.2014 03:43, Andrew Beekhof wrote:
>
>> On 25 Nov 2014, at 10:37 pm, Vladislav Bogdanov <bubble@hoster-ok.com> wrote:
>>
>> Hi,
>>
>> Is there any information how watchdog integration is intended to work?
>> What are currently-evaluated use-cases for that?
>> It seems to be forcibly disabled id SBD is not detected...
>
> Are you referring to no-quorum-policy=suicide?

That too.

But main intention was to understand what value that feature can bring
at all.
I tried to enable it without SBD or no-quorum-policy=suicide and
watchdog was not fired up. Then I looked at sources and realized that it
is enabled only when SBD is detected, and is not actually managed by the
cluster option.

>
>>
>> Also, is there any way to make node (in one-node cluster ;) ) to suicide
>> if it detects fencing is required? Technically, that can be done with
>> IPMI 'power cycle' or 'power reset' commands - but node (and thus the
>> "whole" cluster) will not know about fencing is succeeded, because if it
>> received the answer, then fencing failed. But node will be hard reboot
>> and thus cleaned up otherwise.
>>
>> Best,
>> Vladislav
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>

_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: Suicide fencing and watchdog questions [ In reply to ]

andrew at beekhof

Nov 27, 2014, 3:51 PM

Post #4 of 6 (2556 views)

Permalink

> On 27 Nov 2014, at 4:24 pm, Vladislav Bogdanov <bubble@hoster-ok.com> wrote:
>
> 27.11.2014 03:43, Andrew Beekhof wrote:
>>
>>> On 25 Nov 2014, at 10:37 pm, Vladislav Bogdanov <bubble@hoster-ok.com> wrote:
>>>
>>> Hi,
>>>
>>> Is there any information how watchdog integration is intended to work?
>>> What are currently-evaluated use-cases for that?
>>> It seems to be forcibly disabled id SBD is not detected...
>>
>> Are you referring to no-quorum-policy=suicide?
>
> That too.
>
> But main intention was to understand what value that feature can bring
> at all.
> I tried to enable it without SBD or no-quorum-policy=suicide and
> watchdog was not fired up.

The only interaction with watchdog is via SBD.
Suicide for no-quorum-policy has always relied on fencing.

SBD will look for that value though and tailor its behaviour.

> Then I looked at sources and realized that it
> is enabled only when SBD is detected, and is not actually managed by the
> cluster option.
>
>>
>>>
>>> Also, is there any way to make node (in one-node cluster ;) ) to suicide
>>> if it detects fencing is required? Technically, that can be done with
>>> IPMI 'power cycle' or 'power reset' commands - but node (and thus the
>>> "whole" cluster) will not know about fencing is succeeded, because if it
>>> received the answer, then fencing failed. But node will be hard reboot
>>> and thus cleaned up otherwise.
>>>
>>> Best,
>>> Vladislav
>>>
>>> _______________________________________________
>>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: Suicide fencing and watchdog questions [ In reply to ]

arvidjaar at gmail

Nov 28, 2014, 10:36 PM

Post #5 of 6 (2541 views)

Permalink

Ð’ Thu, 27 Nov 2014 08:24:56 +0300
Vladislav Bogdanov <bubble@hoster-ok.com> Ð¿Ð¸ÑˆÐµÑ‚:

> 27.11.2014 03:43, Andrew Beekhof wrote:
> >
> >> On 25 Nov 2014, at 10:37 pm, Vladislav Bogdanov <bubble@hoster-ok.com> wrote:
> >>
> >> Hi,
> >>
> >> Is there any information how watchdog integration is intended to work?
> >> What are currently-evaluated use-cases for that?
> >> It seems to be forcibly disabled id SBD is not detected...
> >
> > Are you referring to no-quorum-policy=suicide?
>
> That too.
>
> But main intention was to understand what value that feature can bring
> at all.
> I tried to enable it without SBD or no-quorum-policy=suicide and
> watchdog was not fired up. Then I looked at sources and realized that it
> is enabled only when SBD is detected, and is not actually managed by the
> cluster option.
>

It is not enough for a node to kill itself, other nodes need to find out
whether it has done it. What are other options besides SBD in this case?

_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: Suicide fencing and watchdog questions [ In reply to ]

andrew at beekhof

Nov 30, 2014, 3:03 PM

Post #6 of 6 (2512 views)

Permalink

> On 29 Nov 2014, at 5:36 pm, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>
> Ð’ Thu, 27 Nov 2014 08:24:56 +0300
> Vladislav Bogdanov <bubble@hoster-ok.com> Ð¿Ð¸ÑˆÐµÑ‚:
>
>> 27.11.2014 03:43, Andrew Beekhof wrote:
>>>
>>>> On 25 Nov 2014, at 10:37 pm, Vladislav Bogdanov <bubble@hoster-ok.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Is there any information how watchdog integration is intended to work?
>>>> What are currently-evaluated use-cases for that?
>>>> It seems to be forcibly disabled id SBD is not detected...
>>>
>>> Are you referring to no-quorum-policy=suicide?
>>
>> That too.
>>
>> But main intention was to understand what value that feature can bring
>> at all.
>> I tried to enable it without SBD or no-quorum-policy=suicide and
>> watchdog was not fired up. Then I looked at sources and realized that it
>> is enabled only when SBD is detected, and is not actually managed by the
>> cluster option.
>>
>
> It is not enough for a node to kill itself, other nodes need to find out
> whether it has done it. What are other options besides SBD in this case?

Normal fencing devices /could/ be depending on the level of risk you're prepared to accept.

_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org