Mailing List Archive

symmetric anti-collocation
How to I express symmetric anti-collocation in Pacemaker 1.0.9.1?
I'd like to write two rules:
colocation X-Y -2: X Y
colocation Y-X -2: Y X
The idea is that external conditions could place either resource and
I'd like Pacemaker to place the other accordingly.
Unfortunately, Pacemaker will only apply the first such rule specified.
Alan

_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: symmetric anti-collocation [ In reply to ]
On Thu, Nov 11, 2010 at 9:02 PM, Alan Jones <falancluster@gmail.com> wrote:
> How to I express symmetric anti-collocation in Pacemaker 1.0.9.1?
> I'd like to write two rules:
> colocation X-Y -2: X Y
> colocation Y-X -2: Y X

the second one is implied by the first and is therefore redundant

> The idea is that external conditions could place either resource and
> I'd like Pacemaker to place the other accordingly.
> Unfortunately, Pacemaker will only apply the first such rule specified.
> Alan
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>

_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: symmetric anti-collocation [ In reply to ]
On Thu, Nov 11, 2010 at 11:31 PM, Andrew Beekhof <andrew@beekhof.net> wrote:
>> colocation X-Y -2: X Y
>> colocation Y-X -2: Y X
>
> the second one is implied by the first and is therefore redundant

If only that were true!
What happens with the first rule is that other constraints that force
Y to a node will evict X but not the other way around.
What I'm doing is to first apply a "slight" preference for each
resource to each node:
location X-nodeA X 1: nodeA
location Y-nodeB Y 1: nodeB
And then impose absolute constraints that come from the outside environment.
In the particular case that has a problem, the constraint looks like:
location X-not-nodeA X -inf: nodeA
The behavior I expected was for X to be placed on nodeB and Y to
"anti-colocate" onto nodeA because our colocation rule is stronger
than the node preference rule. What happens instead is that both X
and Y run on nodeB.
The similar constraint on Y (by itself) does work:
location Y-not-nodeB Y -inf: nodeB
and results in Y running on nodeA and X running on nodeB. This is the
case whether I have one colocation rule or two, i.e. the second
colocation rule is ignored.

Looking at the code, I think the solution would be to short-circuit
the recursion when you can only run on one node due to -inf rules
rather than on a loop. Obviously, it would not be a simple change and
needs some thought.
If you have any other suggestions let me know.
Alan

_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: symmetric anti-collocation [ In reply to ]
I've looked into the code more and added more logging, etc.
The pengine essentially walks the list of constraints, applying
weights, and then walks the list of resources and tallies the weights.
In my example, it ends up walking the resources backward, i.e. it
assigns a node to Y and then assigns a node to X.
Unfortunately, at the time of assigning a node to Y, X has no assigned
node and so the colocation rule cannot be applied.
What is needed is a backtracking method from the computer science area
of "constraint satisfaction".
Alan

_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: symmetric anti-collocation [ In reply to ]
On Fri, Nov 12, 2010 at 5:27 PM, Alan Jones <falancluster@gmail.com> wrote:
> On Thu, Nov 11, 2010 at 11:31 PM, Andrew Beekhof <andrew@beekhof.net> wrote:
>>> colocation X-Y -2: X Y
>>> colocation Y-X -2: Y X
>>
>> the second one is implied by the first and is therefore redundant
>
> If only that were true!

It is. I know exactly how my code works in this regard.
More than likely a score of -2 is simply too low to have any effect.

> What happens with the first rule is that other constraints that force
> Y to a node will evict X but not the other way around.
> What I'm doing is to first apply a "slight" preference for each
> resource to each node:
> location X-nodeA X 1: nodeA
> location Y-nodeB Y 1: nodeB
> And then impose absolute constraints that come from the outside environment.
> In the particular case that has a problem, the constraint looks like:
> location X-not-nodeA X -inf: nodeA
> The behavior I expected was for X to be placed on nodeB and Y to
> "anti-colocate" onto nodeA because our colocation rule is stronger
> than the node preference rule.  What happens instead is that both X
> and Y run on nodeB.
> The similar constraint on Y (by itself) does work:
> location Y-not-nodeB Y -inf: nodeB
> and results in Y running on nodeA and X running on nodeB.  This is the
> case whether I have one colocation rule or two, i.e. the second
> colocation rule is ignored.
>
> Looking at the code, I think the solution would be to short-circuit
> the recursion when you can only run on one node due to -inf rules
> rather than on a loop.  Obviously, it would not be a simple change and
> needs some thought.
> If you have any other suggestions let me know.
> Alan
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>

_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: symmetric anti-collocation [ In reply to ]
On Sat, Nov 13, 2010 at 3:20 AM, Andrew Beekhof <andrew@beekhof.net> wrote:
> On Fri, Nov 12, 2010 at 5:27 PM, Alan Jones <falancluster@gmail.com> wrote:
>> On Thu, Nov 11, 2010 at 11:31 PM, Andrew Beekhof <andrew@beekhof.net> wrote:
>>>> colocation X-Y -2: X Y
>>>> colocation Y-X -2: Y X
>>>
>>> the second one is implied by the first and is therefore redundant
>>
>> If only that were true!
>
> It is. I know exactly how my code works in this regard.
> More than likely a score of -2 is simply too low to have any effect.

I have tried larger values. If you know of a value that *should*
work, please share it.
I wish to be able to predict the behavior, so arbitrary values are not
acceptable.
It is either large enough to overcome competing values or not.
In this configuration I've set the default resource stickyness to zero and set a
node location rule per resource to one. It is clear from the logs
that the -2 is
not being applied. I also tried unique large numbers to see if I
could trace their
propagation, e.g. -1965, with no effect.
If you have time to look at this, I will have to create sanitized
versions of logs
and configuration, etc. Let me know.
Alan

_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: symmetric anti-collocation [ In reply to ]
On Sat, Nov 13, 2010 at 5:09 PM, Alan Jones <falancluster@gmail.com> wrote:
> On Sat, Nov 13, 2010 at 3:20 AM, Andrew Beekhof <andrew@beekhof.net> wrote:
>> On Fri, Nov 12, 2010 at 5:27 PM, Alan Jones <falancluster@gmail.com> wrote:
>>> On Thu, Nov 11, 2010 at 11:31 PM, Andrew Beekhof <andrew@beekhof.net> wrote:
>>>>> colocation X-Y -2: X Y
>>>>> colocation Y-X -2: Y X
>>>>
>>>> the second one is implied by the first and is therefore redundant
>>>
>>> If only that were true!
>>
>> It is. I know exactly how my code works in this regard.
>> More than likely a score of -2 is simply too low to have any effect.
>
> I have tried larger values.  If you know of a value that *should*
> work, please share it.

INFINITY

> I wish to be able to predict the behavior, so arbitrary values are not
> acceptable.
> It is either large enough to overcome competing values or not.
> In this configuration I've set the default resource stickyness to zero and set a
> node location rule per resource to one.  It is clear from the logs
> that the -2 is
> not being applied.  I also tried unique large numbers to see if I
> could trace their
> propagation, e.g. -1965, with no effect.
> If you have time to look at this, I will have to create sanitized
> versions of logs
> and configuration, etc.  Let me know.
> Alan
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>

_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: symmetric anti-collocation [ In reply to ]
>> I have tried larger values.  If you know of a value that *should*
>> work, please share it.
>
> INFINITY

My understanding is that a colocation score of minus infinity will
prevent the resources from running on the same node, which in my
configuration would result in a loss of availability. The goal is a
value that will overrule the node location preferences of one, allow
for both resources to run on the same node, and provide for
anti-colocation to evict either resource based on the constraints of
the other. I can't claim to understand your code, but I've done
sufficient experiments to understand the code's behavior in 1.0.9.1.
There is simply no combination of colocation rules and values that
will work in both directions. I can get the placement of Y to evict X
-- or -- X to evict Y; but not both.
Alan

_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: symmetric anti-collocation [ In reply to ]
Sent: Sat Nov 13 2010 04:20:56 GMT-0700 (Mountain Standard Time)
From: Andrew Beekhof <andrew@beekhof.net>
To: The Pacemaker cluster resource manager <pacemaker@oss.clusterlabs.org>
Subject: Re: [Pacemaker] symmetric anti-collocation
> On Fri, Nov 12, 2010 at 5:27 PM, Alan Jones <falancluster@gmail.com> wrote:
>
>> On Thu, Nov 11, 2010 at 11:31 PM, Andrew Beekhof <andrew@beekhof.net> wrote:
>>
>>>> colocation X-Y -2: X Y
>>>> colocation Y-X -2: Y X
>>>>
>>> the second one is implied by the first and is therefore redundant
>>>
>> If only that were true!
>>
>
> It is. I know exactly how my code works in this regard.
> More than likely a score of -2 is simply too low to have any effect.
>
I'm going to have to side with Alan here, his pasted statements are
*NOT* redundant and also do *NOT* work. Please be a little more
accepting that your code might have problems.

Exhibit A:
crm(live)# configure show
node nas01 \
attributes standby="off"
node nas02 \
attributes standby="off"
node nas03 \
attributes standby="on"
primitive d1 ocf:pacemaker:Dummy
primitive d2 ocf:pacemaker:Dummy
colocation co1 -inf: d1 d2
colocation co2 -inf: d2 d1
property $id="cib-bootstrap-options" \
dc-version="1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3" \
cluster-infrastructure="openais" \
expected-quorum-votes="3" \
stonith-enabled="false" \
default-resource-stickiness="INFINITY"




crm(live)# status
============
Last updated: Mon Dec 20 21:03:49 2010
Stack: openais
Current DC: nas01 - partition with quorum
Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3
3 Nodes configured, 3 expected votes
2 Resources configured.
============

Node nas03: standby
Online: [ nas01 nas02 ]

d1 (ocf::pacemaker:Dummy): Started nas01
d2 (ocf::pacemaker:Dummy): Started nas02




crm(live)# node standby nas02
crm(live)# status
============
Last updated: Mon Dec 20 21:04:14 2010
Stack: openais
Current DC: nas01 - partition with quorum
Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3
3 Nodes configured, 3 expected votes
2 Resources configured.
============

Node nas02: standby
Node nas03: standby
Online: [ nas01 ]

d2 (ocf::pacemaker:Dummy): Started nas01


Notice how resource 'd1' got evicted from node 'nas01' and 'd2' replaced
it. This would indicate that the 'co2' rule is being completely ignored.
In fact if you delete the co2 rule, you get the exact same behavior. If
you were to delete the co1 rule, and put 'nas02' in standby, d1 would
have stayed running on 'nas01' instead of being replaced. This would
indicate that the constraints are NOT redundant and NOT implied by
eachother.

I'm not trying to be disrespectful or rude, its just that nobody is perfect.
Re: symmetric anti-collocation [ In reply to ]
On Mon, Dec 20, 2010 at 10:25 PM, Patrick H. <pacemaker@feystorm.net> wrote:

> Sent: Sat Nov 13 2010 04:20:56 GMT-0700 (Mountain Standard Time)
> From: Andrew Beekhof <andrew@beekhof.net> <andrew@beekhof.net>
> To: The Pacemaker cluster resource manager <pacemaker@oss.clusterlabs.org><pacemaker@oss.clusterlabs.org>
> Subject: Re: [Pacemaker] symmetric anti-collocation
>
> On Fri, Nov 12, 2010 at 5:27 PM, Alan Jones <falancluster@gmail.com> <falancluster@gmail.com> wrote:
>
>
> On Thu, Nov 11, 2010 at 11:31 PM, Andrew Beekhof <andrew@beekhof.net> <andrew@beekhof.net> wrote:
>
>
> colocation X-Y -2: X Y
> colocation Y-X -2: Y X
>
>
> the second one is implied by the first and is therefore redundant
>
>
> If only that were true!
>
>
> It is. I know exactly how my code works in this regard.
> More than likely a score of -2 is simply too low to have any effect.
>
>
> I'm going to have to side with Alan here, his pasted statements are *NOT*
> redundant and also do *NOT* work. Please be a little more accepting that
> your code might have problems.
>

I think the past 7 years have adequately shown that I am under no illusions
on that score.

However I actually know how colocation is supposed to work, and I'm and
telling you with 100% certainty that the second one is redundant. That the
combination doesn't do what you want is another story.

Unless you use a clone in some manner, there is no way to express:

If nodeA fails, stop rscA
If nodeB fails, stop rscB

In order to complete computations before the end of the universe, Pacemaker
requires you to decide with of rscA and rscB is more important and will
sacrifice the other.

You can of course argue that this requirement is a crime against humanity,
but it remains how Pacemaker is currently designed to work.


> Exhibit A:
> crm(live)# configure show
> node nas01 \
> attributes standby="off"
> node nas02 \
> attributes standby="off"
> node nas03 \
> attributes standby="on"
> primitive d1 ocf:pacemaker:Dummy
> primitive d2 ocf:pacemaker:Dummy
> colocation co1 -inf: d1 d2
> colocation co2 -inf: d2 d1
> property $id="cib-bootstrap-options" \
> dc-version="1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3" \
> cluster-infrastructure="openais" \
> expected-quorum-votes="3" \
> stonith-enabled="false" \
> default-resource-stickiness="INFINITY"
>
>
>
>
> crm(live)# status
> ============
> Last updated: Mon Dec 20 21:03:49 2010
> Stack: openais
> Current DC: nas01 - partition with quorum
> Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3
> 3 Nodes configured, 3 expected votes
> 2 Resources configured.
> ============
>
> Node nas03: standby
> Online: [ nas01 nas02 ]
>
> d1 (ocf::pacemaker:Dummy): Started nas01
> d2 (ocf::pacemaker:Dummy): Started nas02
>
>
>
>
> crm(live)# node standby nas02
> crm(live)# status
> ============
> Last updated: Mon Dec 20 21:04:14 2010
> Stack: openais
> Current DC: nas01 - partition with quorum
> Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3
> 3 Nodes configured, 3 expected votes
> 2 Resources configured.
> ============
>
> Node nas02: standby
> Node nas03: standby
> Online: [ nas01 ]
>
> d2 (ocf::pacemaker:Dummy): Started nas01
>
> Notice how resource 'd1' got evicted from node 'nas01' and 'd2' replaced
> it. This would indicate that the 'co2' rule is being completely ignored. In
> fact if you delete the co2 rule, you get the exact same behavior. If you
> were to delete the co1 rule, and put 'nas02' in standby, d1 would have
> stayed running on 'nas01' instead of being replaced. This would indicate
> that the constraints are NOT redundant and NOT implied by eachother.
>
> I'm not trying to be disrespectful or rude, its just that nobody is
> perfect.
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs:
> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>
>
Re: symmetric anti-collocation [ In reply to ]
On Mon, Dec 20, 2010 at 10:48 PM, Andrew Beekhof <andrew@beekhof.net> wrote:

>
>
> On Mon, Dec 20, 2010 at 10:25 PM, Patrick H. <pacemaker@feystorm.net>wrote:
>
>> Sent: Sat Nov 13 2010 04:20:56 GMT-0700 (Mountain Standard Time)
>> From: Andrew Beekhof <andrew@beekhof.net> <andrew@beekhof.net>
>> To: The Pacemaker cluster resource manager
>> <pacemaker@oss.clusterlabs.org> <pacemaker@oss.clusterlabs.org>
>> Subject: Re: [Pacemaker] symmetric anti-collocation
>>
>> On Fri, Nov 12, 2010 at 5:27 PM, Alan Jones <falancluster@gmail.com> <falancluster@gmail.com> wrote:
>>
>>
>> On Thu, Nov 11, 2010 at 11:31 PM, Andrew Beekhof <andrew@beekhof.net> <andrew@beekhof.net> wrote:
>>
>>
>> colocation X-Y -2: X Y
>> colocation Y-X -2: Y X
>>
>>
>> the second one is implied by the first and is therefore redundant
>>
>>
>> If only that were true!
>>
>>
>> It is. I know exactly how my code works in this regard.
>> More than likely a score of -2 is simply too low to have any effect.
>>
>>
>> I'm going to have to side with Alan here, his pasted statements are *NOT*
>> redundant and also do *NOT* work. Please be a little more accepting that
>> your code might have problems.
>>
>
> I think the past 7 years have adequately shown that I am under no illusions
> on that score.
>
> However I actually know how colocation is supposed to work, and I'm and
> telling you with 100% certainty that the second one is redundant. That the
> combination doesn't do what you want is another story.
>
> Unless you use a clone in some manner, there is no way to express:
>
> If nodeA fails, stop rscA
> If nodeB fails, stop rscB
>

"Unless you use scores of -INFINITY"

It is very clearly documented that scores other than +/-INFINITY are _hints_
to the PE and may be ignored.


>
> In order to complete computations before the end of the universe, Pacemaker
> requires you to decide with of rscA and rscB is more important and will
> sacrifice the other.
>
> You can of course argue that this requirement is a crime against humanity,
> but it remains how Pacemaker is currently designed to work.
>
>
>> Exhibit A:
>> crm(live)# configure show
>> node nas01 \
>> attributes standby="off"
>> node nas02 \
>> attributes standby="off"
>> node nas03 \
>> attributes standby="on"
>> primitive d1 ocf:pacemaker:Dummy
>> primitive d2 ocf:pacemaker:Dummy
>> colocation co1 -inf: d1 d2
>> colocation co2 -inf: d2 d1
>> property $id="cib-bootstrap-options" \
>> dc-version="1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3" \
>> cluster-infrastructure="openais" \
>> expected-quorum-votes="3" \
>> stonith-enabled="false" \
>> default-resource-stickiness="INFINITY"
>>
>>
>>
>>
>> crm(live)# status
>> ============
>> Last updated: Mon Dec 20 21:03:49 2010
>> Stack: openais
>> Current DC: nas01 - partition with quorum
>> Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3
>> 3 Nodes configured, 3 expected votes
>> 2 Resources configured.
>> ============
>>
>> Node nas03: standby
>> Online: [ nas01 nas02 ]
>>
>> d1 (ocf::pacemaker:Dummy): Started nas01
>> d2 (ocf::pacemaker:Dummy): Started nas02
>>
>>
>>
>>
>> crm(live)# node standby nas02
>> crm(live)# status
>> ============
>> Last updated: Mon Dec 20 21:04:14 2010
>> Stack: openais
>> Current DC: nas01 - partition with quorum
>> Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3
>> 3 Nodes configured, 3 expected votes
>> 2 Resources configured.
>> ============
>>
>> Node nas02: standby
>> Node nas03: standby
>> Online: [ nas01 ]
>>
>> d2 (ocf::pacemaker:Dummy): Started nas01
>>
>> Notice how resource 'd1' got evicted from node 'nas01' and 'd2' replaced
>> it. This would indicate that the 'co2' rule is being completely ignored. In
>> fact if you delete the co2 rule, you get the exact same behavior. If you
>> were to delete the co1 rule, and put 'nas02' in standby, d1 would have
>> stayed running on 'nas01' instead of being replaced. This would indicate
>> that the constraints are NOT redundant and NOT implied by eachother.
>>
>> I'm not trying to be disrespectful or rude, its just that nobody is
>> perfect.
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs:
>> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>>
>>
>