Mailing List Archive

On 2011-11-11 09:22, Junko IKEDA wrote:
> Hi,
>
> I am running MySQL replication setting with 2 nodes Master/Slave configuration.
> If Slave status(secs_behind) is lager than Master's parameter(max_slave_lag),
> Slave data is outdated, right?
> check_slave() in mysql RA would run "crm_master -v 0" in this
> situation to mark Slave as "outdated",
> but if Master is shut down in this status,
> Slave will be able to promote instead of its old data.
> (is this correct?)
> It seems that "crm_master -v -INFINITY" is effectual to prevent Slave promotion.

forwarding this to marek and fghaas as i'm not familiar with
multi-state handling inside resource agents.

cheers,
raoul
--
____________________________________________________________________
DI (FH) Raoul Bhatia M.Sc. email. r.bhatia@ipax.at
Technischer Leiter

IPAX - Aloy Bhatia Hava OG web. http://www.ipax.at
Barawitzkagasse 10/2/2/11 email. office@ipax.at
1190 Wien tel. +43 1 3670030
FN 277995t HG Wien fax. +43 1 3670030 15
____________________________________________________________________
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/

On 14.11.2011 09:55, Raoul Bhatia [IPAX] wrote:
> On 2011-11-11 09:22, Junko IKEDA wrote:
>> Hi,
>>
>> I am running MySQL replication setting with 2 nodes Master/Slave
>> configuration.
>> If Slave status(secs_behind) is lager than Master's
>> parameter(max_slave_lag),
>> Slave data is outdated, right?
>> check_slave() in mysql RA would run "crm_master -v 0" in this
>> situation to mark Slave as "outdated",
>> but if Master is shut down in this status,
>> Slave will be able to promote instead of its old data.
>> (is this correct?)
>> It seems that "crm_master -v -INFINITY" is effectual to prevent Slave
>> promotion.
>
> forwarding this to marek and fghaas as i'm not familiar with
> multi-state handling inside resource agents.

Did you set "evict_outdated_slaves"?

--
Pozdrawiam / Best Regards,
Marek Marczykowski | gg:2873965 | RLU #390519
marmarek at staszic waw pl | xmpp:marmarek at staszic waw pl

Hello Ikeda-san!

On 2011-11-11 09:22, Junko IKEDA wrote:
> Hi,
>
> I am running MySQL replication setting with 2 nodes Master/Slave configuration.
> If Slave status(secs_behind) is lager than Master's parameter(max_slave_lag),
> Slave data is outdated, right?

Yes.

> check_slave() in mysql RA would run "crm_master -v 0" in this
> situation to mark Slave as "outdated",
> but if Master is shut down in this status,
> Slave will be able to promote instead of its old data.
> (is this correct?)

Actually, not quite. :)

Andrew will correct me if I'm wrong on this. But as I understand it,

- while a _placement score_ of 0 makes a node eligible for _running_ a
resource (including an instance of a master/slave set),
- only a _promotion score_ or greater than 0 (i.e. a minimum of 1) makes
the node eligible for promoting a resource to the Master role.

So, if a node has a promotion score of 0, then it will node be promoted.
However, your point is entirely valid if you also set a master
preference via a location constraint on the master role. Consider this:

node alice
attributes standby="on"
node bob
attributes standby="off"

primitive p_mysql ocf:heartbeat:mysql
ms ms_myql p_mysql

location l_master_prefers_bob ms_mysql \
rule 200: $role=Master #uname eq bob

In that case, if bob has fallen too far behind (automatic master score:
0), then the location rule still increases that score by 200, so the
total promotion score for bob is 0 + 200 == 200, and bob will be promoted.

> It seems that "crm_master -v -INFINITY" is effectual to prevent Slave promotion.

Yes, that is entirely correct. In the example above, if the outdated
slave sets a promotion preference of -INFINITY, them the total promotion
score would be -INFINITY + 200 == -INFINITY. So the outdated slave, bob,
would never be promoted to master.

But:
> if [ $master_pref -lt 0 ]; then
> # Sanitize a below-zero preference to just zero
> master_pref=0
>
> fi
> $CRM_MASTER -v $master_pref

This if block is unfortunately there for a good reason, namely that (at
least some versions back) the Policy Engine really did not like negative
promotion scores at all. I forget the exact details, but maybe Lars
(Ellenberg) will remember -- I seem to recall him telling me very firmly
something to the effect of "whatever you do, don't use crm_master with a
negative score anywhere". Now, it may be that said issues with the
pengine have since been fixed. If that is the case, I'll be happy to
modify the mysql RA as you suggest.

Surely you have patched your local version of the RA to set a -INFINITY
master preference. If so, does it behave as you expect it? If yes, could
you test it on both a 1.1 and a 1.0 cluster?

Cheers,
Florian
--
Need help with High Availability?
http://www.hastexo.com/now
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/

On 2011-11-14 20:44, Marek Marczykowski wrote:
> On 14.11.2011 09:55, Raoul Bhatia [IPAX] wrote:
>> On 2011-11-11 09:22, Junko IKEDA wrote:
>>> Hi,
>>>
>>> I am running MySQL replication setting with 2 nodes Master/Slave
>>> configuration.
>>> If Slave status(secs_behind) is lager than Master's
>>> parameter(max_slave_lag),
>>> Slave data is outdated, right?
>>> check_slave() in mysql RA would run "crm_master -v 0" in this
>>> situation to mark Slave as "outdated",
>>> but if Master is shut down in this status,
>>> Slave will be able to promote instead of its old data.
>>> (is this correct?)
>>> It seems that "crm_master -v -INFINITY" is effectual to prevent Slave
>>> promotion.
>>
>> forwarding this to marek and fghaas as i'm not familiar with
>> multi-state handling inside resource agents.
>
> Did you set "evict_outdated_slaves"?

In a master/slave set, evict_outdated_slaves will actually kick out (by
failing with $OCF_ERR_INSTALLED) any slave that has fallen behind.

If set to false (the default), then the slave will be allowed to stay in
the cluster, but its master preference will be pushed down so it's not
promoted, and this seems to be Ikeda-san's preferred behavior. The
caveat which I mentioned in my other email in this thread applies here,
though.

For those pulling this thread from the archives: this information is in
the resource agent man page, and in "crm ra info ocf:heartbeat:mysql".

Cheers,
Florian


--
Need help with High Availability?
http://www.hastexo.com/now
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/

Hi Marek, Florian,

Thank you for your comments!

>> Did you set "evict_outdated_slaves"?

No,

> If set to false (the default), then the slave will be allowed to stay in
> the cluster, but its master preference will be pushed down so it's not
> promoted, and this seems to be Ikeda-san's preferred behavior. The
> caveat which I mentioned in my other email in this thread applies here,
> though.

Yes, this is what I expected.
I want to re-start the replication after re-connecting the network.

By the way, I have changed that attribute score from "0" to
"-INFINITY" with Pacemaker 1.0,
but there are still some problems... :(
I will gather the logs and post them here soon.

Thanks,
Junko
_______________________________________________________
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/