Mailing List Archive: 2 Nodes split brain, distant sites

2 Nodes split brain, distant sites

Feb 27, 2014, 6:42 AM

Post #1 of 10 (4852 views)

Hello,

Before starting, my first language is French so I'll try to do my best to explain my problem in English.

1) The situation :

I have 2 servers on 2 distant site.

I need to run openvpn with the same configuration on the 2 servers.
But it must run only on one server at a time.

I want that it start on the second server when the connection with internet is lost on the first node.

I use debian with corosync and pacemaker.

Here is the config :

A) Corosync.conf :
compatibility: whitetank
totem {
version: 2
token: 3000
token_retransmits_before_loss_const: 10
join: 240
consensus: 3600
vsftype: none
max_messages: 20
clear_node_high_bit: yes
secauth: off
threads: 0
nodeid: 1111
rrp_mode: none
interface {
member {
memberaddr: 172.16.135.9
}
member {
memberaddr: 172.16.64.248
}
ringnumber: 0
bindnetaddr: 172.16.135.9
mcastport: 5405
}
transport: udpu
}
amf {
mode: disabled
}
service {
ver: 0
name: pacemaker
}
aisexec {
user: root
group: root
}
logging {
fileline: off
to_stderr: yes
to_logfile: yes
logfile: /var/log/corosync/corosync.log
to_syslog: yes
syslog_facility: daemon
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
tags: enter|leave|trace1|trace2|trace3|trace4|trace6
}
}

B) Pacemaker :
node controle-col
node vpn-air
primitive ClusterMon ocf:pacemaker:ClusterMon \
params user="root" update="30" extra_options="-E /root/PacemakerMailScript.sh -h /tmp/ClusterMon.html" \
op monitor on-fail="restart" interval="60"
primitive openvpn lsb:openvpn \
op monitor interval="30s"
primitive p_ping ocf:pacemaker:ping \
params host_list="8.8.8.8 4.2.2.2" multiplier="100" dampen="5s" \
op monitor interval="60" timeout="60" \
op start interval="0" timeout="60" \
op stop interval="0" timeout="60"
clone ClusterMon-clone ClusterMon
clone c_ping p_ping
location OpenVpnCluster openvpn \
rule $id="OpenVpnCluster-rule" -inf: not_defined pingd or pingd lte 0
location PrefVpnAir openvpn \
rule $id="PrefVpnAir-rule" 50: #uname eq vpn-air
property $id="cib-bootstrap-options" \
dc-version="1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
stonith-enabled="false" \
no-quorum-policy="ignore"

C) Running good crm_mon
============
Last updated: Thu Feb 27 14:54:31 2014
Last change: Wed Jan 15 12:51:35 2014 via crmd on controle-col
Stack: openais
Current DC: controle-col - partition with quorum
Version: 1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff
2 Nodes configured, 2 expected votes
5 Resources configured.
============

Online: [ vpn-air controle-col ]

Clone Set: c_ping [p_ping]
Started: [ controle-col vpn-air ]
openvpn (lsb:openvpn): Started vpn-air
Clone Set: ClusterMon-clone [ClusterMon]
Started: [ controle-col vpn-air ]

2) My problem :

When there is a network problem :

Ex :
a) first-node site lost internet connection ( and communication with second-node at same time due to vpn on internet connection )
b) cluster stop openvpn on first node and launch it on second due to primitive p_ping in config.
c) connection come back on first-node site
d) Problem : first-node and second-node don't bring back cluster, the don't see each other and create a cluster on each node -> split brain I think.
e) Each node has openvpn running which shouldn't happen

I don't have stonith running because I think without quorum it will be problematic
Is there a way to say to corosync to recreate a ring ?

Or have someone another solution ?

Thanks

Tribolet Thomas
ISSeP (Institut Scientifique de Service Public)
th.tribolet@issep.be<mailto:th.tribolet@issep.be>
+32 (0) 4229 83 46

_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Re: 2 Nodes split brain, distant sites [ In reply to ]

fabian.herschel at arcor

Feb 27, 2014, 7:53 AM

Post #2 of 10 (4806 views)

Permalink

Hi,

my first idea would be to fix binnetaddr. It should be the
networkaddress not the machines network address.

Regards
Fabian

On 02/27/2014 03:42 PM, TRIBOLET Thomas wrote:
> Hello,
>
> Before starting, my first language is French so I'll try to do my best to explain my problem in English.
>
>
> 1) The situation :
>
> I have 2 servers on 2 distant site.
>
> I need to run openvpn with the same configuration on the 2 servers.
> But it must run only on one server at a time.
>
> I want that it start on the second server when the connection with internet is lost on the first node.
>
> I use debian with corosync and pacemaker.
>
> Here is the config :
>
>
> A) Corosync.conf :
> compatibility: whitetank
> totem {
> version: 2
> token: 3000
> token_retransmits_before_loss_const: 10
> join: 240
> consensus: 3600
> vsftype: none
> max_messages: 20
> clear_node_high_bit: yes
> secauth: off
> threads: 0
> nodeid: 1111
> rrp_mode: none
> interface {
> member {
> memberaddr: 172.16.135.9
> }
> member {
> memberaddr: 172.16.64.248
> }
> ringnumber: 0
> bindnetaddr: 172.16.135.9
> mcastport: 5405
> }
> transport: udpu
> }
> amf {
> mode: disabled
> }
> service {
> ver: 0
> name: pacemaker
> }
> aisexec {
> user: root
> group: root
> }
> logging {
> fileline: off
> to_stderr: yes
> to_logfile: yes
> logfile: /var/log/corosync/corosync.log
> to_syslog: yes
> syslog_facility: daemon
> debug: off
> timestamp: on
> logger_subsys {
> subsys: AMF
> debug: off
> tags: enter|leave|trace1|trace2|trace3|trace4|trace6
> }
> }
>
> B) Pacemaker :
> node controle-col
> node vpn-air
> primitive ClusterMon ocf:pacemaker:ClusterMon \
> params user="root" update="30" extra_options="-E /root/PacemakerMailScript.sh -h /tmp/ClusterMon.html" \
> op monitor on-fail="restart" interval="60"
> primitive openvpn lsb:openvpn \
> op monitor interval="30s"
> primitive p_ping ocf:pacemaker:ping \
> params host_list="8.8.8.8 4.2.2.2" multiplier="100" dampen="5s" \
> op monitor interval="60" timeout="60" \
> op start interval="0" timeout="60" \
> op stop interval="0" timeout="60"
> clone ClusterMon-clone ClusterMon
> clone c_ping p_ping
> location OpenVpnCluster openvpn \
> rule $id="OpenVpnCluster-rule" -inf: not_defined pingd or pingd lte 0
> location PrefVpnAir openvpn \
> rule $id="PrefVpnAir-rule" 50: #uname eq vpn-air
> property $id="cib-bootstrap-options" \
> dc-version="1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff" \
> cluster-infrastructure="openais" \
> expected-quorum-votes="2" \
> stonith-enabled="false" \
> no-quorum-policy="ignore"
>
>
> C) Running good crm_mon
> ============
> Last updated: Thu Feb 27 14:54:31 2014
> Last change: Wed Jan 15 12:51:35 2014 via crmd on controle-col
> Stack: openais
> Current DC: controle-col - partition with quorum
> Version: 1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff
> 2 Nodes configured, 2 expected votes
> 5 Resources configured.
> ============
>
> Online: [ vpn-air controle-col ]
>
> Clone Set: c_ping [p_ping]
> Started: [ controle-col vpn-air ]
> openvpn (lsb:openvpn): Started vpn-air
> Clone Set: ClusterMon-clone [ClusterMon]
> Started: [ controle-col vpn-air ]
>
>
> 2) My problem :
>
> When there is a network problem :
>
> Ex :
> a) first-node site lost internet connection ( and communication with second-node at same time due to vpn on internet connection )
> b) cluster stop openvpn on first node and launch it on second due to primitive p_ping in config.
> c) connection come back on first-node site
> d) Problem : first-node and second-node don't bring back cluster, the don't see each other and create a cluster on each node -> split brain I think.
> e) Each node has openvpn running which shouldn't happen
>
>
> I don't have stonith running because I think without quorum it will be problematic
> Is there a way to say to corosync to recreate a ring ?
>
> Or have someone another solution ?
>
> Thanks
>
>
> Tribolet Thomas
> ISSeP (Institut Scientifique de Service Public)
> th.tribolet@issep.be<mailto:th.tribolet@issep.be>
> +32 (0) 4229 83 46
>
> _______________________________________________
> Linux-HA mailing list
> Linux-HA@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
>

_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Re: 2 Nodes split brain, distant sites [ In reply to ]

lists at alteeve

Feb 27, 2014, 8:05 AM

Post #3 of 10 (4800 views)

Permalink

On 27/02/14 09:42 AM, TRIBOLET Thomas wrote:
> 2) My problem :
>
> When there is a network problem :
>
> Ex :
> a) first-node site lost internet connection ( and communication with second-node at same time due to vpn on internet connection )
> b) cluster stop openvpn on first node and launch it on second due to primitive p_ping in config.
> c) connection come back on first-node site
> d) Problem : first-node and second-node don't bring back cluster, the don't see each other and create a cluster on each node -> split brain I think.
> e) Each node has openvpn running which shouldn't happen
>
>
> I don't have stonith running because I think without quorum it will be problematic
> Is there a way to say to corosync to recreate a ring ?
>
> Or have someone another solution ?
>
> Thanks

Bonjour,

This is the fundamental problem of "stretch" clusters (or
geo-clusters). There is no way to tell the difference between a site
failure and a network failure. In either case, the link is down, so
fencing can't be used. Without fencing, there is no way to avoid a
split-brain.

As for quorum; When quorum isn't used, fencing becomes *more*
important. Even then, quorum and fencing solve different problems.
Quorum is useful when nodes are acting in a defined manner. Fencing is
needed when a node is in an unknown state (and thus acting in an
undefined manner).

So regardless of quorum, fencing is required. It is the only way to
reliably avoid split-brains. Unfortunately, fencing doesn't work on
stretch clusters.

The pacemaker project is working on something called "booth" which is
designed to deal with this problem, but I don't know much about it, or
whether it's out of testing/dev yet.

So in short, if you must have a stretch cluster, I recommend manual
failover only.

--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Re: 2 Nodes split brain, distant sites [ In reply to ]

th.tribolet at issep

Feb 27, 2014, 9:38 AM

Post #4 of 10 (4796 views)

Permalink

Hi,

Thanks for your response.

I'll try booth plugin. It's in debian repository.

If it don't work I'll try making a fake fencing agent that will only restart corosync. ( It seems it fix the problem )

Thanks
________________________________________
De : linux-ha-bounces@lists.linux-ha.org [linux-ha-bounces@lists.linux-ha.org] de la part de Digimer [lists@alteeve.ca]
Envoyé : jeudi 27 février 2014 17:05
À : General Linux-HA mailing list
Objet : Re: [Linux-HA] 2 Nodes split brain, distant sites

On 27/02/14 09:42 AM, TRIBOLET Thomas wrote:
> 2) My problem :
>
> When there is a network problem :
>
> Ex :
> a) first-node site lost internet connection ( and communication with second-node at same time due to vpn on internet connection )
> b) cluster stop openvpn on first node and launch it on second due to primitive p_ping in config.
> c) connection come back on first-node site
> d) Problem : first-node and second-node don't bring back cluster, the don't see each other and create a cluster on each node -> split brain I think.
> e) Each node has openvpn running which shouldn't happen
>
>
> I don't have stonith running because I think without quorum it will be problematic
> Is there a way to say to corosync to recreate a ring ?
>
> Or have someone another solution ?
>
> Thanks

Bonjour,

This is the fundamental problem of "stretch" clusters (or
geo-clusters). There is no way to tell the difference between a site
failure and a network failure. In either case, the link is down, so
fencing can't be used. Without fencing, there is no way to avoid a
split-brain.

As for quorum; When quorum isn't used, fencing becomes *more*
important. Even then, quorum and fencing solve different problems.
Quorum is useful when nodes are acting in a defined manner. Fencing is
needed when a node is in an unknown state (and thus acting in an
undefined manner).

So regardless of quorum, fencing is required. It is the only way to
reliably avoid split-brains. Unfortunately, fencing doesn't work on
stretch clusters.

The pacemaker project is working on something called "booth" which is
designed to deal with this problem, but I don't know much about it, or
whether it's out of testing/dev yet.

So in short, if you must have a stretch cluster, I recommend manual
failover only.

--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Re: 2 Nodes split brain, distant sites [ In reply to ]

lists at alteeve

Feb 27, 2014, 10:07 AM

Post #5 of 10 (4796 views)

Permalink

For fencing to work, it must actually cut off the other node. Simply
returning a fake success will cause a split-brain.

On 27/02/14 12:38 PM, TRIBOLET Thomas wrote:
> Hi,
>
> Thanks for your response.
>
> I'll try booth plugin. It's in debian repository.
>
> If it don't work I'll try making a fake fencing agent that will only restart corosync. ( It seems it fix the problem )
>
>
> Thanks
> ________________________________________
> De : linux-ha-bounces@lists.linux-ha.org [linux-ha-bounces@lists.linux-ha.org] de la part de Digimer [lists@alteeve.ca]
> Envoyé : jeudi 27 février 2014 17:05
> À : General Linux-HA mailing list
> Objet : Re: [Linux-HA] 2 Nodes split brain, distant sites
>
> On 27/02/14 09:42 AM, TRIBOLET Thomas wrote:
>> 2) My problem :
>>
>> When there is a network problem :
>>
>> Ex :
>> a) first-node site lost internet connection ( and communication with second-node at same time due to vpn on internet connection )
>> b) cluster stop openvpn on first node and launch it on second due to primitive p_ping in config.
>> c) connection come back on first-node site
>> d) Problem : first-node and second-node don't bring back cluster, the don't see each other and create a cluster on each node -> split brain I think.
>> e) Each node has openvpn running which shouldn't happen
>>
>>
>> I don't have stonith running because I think without quorum it will be problematic
>> Is there a way to say to corosync to recreate a ring ?
>>
>> Or have someone another solution ?
>>
>> Thanks
>
> Bonjour,
>
> This is the fundamental problem of "stretch" clusters (or
> geo-clusters). There is no way to tell the difference between a site
> failure and a network failure. In either case, the link is down, so
> fencing can't be used. Without fencing, there is no way to avoid a
> split-brain.
>
> As for quorum; When quorum isn't used, fencing becomes *more*
> important. Even then, quorum and fencing solve different problems.
> Quorum is useful when nodes are acting in a defined manner. Fencing is
> needed when a node is in an unknown state (and thus acting in an
> undefined manner).
>
> So regardless of quorum, fencing is required. It is the only way to
> reliably avoid split-brains. Unfortunately, fencing doesn't work on
> stretch clusters.
>
> The pacemaker project is working on something called "booth" which is
> designed to deal with this problem, but I don't know much about it, or
> whether it's out of testing/dev yet.
>
> So in short, if you must have a stretch cluster, I recommend manual
> failover only.
>
> --
> Digimer
> Papers and Projects: https://alteeve.ca/w/
> What if the cure for cancer is trapped in the mind of a person without
> access to education?
> _______________________________________________
> Linux-HA mailing list
> Linux-HA@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
> _______________________________________________
> Linux-HA mailing list
> Linux-HA@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
>

--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Re: 2 Nodes split brain, distant sites [ In reply to ]

th.tribolet at issep

Feb 28, 2014, 1:17 AM

Post #6 of 10 (4791 views)

Permalink

Hi,

I have taken a look at the documentation about booth. The problem is that it need a third independent arbitrator on a third connection that I don't have, which will grant or remove ticket.

My idea about fencing is to send an ssh command to reboot the server or just corosync but only from one server.

When there is a network problem :

With my constraint on ping I can assume that only one server will be running openvpn.
When network is back node1 will restart node2 to avoid problem.

Do you think that could work ?

I know it's dirty but I don't have another idea.

And how can I configure this sort of fencing on only one server ?

Thanks

-----Message d'origine-----
De : linux-ha-bounces@lists.linux-ha.org [mailto:linux-ha-bounces@lists.linux-ha.org] De la part de Digimer
Envoyé : jeudi 27 février 2014 19:07
À : General Linux-HA mailing list
Objet : Re: [Linux-HA] 2 Nodes split brain, distant sites

For fencing to work, it must actually cut off the other node. Simply returning a fake success will cause a split-brain.

On 27/02/14 12:38 PM, TRIBOLET Thomas wrote:
> Hi,
>
> Thanks for your response.
>
> I'll try booth plugin. It's in debian repository.
>
> If it don't work I'll try making a fake fencing agent that will only
> restart corosync. ( It seems it fix the problem )
>
>
> Thanks
> ________________________________________
> De : linux-ha-bounces@lists.linux-ha.org
> [linux-ha-bounces@lists.linux-ha.org] de la part de Digimer
> [lists@alteeve.ca] Envoyé : jeudi 27 février 2014 17:05 À : General
> Linux-HA mailing list Objet : Re: [Linux-HA] 2 Nodes split brain,
> distant sites
>
> On 27/02/14 09:42 AM, TRIBOLET Thomas wrote:
>> 2) My problem :
>>
>> When there is a network problem :
>>
>> Ex :
>> a) first-node site lost internet connection ( and communication with
>> second-node at same time due to vpn on internet connection )
>> b) cluster stop openvpn on first node and launch it on second due to primitive p_ping in config.
>> c) connection come back on first-node site
>> d) Problem : first-node and second-node don't bring back cluster, the don't see each other and create a cluster on each node -> split brain I think.
>> e) Each node has openvpn running which shouldn't happen
>>
>>
>> I don't have stonith running because I think without quorum it will
>> be problematic Is there a way to say to corosync to recreate a ring ?
>>
>> Or have someone another solution ?
>>
>> Thanks
>
> Bonjour,
>
> This is the fundamental problem of "stretch" clusters (or
> geo-clusters). There is no way to tell the difference between a site
> failure and a network failure. In either case, the link is down, so
> fencing can't be used. Without fencing, there is no way to avoid a
> split-brain.
>
> As for quorum; When quorum isn't used, fencing becomes *more*
> important. Even then, quorum and fencing solve different problems.
> Quorum is useful when nodes are acting in a defined manner. Fencing is
> needed when a node is in an unknown state (and thus acting in an
> undefined manner).
>
> So regardless of quorum, fencing is required. It is the only way
> to reliably avoid split-brains. Unfortunately, fencing doesn't work on
> stretch clusters.
>
> The pacemaker project is working on something called "booth" which
> is designed to deal with this problem, but I don't know much about it,
> or whether it's out of testing/dev yet.
>
> So in short, if you must have a stretch cluster, I recommend
> manual failover only.
>
> --
> Digimer
> Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer
> is trapped in the mind of a person without access to education?
> _______________________________________________
> Linux-HA mailing list
> Linux-HA@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
> _______________________________________________
> Linux-HA mailing list
> Linux-HA@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
>

--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Re: 2 Nodes split brain, distant sites [ In reply to ]

lmb at suse

Feb 28, 2014, 2:55 AM

Post #7 of 10 (4811 views)

Permalink

On 2014-02-27T11:05:21, Digimer <lists@alteeve.ca> wrote:

> So regardless of quorum, fencing is required. It is the only way to
> reliably avoid split-brains. Unfortunately, fencing doesn't work on stretch
> clusters.

For a two node stretch cluster, sbd can also be used reliably as a
fencing mechanism.

It essentially uses the standard iSCSI protocol as a quorum mechanism.
Export one (1MB or so) iSCSI LU from each site to the other, and in the
best case, host one at a 3rd site as tie-breaker. Then run SBD across
these.

booth is striving to address even longer distances, where each site is a
truly separate cluster (e.g., independent corosync/pacemaker setups,
totem not running across the gap).

Regards,
Lars

--
Architect Storage/HA
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Re: 2 Nodes split brain, distant sites [ In reply to ]

lists at alteeve

Feb 28, 2014, 10:13 AM

Post #8 of 10 (4788 views)

Permalink

On 28/02/14 04:17 AM, TRIBOLET Thomas wrote:
> Hi,
>
> I have taken a look at the documentation about booth. The problem is that it need a third independent arbitrator on a third connection that I don't have, which will grant or remove ticket.
>
> My idea about fencing is to send an ssh command to reboot the server or just corosync but only from one server.
>
> When there is a network problem :
>
> With my constraint on ping I can assume that only one server will be running openvpn.
> When network is back node1 will restart node2 to avoid problem.
>
> Do you think that could work ?
>
> I know it's dirty but I don't have another idea.
>
> And how can I configure this sort of fencing on only one server ?
>
> Thanks

You can't assume anything. To do so is to risk a split-brain, that's the
whole goal of fencing; It takes the guess work out. It allows the node
to say, for sure, that the peer is in a known state. As soon as you
guess, you risk problems.

Back to the original problem, you have a single point of failure in the
WAN link between the nodes. It is impossible to distinguish a WAN
failure from a site destruction.

--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Re: 2 Nodes split brain, distant sites [ In reply to ]

lists at alteeve

Feb 28, 2014, 10:16 AM

Post #9 of 10 (4789 views)

Permalink

On 28/02/14 05:55 AM, Lars Marowsky-Bree wrote:
> On 2014-02-27T11:05:21, Digimer <lists@alteeve.ca> wrote:
>
>> So regardless of quorum, fencing is required. It is the only way to
>> reliably avoid split-brains. Unfortunately, fencing doesn't work on stretch
>> clusters.
>
> For a two node stretch cluster, sbd can also be used reliably as a
> fencing mechanism.
>
> It essentially uses the standard iSCSI protocol as a quorum mechanism.
> Export one (1MB or so) iSCSI LU from each site to the other, and in the
> best case, host one at a 3rd site as tie-breaker. Then run SBD across
> these.
>
> booth is striving to address even longer distances, where each site is a
> truly separate cluster (e.g., independent corosync/pacemaker setups,
> totem not running across the gap).
>
>
>
> Regards,
> Lars

Assuming a SAN in each location (otherwise you have a single point of
failure), then isn't it still possible to end up with a split-brain
if/when the WAN link fails?

Something (drbd?) is going to be keeping the data in sync between the
locations. If both assume the other is dead, sure each location's SAN
will block the other node, but then each location will proceed
independently and their data will diverge, right?

--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Re: 2 Nodes split brain, distant sites [ In reply to ]

lmb at suse

Feb 28, 2014, 1:57 PM

Post #10 of 10 (4786 views)

Permalink

On 2014-02-28T13:16:33, Digimer <lists@alteeve.ca> wrote:

> Assuming a SAN in each location (otherwise you have a single point of
> failure), then isn't it still possible to end up with a split-brain if/when
> the WAN link fails?

As I suggested a 3rd tie-breaker site (which, in the case of SBD, can be
any old iSCSI LU somewhere), no, this can't happen. One site would fence
the other.

(The same is true, if a bit more elegantly, for booth.)

> Something (drbd?) is going to be keeping the data in sync between the
> locations. If both assume the other is dead, sure each location's SAN will
> block the other node, but then each location will proceed independently and
> their data will diverge, right?

For a two site setup, perhaps. A manual fail-over would need to make
sure that the other site is really stopped.

Assuming two sites, the currently active site would not be able to
commit new transactions as long as the WAN link is down and the other
site has not been declared dead (through automatic or manual fence
confirmation).

Also, any replication mechanism includes a means to pick a winner and
sync over such divergent changes, should they occur. Which they
shouldn't.

(Asynchronous replication on the other hand always has a RPO > 0, and
can always risk losing "a few" transactions. That is the nature of
disaster recovery. Hopefully, disasters are rare.)

Regards,
Lars

--
Architect Storage/HA
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems