Mailing List Archive

Hi,



Can I use pingd if running openais and pacemaker as I did in Heartbeat? If yes, how can I define ping nodes?

If no, is there some other way to ping hosts in openais?



Thanks,

Atanas

Hi,

Can I use pingd if running openais and pacemaker as I did in Heartbeat? If yes, how can I define ping nodes?
If no, is there some other way to ping hosts in openais?

Thanks,
Atanas

_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

Right, it doesnt work with openais yet.
pingd support for openais isnt far off though... but not much will
happen while i'm on vacation :)

On 2/29/08, Atanas Dyulgerov <atanas.dyulgerov@postpath.com> wrote:
> Hi,
>
>
>
> Can I use pingd if running openais and pacemaker as I did in Heartbeat? If
> yes, how can I define ping nodes?
>
> If no, is there some other way to ping hosts in openais?
>
>
>
> Thanks,
>
> Atanas
>
>
>
>
>
>
>

_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

Great. I'm planning to use OpenAIS instead of Heartbeat so I have to wait for pingd.

Please, consider the following features in the roadmap:
- Locking techniques supported in openais
- openais DR capabilities (quorum server, a ping daemon to influence the quorum or some other fancy method)

Thanks!

Have a great vacation :)


Regards,
Atanas


-----Original Message-----
From: Andrew Beekhof [mailto:beekhof@gmail.com]
Sent: Thursday, March 06, 2008 11:29 AM
To: The Pacemaker cluster resource manager
Subject: Re: [Pacemaker] Does pingd works on openais?

Right, it doesnt work with openais yet.
pingd support for openais isnt far off though... but not much will
happen while i'm on vacation :)

On 2/29/08, Atanas Dyulgerov <atanas.dyulgerov@postpath.com> wrote:
> Hi,
>
>
>
> Can I use pingd if running openais and pacemaker as I did in Heartbeat? If
> yes, how can I define ping nodes?
>
> If no, is there some other way to ping hosts in openais?
>
>
>
> Thanks,
>
> Atanas
>
>
>
>
>
>
>

_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

On 2008-03-06T11:43:49, Atanas Dyulgerov <atanas.dyulgerov@postpath.com> wrote:

> Great. I'm planning to use OpenAIS instead of Heartbeat so I have to
> wait for pingd.

That is laudable, but I hope by "use" you mean "test" and not production
use; I'd not expect to have the openAIS-based stack fully functional
and supportable in under 6 months.

> Please, consider the following features in the roadmap:
> - Locking techniques supported in openais

Hm? openAIS already supports locking.

> - openais DR capabilities (quorum server, a ping daemon to influence the quorum or some other fancy method)

Sure. DR-style deployments are on our roadmap.

quorumd is currently not functional on Heartbeat either, so that's not
really a differentiator so far ;-)


Regards,
Lars

--
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde


_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

-----Original Message-----
From: Lars Marowsky-Bree [mailto:lmb@suse.de]
Sent: Thursday, March 06, 2008 2:52 AM
To: The Pacemaker cluster resource manager
Subject: Re: [Pacemaker] Does pingd works on openais?

>On 2008-03-06T11:43:49, Atanas Dyulgerov <atanas.dyulgerov@postpath.com> >wrote:
>
>> Great. I'm planning to use OpenAIS instead of Heartbeat so I have to
>> wait for pingd.
>
>That is laudable, but I hope by "use" you mean "test" and not production
>use; I'd not expect to have the openAIS-based stack fully functional
>and supportable in under 6 months.
>

Isn't Red Hat Cluster Suite using CMAN on top of openais? Correct me is I'm wrong. They claim it is suitable for production use.

>> Please, consider the following features in the roadmap:
>> - Locking techniques supported in openais
>
>Hm? openAIS already supports locking.

Yes, I know openais supports locking. That's why Pacemaker must support those locking features to lock its resources. Currently Pacemaker cannot lock resources (iscsi, network block devices, filesystems like GFS, OCFS2 and etc.), right?

>
>> - openais DR capabilities (quorum server, a ping daemon to influence the >quorum or some other fancy method)
>
>Sure. DR-style deployments are on our roadmap.

Looking forward to seeing them.

>
>quorumd is currently not functional on Heartbeat either, so that's not
>really a differentiator so far ;-)

I wasn't able to make it work in heartbeat 2.1.3 :(

Btw, what would you say about Red Hat cluster suite? It seems to me like a very completed solution - supports resource locking (GNBD, GFS), cluster filesystem (GFS), quorum disk - things missing in Heartbeat...
GFS is very stable, has superior performance along with GNBD.

What will be the answer of Heartbeat/Pacemaker/OpenAIS? What would say about iSCSI+OCFS2+OpenAIS+Pacemaker solution although
- iscsi software targets does not support locking ("persistent reservation" was the word I think)
- OCFS2 is not completely stable and it is driven by its own cluster

Sorry for being too much interested in your projects :)

Regards,
Lars

--
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde


_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

On 2008-03-06T20:55:50, Atanas Dyulgerov <atanas.dyulgerov@postpath.com> wrote:

> Isn't Red Hat Cluster Suite using CMAN on top of openais? Correct me is I'm wrong. They claim it is suitable for production use.

I don't think CMAN is running on top of openAIS in the shipping distros
just yet.

Even with openAIS being production ready on its own, that doesn't
mean the CRM port will have all glitches ironed out yet. We love early
innovators, just be sure what you're getting ;-)

> >Hm? openAIS already supports locking.
> Yes, I know openais supports locking. That's why Pacemaker must
> support those locking features to lock its resources. Currently
> Pacemaker cannot lock resources (iscsi, network block devices,
> filesystems like GFS, OCFS2 and etc.), right?

That's something the resource agents would need to do. And as far as I
know, openAIS does not do the resource level locking you refer to; but
Pacemaker internally ensures exclusive access to them.

> >quorumd is currently not functional on Heartbeat either, so that's not
> >really a differentiator so far ;-)
> I wasn't able to make it work in heartbeat 2.1.3 :(

I know. quorumd is not functional as I said.


> Btw, what would you say about Red Hat cluster suite? It seems to me like a very completed solution - supports resource locking (GNBD, GFS), cluster filesystem (GFS), quorum disk - things missing in Heartbeat...
> GFS is very stable, has superior performance along with GNBD.

Hm? Heartbeat has support for OCFS2, which also supports locking in more
recent versions. But "resource locking" seems to be something else. That
heartbeat does not need a quorum disk is actually a _feature_, you know.
;-)

The first version of GFS to work with openAIS will likely be GFS2, not
yet shipping either.

> What will be the answer of Heartbeat/Pacemaker/OpenAIS? What would say about iSCSI+OCFS2+OpenAIS+Pacemaker solution although

That is going to work just fine, though if you're using GFS2 or OCFS2 on
top of GNBD or iSCSI (because you don't have a SAN, presumably), I'd
really wonder why you're not directly using NFS?

> - iscsi software targets does not support locking ("persistent reservation" was the word I think)

Why would it need to support that?

> - OCFS2 is not completely stable and it is driven by its own cluster

OCFS2 is not completely stable?

On SLES10, we ship OCFS2 with integration into the heartbeat/CRM
project.


Regards,
Lars

--
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde


_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

-----Original Message-----
From: Lars Marowsky-Bree [mailto:lmb@suse.de]
Sent: Thursday, March 06, 2008 11:08 AM
To: The Pacemaker cluster resource manager
Subject: Re: [Pacemaker] Does pingd works on openais?

On 2008-03-06T20:55:50, Atanas Dyulgerov <atanas.dyulgerov@postpath.com> wrote:

>> Isn't Red Hat Cluster Suite using CMAN on top of openais? Correct me is >I'm wrong. They claim it is suitable for production use.
>
>I don't think CMAN is running on top of openAIS in the shipping distros
>just yet.

Hm, I left with the impression openais is part of the red hat cluster suite in some way: http://sources.redhat.com/cluster/faq.html#what

>
>Even with openAIS being production ready on its own, that doesn't
>mean the CRM port will have all glitches ironed out yet. We love early
>innovators, just be sure what you're getting ;-)
>
>> >Hm? openAIS already supports locking.
>> Yes, I know openais supports locking. That's why Pacemaker must
>> support those locking features to lock its resources. Currently
>> Pacemaker cannot lock resources (iscsi, network block devices,
>> filesystems like GFS, OCFS2 and etc.), right?
>
>That's something the resource agents would need to do. And as far as I
>know, openAIS does not do the resource level locking you refer to; but
>Pacemaker internally ensures exclusive access to them.

Yes, Pacemaker should support resource locking.

>Hm? Heartbeat has support for OCFS2, which also supports locking in more
>recent versions. But "resource locking" seems to be something else. That
>heartbeat does not need a quorum disk is actually a _feature_, you know.
>;-)

This is a new info for me. I knew that ocfs2 includes a simple node information layer which holds a description of the systems which make up the cluster. OCFS2 also includes a simple heartbeat implementation for monitoring which nodes are actually alive - tcp-based messaging system which is used to talk between nodes in a cluster.

So you are saying that Heartbeat replaces successfully all of the above? Can OCFS2 get its membership information from Heartbeat? Does Heartbeat CRM controls OCFS2 distributed lock manager?

If true, could you send me some links for that? Patches, docs, sample configuration in Heartbeat? Thanks!

>
>The first version of GFS to work with openAIS will likely be GFS2, not
>yet shipping either.
>
>> What will be the answer of Heartbeat/Pacemaker/OpenAIS? What would say >about iSCSI+OCFS2+OpenAIS+Pacemaker solution although
>
>That is going to work just fine, though if you're using GFS2 or OCFS2 on
>top of GNBD or iSCSI (because you don't have a SAN, presumably), I'd
>really wonder why you're not directly using NFS?

Hm? GFS and GNBD works with red hat cluster suite only?

I don't have SAN. I'm looking for a cheaper solution. The reason I'm not using NFS is the slower performance compared to the fastest GNBD and iSCSI. Also a very specific service application is running on the cluster and it does not get along with NFS very well.

>
>> - iscsi software targets does not support locking ("persistent >reservation" was the word I think)
>
>Why would it need to support that?

To lock the network block device if node fails to bring down the service application or if for any reason the cluster software fails on that node. In that case the application will be started on a passive node and the cluster might end up with two nodes which try to access the same data.

GNBD supports locking but unfortunately it operates with red hat cluster suite only.

>
>> - OCFS2 is not completely stable and it is driven by its own cluster
>
>OCFS2 is not completely stable?
>
That's what I read on various forums... not my personal experience.

>On SLES10, we ship OCFS2 with integration into the heartbeat/CRM
>project.

Regards,
Atanas

--
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde


_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

On 2008-03-07T20:11:46, Atanas Dyulgerov <atanas.dyulgerov@postpath.com> wrote:

> Hm, I left with the impression openais is part of the red hat cluster suite in some way: http://sources.redhat.com/cluster/faq.html#what

Check question 12 ;-)

> >That's something the resource agents would need to do. And as far as I
> >know, openAIS does not do the resource level locking you refer to; but
> >Pacemaker internally ensures exclusive access to them.
> Yes, Pacemaker should support resource locking.

But it does. Pacemaker already "locks" the resources to one node, and
orders fencing, STONITH, start/stop etc as needed to ensure that
resources are not running where they aren't allowed to.That's what a
cluster resource manager is all about.

I'm not sure what more you are asking for?

> So you are saying that Heartbeat replaces successfully all of the above? Can OCFS2 get its membership information from Heartbeat? Does Heartbeat CRM controls OCFS2 distributed lock manager?
>
> If true, could you send me some links for that? Patches, docs, sample configuration in Heartbeat? Thanks!

We've got that working in SLES10. The patches are all open source (of
course), and the user-space membership code is supposedly merged
upstream in OCFS2 by now.

On the heartbeat/Pacemaker side, it is as easy as configuring the
Filesystem resource for the ocfs2 mount, just as a clone.

> >That is going to work just fine, though if you're using GFS2 or OCFS2 on
> >top of GNBD or iSCSI (because you don't have a SAN, presumably), I'd
> >really wonder why you're not directly using NFS?
> Hm? GFS and GNBD works with red hat cluster suite only?

GFS yes; GFS2 will eventually work with Pacemaker as well.

> I don't have SAN. I'm looking for a cheaper solution. The reason I'm
> not using NFS is the slower performance compared to the fastest GNBD
> and iSCSI.

Have you _benchmarked_ that for your workload? iSCSI/GNBD are
block-level protocols, that means a higher overhead over the network.
And because OCFS2/GFS2 do their own locking, the nodes also have a fair
bit of additional internode communication.

> Also a very specific service application is running on the cluster and
> it does not get along with NFS very well.

What kind of features does it use that don't get along with NFS? In
theory, NFSv4/3 should be 1:1 compatible?

> To lock the network block device if node fails to bring down the
> service application or if for any reason the cluster software fails on
> that node. In that case the application will be started on a passive
> node and the cluster might end up with two nodes which try to access
> the same data.

Pacemaker/Heartbeat handle this through fencing/STONITH. It'd of course
be fatal if not.

> >> - OCFS2 is not completely stable and it is driven by its own cluster
> >OCFS2 is not completely stable?
> That's what I read on various forums... not my personal experience.

Ah. Rumors. ;-)



Regards,
Lars

--
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde


_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

On Fri, Mar 7, 2008 at 12:17 PM, Lars Marowsky-Bree <lmb@suse.de> wrote:
> On 2008-03-07T20:11:46, Atanas Dyulgerov <atanas.dyulgerov@postpath.com> wrote:
>
> > Hm, I left with the impression openais is part of the red hat cluster suite in some way: http://sources.redhat.com/cluster/faq.html#what
>
> Check question 12 ;-)
>
> > >That's something the resource agents would need to do. And as far as I
> > >know, openAIS does not do the resource level locking you refer to; but
> > >Pacemaker internally ensures exclusive access to them.
> > Yes, Pacemaker should support resource locking.
>
> But it does. Pacemaker already "locks" the resources to one node, and
> orders fencing, STONITH, start/stop etc as needed to ensure that
> resources are not running where they aren't allowed to.That's what a
> cluster resource manager is all about.
>
> I'm not sure what more you are asking for?

In fact I'm afraid that CRM lacks one feature that other clustering
projects have. It's that quorum disk. In RedHat ClusterSuite or in HP
ServiceGuard quorum disk helps to fight split brain scenario. It's not
clear how CRM acts when heartbeat link gets broken and nodes can't
communicate to each other. What I see in my logs both nodes try to
STONITH each other which isn't the best way to handle this problem.

Please do not mistake quorum drive (feature for local cluster) with
quorumd server which was designed for geographically spread clusters.

>
> > So you are saying that Heartbeat replaces successfully all of the above? Can OCFS2 get its membership information from Heartbeat? Does Heartbeat CRM controls OCFS2 distributed lock manager?
> >
> > If true, could you send me some links for that? Patches, docs, sample configuration in Heartbeat? Thanks!
>
> We've got that working in SLES10. The patches are all open source (of
> course), and the user-space membership code is supposedly merged
> upstream in OCFS2 by now.
>
> On the heartbeat/Pacemaker side, it is as easy as configuring the
> Filesystem resource for the ocfs2 mount, just as a clone.
>
> > >That is going to work just fine, though if you're using GFS2 or OCFS2 on
> > >top of GNBD or iSCSI (because you don't have a SAN, presumably), I'd
> > >really wonder why you're not directly using NFS?
> > Hm? GFS and GNBD works with red hat cluster suite only?
>
> GFS yes; GFS2 will eventually work with Pacemaker as well.
>
> > I don't have SAN. I'm looking for a cheaper solution. The reason I'm
> > not using NFS is the slower performance compared to the fastest GNBD
> > and iSCSI.
>
> Have you _benchmarked_ that for your workload? iSCSI/GNBD are
> block-level protocols, that means a higher overhead over the network.
> And because OCFS2/GFS2 do their own locking, the nodes also have a fair
> bit of additional internode communication.
>
> > Also a very specific service application is running on the cluster and
> > it does not get along with NFS very well.
>
> What kind of features does it use that don't get along with NFS? In
> theory, NFSv4/3 should be 1:1 compatible?
>
> > To lock the network block device if node fails to bring down the
> > service application or if for any reason the cluster software fails on
> > that node. In that case the application will be started on a passive
> > node and the cluster might end up with two nodes which try to access
> > the same data.
>
> Pacemaker/Heartbeat handle this through fencing/STONITH. It'd of course
> be fatal if not.
>
> > >> - OCFS2 is not completely stable and it is driven by its own cluster
> > >OCFS2 is not completely stable?
> > That's what I read on various forums... not my personal experience.
>
> Ah. Rumors. ;-)
>
>
>
> Regards,
> Lars
>
> --
> Teamlead Kernel, SuSE Labs, Research and Development
> SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
> "Experience is the name everyone gives to their mistakes." -- Oscar Wilde
>
>
> _______________________________________________
> Pacemaker mailing list
> Pacemaker@clusterlabs.org
> http://list.clusterlabs.org/mailman/listinfo/pacemaker
>



--
Serge Dubrouski.

_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

On 2008-03-07T12:30:27, Serge Dubrouski <sergeyfd@gmail.com> wrote:

> In fact I'm afraid that CRM lacks one feature that other clustering
> projects have. It's that quorum disk. In RedHat ClusterSuite or in HP
> ServiceGuard quorum disk helps to fight split brain scenario.

That's not a CRM/Pacemaker-level feature though. That needs to be
provided by the cluster infrastructure below.

You're right that heartbeat does not have a disk quorum plugin. Xinwei
has written a disk-based comm plugin, but that wasn't submitted
upstream. (And, AFAIK, never shipped anywhere.)

> It's not clear how CRM acts when heartbeat link gets broken and nodes
> can't communicate to each other. What I see in my logs both nodes try
> to STONITH each other which isn't the best way to handle this
> problem.

The surviving side would still fence the other, so that's a partially
separate issue.

The fact that links get broken completely, but both side can still reach
the STONITH device, however is statistically very rare. In fact,
connectivity to the STONITH device becomes the "quorum token."

I'm not saying it wouldn't be nice to support disk-based comms/quorum
with heartbeat as well, but really it isn't a crucial feature.

> Please do not mistake quorum drive (feature for local cluster) with
> quorumd server which was designed for geographically spread clusters.

I'm very clear on the distinction ;-)

Though this is not entirely true: it's essentially the same concept. A
3rd party tie-breaker is introduced to decide equal node count splits.
Quorum disk, or quorum server - it's the same idea; one scenario uses
TCP/SSL and the other SCSI reservations, that's the entire difference.
And if, as in this case, the SCSI reservation is in fact handled by the
iSCSI server (over TCP), the distinction becomes pretty much mood.


Regards,
Lars

PS: And please, trim the quotes, if you would be so kind ;-)

--
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde


_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

-----Original Message-----
From: Lars Marowsky-Bree [mailto:lmb@suse.de]
Sent: Friday, March 07, 2008 9:18 PM
To: The Pacemaker cluster resource manager
Subject: Re: [Pacemaker] Does pingd works on openais?

On 2008-03-07T20:11:46, Atanas Dyulgerov <atanas.dyulgerov@postpath.com> wrote:

>> To lock the network block device if node fails to bring down the
>> service application or if for any reason the cluster software fails on
>> that node. In that case the application will be started on a passive
>> node and the cluster might end up with two nodes which try to access
>> the same data.
>
>Pacemaker/Heartbeat handle this through fencing/STONITH. It'd of course
>be fatal if not.
>
>But it does. Pacemaker already "locks" the resources to one node, and
>orders fencing, STONITH, start/stop etc as needed to ensure that
>resources are not running where they aren't allowed to.That's what a
>cluster resource manager is all about.
>
>I'm not sure what more you are asking for?

STONITH brutally shutdowns a node. To do that you need redundant communication lines, smart power devices and definitely a local cluster. For geographically separated cluster with remote nodes STONITH is not applicable.
The method is called Node Fencing and as I said it has too many obstacles.

As for me a better choice is 'resources locking' option aka Resource Fencing. Instead of killing the errant node the cluster CRM just fence/lock its I/O access to the shared storage resource unit cluster messaging system reports back successful service stop on that node. Perfectly suits DR cluster and no need of additional communication lines. More elegant solution!

Heartbeat/Pacemaker does not support resource fencing. To fence resources, the resources have to support locking features by themselves. You cannot lock something that cannot be locked. Software iSCSI targets does not have locking mechanisms whereas GNBD does. However, GNBD locking features work with RedHat ClusterSuite only.

I don't understand what you mean by saying "Pacemaker already "locks" the resources to one node, and orders fencing ...". Which resources does Pacemaker lock?

What I'm trying to say is that the resource fencing is at least as important as node fencing. Pacemaker should be able to support the feature like RedHat Cluster Suite does. Resource locking should be supported in CRM/Pacemaker as well as in the resource by itself (gnbd, iscsi, drbd, etc...).


>> I don't have SAN. I'm looking for a cheaper solution. The reason I'm
>> not using NFS is the slower performance compared to the fastest GNBD
>> and iSCSI.
>
>Have you _benchmarked_ that for your workload?

Yes, I have benchmarked the application performance. With GNBD it had the best score, then comes iSCSI and NFS is at the end.

> iSCSI/GNBD are
>block-level protocols, that means a higher overhead over the network.
>And because OCFS2/GFS2 do their own locking, the nodes also have a fair
>bit of additional internode communication.
>
>> Also a very specific service application is running on the cluster and
>> it does not get along with NFS very well.
>
>What kind of features does it use that don't get along with NFS? In
>theory, NFSv4/3 should be 1:1 compatible?

There are internal application bugs causing problems when running on NFS. I'm not familiar with them in details. NFS is not an option for me.

Regards,
Atanas

--
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde


_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

Hi - just thought I would jump in here

>The fact that links get broken completely, but both side can still reach
>the STONITH device, however is statistically very rare. In fact,
>connectivity to the STONITH device becomes the "quorum token."

This may not always true in a virtual environment if two nodes are running under a VM and the VM in turn runs a virtual Stonith device.



Thanks...

Phil Pinto

-----Original Message-----
From: pacemaker-bounces@clusterlabs.org [mailto:pacemaker-bounces@clusterlabs.org] On Behalf Of Lars Marowsky-Bree
Sent: Friday, March 07, 2008 3:29 PM
To: The Pacemaker cluster resource manager
Subject: Re: [Pacemaker] Does pingd works on openais?

On 2008-03-07T12:30:27, Serge Dubrouski <sergeyfd@gmail.com> wrote:

> In fact I'm afraid that CRM lacks one feature that other clustering
> projects have. It's that quorum disk. In RedHat ClusterSuite or in HP
> ServiceGuard quorum disk helps to fight split brain scenario.

That's not a CRM/Pacemaker-level feature though. That needs to be
provided by the cluster infrastructure below.

You're right that heartbeat does not have a disk quorum plugin. Xinwei
has written a disk-based comm plugin, but that wasn't submitted
upstream. (And, AFAIK, never shipped anywhere.)

> It's not clear how CRM acts when heartbeat link gets broken and nodes
> can't communicate to each other. What I see in my logs both nodes try
> to STONITH each other which isn't the best way to handle this
> problem.

The surviving side would still fence the other, so that's a partially
separate issue.

The fact that links get broken completely, but both side can still reach
the STONITH device, however is statistically very rare. In fact,
connectivity to the STONITH device becomes the "quorum token."

I'm not saying it wouldn't be nice to support disk-based comms/quorum
with heartbeat as well, but really it isn't a crucial feature.

> Please do not mistake quorum drive (feature for local cluster) with
> quorumd server which was designed for geographically spread clusters.

I'm very clear on the distinction ;-)

Though this is not entirely true: it's essentially the same concept. A
3rd party tie-breaker is introduced to decide equal node count splits.
Quorum disk, or quorum server - it's the same idea; one scenario uses
TCP/SSL and the other SCSI reservations, that's the entire difference.
And if, as in this case, the SCSI reservation is in fact handled by the
iSCSI server (over TCP), the distinction becomes pretty much mood.


Regards,
Lars

PS: And please, trim the quotes, if you would be so kind ;-)

--
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde


_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker
--------------------------------------------------------

This message w/attachments (message) may be privileged, confidential or proprietary, and if you are not an intended recipient, please notify the sender, do not use or share it and delete it. Unless specifically indicated, this message is not an offer to sell or a solicitation of any investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Merrill Lynch. Subject to applicable law, Merrill Lynch may monitor, review and retain e-communications (EC) traveling through its networks/systems. The laws of the country of each sender/recipient may impact the handling of EC, and EC may be archived, supervised and produced in countries other than the country in which you are located. This message cannot be guaranteed to be secure or error-free. This message is subject to terms available at the following link: http://www.ml.com/e-communications_terms/. By messaging with Merrill Lynch you consent to the foregoing.
--------------------------------------------------------

_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

Hi - just thought I would jump in here

>The fact that links get broken completely, but both side can still reach
>the STONITH device, however is statistically very rare. In fact,
>connectivity to the STONITH device becomes the "quorum token."

This may not always true in a virtual environment if two nodes are running under a VM and the VM in turn runs a virtual Stonith device.



Thanks...

Phil Pinto

-----Original Message-----
From: pacemaker-bounces@clusterlabs.org [mailto:pacemaker-bounces@clusterlabs.org] On Behalf Of Lars Marowsky-Bree
Sent: Friday, March 07, 2008 3:29 PM
To: The Pacemaker cluster resource manager
Subject: Re: [Pacemaker] Does pingd works on openais?

On 2008-03-07T12:30:27, Serge Dubrouski <sergeyfd@gmail.com> wrote:

> In fact I'm afraid that CRM lacks one feature that other clustering
> projects have. It's that quorum disk. In RedHat ClusterSuite or in HP
> ServiceGuard quorum disk helps to fight split brain scenario.

That's not a CRM/Pacemaker-level feature though. That needs to be
provided by the cluster infrastructure below.

You're right that heartbeat does not have a disk quorum plugin. Xinwei
has written a disk-based comm plugin, but that wasn't submitted
upstream. (And, AFAIK, never shipped anywhere.)

> It's not clear how CRM acts when heartbeat link gets broken and nodes
> can't communicate to each other. What I see in my logs both nodes try
> to STONITH each other which isn't the best way to handle this
> problem.

The surviving side would still fence the other, so that's a partially
separate issue.

The fact that links get broken completely, but both side can still reach
the STONITH device, however is statistically very rare. In fact,
connectivity to the STONITH device becomes the "quorum token."

I'm not saying it wouldn't be nice to support disk-based comms/quorum
with heartbeat as well, but really it isn't a crucial feature.

> Please do not mistake quorum drive (feature for local cluster) with
> quorumd server which was designed for geographically spread clusters.

I'm very clear on the distinction ;-)

Though this is not entirely true: it's essentially the same concept. A
3rd party tie-breaker is introduced to decide equal node count splits.
Quorum disk, or quorum server - it's the same idea; one scenario uses
TCP/SSL and the other SCSI reservations, that's the entire difference.
And if, as in this case, the SCSI reservation is in fact handled by the
iSCSI server (over TCP), the distinction becomes pretty much mood.


Regards,
Lars

PS: And please, trim the quotes, if you would be so kind ;-)

--
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde


_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker
--------------------------------------------------------

This message w/attachments (message) may be privileged, confidential or proprietary, and if you are not an intended recipient, please notify the sender, do not use or share it and delete it. Unless specifically indicated, this message is not an offer to sell or a solicitation of any investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Merrill Lynch. Subject to applicable law, Merrill Lynch may monitor, review and retain e-communications (EC) traveling through its networks/systems. The laws of the country of each sender/recipient may impact the handling of EC, and EC may be archived, supervised and produced in countries other than the country in which you are located. This message cannot be guaranteed to be secure or error-free. This message is subject to terms available at the following link: http://www.ml.com/e-communications_terms/. By messaging with Merrill Lynch you consent to the foregoing.
--------------------------------------------------------

_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

On Mar 10, 2008, at 10:13 AM, Atanas Dyulgerov wrote:

>
>
> -----Original Message-----
> From: Lars Marowsky-Bree [mailto:lmb@suse.de]
> Sent: Friday, March 07, 2008 9:18 PM
> To: The Pacemaker cluster resource manager
> Subject: Re: [Pacemaker] Does pingd works on openais?
>
> On 2008-03-07T20:11:46, Atanas Dyulgerov <atanas.dyulgerov@postpath.com
> > wrote:
>
>>> To lock the network block device if node fails to bring down the
>>> service application or if for any reason the cluster software
>>> fails on
>>> that node. In that case the application will be started on a passive
>>> node and the cluster might end up with two nodes which try to access
>>> the same data.
>>
>> Pacemaker/Heartbeat handle this through fencing/STONITH. It'd of
>> course
>> be fatal if not.
>>
>> But it does. Pacemaker already "locks" the resources to one node, and
>> orders fencing, STONITH, start/stop etc as needed to ensure that
>> resources are not running where they aren't allowed to.That's what a
>> cluster resource manager is all about.
>>
>> I'm not sure what more you are asking for?
>
> STONITH brutally shutdowns a node. To do that you need redundant
> communication lines, smart power devices and definitely a local
> cluster. For geographically separated cluster with remote nodes
> STONITH is not applicable.
> The method is called Node Fencing and as I said it has too many
> obstacles.
>
> As for me a better choice is 'resources locking' option aka Resource
> Fencing. Instead of killing the errant node the cluster CRM just
> fence/lock its I/O access to the shared storage resource unit
> cluster messaging system reports back successful service stop on
> that node. Perfectly suits DR cluster and no need of additional
> communication lines. More elegant solution!
>
> Heartbeat/Pacemaker does not support resource fencing.

Technically it does (or could)... you just need to write an RA that
implements the locking you want.

a) write the locking RA
b) run it everywhere as a clone
c) have the RA "fail" when the node/resource/whatever (a fancy version
might implement more than one zone) is "fenced"
d) have any resource that needs resource fencing depend on the lock
resource (colocation)

At least thats how I remember the discussion.
I forget exactly how the exclusion of the node (so that the locking
resource fails) was going to work... maybe a new stonith plugin would
do the trick.

> To fence resources, the resources have to support locking features
> by themselves. You cannot lock something that cannot be locked.
> Software iSCSI targets does not have locking mechanisms whereas GNBD
> does. However, GNBD locking features work with RedHat ClusterSuite
> only.
>
> I don't understand what you mean by saying "Pacemaker already
> "locks" the resources to one node, and orders fencing ...". Which
> resources does Pacemaker lock?
>
> What I'm trying to say is that the resource fencing is at least as
> important as node fencing. Pacemaker should be able to support the
> feature like RedHat Cluster Suite does. Resource locking should be
> supported in CRM/Pacemaker as well as in the resource by itself
> (gnbd, iscsi, drbd, etc...).
>
>
>>> I don't have SAN. I'm looking for a cheaper solution. The reason I'm
>>> not using NFS is the slower performance compared to the fastest GNBD
>>> and iSCSI.
>>
>> Have you _benchmarked_ that for your workload?
>
> Yes, I have benchmarked the application performance. With GNBD it
> had the best score, then comes iSCSI and NFS is at the end.
>
>> iSCSI/GNBD are
>> block-level protocols, that means a higher overhead over the network.
>> And because OCFS2/GFS2 do their own locking, the nodes also have a
>> fair
>> bit of additional internode communication.
>>
>>> Also a very specific service application is running on the cluster
>>> and
>>> it does not get along with NFS very well.
>>
>> What kind of features does it use that don't get along with NFS? In
>> theory, NFSv4/3 should be 1:1 compatible?
>
> There are internal application bugs causing problems when running on
> NFS. I'm not familiar with them in details. NFS is not an option for
> me.
>
> Regards,
> Atanas
>
> --
> Teamlead Kernel, SuSE Labs, Research and Development
> SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
> "Experience is the name everyone gives to their mistakes." -- Oscar
> Wilde
>
>
> _______________________________________________
> Pacemaker mailing list
> Pacemaker@clusterlabs.org
> http://list.clusterlabs.org/mailman/listinfo/pacemaker
>
> _______________________________________________
> Pacemaker mailing list
> Pacemaker@clusterlabs.org
> http://list.clusterlabs.org/mailman/listinfo/pacemaker


_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

On 2008-03-10T11:13:51, Atanas Dyulgerov <atanas.dyulgerov@postpath.com> wrote:

> STONITH brutally shutdowns a node. To do that you need redundant
> communication lines, smart power devices and definitely a local
> cluster. For geographically separated cluster with remote nodes
> STONITH is not applicable. The method is called Node Fencing and as I
> said it has too many obstacles.

All fencing methods require a means of communicating with the device; in
case of WAN clusters, the link between the (replicating) storage arrays
(for example) will also be cut - resource fencing is unavailable for the
same reasons as STONITH is.

WAN clusters require the concept of self-fencing after loss of site
quorum.

All discussions regarding node versus resource level fencing are
correct and hold true, but as Andrew explained, resource fencing we
already could support if the RAs implemented it - if you set
no-quorum-policy=ignore, you've turned the cluster into a
resource-driven model, like SteelEye's LifeKeeper.

Yet, the wide-area versus metro-area versus local data center cluster
discussion is quite separate from this.

> As for me a better choice is 'resources locking' option aka Resource
> Fencing. Instead of killing the errant node the cluster CRM just
> fence/lock its I/O access to the shared storage resource unit cluster
> messaging system reports back successful service stop on that node.
> Perfectly suits DR cluster and no need of additional communication
> lines. More elegant solution!

Again, that's not quite true; see above. How does the resource itself
ensure fencing if the links are all cut? If you have EMC Symmetrix with
builtin replication, that has exactly the same problem - you need a
tie-breaker to decide how to continue and where. Whether you then do
resource fencing or node fencing is not necessarily much different.

> Heartbeat/Pacemaker does not support resource fencing. To fence
> resources, the resources have to support locking features by
> themselves. You cannot lock something that cannot be locked. Software
> iSCSI targets does not have locking mechanisms whereas GNBD does.
> However, GNBD locking features work with RedHat ClusterSuite only.

iSCSI targets in theory support SCSI reservations. Yet that only works
if the target is centralized (and thus a SPoF) - if you replicate it,
see above.

> I don't understand what you mean by saying "Pacemaker already "locks"
> the resources to one node, and orders fencing ...". Which resources
> does Pacemaker lock?

It's internal, and not exposed.

> What I'm trying to say is that the resource fencing is at least as
> important as node fencing. Pacemaker should be able to support the
> feature like RedHat Cluster Suite does. Resource locking should be
> supported in CRM/Pacemaker as well as in the resource by itself (gnbd,
> iscsi, drbd, etc...).

True, resource level fencing is desirable, and the RAs should do it
automatically. Possibly, certain extensions to Pacemaker might also be
useful here, though I think that's not the biggest obstacle.


> >> I don't have SAN. I'm looking for a cheaper solution. The reason
> >> I'm not using NFS is the slower performance compared to the fastest
> >> GNBD and iSCSI.
> >Have you _benchmarked_ that for your workload?
> Yes, I have benchmarked the application performance. With GNBD it had
> the best score, then comes iSCSI and NFS is at the end.

That I find quite surprising. I'd like to duplicate that benchmark
eventually; do you have the benchmark description and results available
somewhere?


Regards,
Lars

--
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde


_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

On 2008-03-12T16:22:34, "Pinto, Phil (GTI)" <phil_pinto@ml.com> wrote:

> >The fact that links get broken completely, but both side can still
> >reach the STONITH device, however is statistically very rare. In
> >fact, connectivity to the STONITH device becomes the "quorum token."
> This may not always true in a virtual environment if two nodes are
> running under a VM and the VM in turn runs a virtual Stonith device.

I didn't say "impossible", I said "statistically very rare". And that it
is - almost always, the broken link means that one side has crashed, or
in the case of a hypervisor, the HVM has broken down.

And, the worst that will happen is that both sides reboot (brief outage)
or shutdown (loss-of-service); while highly undesirable, it is not as
bad as sacrificing correctness.

In any case, almost all the special considerations we discuss here apply
to quorum disks as well - token is token. There is no magic weapon which
never fails - we are always discussing probabilities here, and anyone
who claims 100% certainity is lieing.

(I only do that in executive level presentations at customers, because
they don't like to hear about the theoretical possibilities; but when
we're having an engineering/research level discussion like here, I feel
I must insist on being true about all possible approaches.)


Regards,
Lars

--
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde


_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

-----Original Message-----
From: Lars Marowsky-Bree [mailto:lmb@suse.de]
Sent: Tuesday, March 18, 2008 4:59 PM
To: The Pacemaker cluster resource manager
Subject: Re: [Pacemaker] Does pingd works on openais?

On 2008-03-10T11:13:51, Atanas Dyulgerov <atanas.dyulgerov@postpath.com> wrote:

>> STONITH brutally shutdowns a node. To do that you need redundant
>> communication lines, smart power devices and definitely a local
>> cluster. For geographically separated cluster with remote nodes
>> STONITH is not applicable. The method is called Node Fencing and as I
>> said it has too many obstacles.
>
>All fencing methods require a means of communicating with the device; in
>case of WAN clusters, the link between the (replicating) storage arrays
>(for example) will also be cut - resource fencing is unavailable for the
>same reasons as STONITH is.

The case when node loses connectivity to the cluster but it still remains
connected to the shared resource. Then the other nodes which retain quorum
can lock the shared storage resource to stop the errant node from accessing
it. This fencing method does not require communication with the failed node.
That's what RHCS do I believe.

Shutting down a node seems inconvenient for me. You have to manually power
it on. What happens if the network link disconnects two times a day?

>WAN clusters require the concept of self-fencing after loss of site
>quorum.

Any self-fencing method for production use implemented so far? I would
like to test...

>All discussions regarding node versus resource level fencing are
>correct and hold true, but as Andrew explained, resource fencing we
>already could support if the RAs implemented it - if you set
>no-quorum-policy=ignore, you've turned the cluster into a
>resource-driven model, like SteelEye's LifeKeeper.
>
>Yet, the wide-area versus metro-area versus local data center cluster
>discussion is quite separate from this.
>
>> As for me a better choice is 'resources locking' option aka Resource
>> Fencing. Instead of killing the errant node the cluster CRM just
>> fence/lock its I/O access to the shared storage resource unit cluster
>> messaging system reports back successful service stop on that node.
>> Perfectly suits DR cluster and no need of additional communication
>> lines. More elegant solution!
>
>Again, that's not quite true; see above. How does the resource itself
>ensure fencing if the links are all cut?

The sub-cluster that retains quorum order the resource to lock i/o from
the other sub-cluster node(s) without quorum. For e.g. GNBD exporter
fence i/o access for nodes failed/disconnected from the cluster.

>> Heartbeat/Pacemaker does not support resource fencing. To fence
>> resources, the resources have to support locking features by
>> themselves. You cannot lock something that cannot be locked. Software
>> iSCSI targets does not have locking mechanisms whereas GNBD does.
>> However, GNBD locking features work with RedHat ClusterSuite only.
>
>iSCSI targets in theory support SCSI reservations. Yet that only works
>if the target is centralized (and thus a SPoF) - if you replicate it,
>see above.

Which iSCSI target software implementation support reservation? Does the
iSCSI ocf RA support that reservation?


>True, resource level fencing is desirable, and the RAs should do it
>automatically. Possibly, certain extensions to Pacemaker might also be
>useful here, though I think that's not the biggest obstacle.

I'm looking forward to those extensions in Pacemaker.

>> >Have you _benchmarked_ that for your workload?
>> Yes, I have benchmarked the application performance. With GNBD it had
>> the best score, then comes iSCSI and NFS is at the end.
>
>That I find quite surprising. I'd like to duplicate that benchmark
>eventually; do you have the benchmark description and results available
>somewhere?

The application I'm running in the cluster is mail server. I benchmarked
the server performance running on GNBD(ext3), iSCSI(ext3) and NFS.
Performance is measured with Loadsim. http://www.msexchange.org/tutorials/Simulating-Stress-Exchange-2003-LoadSim.html
If Loadsim results will have meaning for I can share them with you.


Regards,
Atanas

--
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde


_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

-----Original Message-----
From: Andrew Beekhof [mailto:beekhof@gmail.com]
Sent: Tuesday, March 18, 2008 11:59 AM
To: The Pacemaker cluster resource manager
Subject: Re: [Pacemaker] Does pingd works on openais?

>> Heartbeat/Pacemaker does not support resource fencing.
>
>Technically it does (or could)... you just need to write an RA that
>implements the locking you want.
>
>a) write the locking RA

Do you plan to implement such locking RA?

>b) run it everywhere as a clone
>c) have the RA "fail" when the node/resource/whatever (a fancy version
>might implement more than one zone) is "fenced"

I don't understand how to configure that. If I use on_fail=fence,
Pacemaker will fence the node? How to tell Pacemaker to fence resource?

>d) have any resource that needs resource fencing depend on the lock
>resource (colocation)
>

Could please explain how to configure such collocation? Or it is not
yet possible?

How to order resource fencing? OCF RA should support resource fencing.
Which resources OCF can do that? (GNBD, iSCSI, ...). Resources which
support locking I guess?


Thanks for the help.

Regards,
Atanas
>
>
> _______________________________________________
> Pacemaker mailing list
> Pacemaker@clusterlabs.org
> http://list.clusterlabs.org/mailman/listinfo/pacemaker
>
> _______________________________________________
> Pacemaker mailing list
> Pacemaker@clusterlabs.org
> http://list.clusterlabs.org/mailman/listinfo/pacemaker


_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

On Wed, Mar 19, 2008 at 3:20 AM, Atanas Dyulgerov
<atanas.dyulgerov@postpath.com> wrote:
>
>
> -----Original Message-----
> From: Lars Marowsky-Bree [mailto:lmb@suse.de]
> Sent: Tuesday, March 18, 2008 4:59 PM
> To: The Pacemaker cluster resource manager
> Subject: Re: [Pacemaker] Does pingd works on openais?
>
> On 2008-03-10T11:13:51, Atanas Dyulgerov <atanas.dyulgerov@postpath.com> wrote:
>
> >> STONITH brutally shutdowns a node. To do that you need redundant
> >> communication lines, smart power devices and definitely a local
> >> cluster. For geographically separated cluster with remote nodes
> >> STONITH is not applicable. The method is called Node Fencing and as I
> >> said it has too many obstacles.
> >
> >All fencing methods require a means of communicating with the device; in
> >case of WAN clusters, the link between the (replicating) storage arrays
> >(for example) will also be cut - resource fencing is unavailable for the
> >same reasons as STONITH is.
>
> The case when node loses connectivity to the cluster but it still remains
> connected to the shared resource. Then the other nodes which retain quorum
> can lock the shared storage resource to stop the errant node from accessing
> it. This fencing method does not require communication with the failed node.
> That's what RHCS do I believe.

I don't know about RHCS 4.0 but 3.0 used to kill a diconnected node.
In fact it was shipped with a STONITH module :-) HP
ServiceGuard,Veritas VCS and Oracle RAC killed a failed node as well.
Though it was done through halt on a disconnected node when it wasn't
able to lock a shared device.

--
Serge Dubrouski.

_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

On 2008-03-19T11:20:46, Atanas Dyulgerov <atanas.dyulgerov@postpath.com> wrote:

> The case when node loses connectivity to the cluster but it still remains
> connected to the shared resource. Then the other nodes which retain quorum
> can lock the shared storage resource to stop the errant node from accessing
> it. This fencing method does not require communication with the failed node.
> That's what RHCS do I believe.

Right, but that does only work if the resource supports that. If you're
looking at DR style setups with replicated storage, that doesn't really
apply - the storage links will likely be severed as well.

Again, I'm not saying this isn't a nice feature, but not as important
technically. We'll implement it in the future, but you can have working
clusters even without.

> >WAN clusters require the concept of self-fencing after loss of site
> >quorum.
>
> Any self-fencing method for production use implemented so far? I would
> like to test...

No. I didn't say we had that implemented. DR - metro- and wide-area
clusters - are not one of heartbeat/Pacemaker's strengths right now. We
plan to address this in the future.

> The application I'm running in the cluster is mail server. I benchmarked
> the server performance running on GNBD(ext3), iSCSI(ext3) and NFS.
> Performance is measured with Loadsim. http://www.msexchange.org/tutorials/Simulating-Stress-Exchange-2003-LoadSim.html
> If Loadsim results will have meaning for I can share them with you.

Thanks, that's interesting. I'll try to find some resources to run
benchmarks on our own, because this _is_ a little bit counterintuitive.


Regards,
Lars

--
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde


_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

On 2008-03-19T11:38:41, Atanas Dyulgerov <atanas.dyulgerov@postpath.com> wrote:

> >a) write the locking RA
> Do you plan to implement such locking RA?

Yes, eventually we will implement locking in some SCSI RA.

> I don't understand how to configure that. If I use on_fail=fence,
> Pacemaker will fence the node? How to tell Pacemaker to fence resource?

The fencing of the resource would be implicit in the "start" - other
nodes are denied access.

> >d) have any resource that needs resource fencing depend on the lock
> >resource (colocation)
> >
> Could please explain how to configure such collocation? Or it is not
> yet possible?

That would actually be possible right now, simply using rsc_order and
rsc_colocation constraints.


Regards,
Lars

--
Teamlead Kernel, SuSE Labs, Research and Development
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde


_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker

On Wed, Mar 19, 2008 at 10:38 AM, Atanas Dyulgerov
<atanas.dyulgerov@postpath.com> wrote:
>
>
> -----Original Message-----
> From: Andrew Beekhof [mailto:beekhof@gmail.com]
>
> Sent: Tuesday, March 18, 2008 11:59 AM
> To: The Pacemaker cluster resource manager
> Subject: Re: [Pacemaker] Does pingd works on openais?
>
>
> >> Heartbeat/Pacemaker does not support resource fencing.
> >
>
> >Technically it does (or could)... you just need to write an RA that
> >implements the locking you want.
> >
> >a) write the locking RA
>
> Do you plan to implement such locking RA?

Maybe one day - I have quite a lot of other things that require my
attention before then

> >b) run it everywhere as a clone
> >c) have the RA "fail" when the node/resource/whatever (a fancy version
> >might implement more than one zone) is "fenced"
>
> I don't understand how to configure that.

You dont configure it, it would be a property of the resource agent
that the monitor action fails in this case.

> If I use on_fail=fence,
> Pacemaker will fence the node? How to tell Pacemaker to fence resource?
>
>
> >d) have any resource that needs resource fencing depend on the lock
> >resource (colocation)
> >
>
> Could please explain how to configure such collocation? Or it is not
> yet possible?

http://clusterlabs.org/mw/Image:Colocation_Explained_-_White.pdf

> How to order resource fencing?

Resource ordering constraints.

> OCF RA should support resource fencing.

No, the idea is that the other resources have no idea that its happening.
All the details are hidden in the locking resource

> Which resources OCF can do that? (GNBD, iSCSI, ...). Resources which
> support locking I guess?
>
>
> Thanks for the help.
>
> Regards,
> Atanas
>
>
> >
> >
> > _______________________________________________
> > Pacemaker mailing list
> > Pacemaker@clusterlabs.org
> > http://list.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > _______________________________________________
> > Pacemaker mailing list
> > Pacemaker@clusterlabs.org
> > http://list.clusterlabs.org/mailman/listinfo/pacemaker
>
>
> _______________________________________________
> Pacemaker mailing list
> Pacemaker@clusterlabs.org
> http://list.clusterlabs.org/mailman/listinfo/pacemaker
>
> _______________________________________________
> Pacemaker mailing list
> Pacemaker@clusterlabs.org
> http://list.clusterlabs.org/mailman/listinfo/pacemaker
>

_______________________________________________
Pacemaker mailing list
Pacemaker@clusterlabs.org
http://list.clusterlabs.org/mailman/listinfo/pacemaker