Mailing List Archive

On 15/11/14 05:06 AM, Andras POTOCZKY wrote:
> Hi List,
>
> I would like to configure an application based HA solution. I mean if my
> important application stops working (but the active server is still
> alive), the HA configuration is passing the shared IP to the standby node.
>
> I've found pacemaker is the tool for it but I couldnt find any document
> or example related to the application level HA.
>
> Can somebody point me how can I figure this out?
>
> thanks,
> Andras

There are many ways to do this, depending on your application.

If your application supports sysvinit style start/stop/status, then it
will "just work". When the app fails, the next '/etc/init.d/foo status'
call (or it's systemd equiv.) returns a bad return code, pacemaker will
treat it as failed. colocate your virtual IP with your app and they
should move together.

If your app doesn't work this way, then you'll need to write a "resource
agent", which acts as a "translator" layer between pacemaker and your
application. You can look at the many other RAs and you will see most
(all?) are simple scripts in various languages.

hth

--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Andras POTOCZKY wrote:
> Hi List,
>
> I would like to configure an application based HA solution. I mean if
> my important application stops working (but the active server is still
> alive), the HA configuration is passing the shared IP to the standby
> node.
>
> I've found pacemaker is the tool for it but I couldnt find any
> document or example related to the application level HA.
>
> Can somebody point me how can I figure this out?
>
>

The short answer: Look for documentation on resource agents. If your
application has a pre-existing resource agent defined, then you set that
up and it handles failover.

The longer answer is that application-level failover is difficult,
because it involves things like:
- IP addresses
- process state
- work queues
- data files
- etc.
All of which are unique to a specific application.

Hence, it becomes a lot easier to:
- address failover at the virtual machine level (IP addresses,
applications, processes)
- use either a storage area network, or DRBD to make sure that your data
remains available and consistent

Some applications provide their own application-layer failover - such as
databases with replication logic built in:
- put them behind a load leveler, and you're done (assuming your
load-leveler isn't a single point of failure), or,
- if your public-facing DNS name points to multiple IP addresses, then
you have a level of failover there, or,
- use CARP to support IP address failover

For other applications, it's quite a bit harder: I've been trying to
come up with a way to make our email and mailing-list infrastructure HA,
at the application layer, but there are so many distinct processes
running, and ques, and such, that just VM-level failover, with DRBD disk
replication has always been far simpler. (The reason I keep looking is
that I'd like to migrate to either a BSD or illumos based platform, but
there's no Xen and no DRBD.)

Miles Fidelman



HAST
I've been wrestling with this one for a while - more in terms of setting
up HA applications


--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra

_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

On 11/15/2014 10:21 AM, Miles Fidelman wrote:
> Andras POTOCZKY wrote:
...
>> I've found pacemaker is the tool for it but I couldnt find any
>> document or example related to the application level HA.
>>
>> Can somebody point me how can I figure this out?
...
> The short answer: Look for documentation on resource agents. If your
> application has a pre-existing resource agent defined, then you set that
> up and it handles failover.

The shorter answer: Clusters from Scratch (google it). If your
application is "stateless", like httpd, there is a chapter on apache. If
it keeps state, that's typically stored on a drbd filesystem and there's
a chapter on drbd. That should give you a reasonable starting point.

Dima

_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Hi

Thanks to all for the answers.
The application supports sysvinit style start/stop/status and the
application has failover mechanism. It is working well by active/standby
state but at this time the shared IP is picked up by the standby server
if the active server is unavailable.

Andras


On 2014.11.15. 17:14, Digimer wrote:
> If your application supports sysvinit style start/stop/status, then it
> will "just work". When the app fails, the next '/etc/init.d/foo
> status' call (or it's systemd equiv.) returns a bad return code,
> pacemaker will treat it as failed. colocate your virtual IP with your
> app and they should move together.

_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

On 15/11/14 12:43 PM, Andras POTOCZKY wrote:
> Hi
>
> Thanks to all for the answers.
> The application supports sysvinit style start/stop/status and the
> application has failover mechanism. It is working well by active/standby
> state but at this time the shared IP is picked up by the standby server
> if the active server is unavailable.
>
> Andras

Glad it works!

Now, you have stonith configured and tested, right? :)

--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

My problem is a bit easier then you mentioned with the mail server
example because I dont have many processes to check. The application is
handling well the active/passive state and I dont have to care about the
data incosistency.
The problem comes from this situation. Both of the servers are alive and
heartbeat (that doesnt support app level HA) is changing the IP address
when the active server failed.

Andras


On 2014.11.15. 17:21, Miles Fidelman wrote:
> Andras POTOCZKY wrote:
>> Hi List,
>>
>> I would like to configure an application based HA solution. I mean if
>> my important application stops working (but the active server is
>> still alive), the HA configuration is passing the shared IP to the
>> standby node.
>>
>> I've found pacemaker is the tool for it but I couldnt find any
>> document or example related to the application level HA.
>>
>> Can somebody point me how can I figure this out?
>>
>>
>
> The short answer: Look for documentation on resource agents. If your
> application has a pre-existing resource agent defined, then you set
> that up and it handles failover.
>
> The longer answer is that application-level failover is difficult,
> because it involves things like:
> - IP addresses
> - process state
> - work queues
> - data files
> - etc.
> All of which are unique to a specific application.
>
> Hence, it becomes a lot easier to:
> - address failover at the virtual machine level (IP addresses,
> applications, processes)
> - use either a storage area network, or DRBD to make sure that your
> data remains available and consistent
>
> Some applications provide their own application-layer failover - such
> as databases with replication logic built in:
> - put them behind a load leveler, and you're done (assuming your
> load-leveler isn't a single point of failure), or,
> - if your public-facing DNS name points to multiple IP addresses, then
> you have a level of failover there, or,
> - use CARP to support IP address failover
>
> For other applications, it's quite a bit harder: I've been trying to
> come up with a way to make our email and mailing-list infrastructure
> HA, at the application layer, but there are so many distinct processes
> running, and ques, and such, that just VM-level failover, with DRBD
> disk replication has always been far simpler. (The reason I keep
> looking is that I'd like to migrate to either a BSD or illumos based
> platform, but there's no Xen and no DRBD.)
>
> Miles Fidelman
>
>
>
> HAST
> I've been wrestling with this one for a while - more in terms of
> setting up HA applications
>
>

_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Yes thanks, I'l try it soon.

Andras


On 2014.11.15. 18:48, Digimer wrote:
> On 15/11/14 12:43 PM, Andras POTOCZKY wrote:
>> Hi
>>
>> Thanks to all for the answers.
>> The application supports sysvinit style start/stop/status and the
>> application has failover mechanism. It is working well by active/standby
>> state but at this time the shared IP is picked up by the standby server
>> if the active server is unavailable.
>>
>> Andras
>
> Glad it works!
>
> Now, you have stonith configured and tested, right? :)
>

_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Without it, you will get split-brains. That is, both nodes running
thinking the other is dead. If it was OK to run both nodes, I doubt
you'd need failover. ;)

So I'd make it a priority.

cheers

On 15/11/14 12:54 PM, Andras POTOCZKY wrote:
> Yes thanks, I'l try it soon.
>
> Andras
>
>
> On 2014.11.15. 18:48, Digimer wrote:
>> On 15/11/14 12:43 PM, Andras POTOCZKY wrote:
>>> Hi
>>>
>>> Thanks to all for the answers.
>>> The application supports sysvinit style start/stop/status and the
>>> application has failover mechanism. It is working well by active/standby
>>> state but at this time the shared IP is picked up by the standby server
>>> if the active server is unavailable.
>>>
>>> Andras
>>
>> Glad it works!
>>
>> Now, you have stonith configured and tested, right? :)
>>
>
> _______________________________________________
> Linux-HA mailing list
> Linux-HA@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems


--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

At this time it is a testing environment so I can sleep well tonight
without it :) but i'll try it soon.

Andras

On 2014.11.15. 19:08, Digimer wrote:
> Without it, you will get split-brains. That is, both nodes running
> thinking the other is dead. If it was OK to run both nodes, I doubt
> you'd need failover. ;)
>
> So I'd make it a priority.
>
> cheers
>
> On 15/11/14 12:54 PM, Andras POTOCZKY wrote:
>> Yes thanks, I'l try it soon.
>>
>> Andras
>>
>>
>> On 2014.11.15. 18:48, Digimer wrote:
>>> On 15/11/14 12:43 PM, Andras POTOCZKY wrote:
>>>> Hi
>>>>
>>>> Thanks to all for the answers.
>>>> The application supports sysvinit style start/stop/status and the
>>>> application has failover mechanism. It is working well by
>>>> active/standby
>>>> state but at this time the shared IP is picked up by the standby
>>>> server
>>>> if the active server is unavailable.
>>>>
>>>> Andras
>>>
>>> Glad it works!
>>>
>>> Now, you have stonith configured and tested, right? :)
>>>
>>
>> _______________________________________________
>> Linux-HA mailing list
>> Linux-HA@lists.linux-ha.org
>> http://lists.linux-ha.org/mailman/listinfo/linux-ha
>> See also: http://linux-ha.org/ReportingProblems
>
>

_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

On 11/15/2014 09:14 AM, Digimer wrote:
> On 15/11/14 05:06 AM, Andras POTOCZKY wrote:
>> Hi List,
>>
>> I would like to configure an application based HA solution. I mean if my
>> important application stops working (but the active server is still
>> alive), the HA configuration is passing the shared IP to the standby
>> node.
>>
>> I've found pacemaker is the tool for it but I couldnt find any document
>> or example related to the application level HA.
>>
>> Can somebody point me how can I figure this out?
>>
>> thanks,
>> Andras
>
> There are many ways to do this, depending on your application.
>
> If your application supports sysvinit style start/stop/status, then it
> will "just work". When the app fails, the next '/etc/init.d/foo
> status' call (or it's systemd equiv.) returns a bad return code,
> pacemaker will treat it as failed. colocate your virtual IP with your
> app and they should move together.
>
> If your app doesn't work this way, then you'll need to write a
> "resource agent", which acts as a "translator" layer between pacemaker
> and your application. You can look at the many other RAs and you will
> see most (all?) are simple scripts in various languages.

I strongly recommend writing a resource agent which can tell if your
application is working correctly. Applications crash, but far more
often they hang or stop giving correct results. An init script for your
resource agent won't be able to tell you that.

_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Hi

I tried pacemaker and I could use it with "lsb syntax" but I realized it
is not the best solution. The application doesnt give correct result all
the time (as you said too) so I decided to make an agent.

Few days ago I made the agent and it is more beautiful and works well.

thanks,
Andras


On 2014.11.20. 14:22, Alan Robertson wrote:
> I strongly recommend writing a resource agent which can tell if your
> application is working correctly. Applications crash, but far more
> often they hang or stop giving correct results. An init script for your
> resource agent won't be able to tell you that.
>
> _______________________________________________
> Linux-HA mailing list
> Linux-HA@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems

_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

When everything works right, it truly is beautiful!



On 11/20/2014 07:11 AM, Andras POTOCZKY wrote:
> Hi
>
> I tried pacemaker and I could use it with "lsb syntax" but I realized
> it is not the best solution. The application doesnt give correct
> result all the time (as you said too) so I decided to make an agent.
>
> Few days ago I made the agent and it is more beautiful and works well.
>
> thanks,
> Andras
>
>
> On 2014.11.20. 14:22, Alan Robertson wrote:
>> I strongly recommend writing a resource agent which can tell if your
>> application is working correctly. Applications crash, but far more
>> often they hang or stop giving correct results. An init script for your
>> resource agent won't be able to tell you that.
>>
>> _______________________________________________
>> Linux-HA mailing list
>> Linux-HA@lists.linux-ha.org
>> http://lists.linux-ha.org/mailman/listinfo/linux-ha
>> See also: http://linux-ha.org/ReportingProblems
>
> _______________________________________________
> Linux-HA mailing list
> Linux-HA@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems

_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

I have used heartbeat, keepalived, CARP and few other softwares but I
have never tried pacemaker before. They are covered every needs for me
till last week.
Maybe that was the reason why I didnt have solution for app level HA and
I asked the list for solution.

I tried pacemaker this week and now I can use one more HA weapon :)

Andras


On 2014.11.20. 21:29, Alan Robertson wrote:
> When everything works right, it truly is beautiful!
>
>
>
> On 11/20/2014 07:11 AM, Andras POTOCZKY wrote:
>> Hi
>>
>> I tried pacemaker and I could use it with "lsb syntax" but I realized
>> it is not the best solution. The application doesnt give correct
>> result all the time (as you said too) so I decided to make an agent.
>>
>> Few days ago I made the agent and it is more beautiful and works well.
>>
>> thanks,
>> Andras
>>
>>
>> On 2014.11.20. 14:22, Alan Robertson wrote:
>>> I strongly recommend writing a resource agent which can tell if your
>>> application is working correctly. Applications crash, but far more
>>> often they hang or stop giving correct results. An init script for your
>>> resource agent won't be able to tell you that.
>>>
>>>

_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems