Mailing List Archive: resource monitor priority

resource monitor priority - timeout on high load

Oct 5, 2015, 10:53 AM

Post #1 of 5 (2086 views)

Hi,

I have a situation, when resource monitor operations timeout on system high
load.

One thing to do is to increase timeouts, but there can always be load, at
which the timeout event would happen, and on the other hand too long
timeouts would not be very useful, so I would like to increase the priority
of monitoring processes (GNU Linux), but I do not want to be too broad.

So my question is, which pacemaker process should I target to ensure timely
monitoring operations, or maybe even there is a pacemaker configuration
parameter, which would help in such a case.

I know for sure, that agents finish they monitoring requests on time, and
the timeout is due to high load. I have several agents, so I will provide
just some details to illustrate the case:
- timeout: ~60s
- interval: ~60s
- normal (low load) monitor request time: 0.2s
- high load monitor request time: 8s - when pacemaker reports resource
operation timeout

Thanks,

--
Best Regards,

Radoslaw Garbacz
XtremeData Incorporation

Re: resource monitor priority - timeout on high load [ In reply to ]

emi2fast at gmail

Oct 5, 2015, 11:08 AM

Post #2 of 5 (2043 views)

Permalink

If I remeber, there is a way to make real time pacemaker process, but
I don't know if this can help you, you need to tell us if your server
high load is system space or userspace.

2015-10-05 19:53 GMT+02:00 Radoslaw Garbacz
<radoslaw.garbacz@xtremedatainc.com>:
> Hi,
>
> I have a situation, when resource monitor operations timeout on system high
> load.
>
> One thing to do is to increase timeouts, but there can always be load, at
> which the timeout event would happen, and on the other hand too long
> timeouts would not be very useful, so I would like to increase the priority
> of monitoring processes (GNU Linux), but I do not want to be too broad.
>
> So my question is, which pacemaker process should I target to ensure timely
> monitoring operations, or maybe even there is a pacemaker configuration
> parameter, which would help in such a case.
>
> I know for sure, that agents finish they monitoring requests on time, and
> the timeout is due to high load. I have several agents, so I will provide
> just some details to illustrate the case:
> - timeout: ~60s
> - interval: ~60s
> - normal (low load) monitor request time: 0.2s
> - high load monitor request time: 8s - when pacemaker reports resource
> operation timeout
>
>
> Thanks,
>
> --
> Best Regards,
>
> Radoslaw Garbacz
> XtremeData Incorporation
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>

--
.~.
/V\
// \\
/( )\
^`~'^

_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: resource monitor priority - timeout on high load [ In reply to ]

radoslaw.garbacz at xtremedatainc

Oct 5, 2015, 12:09 PM

Post #3 of 5 (2043 views)

Permalink

I think it would be helpful, my load is definitely user-space.
My goal is for pacemaker to survive high user load, and not shut down
resources on monitoring timeouts, additionally resources should not be
restarted either, i.e. they have to be left alone, unless - of course -
there is a real resource failure.

Thanks,

On Mon, Oct 5, 2015 at 1:08 PM, emmanuel segura <emi2fast@gmail.com> wrote:

> If I remeber, there is a way to make real time pacemaker process, but
> I don't know if this can help you, you need to tell us if your server
> high load is system space or userspace.
>
> 2015-10-05 19:53 GMT+02:00 Radoslaw Garbacz
> <radoslaw.garbacz@xtremedatainc.com>:
> > Hi,
> >
> > I have a situation, when resource monitor operations timeout on system
> high
> > load.
> >
> > One thing to do is to increase timeouts, but there can always be load, at
> > which the timeout event would happen, and on the other hand too long
> > timeouts would not be very useful, so I would like to increase the
> priority
> > of monitoring processes (GNU Linux), but I do not want to be too broad.
> >
> > So my question is, which pacemaker process should I target to ensure
> timely
> > monitoring operations, or maybe even there is a pacemaker configuration
> > parameter, which would help in such a case.
> >
> > I know for sure, that agents finish they monitoring requests on time, and
> > the timeout is due to high load. I have several agents, so I will provide
> > just some details to illustrate the case:
> > - timeout: ~60s
> > - interval: ~60s
> > - normal (low load) monitor request time: 0.2s
> > - high load monitor request time: 8s - when pacemaker reports resource
> > operation timeout
> >
> >
> > Thanks,
> >
> > --
> > Best Regards,
> >
> > Radoslaw Garbacz
> > XtremeData Incorporation
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> >
>
>
>
> --
> .~.
> /V\
> // \\
> /( )\
> ^`~'^
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>

--
Best Regards,

Radoslaw Garbacz
XtremeData Incorporation

Re: resource monitor priority - timeout on high load [ In reply to ]

dejanmm at fastmail

Oct 6, 2015, 1:37 AM

Post #4 of 5 (2033 views)

Permalink

Hi,

On Mon, Oct 05, 2015 at 12:53:25PM -0500, Radoslaw Garbacz wrote:
> Hi,
>
> I have a situation, when resource monitor operations timeout on system high
> load.
>
> One thing to do is to increase timeouts, but there can always be load, at
> which the timeout event would happen, and on the other hand too long
> timeouts would not be very useful,

Why do you think so? Did you see actual failures due to a
timeout, i.e. a process just hanging in limbo?

> so I would like to increase the priority
> of monitoring processes (GNU Linux), but I do not want to be too broad.

If that could make the server a bit bigger too ;-)

Seriously, IMO you have two options: either add more resources
(as in hardware) or increase the monitor timeouts.

Thanks,

Dejan

> So my question is, which pacemaker process should I target to ensure timely
> monitoring operations, or maybe even there is a pacemaker configuration
> parameter, which would help in such a case.
>
> I know for sure, that agents finish they monitoring requests on time, and
> the timeout is due to high load. I have several agents, so I will provide
> just some details to illustrate the case:
> - timeout: ~60s
> - interval: ~60s
> - normal (low load) monitor request time: 0.2s
> - high load monitor request time: 8s - when pacemaker reports resource
> operation timeout
>
>
> Thanks,
>
> --
> Best Regards,
>
> Radoslaw Garbacz
> XtremeData Incorporation

> _______________________________________________
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: resource monitor priority - timeout on high load [ In reply to ]

radoslaw.garbacz at xtremedatainc

Oct 13, 2015, 9:12 AM

Post #5 of 5 (2001 views)

Permalink

Indeed, I do not see actual failures due to a timeout.

Thanks Dejan,

On Tue, Oct 6, 2015 at 3:37 AM, Dejan Muhamedagic <dejanmm@fastmail.fm>
wrote:

> Hi,
>
> On Mon, Oct 05, 2015 at 12:53:25PM -0500, Radoslaw Garbacz wrote:
> > Hi,
> >
> > I have a situation, when resource monitor operations timeout on system
> high
> > load.
> >
> > One thing to do is to increase timeouts, but there can always be load, at
> > which the timeout event would happen, and on the other hand too long
> > timeouts would not be very useful,
>
> Why do you think so? Did you see actual failures due to a
> timeout, i.e. a process just hanging in limbo?
>
> > so I would like to increase the priority
> > of monitoring processes (GNU Linux), but I do not want to be too broad.
>
> If that could make the server a bit bigger too ;-)
>
> Seriously, IMO you have two options: either add more resources
> (as in hardware) or increase the monitor timeouts.
>
> Thanks,
>
> Dejan
>
> > So my question is, which pacemaker process should I target to ensure
> timely
> > monitoring operations, or maybe even there is a pacemaker configuration
> > parameter, which would help in such a case.
> >
> > I know for sure, that agents finish they monitoring requests on time, and
> > the timeout is due to high load. I have several agents, so I will provide
> > just some details to illustrate the case:
> > - timeout: ~60s
> > - interval: ~60s
> > - normal (low load) monitor request time: 0.2s
> > - high load monitor request time: 8s - when pacemaker reports resource
> > operation timeout
> >
> >
> > Thanks,
> >
> > --
> > Best Regards,
> >
> > Radoslaw Garbacz
> > XtremeData Incorporation
>
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>

--
Best Regards,

Radoslaw Garbacz
XtremeData Incorporation