Mailing List Archive

handler timeout
Hi,

there are a lot of Floating point operations in my modperl handlers.
(i.e, considering something like OpenPose).
the client is easy to get timeout from the server.
how to fix up this?

Thanks.
Re: handler timeout [ In reply to ]
* PANG J. <pangj@uk2.net> wrote:

> Date: Wed, 28 Mar 2018 09:30:17 +0800
> From: "PANG J." <pangj@uk2.net>
> To: modperl@perl.apache.org
> Subject: handler timeout
> User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101
> Thunderbird/52.6.0
>
> Hi,
>
> there are a lot of Floating point operations in my modperl handlers. (i.e,
> considering something like OpenPose).
> the client is easy to get timeout from the server.
> how to fix up this?

There can be many things that can cause this. What have you done so far in trouble-shooting?

BTW, good to know mod_perl is used for this kind of projects. :-)

Regards,


Jie
Re: handler timeout [ In reply to ]
Hi,

what the primary reason is handler computes for long time, so client
gets timeout.


On 2018/3/28 ??? AM 10:08, Jie Gao wrote:
> * PANG J. <pangj@uk2.net> wrote:
>
>> Date: Wed, 28 Mar 2018 09:30:17 +0800
>> From: "PANG J." <pangj@uk2.net>
>> To: modperl@perl.apache.org
>> Subject: handler timeout
>> User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101
>> Thunderbird/52.6.0
>>
>> Hi,
>>
>> there are a lot of Floating point operations in my modperl handlers. (i.e,
>> considering something like OpenPose).
>> the client is easy to get timeout from the server.
>> how to fix up this?
>
> There can be many things that can cause this. What have you done so far in trouble-shooting?
>
> BTW, good to know mod_perl is used for this kind of projects. :-)
>
> Regards,
>
>
> Jie
>
Re: handler timeout [ In reply to ]
* PANG J. <pangj@uk2.net> wrote:

> Date: Wed, 28 Mar 2018 10:17:33 +0800
> From: "PANG J." <pangj@uk2.net>
> To: modperl@perl.apache.org
> Subject: Re: handler timeout
> User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101
> Thunderbird/52.6.0
>
> Hi,
>
> what the primary reason is handler computes for long time, so client gets
> timeout.

To start with, is your hosting machine running out of resources (CPU cycles, memory, etc)?

And what is the timeout value you have configured for Apache? The defaults might be too low.


Regards,


Jie


>
> On 2018/3/28 ??? AM 10:08, Jie Gao wrote:
> >* PANG J. <pangj@uk2.net> wrote:
> >
> >>Date: Wed, 28 Mar 2018 09:30:17 +0800
> >>From: "PANG J." <pangj@uk2.net>
> >>To: modperl@perl.apache.org
> >>Subject: handler timeout
> >>User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101
> >> Thunderbird/52.6.0
> >>
> >>Hi,
> >>
> >>there are a lot of Floating point operations in my modperl handlers. (i.e,
> >>considering something like OpenPose).
> >>the client is easy to get timeout from the server.
> >>how to fix up this?
> >
> >There can be many things that can cause this. What have you done so far in trouble-shooting?
> >
> >BTW, good to know mod_perl is used for this kind of projects. :-)
> >
> >Regards,
> >
> >
> >Jie
> >
>
Re: handler timeout [ In reply to ]
On 2018/3/28 ??? AM 10:41, Jie Gao wrote:

>
> To start with, is your hosting machine running out of resources (CPU cycles, memory, etc)?
>
No. resources are enough. we have about 100 servers for computing, each
with 24 physical cores. But yes, the system load most time is high.

> And what is the timeout value you have configured for Apache? The defaults might be too low.
>
Timeout 300
Re: handler timeout [ In reply to ]
* PANG J. <pangj@uk2.net> wrote:

> Date: Wed, 28 Mar 2018 10:49:46 +0800
> From: "PANG J." <pangj@uk2.net>
> To: modperl@perl.apache.org
> Subject: Re: handler timeout
> User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101
> Thunderbird/52.6.0
>
>
>
> On 2018/3/28 ??? AM 10:41, Jie Gao wrote:
>
> >To start with, is your hosting machine running out of resources (CPU cycles, memory, etc)?
> >
> No. resources are enough. we have about 100 servers for computing, each with
> 24 physical cores. But yes, the system load most time is high.

Your system load probably would explain it. Your tasks are computation intensive, and unless your programmes are multi-threaded, they each will be limited by the speed of the single CPU core it runs on.

> >And what is the timeout value you have configured for Apache? The defaults might be too low.
> >
> Timeout 300

I'd increase this timeout value gradually to suit the need.

Regards,


Jie
Re: handler timeout [ In reply to ]
Hi.
Are we not a bit on the wrong track here ?

I believe that the timeout which Pang J. is mentioning, may be the browser-side timeout,
which is fixed at the browser level at about 5 minutes or so.
When a browser sends a request to a server, and it does receive /some/ response within the
next +-5 minutes, then the browser will drop the connection to the server, and pop up a
message saying "sorry, the server appears not to respond.."
In other words, it is not a server timeout, it is a client timeout.
The only way to avoid this, is to insure that the server sends at least /some/ temporary
response to the client (*), regularly, so that this browser timeout does not occur.
Unfortunately, that is a bit more complicated to set up, than just some parameter somewhere.
But there must be plenty of past discussions of this issue already on the www, and
solution guidelines.

(*) Something like a message "Please wait, we are still calculating.."
As long as the browser receives /some/ response regularly, it will not time out.

On 28.03.2018 05:11, Jie Gao wrote:
> * PANG J. <pangj@uk2.net> wrote:
>
>> Date: Wed, 28 Mar 2018 10:49:46 +0800
>> From: "PANG J." <pangj@uk2.net>
>> To: modperl@perl.apache.org
>> Subject: Re: handler timeout
>> User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101
>> Thunderbird/52.6.0
>>
>>
>>
>> On 2018/3/28 ??? AM 10:41, Jie Gao wrote:
>>
>>> To start with, is your hosting machine running out of resources (CPU cycles, memory, etc)?
>>>
>> No. resources are enough. we have about 100 servers for computing, each with
>> 24 physical cores. But yes, the system load most time is high.
>
> Your system load probably would explain it. Your tasks are computation intensive, and unless your programmes are multi-threaded, they each will be limited by the speed of the single CPU core it runs on.
>
>>> And what is the timeout value you have configured for Apache? The defaults might be too low.
>>>
>> Timeout 300
>
> I'd increase this timeout value gradually to suit the need.
>
> Regards,
>
>
> Jie
>
Re: handler timeout [ In reply to ]
Addendum : searching Google for "avoid browser timeout" gets plenty of answers, among
which this is a perl-oriented one : https://www.perlmonks.org/bare/?node_id=252682
with further links.

On 28.03.2018 12:11, André Warnier (tomcat) wrote:
> Hi.
> Are we not a bit on the wrong track here ?
>
> I believe that the timeout which Pang J. is mentioning, may be the browser-side timeout,
> which is fixed at the browser level at about 5 minutes or so.
> When a browser sends a request to a server, and it does receive /some/ response within the
> next +-5 minutes, then the browser will drop the connection to the server, and pop up a
> message saying "sorry, the server appears not to respond.."
> In other words, it is not a server timeout, it is a client timeout.
> The only way to avoid this, is to insure that the server sends at least /some/ temporary
> response to the client (*), regularly, so that this browser timeout does not occur.
> Unfortunately, that is a bit more complicated to set up, than just some parameter somewhere.
> But there must be plenty of past discussions of this issue already on the www, and
> solution guidelines.
>
> (*) Something like a message "Please wait, we are still calculating.."
> As long as the browser receives /some/ response regularly, it will not time out.
>
> On 28.03.2018 05:11, Jie Gao wrote:
>> * PANG J. <pangj@uk2.net> wrote:
>>
>>> Date: Wed, 28 Mar 2018 10:49:46 +0800
>>> From: "PANG J." <pangj@uk2.net>
>>> To: modperl@perl.apache.org
>>> Subject: Re: handler timeout
>>> User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101
>>> Thunderbird/52.6.0
>>>
>>>
>>>
>>> On 2018/3/28 ??? AM 10:41, Jie Gao wrote:
>>>
>>>> To start with, is your hosting machine running out of resources (CPU cycles, memory,
>>>> etc)?
>>>>
>>> No. resources are enough. we have about 100 servers for computing, each with
>>> 24 physical cores. But yes, the system load most time is high.
>>
>> Your system load probably would explain it. Your tasks are computation intensive, and
>> unless your programmes are multi-threaded, they each will be limited by the speed of the
>> single CPU core it runs on.
>>
>>>> And what is the timeout value you have configured for Apache? The defaults might be
>>>> too low.
>>>>
>>> Timeout 300
>>
>> I'd increase this timeout value gradually to suit the need.
>>
>> Regards,
>>
>>
>> Jie
>>
>
Re: handler timeout [ In reply to ]
what the client I meant is mobile App.
mobile App gets the result from server via SDK.
in future we may move the computing task into App itself.
But currently they are running on server side.

thanks.

On 2018/3/28 ??? PM 6:11, André Warnier (tomcat) wrote:
> I believe that the timeout which Pang J. is mentioning, may be the
> browser-side timeout, which is fixed at the browser level at about 5
> minutes or so.
> When a browser sends a request to a server, and it does receive /some/
> response within the next +-5 minutes, then the browser will drop the
> connection to the server, and pop up a message saying "sorry, the server
> appears not to respond.."
> In other words, it is not a server timeout, it is a client timeout.
> The only way to avoid this, is to insure that the server sends at least
> /some/ temporary response to the client (*), regularly, so that this
> browser timeout does not occur.
> Unfortunately, that is a bit more complicated to set up, than just some
> parameter somewhere.
> But there must be plenty of past discussions of this issue already on
> the www, and solution guidelines.
Re: handler timeout [ In reply to ]
for long run stuff, people are usually using job queue like gearman,
theschwartz or minion etc.

you can send a ID back to app first, then init another request with
that id to check if it's finished.

Thanks

On Wed, Mar 28, 2018 at 6:31 PM, PANG J. <pangj@uk2.net> wrote:
> what the client I meant is mobile App.
> mobile App gets the result from server via SDK.
> in future we may move the computing task into App itself.
> But currently they are running on server side.
>
> thanks.
>
>
> On 2018/3/28 ??? PM 6:11, André Warnier (tomcat) wrote:
>>
>> I believe that the timeout which Pang J. is mentioning, may be the
>> browser-side timeout, which is fixed at the browser level at about 5 minutes
>> or so.
>> When a browser sends a request to a server, and it does receive /some/
>> response within the next +-5 minutes, then the browser will drop the
>> connection to the server, and pop up a message saying "sorry, the server
>> appears not to respond.."
>> In other words, it is not a server timeout, it is a client timeout.
>> The only way to avoid this, is to insure that the server sends at least
>> /some/ temporary response to the client (*), regularly, so that this browser
>> timeout does not occur.
>> Unfortunately, that is a bit more complicated to set up, than just some
>> parameter somewhere.
>> But there must be plenty of past discussions of this issue already on the
>> www, and solution guidelines.



--
Fayland Lam // http://www.fayland.me/
Re: handler timeout [ In reply to ]
On 28.03.2018 12:31, PANG J. wrote:
> what the client I meant is mobile App.
> mobile App gets the result from server via SDK.

Ok. But it is very likely that your "mobile app SDK", also has a timeout after it sends a
request to a server. Or are you /sure/ that it waits forever ?
/Precisely what/ makes you think that it is a server-side timeout ?

> in future we may move the computing task into App itself.
> But currently they are running on server side.
>
> thanks.
>
> On 2018/3/28 ??? PM 6:11, André Warnier (tomcat) wrote:
>> I believe that the timeout which Pang J. is mentioning, may be the browser-side timeout,
>> which is fixed at the browser level at about 5 minutes or so.
>> When a browser sends a request to a server, and it does receive /some/ response within
>> the next +-5 minutes, then the browser will drop the connection to the server, and pop
>> up a message saying "sorry, the server appears not to respond.."
>> In other words, it is not a server timeout, it is a client timeout.
>> The only way to avoid this, is to insure that the server sends at least /some/ temporary
>> response to the client (*), regularly, so that this browser timeout does not occur.
>> Unfortunately, that is a bit more complicated to set up, than just some parameter
>> somewhere.
>> But there must be plenty of past discussions of this issue already on the www, and
>> solution guidelines.
Re: handler timeout [ In reply to ]
I do think it's a client timeout.

regards.

On 2018/3/28 ??? PM 6:37, André Warnier (tomcat) wrote:
>
> Ok. But it is very likely that your "mobile app SDK", also has a timeout
> after it sends a request to a server. Or are you /sure/ that it waits
> forever ?
> /Precisely what/ makes you think that it is a server-side timeout ?
Re: handler timeout [ In reply to ]
As shown below,


Last day total requests are 42,368,982,  not all are successful, but
42,362,363 are right.

The failed requests are timeout.

Thanks.



On 2018/3/28 ??? PM 6:37, André Warnier (tomcat) wrote:
> On 28.03.2018 12:31, PANG J. wrote:
>> what the client I meant is mobile App.
>> mobile App gets the result from server via SDK.
>
> Ok. But it is very likely that your "mobile app SDK", also has a
> timeout after it sends a request to a server. Or are you /sure/ that
> it waits forever ?
> /Precisely what/ makes you think that it is a server-side timeout ?
>
>> in future we may move the computing task into App itself.
>> But currently they are running on server side.
>>
>> thanks.
>>
>> On 2018/3/28 ??? PM 6:11, André Warnier (tomcat) wrote:
>>> I believe that the timeout which Pang J. is mentioning, may be the
>>> browser-side timeout,
>>> which is fixed at the browser level at about 5 minutes or so.
>>> When a browser sends a request to a server, and it does receive
>>> /some/ response within
>>> the next +-5 minutes, then the browser will drop the connection to
>>> the server, and pop
>>> up a message saying "sorry, the server appears not to respond.."
>>> In other words, it is not a server timeout, it is a client timeout.
>>> The only way to avoid this, is to insure that the server sends at
>>> least /some/ temporary
>>> response to the client (*), regularly, so that this browser timeout
>>> does not occur.
>>> Unfortunately, that is a bit more complicated to set up, than just
>>> some parameter
>>> somewhere.
>>> But there must be plenty of past discussions of this issue already
>>> on the www, and
>>> solution guidelines.
>
Re: handler timeout [ In reply to ]
On 28.03.2018 12:54, PANG J. wrote:
> I do think it's a client timeout.
>
Ok, then there are many kinds of answers to your question..

1) the quick and dirty solution, is to find out how to set the timeout higher in your app
SDK, and set it high enough so that the timeout never happens.
As far as I know, there is no timeout at the server side. The server-side app will just
take whatever time it takes to provide the response, and then the server will send it.
If the connection to the client is still open when that happens, then everything will be
fine. If by that time the client has closed the connection (because it was tired of
waiting), then you will see an error both on the client side (the timeout), and on the
server side : something in the error log like "cannot send response : connection closed by
client".

But that is not a "good" solution, for a number of reasons. One reason is that then, if
there really is a problem on the server, the client will have to wait for a very long
time, before it sees the error.
Another reason is that you will "block" one server-side webserver process or thread, for
as long as it takes to generate one response. That will use up a lot of resources on your
server (which maybe is not very important to you, but it is still quite inefficient).

2) a slightly better solution, would be looking at some of the links found in Google, for
"tricks" to get around the browser timeout (your app is not a browser, but it is a HTTP
client, so maybe some of these tricks apply to you also).

3) a better solution would be to change the logic of your applications, and use one of the
"job queue frameworks" which a previous person mentioned.
That would avoid these timeout issues, and is more efficient both for the client and the
server. But it does require changes in the design of your application, and the user
interface.

4) an even higher level question, is whether HTTP 1.x is really the appropriate protocol
to use, for such a type of application. HTTP was fundamentally designed (some 30 years
ago) to handle a case of "one request, one response", and the requests were supposed to be
for something like a single HTML document. So the request-response cycle was not supposed
to take more than a few seconds.
And that is why HTTP clients time out after a couple of minutes waiting, because in HTTP,
this is fundamentally considered as possible only if there is a network problem or a
server problem, and it is thus considered /as an error/.

HTTP was not really designed for handling long-processing-time requests, and that
fundamental architecture is still the base of HTTP today. There have been lots of clever
developments since then to "get around" that kind of limitation (such as the job queues
mentioned above), and javascript applets and whatever, but the fundamental truth is that
all of those are like ways to just get around the basic HTTP (1.x) protocol design, and
they are always somewhat clunky or delicate, just because they have to "fight" against
this basic HTTP design.

Maybe you should have a look at HTTP 2.0, if your client-side applet SDK supports it.

And if it does not support it, maybe you should look at alternatives to standard HTTP
servers, on the server side.
If you want to stay in the perl world for that, you may want to have a look at
http://search.cpan.org/~miyagawa/Starman-0.1000/lib/Starman.pm



> regards.
>
> On 2018/3/28 ??? PM 6:37, André Warnier (tomcat) wrote:
>>
>> Ok. But it is very likely that your "mobile app SDK", also has a timeout after it sends
>> a request to a server. Or are you /sure/ that it waits forever ?
>> /Precisely what/ makes you think that it is a server-side timeout ?
Re: handler timeout [ In reply to ]
On 28.03.2018 13:13, PANG J. wrote:
> As shown below,
>
>
> Last day total requests are 42,368,982, not all are successful, but 42,362,363 are right.
>
> The failed requests are timeout.
>

OK, that is quite an impressive number of requests, and a good answer to people who always
wonder if an Apache/mod_perl/perl solution is really scaleable..
(Perl/mod_perl advocates, take note)

5619 failed requests, on a total of 42,368,982 is approximately a 0,013% failure rate.

That does no really look like justifying a complete architecture change.

Can you maybe reprogram the client-side app, to wait for a "normal" time, and if there is
a timeout, ask the user if they want to continue waiting, or abort the call ?

(According to the numbers above, such a message should only appear in 0,013% of the cases,
and the other 42,362,363 (99.987%) clients will never see it.)


> Thanks.
>
>
>
> On 2018/3/28 ??? PM 6:37, André Warnier (tomcat) wrote:
>> On 28.03.2018 12:31, PANG J. wrote:
>>> what the client I meant is mobile App.
>>> mobile App gets the result from server via SDK.
>>
>> Ok. But it is very likely that your "mobile app SDK", also has a timeout after it sends
>> a request to a server. Or are you /sure/ that it waits forever ?
>> /Precisely what/ makes you think that it is a server-side timeout ?
>>
>>> in future we may move the computing task into App itself.
>>> But currently they are running on server side.
>>>
>>> thanks.
>>>
>>> On 2018/3/28 ??? PM 6:11, André Warnier (tomcat) wrote:
>>>> I believe that the timeout which Pang J. is mentioning, may be the browser-side timeout,
>>>> which is fixed at the browser level at about 5 minutes or so.
>>>> When a browser sends a request to a server, and it does receive /some/ response within
>>>> the next +-5 minutes, then the browser will drop the connection to the server, and pop
>>>> up a message saying "sorry, the server appears not to respond.."
>>>> In other words, it is not a server timeout, it is a client timeout.
>>>> The only way to avoid this, is to insure that the server sends at least /some/ temporary
>>>> response to the client (*), regularly, so that this browser timeout does not occur.
>>>> Unfortunately, that is a bit more complicated to set up, than just some parameter
>>>> somewhere.
>>>> But there must be plenty of past discussions of this issue already on the www, and
>>>> solution guidelines.
>>
>
Re: handler timeout [ In reply to ]
You can also turn on "keep-alive" and tune it to suit your current, immediate need.

-Jie