Mailing List Archive

Hi.

On 04.10.2016 10:17, SUZUKI Arthur wrote:
[...]

>
> On 03.10.2016 16:57, SUZUKI Arthur wrote:
>>> Hello List,
>>> I'm sending this message to know if there are some hints/tips to help with the problem
>>> we're facing.
>>> The problem is that for a same query repeated over time,
>>> reply time can be as short as 5ms and as long as 1s.
>>>
>>> Since there is neither correlation with CPU load or RAM usage, nor any networking
>>> constraints,
>>> we think it is due to the cache refresh mechanism.
>>>
>>> The problem occurs at random times, with a probability of around 1/50 queries (empirical
>>> data).
>>>
>>> Is there any configuration option which could help?
>>>
>>> Our current httpd.conf contains the following:
>>>
>>> PerlModule ModPerl::Registry
>>> PerlOptions +Parent
>>> PerlSwitches -I/home/koha/src
>>>
>>> <Files "ilsdi.pl">
>>> SetHandler perl-script
>>> # more faster, link with worker
>>> PerlResponseHandler ModPerl::Registry
>>> # less faster, link with prefork
>>> #PerlResponseHandler ModPerl::PerlRun
>>> Options +ExecCGI
>>> PerlOptions +ParseHeaders
>>> </Files>
>>>
>>> Thanks a lot in advance for your replies.
>>> Best regards,
>

You still do not provide enough information about your server configuration (the part not
mod_perl specific) to really make a narrow guess, but let me try anyway :

One thing that might explain a seemingly-long processing time for a request approximately
once in 50 calls, is if there was something special and resource-intensive which happens
once per approximately 50 calls, right ?

One thing which /might/ explain this, /perhaps/ and depending on your configuration, is if
approximately once in 50 calls, Apache had to fork a new child process, to handle the
call. This new child process would then start with a brand-new, fresh Perl interpreter,
which might need to recompile some modules before it even starts handling the request.

Look for example here : http://perl.apache.org/docs/2.0/user/config/config.html#C_Parent_
and your configuration line above :
PerlOptions +Parent

In that configuration, the cgi-bin script which you are running (and all its
dependencies), is not compiled by the embedded perl until it has been run at least once.
So the first execution will take more time than the following ones.
Now if the rest of your configuration is so that every 50 requests or so, Apache starts a
new child with a new perl interpreter, that may explain the symptoms that you are observing.

When Apache handles a request (if it is a "prefork" MPM), it will look for a child Apache
which is free, to pass the request for execution. If the server is lightly loaded, it may
be that the child to which it passes the next request is always the same, because it is
always done with the previous request, and free, when the next request comes in.
But if this child process somehow has a limit to how many requests it may process before
it dies, and this limit was around 50, then somewhere around the 50th request, another
child would get the next request, and for this child it would be the first one (or the
first one with /this/ cgi-bin script).

There are roughly similar phenomenons that might happen in other kinds of Apache MPM's,
which is why it is important to know which one you are using, and with which setup parameters.

And yes, there are ways to improve this. But again, we would need to know more about your
configuration in order to make suggestions.

Sounds like André is on the right track. I've certainly run into a similar
issue (with a non-Koha app).

However, because I was using the Catalyst framework, I was able to just
preload the entire app, so that the Perl modules were loaded into the Apache
master process before forking, and that did the trick. That app is a lot
smaller than Koha though too. This case is a bit more complicated since Koha
isn't really a MVC app, but you could look at the Koha Plack examples and
see which modules they pre-load.

You might also want to run ilsdi.pl with Devel::NYTProf to identify what
exactly is slowing down that 1s long query.

I've only played with PerlOptions +Parent a bit, but I'm guessing that you
have multiple mod_perl apps, and that we're not actually seeing your entire
httpd.conf configuration relating to Koha, right?

David Cook
Systems Librarian
Prosentient Systems
72/330 Wattle St
Ultimo, NSW 2007
Australia

Office: 02 9212 0899
Direct: 02 8005 0595


> -----Original Message-----
> From: André Warnier [mailto:aw@ice-sa.com]
> Sent: Tuesday, 4 October 2016 10:02 PM
> To: modperl@perl.apache.org
> Subject: Re: Cache refresh each 50 queries?
>
> Hi.
>
> On 04.10.2016 10:17, SUZUKI Arthur wrote:
> [...]
>
> >
> > On 03.10.2016 16:57, SUZUKI Arthur wrote:
> >>> Hello List,
> >>> I'm sending this message to know if there are some hints/tips to
> >>> help with the problem we're facing.
> >>> The problem is that for a same query repeated over time, reply time
> >>> can be as short as 5ms and as long as 1s.
> >>>
> >>> Since there is neither correlation with CPU load or RAM usage, nor
> >>> any networking constraints, we think it is due to the cache refresh
> >>> mechanism.
> >>>
> >>> The problem occurs at random times, with a probability of around
> >>> 1/50 queries (empirical data).
> >>>
> >>> Is there any configuration option which could help?
> >>>
> >>> Our current httpd.conf contains the following:
> >>>
> >>> PerlModule ModPerl::Registry
> >>> PerlOptions +Parent
> >>> PerlSwitches -I/home/koha/src
> >>>
> >>> <Files "ilsdi.pl">
> >>> SetHandler perl-script
> >>> # more faster, link with worker
> >>> PerlResponseHandler ModPerl::Registry
> >>> # less faster, link with prefork
> >>> #PerlResponseHandler ModPerl::PerlRun
> >>> Options +ExecCGI
> >>> PerlOptions +ParseHeaders
> >>> </Files>
> >>>
> >>> Thanks a lot in advance for your replies.
> >>> Best regards,
> >
>
> You still do not provide enough information about your server
configuration
> (the part not mod_perl specific) to really make a narrow guess, but let me
try
> anyway :
>
> One thing that might explain a seemingly-long processing time for a
request
> approximately once in 50 calls, is if there was something special and
> resource-intensive which happens once per approximately 50 calls, right ?
>
> One thing which /might/ explain this, /perhaps/ and depending on your
> configuration, is if approximately once in 50 calls, Apache had to fork a
new
> child process, to handle the call. This new child process would then start
with
> a brand-new, fresh Perl interpreter, which might need to recompile some
> modules before it even starts handling the request.
>
> Look for example here :
> http://perl.apache.org/docs/2.0/user/config/config.html#C_Parent_
> and your configuration line above :
> PerlOptions +Parent
>
> In that configuration, the cgi-bin script which you are running (and all
its
> dependencies), is not compiled by the embedded perl until it has been run
at
> least once.
> So the first execution will take more time than the following ones.
> Now if the rest of your configuration is so that every 50 requests or so,
> Apache starts a new child with a new perl interpreter, that may explain
the
> symptoms that you are observing.
>
> When Apache handles a request (if it is a "prefork" MPM), it will look for
a
> child Apache which is free, to pass the request for execution. If the
server is
> lightly loaded, it may be that the child to which it passes the next
request is
> always the same, because it is always done with the previous request, and
> free, when the next request comes in.
> But if this child process somehow has a limit to how many requests it may
> process before it dies, and this limit was around 50, then somewhere
around
> the 50th request, another child would get the next request, and for this
child
> it would be the first one (or the first one with /this/ cgi-bin script).
>
> There are roughly similar phenomenons that might happen in other kinds of
> Apache MPM's, which is why it is important to know which one you are
using,
> and with which setup parameters.
>
> And yes, there are ways to improve this. But again, we would need to know
> more about your configuration in order to make suggestions.

Hello David, André,

the server we're working with is fully dedicated to Koha.

Thank you André for your hints about Apache configurations.

@David, same thing, I'll try to preload C4::ILSDI:Services and see if it
helps.

Thanks to both of you, I'll let you know about the results.
Cheers,
Arthur

Le 05/10/2016 à 01:20, David Cook a écrit :
> Sounds like André is on the right track. I've certainly run into a similar
> issue (with a non-Koha app).
>
> However, because I was using the Catalyst framework, I was able to just
> preload the entire app, so that the Perl modules were loaded into the Apache
> master process before forking, and that did the trick. That app is a lot
> smaller than Koha though too. This case is a bit more complicated since Koha
> isn't really a MVC app, but you could look at the Koha Plack examples and
> see which modules they pre-load.
>
> You might also want to run ilsdi.pl with Devel::NYTProf to identify what
> exactly is slowing down that 1s long query.
>
> I've only played with PerlOptions +Parent a bit, but I'm guessing that you
> have multiple mod_perl apps, and that we're not actually seeing your entire
> httpd.conf configuration relating to Koha, right?
>
> David Cook
> Systems Librarian
> Prosentient Systems
> 72/330 Wattle St
> Ultimo, NSW 2007
> Australia
>
> Office: 02 9212 0899
> Direct: 02 8005 0595
>
>
>> -----Original Message-----
>> From: André Warnier [mailto:aw@ice-sa.com]
>> Sent: Tuesday, 4 October 2016 10:02 PM
>> To: modperl@perl.apache.org
>> Subject: Re: Cache refresh each 50 queries?
>>
>> Hi.
>>
>> On 04.10.2016 10:17, SUZUKI Arthur wrote:
>> [...]
>>
>>> On 03.10.2016 16:57, SUZUKI Arthur wrote:
>>>>> Hello List,
>>>>> I'm sending this message to know if there are some hints/tips to
>>>>> help with the problem we're facing.
>>>>> The problem is that for a same query repeated over time, reply time
>>>>> can be as short as 5ms and as long as 1s.
>>>>>
>>>>> Since there is neither correlation with CPU load or RAM usage, nor
>>>>> any networking constraints, we think it is due to the cache refresh
>>>>> mechanism.
>>>>>
>>>>> The problem occurs at random times, with a probability of around
>>>>> 1/50 queries (empirical data).
>>>>>
>>>>> Is there any configuration option which could help?
>>>>>
>>>>> Our current httpd.conf contains the following:
>>>>>
>>>>> PerlModule ModPerl::Registry
>>>>> PerlOptions +Parent
>>>>> PerlSwitches -I/home/koha/src
>>>>>
>>>>> <Files "ilsdi.pl">
>>>>> SetHandler perl-script
>>>>> # more faster, link with worker
>>>>> PerlResponseHandler ModPerl::Registry
>>>>> # less faster, link with prefork
>>>>> #PerlResponseHandler ModPerl::PerlRun
>>>>> Options +ExecCGI
>>>>> PerlOptions +ParseHeaders
>>>>> </Files>
>>>>>
>>>>> Thanks a lot in advance for your replies.
>>>>> Best regards,
>> You still do not provide enough information about your server
> configuration
>> (the part not mod_perl specific) to really make a narrow guess, but let me
> try
>> anyway :
>>
>> One thing that might explain a seemingly-long processing time for a
> request
>> approximately once in 50 calls, is if there was something special and
>> resource-intensive which happens once per approximately 50 calls, right ?
>>
>> One thing which /might/ explain this, /perhaps/ and depending on your
>> configuration, is if approximately once in 50 calls, Apache had to fork a
> new
>> child process, to handle the call. This new child process would then start
> with
>> a brand-new, fresh Perl interpreter, which might need to recompile some
>> modules before it even starts handling the request.
>>
>> Look for example here :
>> http://perl.apache.org/docs/2.0/user/config/config.html#C_Parent_
>> and your configuration line above :
>> PerlOptions +Parent
>>
>> In that configuration, the cgi-bin script which you are running (and all
> its
>> dependencies), is not compiled by the embedded perl until it has been run
> at
>> least once.
>> So the first execution will take more time than the following ones.
>> Now if the rest of your configuration is so that every 50 requests or so,
>> Apache starts a new child with a new perl interpreter, that may explain
> the
>> symptoms that you are observing.
>>
>> When Apache handles a request (if it is a "prefork" MPM), it will look for
> a
>> child Apache which is free, to pass the request for execution. If the
> server is
>> lightly loaded, it may be that the child to which it passes the next
> request is
>> always the same, because it is always done with the previous request, and
>> free, when the next request comes in.
>> But if this child process somehow has a limit to how many requests it may
>> process before it dies, and this limit was around 50, then somewhere
> around
>> the 50th request, another child would get the next request, and for this
> child
>> it would be the first one (or the first one with /this/ cgi-bin script).
>>
>> There are roughly similar phenomenons that might happen in other kinds of
>> Apache MPM's, which is why it is important to know which one you are
> using,
>> and with which setup parameters.
>>
>> And yes, there are ways to improve this. But again, we would need to know
>> more about your configuration in order to make suggestions.
>
>

--
Arthur SUZUKI
Service informatique des bibliothèques
BIBLIOTHÈQUES UNIVERSITAIRES
Université Jean Moulin Lyon 3
6 Cours Albert Thomas - B.P. 8242 – 69355 Lyon Cedex 08
ligne directe : +33 (0)4 78 78 79 16 | http://bu.univ-lyon3.fr
L'Université Jean Moulin est membre fondateur de l'Université de Lyon

Additional hint :

Normally, when Apache goes through the "effort" of setting up a new child process, you
would want that this child process then runs for as long as possible (for as many requests
as possible), to avoid the repeated overhead of restarting another child process.
In an Apache prefork configuration, you would do this by setting

MaxRequestsPerChild 0 (or a very high number)

(as per http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild)

However, some perl modules/applications are known to "leak memory" (meaning that each time
they are called, the Apache child memory footprint increases a bit, and never goes down
anymore). And after a while, depending on how busy the site is and how many times that
application is called, Apache may crash if it does not have enough memory anymore.

That is why, in some cases, you would set this MaxRequestsPerChild to a value different
from 0, to limit the number of requests that any Apache child would process, before it is
forcefully terminated by Apache, and replaced by a new one.
(The exact number used, would depend on how much additional memory it uses after each
request).
Terminating this child process would return the used memory to the OS, and the new child
would start again fresh, with a minimum amount of memory.
Then it will start growing again, until it hits the limit again, etc..

So, setting MaxRequestsPerChild <> 0, is in fact a "quick-and-dirty fix" for a problem of
the application (it should not do that, leaking memory). And it would be better to fix the
application, so that it can run without leaking memory..

P.S.
You still have not indicated what your OS is, or what kind of Apache MPM you are using.
If you need more help, please do that the next time you post a message here.


On 05.10.2016 10:53, SUZUKI Arthur wrote:
> Hello David, André,
>
> the server we're working with is fully dedicated to Koha.
>
> Thank you André for your hints about Apache configurations.
>
> @David, same thing, I'll try to preload C4::ILSDI:Services and see if it helps.
>
> Thanks to both of you, I'll let you know about the results.
> Cheers,
> Arthur
>
> Le 05/10/2016 à 01:20, David Cook a écrit :
>> Sounds like André is on the right track. I've certainly run into a similar
>> issue (with a non-Koha app).
>>
>> However, because I was using the Catalyst framework, I was able to just
>> preload the entire app, so that the Perl modules were loaded into the Apache
>> master process before forking, and that did the trick. That app is a lot
>> smaller than Koha though too. This case is a bit more complicated since Koha
>> isn't really a MVC app, but you could look at the Koha Plack examples and
>> see which modules they pre-load.
>>
>> You might also want to run ilsdi.pl with Devel::NYTProf to identify what
>> exactly is slowing down that 1s long query.
>>
>> I've only played with PerlOptions +Parent a bit, but I'm guessing that you
>> have multiple mod_perl apps, and that we're not actually seeing your entire
>> httpd.conf configuration relating to Koha, right?
>>
>> David Cook
>> Systems Librarian
>> Prosentient Systems
>> 72/330 Wattle St
>> Ultimo, NSW 2007
>> Australia
>>
>> Office: 02 9212 0899
>> Direct: 02 8005 0595
>>
>>
>>> -----Original Message-----
>>> From: André Warnier [mailto:aw@ice-sa.com]
>>> Sent: Tuesday, 4 October 2016 10:02 PM
>>> To: modperl@perl.apache.org
>>> Subject: Re: Cache refresh each 50 queries?
>>>
>>> Hi.
>>>
>>> On 04.10.2016 10:17, SUZUKI Arthur wrote:
>>> [...]
>>>
>>>> On 03.10.2016 16:57, SUZUKI Arthur wrote:
>>>>>> Hello List,
>>>>>> I'm sending this message to know if there are some hints/tips to
>>>>>> help with the problem we're facing.
>>>>>> The problem is that for a same query repeated over time, reply time
>>>>>> can be as short as 5ms and as long as 1s.
>>>>>>
>>>>>> Since there is neither correlation with CPU load or RAM usage, nor
>>>>>> any networking constraints, we think it is due to the cache refresh
>>>>>> mechanism.
>>>>>>
>>>>>> The problem occurs at random times, with a probability of around
>>>>>> 1/50 queries (empirical data).
>>>>>>
>>>>>> Is there any configuration option which could help?
>>>>>>
>>>>>> Our current httpd.conf contains the following:
>>>>>>
>>>>>> PerlModule ModPerl::Registry
>>>>>> PerlOptions +Parent
>>>>>> PerlSwitches -I/home/koha/src
>>>>>>
>>>>>> <Files "ilsdi.pl">
>>>>>> SetHandler perl-script
>>>>>> # more faster, link with worker
>>>>>> PerlResponseHandler ModPerl::Registry
>>>>>> # less faster, link with prefork
>>>>>> #PerlResponseHandler ModPerl::PerlRun
>>>>>> Options +ExecCGI
>>>>>> PerlOptions +ParseHeaders
>>>>>> </Files>
>>>>>>
>>>>>> Thanks a lot in advance for your replies.
>>>>>> Best regards,
>>> You still do not provide enough information about your server
>> configuration
>>> (the part not mod_perl specific) to really make a narrow guess, but let me
>> try
>>> anyway :
>>>
>>> One thing that might explain a seemingly-long processing time for a
>> request
>>> approximately once in 50 calls, is if there was something special and
>>> resource-intensive which happens once per approximately 50 calls, right ?
>>>
>>> One thing which /might/ explain this, /perhaps/ and depending on your
>>> configuration, is if approximately once in 50 calls, Apache had to fork a
>> new
>>> child process, to handle the call. This new child process would then start
>> with
>>> a brand-new, fresh Perl interpreter, which might need to recompile some
>>> modules before it even starts handling the request.
>>>
>>> Look for example here :
>>> http://perl.apache.org/docs/2.0/user/config/config.html#C_Parent_
>>> and your configuration line above :
>>> PerlOptions +Parent
>>>
>>> In that configuration, the cgi-bin script which you are running (and all
>> its
>>> dependencies), is not compiled by the embedded perl until it has been run
>> at
>>> least once.
>>> So the first execution will take more time than the following ones.
>>> Now if the rest of your configuration is so that every 50 requests or so,
>>> Apache starts a new child with a new perl interpreter, that may explain
>> the
>>> symptoms that you are observing.
>>>
>>> When Apache handles a request (if it is a "prefork" MPM), it will look for
>> a
>>> child Apache which is free, to pass the request for execution. If the
>> server is
>>> lightly loaded, it may be that the child to which it passes the next
>> request is
>>> always the same, because it is always done with the previous request, and
>>> free, when the next request comes in.
>>> But if this child process somehow has a limit to how many requests it may
>>> process before it dies, and this limit was around 50, then somewhere
>> around
>>> the 50th request, another child would get the next request, and for this
>> child
>>> it would be the first one (or the first one with /this/ cgi-bin script).
>>>
>>> There are roughly similar phenomenons that might happen in other kinds of
>>> Apache MPM's, which is why it is important to know which one you are
>> using,
>>> and with which setup parameters.
>>>
>>> And yes, there are ways to improve this. But again, we would need to know
>>> more about your configuration in order to make suggestions.
>>
>>
>

Hi again.

I want to correct somewhat what I was saying below, in "Additional hint".
See : http://perl.apache.org/docs/1.0/guide/performance.html#Sharing_Memory

What I was saying about "leaking memory" below remains true.

But even with applications which do not "leak memory" per se, there will anyway be some
increase in the amount of memory that each Apache child is using, over time.
So the "best" setting for "MaxRequestsPerChild", in an Apache/mod_perl scenario, may not
be 0 ( = unlimited). Instead, use some relatively large number, such as 1000 for example.
That will cause a child to serve 1000 requests before it gets replaced by a new one.
And having a slightly longer delay every 1000 requests, would probably not even be
noticed. (Of course, if your server crashes with an OOM before you reach 1000, then you
will have to reduce that number, experimentally).

The reason is well explained in that page above.


On 05.10.2016 11:30, A. Warnier wrote:
> Additional hint :
>
> Normally, when Apache goes through the "effort" of setting up a new child process, you
> would want that this child process then runs for as long as possible (for as many requests
> as possible), to avoid the repeated overhead of restarting another child process.
> In an Apache prefork configuration, you would do this by setting
>
> MaxRequestsPerChild 0 (or a very high number)
>
> (as per http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild)
>
> However, some perl modules/applications are known to "leak memory" (meaning that each time
> they are called, the Apache child memory footprint increases a bit, and never goes down
> anymore). And after a while, depending on how busy the site is and how many times that
> application is called, Apache may crash if it does not have enough memory anymore.
>
> That is why, in some cases, you would set this MaxRequestsPerChild to a value different
> from 0, to limit the number of requests that any Apache child would process, before it is
> forcefully terminated by Apache, and replaced by a new one.
> (The exact number used, would depend on how much additional memory it uses after each
> request).
> Terminating this child process would return the used memory to the OS, and the new child
> would start again fresh, with a minimum amount of memory.
> Then it will start growing again, until it hits the limit again, etc..
>
> So, setting MaxRequestsPerChild <> 0, is in fact a "quick-and-dirty fix" for a problem of
> the application (it should not do that, leaking memory). And it would be better to fix the
> application, so that it can run without leaking memory..
>
> P.S.
> You still have not indicated what your OS is, or what kind of Apache MPM you are using.
> If you need more help, please do that the next time you post a message here.
>
>
> On 05.10.2016 10:53, SUZUKI Arthur wrote:
>> Hello David, André,
>>
>> the server we're working with is fully dedicated to Koha.
>>
>> Thank you André for your hints about Apache configurations.
>>
>> @David, same thing, I'll try to preload C4::ILSDI:Services and see if it helps.
>>
>> Thanks to both of you, I'll let you know about the results.
>> Cheers,
>> Arthur
>>
>> Le 05/10/2016 à 01:20, David Cook a écrit :
>>> Sounds like André is on the right track. I've certainly run into a similar
>>> issue (with a non-Koha app).
>>>
>>> However, because I was using the Catalyst framework, I was able to just
>>> preload the entire app, so that the Perl modules were loaded into the Apache
>>> master process before forking, and that did the trick. That app is a lot
>>> smaller than Koha though too. This case is a bit more complicated since Koha
>>> isn't really a MVC app, but you could look at the Koha Plack examples and
>>> see which modules they pre-load.
>>>
>>> You might also want to run ilsdi.pl with Devel::NYTProf to identify what
>>> exactly is slowing down that 1s long query.
>>>
>>> I've only played with PerlOptions +Parent a bit, but I'm guessing that you
>>> have multiple mod_perl apps, and that we're not actually seeing your entire
>>> httpd.conf configuration relating to Koha, right?
>>>
>>> David Cook
>>> Systems Librarian
>>> Prosentient Systems
>>> 72/330 Wattle St
>>> Ultimo, NSW 2007
>>> Australia
>>>
>>> Office: 02 9212 0899
>>> Direct: 02 8005 0595
>>>
>>>
>>>> -----Original Message-----
>>>> From: André Warnier [mailto:aw@ice-sa.com]
>>>> Sent: Tuesday, 4 October 2016 10:02 PM
>>>> To: modperl@perl.apache.org
>>>> Subject: Re: Cache refresh each 50 queries?
>>>>
>>>> Hi.
>>>>
>>>> On 04.10.2016 10:17, SUZUKI Arthur wrote:
>>>> [...]
>>>>
>>>>> On 03.10.2016 16:57, SUZUKI Arthur wrote:
>>>>>>> Hello List,
>>>>>>> I'm sending this message to know if there are some hints/tips to
>>>>>>> help with the problem we're facing.
>>>>>>> The problem is that for a same query repeated over time, reply time
>>>>>>> can be as short as 5ms and as long as 1s.
>>>>>>>
>>>>>>> Since there is neither correlation with CPU load or RAM usage, nor
>>>>>>> any networking constraints, we think it is due to the cache refresh
>>>>>>> mechanism.
>>>>>>>
>>>>>>> The problem occurs at random times, with a probability of around
>>>>>>> 1/50 queries (empirical data).
>>>>>>>
>>>>>>> Is there any configuration option which could help?
>>>>>>>
>>>>>>> Our current httpd.conf contains the following:
>>>>>>>
>>>>>>> PerlModule ModPerl::Registry
>>>>>>> PerlOptions +Parent
>>>>>>> PerlSwitches -I/home/koha/src
>>>>>>>
>>>>>>> <Files "ilsdi.pl">
>>>>>>> SetHandler perl-script
>>>>>>> # more faster, link with worker
>>>>>>> PerlResponseHandler ModPerl::Registry
>>>>>>> # less faster, link with prefork
>>>>>>> #PerlResponseHandler ModPerl::PerlRun
>>>>>>> Options +ExecCGI
>>>>>>> PerlOptions +ParseHeaders
>>>>>>> </Files>
>>>>>>>
>>>>>>> Thanks a lot in advance for your replies.
>>>>>>> Best regards,
>>>> You still do not provide enough information about your server
>>> configuration
>>>> (the part not mod_perl specific) to really make a narrow guess, but let me
>>> try
>>>> anyway :
>>>>
>>>> One thing that might explain a seemingly-long processing time for a
>>> request
>>>> approximately once in 50 calls, is if there was something special and
>>>> resource-intensive which happens once per approximately 50 calls, right ?
>>>>
>>>> One thing which /might/ explain this, /perhaps/ and depending on your
>>>> configuration, is if approximately once in 50 calls, Apache had to fork a
>>> new
>>>> child process, to handle the call. This new child process would then start
>>> with
>>>> a brand-new, fresh Perl interpreter, which might need to recompile some
>>>> modules before it even starts handling the request.
>>>>
>>>> Look for example here :
>>>> http://perl.apache.org/docs/2.0/user/config/config.html#C_Parent_
>>>> and your configuration line above :
>>>> PerlOptions +Parent
>>>>
>>>> In that configuration, the cgi-bin script which you are running (and all
>>> its
>>>> dependencies), is not compiled by the embedded perl until it has been run
>>> at
>>>> least once.
>>>> So the first execution will take more time than the following ones.
>>>> Now if the rest of your configuration is so that every 50 requests or so,
>>>> Apache starts a new child with a new perl interpreter, that may explain
>>> the
>>>> symptoms that you are observing.
>>>>
>>>> When Apache handles a request (if it is a "prefork" MPM), it will look for
>>> a
>>>> child Apache which is free, to pass the request for execution. If the
>>> server is
>>>> lightly loaded, it may be that the child to which it passes the next
>>> request is
>>>> always the same, because it is always done with the previous request, and
>>>> free, when the next request comes in.
>>>> But if this child process somehow has a limit to how many requests it may
>>>> process before it dies, and this limit was around 50, then somewhere
>>> around
>>>> the 50th request, another child would get the next request, and for this
>>> child
>>>> it would be the first one (or the first one with /this/ cgi-bin script).
>>>>
>>>> There are roughly similar phenomenons that might happen in other kinds of
>>>> Apache MPM's, which is why it is important to know which one you are
>>> using,
>>>> and with which setup parameters.
>>>>
>>>> And yes, there are ways to improve this. But again, we would need to know
>>>> more about your configuration in order to make suggestions.
>>>
>>>
>>
>

You can look at Apache::SizeLimit as an alternative - this is designed
to cope with applications which occassionaly "leak memory"..

If one requests uses a lot of memory it will not be recovered -- Perl
doesn't hand this memory back - so subsequent requests are handled by
the inflated process.

We use mod_perl handlers to track this to see which requests cause the
memory inflation to happen - these can then be looked at to see if they
can be optimized! Check that large datastructures are not being copied
around - avoiding processing one large data structure into another large
data structure etc....

On 10/5/2016 11:39 AM, A. Warnier wrote:
> Hi again.
>
> I want to correct somewhat what I was saying below, in "Additional hint".
> See :
> http://perl.apache.org/docs/1.0/guide/performance.html#Sharing_Memory
>
> What I was saying about "leaking memory" below remains true.
>
> But even with applications which do not "leak memory" per se, there
> will anyway be some increase in the amount of memory that each Apache
> child is using, over time.
> So the "best" setting for "MaxRequestsPerChild", in an Apache/mod_perl
> scenario, may not be 0 ( = unlimited). Instead, use some relatively
> large number, such as 1000 for example.
> That will cause a child to serve 1000 requests before it gets replaced
> by a new one.
> And having a slightly longer delay every 1000 requests, would probably
> not even be noticed. (Of course, if your server crashes with an OOM
> before you reach 1000, then you will have to reduce that number,
> experimentally).
>
> The reason is well explained in that page above.
>
>
> On 05.10.2016 11:30, A. Warnier wrote:
>> Additional hint :
>>
>> Normally, when Apache goes through the "effort" of setting up a new
>> child process, you
>> would want that this child process then runs for as long as possible
>> (for as many requests
>> as possible), to avoid the repeated overhead of restarting another
>> child process.
>> In an Apache prefork configuration, you would do this by setting
>>
>> MaxRequestsPerChild 0 (or a very high number)
>>
>> (as per
>> http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild)
>>
>> However, some perl modules/applications are known to "leak memory"
>> (meaning that each time
>> they are called, the Apache child memory footprint increases a bit,
>> and never goes down
>> anymore). And after a while, depending on how busy the site is and
>> how many times that
>> application is called, Apache may crash if it does not have enough
>> memory anymore.
>>
>> That is why, in some cases, you would set this MaxRequestsPerChild to
>> a value different
>> from 0, to limit the number of requests that any Apache child would
>> process, before it is
>> forcefully terminated by Apache, and replaced by a new one.
>> (The exact number used, would depend on how much additional memory it
>> uses after each
>> request).
>> Terminating this child process would return the used memory to the
>> OS, and the new child
>> would start again fresh, with a minimum amount of memory.
>> Then it will start growing again, until it hits the limit again, etc..
>>
>> So, setting MaxRequestsPerChild <> 0, is in fact a "quick-and-dirty
>> fix" for a problem of
>> the application (it should not do that, leaking memory). And it would
>> be better to fix the
>> application, so that it can run without leaking memory..
>>
>> P.S.
>> You still have not indicated what your OS is, or what kind of Apache
>> MPM you are using.
>> If you need more help, please do that the next time you post a
>> message here.
>>
>>
>> On 05.10.2016 10:53, SUZUKI Arthur wrote:
>>> Hello David, André,
>>>
>>> the server we're working with is fully dedicated to Koha.
>>>
>>> Thank you André for your hints about Apache configurations.
>>>
>>> @David, same thing, I'll try to preload C4::ILSDI:Services and see
>>> if it helps.
>>>
>>> Thanks to both of you, I'll let you know about the results.
>>> Cheers,
>>> Arthur
>>>
>>> Le 05/10/2016 à 01:20, David Cook a écrit :
>>>> Sounds like André is on the right track. I've certainly run into a
>>>> similar
>>>> issue (with a non-Koha app).
>>>>
>>>> However, because I was using the Catalyst framework, I was able to
>>>> just
>>>> preload the entire app, so that the Perl modules were loaded into
>>>> the Apache
>>>> master process before forking, and that did the trick. That app is
>>>> a lot
>>>> smaller than Koha though too. This case is a bit more complicated
>>>> since Koha
>>>> isn't really a MVC app, but you could look at the Koha Plack
>>>> examples and
>>>> see which modules they pre-load.
>>>>
>>>> You might also want to run ilsdi.pl with Devel::NYTProf to identify
>>>> what
>>>> exactly is slowing down that 1s long query.
>>>>
>>>> I've only played with PerlOptions +Parent a bit, but I'm guessing
>>>> that you
>>>> have multiple mod_perl apps, and that we're not actually seeing
>>>> your entire
>>>> httpd.conf configuration relating to Koha, right?
>>>>
>>>> David Cook
>>>> Systems Librarian
>>>> Prosentient Systems
>>>> 72/330 Wattle St
>>>> Ultimo, NSW 2007
>>>> Australia
>>>>
>>>> Office: 02 9212 0899
>>>> Direct: 02 8005 0595
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: André Warnier [mailto:aw@ice-sa.com]
>>>>> Sent: Tuesday, 4 October 2016 10:02 PM
>>>>> To: modperl@perl.apache.org
>>>>> Subject: Re: Cache refresh each 50 queries?
>>>>>
>>>>> Hi.
>>>>>
>>>>> On 04.10.2016 10:17, SUZUKI Arthur wrote:
>>>>> [...]
>>>>>
>>>>>> On 03.10.2016 16:57, SUZUKI Arthur wrote:
>>>>>>>> Hello List,
>>>>>>>> I'm sending this message to know if there are some hints/tips to
>>>>>>>> help with the problem we're facing.
>>>>>>>> The problem is that for a same query repeated over time, reply
>>>>>>>> time
>>>>>>>> can be as short as 5ms and as long as 1s.
>>>>>>>>
>>>>>>>> Since there is neither correlation with CPU load or RAM usage, nor
>>>>>>>> any networking constraints, we think it is due to the cache
>>>>>>>> refresh
>>>>>>>> mechanism.
>>>>>>>>
>>>>>>>> The problem occurs at random times, with a probability of around
>>>>>>>> 1/50 queries (empirical data).
>>>>>>>>
>>>>>>>> Is there any configuration option which could help?
>>>>>>>>
>>>>>>>> Our current httpd.conf contains the following:
>>>>>>>>
>>>>>>>> PerlModule ModPerl::Registry
>>>>>>>> PerlOptions +Parent
>>>>>>>> PerlSwitches -I/home/koha/src
>>>>>>>>
>>>>>>>> <Files "ilsdi.pl">
>>>>>>>> SetHandler perl-script
>>>>>>>> # more faster, link with worker
>>>>>>>> PerlResponseHandler ModPerl::Registry
>>>>>>>> # less faster, link with prefork
>>>>>>>> #PerlResponseHandler ModPerl::PerlRun
>>>>>>>> Options +ExecCGI
>>>>>>>> PerlOptions +ParseHeaders
>>>>>>>> </Files>
>>>>>>>>
>>>>>>>> Thanks a lot in advance for your replies.
>>>>>>>> Best regards,
>>>>> You still do not provide enough information about your server
>>>> configuration
>>>>> (the part not mod_perl specific) to really make a narrow guess,
>>>>> but let me
>>>> try
>>>>> anyway :
>>>>>
>>>>> One thing that might explain a seemingly-long processing time for a
>>>> request
>>>>> approximately once in 50 calls, is if there was something special and
>>>>> resource-intensive which happens once per approximately 50 calls,
>>>>> right ?
>>>>>
>>>>> One thing which /might/ explain this, /perhaps/ and depending on your
>>>>> configuration, is if approximately once in 50 calls, Apache had to
>>>>> fork a
>>>> new
>>>>> child process, to handle the call. This new child process would
>>>>> then start
>>>> with
>>>>> a brand-new, fresh Perl interpreter, which might need to recompile
>>>>> some
>>>>> modules before it even starts handling the request.
>>>>>
>>>>> Look for example here :
>>>>> http://perl.apache.org/docs/2.0/user/config/config.html#C_Parent_
>>>>> and your configuration line above :
>>>>> PerlOptions +Parent
>>>>>
>>>>> In that configuration, the cgi-bin script which you are running
>>>>> (and all
>>>> its
>>>>> dependencies), is not compiled by the embedded perl until it has
>>>>> been run
>>>> at
>>>>> least once.
>>>>> So the first execution will take more time than the following ones.
>>>>> Now if the rest of your configuration is so that every 50 requests
>>>>> or so,
>>>>> Apache starts a new child with a new perl interpreter, that may
>>>>> explain
>>>> the
>>>>> symptoms that you are observing.
>>>>>
>>>>> When Apache handles a request (if it is a "prefork" MPM), it will
>>>>> look for
>>>> a
>>>>> child Apache which is free, to pass the request for execution. If
>>>>> the
>>>> server is
>>>>> lightly loaded, it may be that the child to which it passes the next
>>>> request is
>>>>> always the same, because it is always done with the previous
>>>>> request, and
>>>>> free, when the next request comes in.
>>>>> But if this child process somehow has a limit to how many requests
>>>>> it may
>>>>> process before it dies, and this limit was around 50, then somewhere
>>>> around
>>>>> the 50th request, another child would get the next request, and
>>>>> for this
>>>> child
>>>>> it would be the first one (or the first one with /this/ cgi-bin
>>>>> script).
>>>>>
>>>>> There are roughly similar phenomenons that might happen in other
>>>>> kinds of
>>>>> Apache MPM's, which is why it is important to know which one you are
>>>> using,
>>>>> and with which setup parameters.
>>>>>
>>>>> And yes, there are ways to improve this. But again, we would need
>>>>> to know
>>>>> more about your configuration in order to make suggestions.
>>>>
>>>>
>>>
>>
>



--
The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.