Mailing List Archive: Memory usage with FCGI? Server running out of RAM

Memory usage with FCGI? Server running out of RAM

Jun 11, 2015, 5:31 AM

Post #1 of 3 (2113 views)

Sysadmin-fu needed: I have a personal server in the cloud that has 1 GB
of RAM. It's running Debian, and the usual server basics--MySQL, nginx,
etc. No X, of course.

There are three Catalyst apps running, all of them fairly small, using
nginx and FCGI (I run the MyApp_fastcgi.pl script to a socket, that
nginx then responds to). But they're taking up all the memory on the
system, and I can't run another Cat app (and sometimes even system
commands hang for lack of memory). I tried reducing --nproc to 1 for two
of the three apps that I'm the only one using (the other has --nproc 3).
The output of "top" with "M" (sorting by RES) looks like (I removed a
few columns for spacing):

PID USER VIRT RES SHR S %MEM TIME+ COMMAND

23766 www-data 440m 230m 2584 S 23.0 0:13.40 perl-fcgi
16958 mysql 428m 188m 0 S 18.8 118:14.98 mysqld
25992 www-data 375m 158m 2608 S 15.9 0:06.80 perl-fcgi
26884 www-data 238m 80m 2616 S 8.0 0:01.61 perl-fcgi
2513 www-data 282m 76m 0 S 7.7 0:06.01 perl-fcgi
2512 www-data 182m 66m 0 S 6.6 0:00.00 perl-fcgi-pm [Library]
1825 www-data 172m 61m 0 S 6.1 0:03.44 perl
2487 www-data 163m 46m 60 S 4.6 0:00.18 perl-fcgi-pm [CatSF]
1692 root 119m 5128 140 S 0.5 0:56.32 rsyslogd
19998 www-data 79452 3296 1136 S 0.3 6:10.61 nginx
2265 www-data 79768 3148 504 S 0.3 15:57.81 nginx
[...]

Is this an expected or appropriate amount of memory usage? I appreciate
that 1GB isn't all that much memory nowadays, but this is a very lightly
loaded machine, and I don't want to pay a lot more for a larger cloud
server to handle this if it's not necessary.

Thanks.

_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: Memory usage with FCGI? Server running out of RAM [ In reply to ]

2015 at denny

Jun 11, 2015, 6:03 AM

Post #2 of 3 (2042 views)

Permalink

I run a server with 18 separate installs of ShinyCMS running, with at least 3 and sometimes 20 processes per install. 122 perl-fcgi processes in total. They look a bit bigger than most of your processes. I've got 8GB of RAM, and I'm pushing up against the edges of it a bit, but I don't usually have any problems.

Hope that's useful info!

On 11 June 2015 13:31:16 BST, Jesse Sheidlower <jester@panix.com> wrote:
>
>Sysadmin-fu needed: I have a personal server in the cloud that has 1 GB
>of RAM. It's running Debian, and the usual server basics--MySQL, nginx,
>etc. No X, of course.
>
>There are three Catalyst apps running, all of them fairly small, using
>nginx and FCGI (I run the MyApp_fastcgi.pl script to a socket, that
>nginx then responds to). But they're taking up all the memory on the
>system, and I can't run another Cat app (and sometimes even system
>commands hang for lack of memory). I tried reducing --nproc to 1 for
>two
>of the three apps that I'm the only one using (the other has --nproc
>3).
>The output of "top" with "M" (sorting by RES) looks like (I removed a
>few columns for spacing):
>
> PID USER VIRT RES SHR S %MEM TIME+ COMMAND
>
>23766 www-data 440m 230m 2584 S 23.0 0:13.40 perl-fcgi
>16958 mysql 428m 188m 0 S 18.8 118:14.98 mysqld
>25992 www-data 375m 158m 2608 S 15.9 0:06.80 perl-fcgi
>26884 www-data 238m 80m 2616 S 8.0 0:01.61 perl-fcgi
> 2513 www-data 282m 76m 0 S 7.7 0:06.01 perl-fcgi
> 2512 www-data 182m 66m 0 S 6.6 0:00.00 perl-fcgi-pm [Library]
> 1825 www-data 172m 61m 0 S 6.1 0:03.44 perl
> 2487 www-data 163m 46m 60 S 4.6 0:00.18 perl-fcgi-pm [CatSF]
> 1692 root 119m 5128 140 S 0.5 0:56.32 rsyslogd
>19998 www-data 79452 3296 1136 S 0.3 6:10.61 nginx
> 2265 www-data 79768 3148 504 S 0.3 15:57.81 nginx
>[...]
>
>Is this an expected or appropriate amount of memory usage? I appreciate
>that 1GB isn't all that much memory nowadays, but this is a very
>lightly
>loaded machine, and I don't want to pay a lot more for a larger cloud
>server to handle this if it's not necessary.
>
>Thanks.
>
>_______________________________________________
>List: Catalyst@lists.scsys.co.uk
>Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>Searchable archive:
>http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>Dev site: http://dev.catalyst.perl.org/

--
Sent from my phone. Please excuse typos, terseness, and top-posting.

Re: Memory usage with FCGI? Server running out of RAM [ In reply to ]

lasse at unity3d

Jun 11, 2015, 6:53 AM

Post #3 of 3 (2049 views)

Permalink

In my experience, it's usually a few naughty requests that gobble up a lot
of memory. We found it useful to record process vsize and rss at the
beginning and end of the request cycle to generate warnings like this:

Warning: PID 28710 grew rss by 20% from 42 MiB to 51 MiB serving GET
/api/search?q=foo

One of the common mistakes in our app was inflating all row objects in a
result set at the same time when it wasn't needed. We prefetch a lot so row
objects and their related objects get big. In those cases, this:

foreach my $row ($rs->all)
{
push @results, $row->to_json,
}

Will use a lot more memory than this:

while (my $row = $rs->next)
{
push @results, $row->to_json,
}

DBIx::Class's HashRefInflator is often touted for its speed but it's also a
lot lighter on memory...

Another thing you could consider is using Plack to run all your Perl apps
in the same server/process. Memory for all common dependencies would be
shared among the apps in that case which might get you a decent gain...

There's also a bunch of Plack::MiddleWare's on CPAN to help with
debugging/profiling of apps...

/L

On Thu, Jun 11, 2015 at 3:03 PM, Denny <2015@denny.me> wrote:

> I run a server with 18 separate installs of ShinyCMS running, with at
> least 3 and sometimes 20 processes per install. 122 perl-fcgi processes in
> total. They look a bit bigger than most of your processes. I've got 8GB of
> RAM, and I'm pushing up against the edges of it a bit, but I don't usually
> have any problems.
>
> Hope that's useful info!
>
>
>
> On 11 June 2015 13:31:16 BST, Jesse Sheidlower <jester@panix.com> wrote:
>
>>
>> Sysadmin-fu needed: I have a personal server in the cloud that has 1 GB
>> of RAM. It's running Debian, and the usual server basics--MySQL, nginx,
>> etc. No X, of course.
>>
>> There are three Catalyst apps running, all of them fairly small, using
>> nginx and FCGI (I run the MyApp_fastcgi.pl script to a socket, that
>> nginx then responds to). But they're taking up all the memory on the
>> system, and I can't run another Cat app (and sometimes even system
>> commands hang for lack of memory). I tried reducing --nproc to 1 for two
>> of the three apps that I'm the only one using (the other has --nproc 3).
>> The output of "top" with "M" (sorting by RES) looks like (I removed a
>> few columns for spacing):
>>
>> PID USER VIRT RES SHR S %MEM TIME+ COMMAND
>>
>> 23766 www-data 440m 230m 2584 S 23.0 0:13.40 perl-fcgi
>> 16958 mysql 428m 188m 0 S 18.8 118:14.98
>> mysqld
>> 25992 www-data 375m 158m 2608 S 15.9 0:06.80 perl-fcgi
>> 26884 www-data 238m 80m 2616 S 8.0 0:01.61 perl-fcgi
>> 2513 www-data 282m 76m 0 S 7.7 0:06.01 perl-fcgi
>> 2512 www-data 182m 66m 0 S 6.6 0:00.00 perl-fcgi-pm [Library]
>> 1825 www-data 172m 61m 0 S 6.1 0:03.44 perl
>> 2487 www-data 163m 46m 60 S 4.6 0:00.18 perl-fcgi-pm [CatSF]
>> 1692 root 119m 5128 140 S 0.5 0:56.32 rsyslogd
>> 19998 www-data 79452 3296 1136 S 0.3 6:10.61 nginx
>> 2265 www-data 79768 3148 504 S 0.3 15:57.81 nginx
>> [...]
>>
>> Is this an expected or appropriate amount of memory usage? I appreciate
>> that 1GB isn't all that much memory nowadays, but this is a very lightly
>> loaded machine, and I don't want to pay a lot more for a larger cloud
>> server to handle this if it's not necessary.
>>
>> Thanks.
>>
>> ------------------------------
>>
>> List: Catalyst@lists.scsys.co.uk
>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>> Dev site: http://dev.catalyst.perl.org/
>>
>>
> --
> Sent from my phone. Please excuse typos, terseness, and top-posting.
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>