Mailing List Archive: Catalyst Performance Testing

Catalyst Performance Testing

Apr 22, 2009, 5:50 PM

Post #1 of 9 (5888 views)

Hello Crew,

I've been using Catalyst for a while now, thank you great framework. I
have a new project that requires a much higher performance than my
previous projects and I decided to do some load testing and would like
to share my results for a sanity check.
I tested two flavors: nginx + fastcgi and apache + modperl and I'm
using version 5.71 of Catalyst

Baseline: Just using the webserver (apache or nginx) to download a
static file
Static Catalyst: Running the same query but going through Catalyst to
retrieve the same static file (using Static::Simple)
Noop Catalyst: Processing a request on a empty controller

* all numbers in request per second
Nginx + Fastcgi + Catalyst
---------------------------
Baseline: 1553 req/s
Static Catalyst: 134
Noop Catalyst: 91

Apache + Modperl + Catalyst
-----------------------------
Baseline: 976
Static Catalyst: 120
Noop Catalyst: 84

So a couple questions:
* Looking at these results. Do they seem consistent in what the
overhead of Catalyst should be?
* Is there advice that can be done to disable feature in Catalyst I
might not need in order to increase performance?
- I've took out StackTrace and -Debug from the main pm file
- I've removed the $log as well
* Does upgrading to 5.8 increase performance?

I wrote a very simple 10 line modperl app that just grabs a static file
and dumps it to STDOUT just for a little more comparison:
Static Modperl: 472 reqs/sec

So there is ~71% efficiency hit in using Catalyst to serve a static file
and simple modperl script. Granted Catalyst is doing a lot more stuff
but seems like a humongous hit to take.

How was the load testing performed: apache ab from multiple client servers
ex: ab -t 10 -c 40 "http://x-x-x-x.compute-1.amazonaws.com/foo/"
* Each test was run three times and the results where averaged
* modperl and fastcgi process memory was preloaded prior to test execution

Hardware: AWS EC2 - small instance

Apache Settings:
<IfModule mpm_prefork_module>
StartServers 45
MinSpareServers 30
MaxSpareServers 45
MaxClients 45
MaxRequestsPerChild 0
</IfModule>

FastCGI Setttings:
..../script/xxxx_fastcgi.pl -n 35 -d

Also completely unrelated. I could not get Catalyst working on the
mpm_worker_module. Is Catalyst not thread safe?

Thanks in advance for your time,

Jason Wieland

_______________________________________________
Catalyst-dev mailing list
Catalyst-dev@lists.scsys.co.uk
http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst-dev

Re: Catalyst Performance Testing [ In reply to ]

peter at peknet

Apr 22, 2009, 6:20 PM

Post #2 of 9 (5747 views)

Permalink

Jason Wieland wrote on 4/22/09 7:50 PM:

> So a couple questions:
> * Looking at these results. Do they seem consistent in what the
> overhead of Catalyst should be?
> * Is there advice that can be done to disable feature in Catalyst I
> might not need in order to increase performance?
> - I've took out StackTrace and -Debug from the main pm file
> - I've removed the $log as well
> * Does upgrading to 5.8 increase performance?
>
> I wrote a very simple 10 line modperl app that just grabs a static file
> and dumps it to STDOUT just for a little more comparison:
> Static Modperl: 472 reqs/sec
>
> So there is ~71% efficiency hit in using Catalyst to serve a static file
> and simple modperl script. Granted Catalyst is doing a lot more stuff
> but seems like a humongous hit to take.

Those numbers seem very reasonable to me.

But I never serve static content with Catalyst in production. Static::Simple is
for development only.

Your benchmark is accurate but it's measuring apples and oranges. Catalyst is
not optimized for serving static content, because there's already good tools
that do that. It's optimized for development of dynamic content.

Why not measure how many requests your web server can serve for dynamic content?

--
Peter Karman . http://peknet.com/ . peter@peknet.com

_______________________________________________
Catalyst-dev mailing list
Catalyst-dev@lists.scsys.co.uk
http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst-dev

Re: Catalyst Performance Testing [ In reply to ]

jwieland at gmail

Apr 22, 2009, 8:12 PM

Post #3 of 9 (5741 views)

Permalink

Peter Karman wrote:
> Jason Wieland wrote on 4/22/09 7:50 PM:
>
>
>> So a couple questions:
>> * Looking at these results. Do they seem consistent in what the
>> overhead of Catalyst should be?
>> * Is there advice that can be done to disable feature in Catalyst I
>> might not need in order to increase performance?
>> - I've took out StackTrace and -Debug from the main pm file
>> - I've removed the $log as well
>> * Does upgrading to 5.8 increase performance?
>>
>> I wrote a very simple 10 line modperl app that just grabs a static file
>> and dumps it to STDOUT just for a little more comparison:
>> Static Modperl: 472 reqs/sec
>>
>> So there is ~71% efficiency hit in using Catalyst to serve a static file
>> and simple modperl script. Granted Catalyst is doing a lot more stuff
>> but seems like a humongous hit to take.
>>
>
> Those numbers seem very reasonable to me.
>
> But I never serve static content with Catalyst in production. Static::Simple is
> for development only.
>
> Your benchmark is accurate but it's measuring apples and oranges. Catalyst is
> not optimized for serving static content, because there's already good tools
> that do that. It's optimized for development of dynamic content.
>
> Why not measure how many requests your web server can serve for dynamic content?
>
>

Yes Peter I agree, I do serve all my static content from outside
Catalyst. I was just trying to get a feel of the type of overhead that
a minimal install of the framework. Or the difference between running a
simple script in vanilla modperl and Catalyst modperl. Tomorrow I will
create an app that has the same functionality in Catalyst and vanilla
modperl and rerun the benchmark to find out a better idea of the overhead.

Jason

Re: Catalyst Performance Testing [ In reply to ]

hdp.perl.catalyst.dev at weftsoar

Apr 22, 2009, 8:19 PM

Post #4 of 9 (5726 views)

Permalink

On Wed, Apr 22, 2009 at 08:12:15PM -0700, Jason Wieland wrote:
> Yes Peter I agree, I do serve all my static content from outside Catalyst. I
> was just trying to get a feel of the type of overhead that a minimal install of
> the framework.

Which is useless at best and misleading at worst; it's like printing repeatedly
to /dev/null and expecting that to tell you anything significant about what OS
will run your database faster.

hdp.

_______________________________________________
Catalyst-dev mailing list
Catalyst-dev@lists.scsys.co.uk
http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst-dev

Re: Catalyst Performance Testing [ In reply to ]

peter at peknet

Apr 22, 2009, 8:45 PM

Post #5 of 9 (5737 views)

Permalink

> Peter Karman wrote:
>> Catalyst is
>> not optimized for serving static content, because there's already good
>> tools
>> that do that. It's optimized for development of dynamic content.
>>

I just want to re-emphasize what I wrote earlier. Catalyst is optimized for the
*development* of dynamic content. Not necessarily for the delivery of said
content. Sure, you could write a dynamic site for "pure" mod_perl, or even in C,
but it'll take a lot longer to write, and you won't be able to leverage the
[ever-increasing] number of plugins, base controllers/models/views, etc., that
Catalyst offers.

The most expensive part of any web site development is developer time, including
maintenance.

That said, the most useful benchmarks you'll be able to develop are going to be
those specific to *your* project, and such benchmarks should take into account
time-to-develop as well as requests-per-second. Hardware is cheap; knowledgeable
people are not.

--
Peter Karman . http://peknet.com/ . peter@peknet.com

_______________________________________________
Catalyst-dev mailing list
Catalyst-dev@lists.scsys.co.uk
http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst-dev

Re: Catalyst Performance Testing [ In reply to ]

diment at gmail

Apr 22, 2009, 8:51 PM

Post #6 of 9 (5754 views)

Permalink

Bear in mind that we know of Catalyst sites with hundreds of bandwidth
intensive hits per second that hasn't had to be rewritten in scala or
whatever in order to scale ;)

On 23/04/2009, at 1:45 PM, Peter Karman wrote:

>> Peter Karman wrote:
>>> Catalyst is
>>> not optimized for serving static content, because there's already
>>> good
>>> tools
>>> that do that. It's optimized for development of dynamic content.
>>>
>
> I just want to re-emphasize what I wrote earlier. Catalyst is
> optimized for the
> *development* of dynamic content. Not necessarily for the delivery
> of said
> content. Sure, you could write a dynamic site for "pure" mod_perl,
> or even in C,
> but it'll take a lot longer to write, and you won't be able to
> leverage the
> [ever-increasing] number of plugins, base controllers/models/views,
> etc., that
> Catalyst offers.
>
> The most expensive part of any web site development is developer
> time, including
> maintenance.
>
> That said, the most useful benchmarks you'll be able to develop are
> going to be
> those specific to *your* project, and such benchmarks should take
> into account
> time-to-develop as well as requests-per-second. Hardware is cheap;
> knowledgeable
> people are not.
>
> --
> Peter Karman . http://peknet.com/ . peter@peknet.com
>
> _______________________________________________
> Catalyst-dev mailing list
> Catalyst-dev@lists.scsys.co.uk
> http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst-dev

_______________________________________________
Catalyst-dev mailing list
Catalyst-dev@lists.scsys.co.uk
http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst-dev

Re: Catalyst Performance Testing [ In reply to ]

dbix-class at trout

Apr 23, 2009, 10:56 AM

Post #7 of 9 (5732 views)

Permalink

On Wed, Apr 22, 2009 at 05:50:02PM -0700, Jason Wieland wrote:
> Baseline: Just using the webserver (apache or nginx) to download a
> static file
> Static Catalyst: Running the same query but going through Catalyst to
> retrieve the same static file (using Static::Simple)
> Noop Catalyst: Processing a request on a empty controller

So, you've benchmarked Hello World. Why do you care?

I've worked on a lot of large scale Catalyst applications, and done a
fair amount of optimisation work, and I have *never* found Catalyst to
be the bottleneck in my performance testing. Better still, because
the teams had used Catalyst they had more time at the end of the
project for performance testing and optimisation and it was easier
to clean up the code because the architecture of the application was
flexible. That really, massively beats caring about how fast Hello
World is :)

I'd point out that BBC iPlayer is built on Catalyst and DBIx::Class
- are you going to tell me you're going to need to scale further than
they do? :)

> Also completely unrelated. I could not get Catalyst working on the
> mpm_worker_module. Is Catalyst not thread safe?

Catalyst is threadsafe. Your DBD:: may not be, or something else you
load. But FastCGI is a much better production environment than mod_perl
anyway.

--
Matt S Trout Need help with your Catalyst or DBIx::Class project?
Technical Director http://www.shadowcat.co.uk/catalyst/
Shadowcat Systems Ltd. Want a managed development or deployment platform?
http://chainsawblues.vox.com/ http://www.shadowcat.co.uk/servers/

_______________________________________________
Catalyst-dev mailing list
Catalyst-dev@lists.scsys.co.uk
http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst-dev

Re: Catalyst Performance Testing [ In reply to ]

jwieland at gmail

Apr 23, 2009, 1:41 PM

Post #8 of 9 (5728 views)

Permalink

Matt S Trout wrote:
> On Wed, Apr 22, 2009 at 05:50:02PM -0700, Jason Wieland wrote:
>
>> Baseline: Just using the webserver (apache or nginx) to download a
>> static file
>> Static Catalyst: Running the same query but going through Catalyst to
>> retrieve the same static file (using Static::Simple)
>> Noop Catalyst: Processing a request on a empty controller
>>
>
> So, you've benchmarked Hello World. Why do you care?
>
> I've worked on a lot of large scale Catalyst applications, and done a
> fair amount of optimisation work, and I have *never* found Catalyst to
> be the bottleneck in my performance testing. Better still, because
> the teams had used Catalyst they had more time at the end of the
> project for performance testing and optimisation and it was easier
> to clean up the code because the architecture of the application was
> flexible. That really, massively beats caring about how fast Hello
> World is :)
>
I completely agree, one of my most recent projects was creating
http://notify.me pure Catalyst we get 10k+ hits a day, rapid
development, no issues, its a joy. Catalyst is wonderful technology for
all the reasons above and previous emails, hats off to the developers.

However I'm working on a different project for an ISP (ondemand
anti-phishing protection) which requires us to handle a peak of 10,000
reqs/second and response that takes no less then 250milliseconds. We
have an backend engine that can do that all what we need to put in front
is a restful web layer. Because of the beauti of Catalyst got this
working in a day or two. However to scaling is not going to work in the
current state.

I'm not using any of Catalyst plugins, just the vanilla Catalyst and its
MVC framework. I don't know if it would be better to hack through
Catalyst and remove the parts I don't need for this project. Or write
another very simple webframework re-create the parts of Catalyst I do use.

> I'd point out that BBC iPlayer is built on Catalyst and DBIx::Class
> - are you going to tell me you're going to need to scale further than
> they do? :)
>
>
>> Also completely unrelated. I could not get Catalyst working on the
>> mpm_worker_module. Is Catalyst not thread safe?
>>
>
> Catalyst is threadsafe. Your DBD:: may not be, or something else you
> load. But FastCGI is a much better production environment than mod_perl
> anyway.
>
Not using a DBD:: Just try out the base Catalyst example, created when
you start a new project.

I do agree fastcgi is a better production environment though. Just did
the modperl stuff for comparisons and also there are some old timers in
the office that are determined to think modperl has to be faster then
fcgi. Not withstanding the benefits of a two tiered response layer.

Thanks for everyone's time and responses.

Jason Wieland

Re: Catalyst Performance Testing [ In reply to ]

dbix-class at trout

Apr 27, 2009, 4:17 AM

Post #9 of 9 (5707 views)

Permalink

On Thu, Apr 23, 2009 at 01:41:55PM -0700, Jason Wieland wrote:
> However I'm working on a different project for an ISP (ondemand
> anti-phishing protection) which requires us to handle a peak of 10,000
> reqs/second and response that takes no less then 250milliseconds. We
> have an backend engine that can do that all what we need to put in front
> is a restful web layer. Because of the beauti of Catalyst got this
> working in a day or two. However to scaling is not going to work in the
> current state.
>
> I'm not using any of Catalyst plugins, just the vanilla Catalyst and its
> MVC framework. I don't know if it would be better to hack through
> Catalyst and remove the parts I don't need for this project. Or write
> another very simple webframework re-create the parts of Catalyst I do use.

Right, ok, you're actually one of the special cases where the framework
overhead actually does matter then. Sorry, 99% of the people benching
hello world aren't so I tend to assume :)

There are two things you can do then - (1) throw Devel::NYTProf at your
app and see if there's a specific bit of Catalyst that needs optimising,
which we'd be happy to help with, (2) take HTTP::Engine and Path::Dispatcher
and roll your own, treating Catalyst as your fast prototyping tool.

> >Catalyst is threadsafe. Your DBD:: may not be, or something else you
> >load. But FastCGI is a much better production environment than mod_perl
> >anyway.
> >
> Not using a DBD:: Just try out the base Catalyst example, created when
> you start a new project.

Well, if Catalyst wasn't threadsafe we'd hear screaming from the people
on win32, so in the absence of screaming I'm happy to assume it is :)

--
Matt S Trout Need help with your Catalyst or DBIx::Class project?
Technical Director http://www.shadowcat.co.uk/catalyst/
Shadowcat Systems Ltd. Want a managed development or deployment platform?
http://chainsawblues.vox.com/ http://www.shadowcat.co.uk/servers/

_______________________________________________
Catalyst-dev mailing list
Catalyst-dev@lists.scsys.co.uk
http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst-dev