Many of you have probably already seen this, but in case you haven't, I'll post it here. Props to my co-worker, Michael Peters, for pointing this out to me.

This is an article about benchmarking web frameworks in PHP, Ruby, and Python against each other. It's a nicely done test and my only complaint about it is that the code for the test apps and the benchmarking don't seem to be there.

Obviously, it would be interesting to see how CGI::Application, Jifty, and Catalyst would do in this comparison. It would require a willingness to learn enough about at least one of the other frameworks to do a fair test though, i.e. install and configure properly, and use the same code and benchmarking tools used in this article.

  • Comment on an interesting web frameworks benchmark

Replies are listed 'Best First'.
Re: an interesting web frameworks benchmark
by rhesa (Vicar) on Jul 17, 2006 at 00:03 UTC
    As far as I can tell, they just pointed Siege at a couple of the urls in the app, with two sets of configured durations and concurrency levels. It would be helpful to have more details on it though, I agree.

    It'd also make the task slightly easier if they had supplied the full source code to the Ruby app. The OnLAMP article only describes the steps required to build it. For those of you who are too lazy1 to RTFA, here's the short description:

    We'll create an online collaborative cookbook for holding and sharing everyone's favorite recipes. We want our cookbook to:
    • Display a list of all recipes.
    • Create new recipes and edit existing recipes.
    • Assign a recipe to a category (like "dessert" or "soup").

    As for the Perl frameworks you mention, I think the toughest will be Jifty. CGI::Application and Catalyst can be configured to use the same rendering engine and database abstraction layer, but I don't believe this is true for Jifty. I suspect that will make it rather difficult to create a meaningful benchmark.

    That being said, I don't expect wildly varying outcomes. What would interest me more is developer speed2. The most important reason to choose a framework is not for its performance, but for how fast you can get work done, and how much code you need to write yourself to accomplish your goals. In my opinion, if an application has performance problems, it's either because of obvious design flaws, or a lack of hardware resources. The former should be easily fixed by any reasonably experienced developer, and the latter is a simple matter of upgrading your machines3. The real issue is how fast you can add new features, and how much work it takes to refactor your code.

    I just realised while writing this up: you may not be after a comparison between these three Perl frameworks, but their performance relative to the three in the original article. In that case we'll need similar hardware and the same software versions for a fair comparison.

    Update: The ruby source code for the example is available from page 2 of the OnLAMP article, right below this short description. I should learn to read...

    1 Just kidding :)
    2Hat tip to dragonchild for pointing this out in #cgiapp.
    3Obviously this is a simplification
[reply]
      That's right, I don't really care which of the Perl frameworks is fastest, but I'd like to see that they are keeping up with other languages in terms of speed. I'd expect Perl to do very well in this type of comparison, since the Perl 5 implementation has been tuned a great deal over the years.

      It's not important to use the same hardware as the original test, as long as one can recreate at least one of the apps tested in that one as a reference.

[reply]
Re: an interesting web frameworks benchmark
by Thilosophy (Curate) on Jul 17, 2006 at 02:06 UTC
    I am wondering how important these benchmarks are. I am under the impression that response time of any reasonably-sized application is determined mostly by the time you need to run business logic (such as database queries). That part of the application is always custom code outside of whatever framework you use. If this assumption is correct, the raw performance of the framework is not too important. It becomes much more relevant how easy, fast, and flexible the framework is in supporting you in writing that business logic.

    On the other hand, the above statement applies more to traditional frameworks like CGI::Application, whereas "full-stack" frameworks such as Jifty or Rails also aim to provide the underpinnings for the business logic part (such as object-relational mappers), in the extreme case completely and exclusively controlling the database. If you go this way, it is probably quite essential to look at how this impacts on performance.

    The article states that although "we didn’t come up with complex queries", "when connecting rails to Oracle the performance dropped to the extent it made any production use of the product useless". Anyone has any idea what went wrong there?

[reply]

      On the other hand, the above statement applies more to traditional frameworks like CGI::Application, whereas "full-stack" frameworks such as Jifty or Rails also aim to provide the underpinnings for the business logic part (such as object-relational mappers)

      Exactly. These are tools that are trying to supply a lot of pre-built code to you, so it matters more how much overhead they add. You will, in theory, run quite a lot of Rails code on every request.

      The article states that although "we didn’t come up with complex queries", "when connecting rails to Oracle the performance dropped to the extent it made any production use of the product useless". Anyone has any idea what went wrong there?

      Yes. I have read that Ruby has weak Oracle support in it's database libraries. Most people are using PostgreSQL or MySQL, so it doesn't matter to them. Perl is lucky to have really great Oracle support. The DBD::Oracle module has great performance.

[reply]
Re: an interesting web frameworks benchmark
by rhesa (Vicar) on Jul 24, 2006 at 00:04 UTC
    Over the past week, I've been trying to setup a similar benchmark between the Ruby app and a CGI::Application version. I haven't been able to setup mod_ruby at all, so I've limited myself to comparing the plain cgi versions, and the mod_perl version. I'm afraid this doesn't do justice to Ruby, but since my wife and I are expecting our first child in the coming weeks, I haven't been as eager to persue this. I'd appreciate it if someone else could lend me a hand with the ruby setup.

    My CGI::Application version uses the following main components:

    I made it into a framework-like application by using the following plugins and modules:

    If anyone is interested, I will put my code online somewhere.

    Even though the numbers are not as informative as I'd like them to be -- due to my lack of Ruby skills -- I still want to show you the benchmark results I got sofar:

    Summary - 1 minute test with 15 concurrent users
      Ruby/CGIPerl/CGIPerl/mod_perl
    Transactions 70 212 1685
    Transaction rate 1.17 t/s3.53 t/s28.11 t/s
    Longest transaction13.53 s 10.40 s 0.19 s

    Summary - 3 minute test with 50 concurrent users
      Ruby/CGIPerl/CGIPerl/mod_perl
    Transactions 159 607 14245
    Transaction rate 0.88 t/s3.37 t/s79.23 t/s
    Longest transaction88.64 s 22.76 s 5.24 s

    Read more... Raw Siege results (4 kB)
[reply]
[d/l]

Back to Meditations