A Simpleton Benchmark of PSGI App Frameworks

Introduction

This papers discusses the topic of benchmarks in the context of PSGI application frameworks. It is not meant to incite violence over the results as they are to be taken with a heavy grain of salt. The reason being that even the slowest performing framework can serve up requests in approximately two milliseconds. Generally this amount of time contributes very little to the overall equation of web page response time, because other factors like IO and the network latency play a more dominant role.

Route Setup

I took five different frameworks and built a minimal application with each. I created a route /hola/${name} that would simply return “Hola $name” when requested. Here is how it looks in each framework:

Web::Simple

sub (GET + /hola/* ) {
    my ( $self, $name ) = @_;
    [ 200, [ 'Content-type', 'text/plain' ], ["Hola $name"] ];
},

Dancer

get '/hola/:name' => sub {
    return "Hola " . params->{name};
};

Mojolicious

get '/hola/:name' => sub {
    my $self = shift;
    $self->render( text => "Hola " . $self->param('name') );
};

Amon2

sub hola {
    my ( $class, $c ) = @_;
    my $output = "hola " . $c->req->param('name');
    return $c->create_response(
        200,
        ['Content-Type' => 'text/plain', 'Content-Length' => length($output)],
        $output,
    );
}

Note that in addition to defining the sub above, one has to map the route to this sub. Assuming this action is in the controller Root.pm then we define the dispatch map as:

connect '/hola'  => 'Root#hola';

Tatsumaki

sub get {
    my ($self, $name) = @_;
    $self->write("Hola $name");
}

Tatsumaki is similar to Amon in that we have to define the route separate from the function that handles it. Specifically one passes the route map to the constructor of a Tatsumaki app like so:

my $app = Tatsumaki::Application->new([
    '/hola/(\w+)' => 'HolaNameHandler',
]);

The Method

Approach

It is certainly not scientific, but I created the simplest application I could for each framework based on the documentation I found. One could optimize the be-jezus out of any benchmark, but that was not my intent. Rather I wanted to see what I got out of the box for performance with the default config against Starman.

Web Server Backend

For all of the frameworks except Tatsumaki I used the Starman backend. Applications were run in production in mode with 10 workers.

plackup -s Starman -E production --workers=10 -a app.pl

The Tatsumaki was run against the Twiggy backend since it’s an AnyEvent based framework.

Benchmark Tool

I used the Apache Benchmark tool ab in the following manner:

ab -n 1000 -c 10 http://$host:5000/hola/cabron

I did testing both directly on localhost and over a LAN. The results are the average of three runs for each framework with the requests being done over a LAN¹

The Results

Follow Ups

[dated: Feb. 14, 2011]
After releasing this article I received some feedback worth mentioning.

Mojolicious Speedups

It was pointed out to me (thanks sri) that Mojolicious compares better using the following mini-app:

use Mojo::Server::PSGI;
use FindBin;
my $psgi = Mojo::Server::PSGI->new(
  on_handler => sub {
    my ($self, $tx) = @_;
    my $res = $tx->res;
    $res->code(200);
    $res->body('Hola ' . $tx->req->url->path->parts->[-1]);
    $tx->resume;
  }
);
return sub { $psgi->run(@_) };

This did indeed crank up the rps to 1139

Tatsumaki using Starman

miyagawa graciously pointed out that I could run Tatsumaki against the Starman backend since I wasn’t doing any IO (which would benefit from an AnyEvent library when using the Twiggy backend.)

Upon doing so, it turns out that Tatsumaki using Starman is the fastest of all coming in at 2751 rps.

Conclusion

I’ll let you drawn your own.

Framework Versions Used

For reference here is a table of the versions of each framework that I used in testing:

Framework	Version
Mojolicious	1.01
Dancer	1.3001
Amon2	2.14
Web::Simple	0.005
Tatsumaki	0.1011

Footnotes

¹ Thanks to chansen for suggesting to separate client and server workloads.