In response to Wil’s comment regarding the PHP framework performance comparisons I made in my previous post, I have decided to post the results I got.
My decision not post them initially was due to the benchmarks not being done in complete isolation (a seperate client and server machine) but the scores relative to each should still be accurate and that’s what I am testing for.
Furthermore, while I only used requests/sec as a performance yardstick, I do realize that other factors do affect a web application’s response in the real world.
The Tools
The tests were run on a 1.8Ghz AMD sempron computer with 512Mb RAM running Ubuntu Gutsy (7.10) Desktop.
I used httperf to run the tests although I did run some of them again using apachebench and the results were consistent.
I created the same front page with CakePHP, CodeIgniter, Zend Framework and Ruby on Rails. I also duplicated the same functionality in HTML and in PHP using procedural code (aka. spaghetti code) to act as baselines.
I decided to include data access in the tests and the tests involve the application fetching four rows from the database and rendering the results.
The test files are also available for download for those interested in them. The sql file for the mysql database is also included. The code assumes a database root user with no password.
The Results
All scores are requests per second. Higher is better.
- No PHP code cache
All frameworks used an ORM (of sorts, in the case of codeigniter). The Zend Framework used Zend_DB_Table and CodeIgniter used ActiveRecord.Run 1
Run 2
Run 3
Run 4
Average
Baseline HTML 1327.5
1326.5
1328.6
1329.1
1327.9
Baseline PHP 331.6
332.1
331.4
332.0
331.8
CakePHP 3.6
3.7
3.8
3.5
3.7
CodeIgniter 21.5
21.2
21.7
21.7
21.5
Zend Framework 9.3
9.1
9.2
9.3
9.2
- With eAccelerator PHP code cache
All frameworks used an ORM (of sorts, in the case of CodeIgniter). The Zend Framework used Zend_DB_Table and CodeIgniter used ActiveRecord.Run 1
Run 2
Run 3
Run 4
Average
CakePHP 8.0
8.0
8.0
7.2
7.8
CodeIgniter 98.2
98.1
98.3
98.3
98.2
Zend Framework 33.2
33.3
33.5
33.6
33.4
- With APC PHP code cache
All frameworks used an ORM (of sorts, in the case of CodeIgniter). The Zend Framework used Zend_DB_Table and CodeIgniter used ActiveRecord.Run 1
Run 2
Run 3
Run 4
Average
CakePHP 7.3
7.3
7.3
7.3
7.3
CodeIgniter 97.5
98.0
96.6
98.3
97.6
Zend Framework 32.8
33.3
31.8
32.7
32.7
- With APC PHP code cache, No ORM
The Zend Framework used Zend_DB and I disabled ActiveRecord in CodeIgniter. I couldn’t figure out how to disable the ORM in cakePHP so I left it out.Run 1
Run 2
Run 3
Run 4
Average
CodeIgniter 106.4
105.6
106.3
106.6
106.2
Zend Framework 42.6
42.8
42.9
43.0
42.8
- With APC PHP code cache, No database calls
This tests the impact of the database call on the overall performance. I sent an empty result set to the view directly from the controller, thus bypassing the model.Run 1
Run 2
Run 3
Run 4
Average
CodeIgniter 118.1
118.3
117.3
118.2
118.0
Zend Framework 51.9
52.1
52.1
52.2
52.0
- Ruby on Rails comparison
I added this test to see how the PHP frameworks stcked up against Ruby on Rails. I also used the chance to try out Passenger (mod_rails) and Ruby Enterprise.Run 1
Run 2
Run 3
Run 4
Average
ROR with 1 Mongrel 88.1
85.1
84.9
84.8
85.7
ROR with Passenger 85.2
97.3
86.3
84.2
88.2
ROR with Passenger and Ruby Enterprise 89.0
99.2
98.9
98.6
96.4
Conclusion
CodeIgniter is over twice the speed of the Zend framework in all cases and CakePHP is a lot slower than the other two PHP frameworks. I do admit that considering my experience with CodeIgniter, I might have inadvertently set it up optimally without doing the same to the other two frameworks. I have posted the files and I welcome comments from “the experts”.
Update
We have decided to use the Zend framework so obviously outright performance is not the only factor in the choice of framework. The results we are currently getting are fast enough for us and in our existing application, the database is the bottleneck not the PHP code.
I have also done a few more tests on a production-grade dual-core server with 2Gb of RAM running Centos 5. This time, all benchmarks were from a seperate client computer on the same network. I used the exact same files as the previous results except for CakePHP where I used the recently released RC2. The results follow.
- No PHP code cache
All frameworks used an ORM (of sorts, in the case of codeigniter). The Zend Framework used Zend_DB_Table and CodeIgniter used ActiveRecord.
Run 1
Run 2
Run 3
Run 4
Average
Baseline HTML 3431.2
3311.8
3427.7
3395.0
3391.4
Baseline PHP 1912.1
1932.3
1983.3
1911.3
1934.7
CakePHP 15.6
15.6
15.6
15.6
15.6
CodeIgniter 83.5
83.0
82.0
83.2
82.9
Zend Framework 34.7
34.6
34.6
34.6
34.6
- With eAccelerator PHP code cache
All frameworks used an ORM (of sorts, in the case of CodeIgniter). The Zend Framework used Zend_DB_Table and CodeIgniter used ActiveRecord.
Run 1
Run 2
Run 3
Run 4
Average
CakePHP 36.0
36.1
36.1
36.2
36.1
CodeIgniter 383.3
377.9
371.8
385.2
379.5
Zend Framework 129.2
128.5
129.0
128.9
128.9
- With the Zend Platform
All frameworks used an ORM (of sorts, in the case of CodeIgniter). The Zend Framework used Zend_DB_Table and CodeIgniter used ActiveRecord.
Run 1
Run 2
Run 3
Run 4
Average
CakePHP 24.7
24.8
24.9
24.8
24.8
CodeIgniter 255.0
256.6
254.1
253.8
254.9
Zend Framework 83.5
84.4
83.8
83.4
83.8
Related posts:
Pingback: PHP Coding School » Blog Archive » php code [2008-06-30 08:28:31]
Was there any caching on the ROR tests?
@Brad,
No, I didn’t use caching on any of the frameworks.
That said, Rails was run in production mode which caches the framework code.
Thanks for benchmarking the frameworks!
What were the specific versions of each framework. I’m kinda of scared since I’ve been tooling around with Cake!
@Manny,
The framework versions I used are:
– CakePHP: 1.2.0.7125 rc1
– CodeIgniter: 1.6.3
– Zend Framework: 1.5.2
– Ruby on Rails: 2.1.0
However, you don’t need to have reservations about using CakePHP. It is a good framework if your project is a good fit and you can spare the time to learn it.
It is probably the framework with the shortest development time (if you know how to use it) of all the PHP frameworks I tested because a lot is assumed and so you write less code to get stuff done.
Also, in a production environment using caching is recommended. I didn’t use that in this test but caching will dramatically speed up the application.
Finally, the specs for the computer I used are wimpy (an old PC I use to make sure my code works in a linux environment before uploading – I use Windows for development). I’m sure the scores from a typical production server will be better.
Yes, CakePHP will still be slower but unless you are working on a very high traffic site (which is the case with the project we are about to start) you will be hard pressed to tell the difference.
@Ekerete,
Thanks for the version info.
I’ve downloaded the code and will review Cake stuff!
Hi Ekeret,
Thanks for doing these. I am wondering if you would mind running it again with RC2 which was released a few days ago. We are beginning to optimize the 1.2 series, so you should see a nice change.
Looking forward to updates.
Pingback: Which PHP framework is slower? « PHP::Impact ( [str Blog] )
Now this is more like it!
Great tests, full transparency, multiple environments, with layouts even! I’m surprised that it came out so close to your original assertion, but your methods seem bulletproof. The only better request/sec comparison that I can think of is one against a real world app, which is highly unlikely considering you’d have to implement it in 4 different frameworks.
Do you mind if I use your test apps for some internal testing? These would really help us in the performance initiative I mentioned before. I hope the other framework contributors get a chance to see these results; they’re certainly making me take a closer look at our performance characteristics.
Thanks for taking us to task and publishing all the evidence!
Sorry, I kinda skipped over the first paragraph to get to all the meaty results. I have to say you seriously underplayed your tests in the first post; now I know where you got the ‘twice as fast’ from. Now, why in the world would you keep such great tests to yourself?
,Wil
ave you considered looking at Kohana, the open-source spin-off to CodeIgnitor. I’d be interested as to how it performs as it’s PHP5 OO only
Pingback: michaelwales.com » CodeIgniter vs. Zend Framework
@gwoo,
Didn’t expect this much interest when I posted it else would have done it better. I intend to redo the tests on a setup typical of an actual production server – think 64 bits and at least 2Gb of RAM (need to do it for work anyway as we need to also test out the frameworks using the Zend Platform). I will definitely include the new version of CakePHP then and post the updated results.
@Wil,
I’m happy you’re satisfied and of course, you can use the test apps. I promise not keep the goodies all to myself next time.
I would have loved to see symfony right here. Nice work, however.
@Howie,
The frameworks I benchmarked were those we considered for our next application.
As much as we would like to have looked at a lot more of them, we obviously have to balance that with the time constraint.
I think Prado framework (http://www.pradosoft.com) is the best php framework.
Can you compare Prado with others frameworks ?
Pingback: CodeIgniter is the fastest PHP framework | am i works?
What about application versions? LAMP Rails etc.
Pingback: Web 2.0 Announcer
cool!!!
thanks…
codeigniter is pretty much the micro framework here..
Been benching CakePHP 1.2 (rc1, rc2) a lot lately for a application about to launch at work. the most i can sqeeze out of rc2 at the moment is around 80-100 request/second.. Regression in the caching subsystem affects version rc1, i got 2 requests/second.. Now its fixed.
that being said, try Rails 2.1 with 3 instance of thin or ebb http daemon, you won’t regret it. I got a few hundred request/s for a hello world.
Comparing PHP and Ruby are two different beast, PHP are stateless where code needs to be interpreted and executed by the Zend Engine every request, Where in ruby classes are just objects, loaded once..
“File IO effects PHP” vs “Memory effects rails”
That is from a fulltime rails and cake hacker..
If you want screamer frameworks why don’t you include django and merb..
they are both know being faster than any of these above..
The end of the day, you need to ask what you want to get out of framework,
if you want elegance, some thing build big application go rails..
if you want to try ruby but need speed and stay slim go Merb + ActiveRecord,
if want stay PHP but the agile way go Cake
if you want AppEngine and speed go Django..
otherwise if u happy about the minimal CI, stay with it..
Zend as a component library, its awesome .. being a agile development framework = aweful (where is code generation?)
Throw CoughPHP in there — supposed to be the fastest of the ORMs.
It would be interesting to see what happens with caching and the Zend encoding engine.
Have you ever considered doing other tests on the frameworks with different accelerators?
I would say if you want portability (run everywhere) + google app engine + speed run web2py.
If you’re going to include other Ruby frameworks, you should include Ramaze.
Pingback: Benchmarks entre os mais famosos framework PHP | Cairo’sBlog
Pingback: PHP framework ??? « Oceanic | ????
Pingback: Comparación entre frameworks de PHP | eleZeta - Lucas Zallio
Symfony would be a great contender to see in this benchmark.
If you’re interested in minimalistic yet comfortable support library for PHP have a look at http://itools.search.ch
Quick size comparison of code needed (lines & chars):
itools 28 589
php 61 1443
html 64 1380
Quick comparison of req/s (with eAccelerator on my machine):
html 1157.3 (10 times faster, not 3 times like in article!?)
php 117.4
itools 62.4 (still half the speed of raw PHP)
The full (yes, full, no setup code necessary) source code can be found at http://pastebin.com/f2b030383
http://www.alrond.com/en/2007/feb/04/in-addition-to-the-test-of-mvc-frameworks/
Django’s the fastest and has about the nicest orm interface.
” I do admit that considering my experience with CodeIgniter, I might have inadvertently set it up optimally without doing the same to the other two frameworks.” This guys doesn’t know a thing, and you’d do well to take his results with a salt shaker.
@Max: Man, you are everywhere! web2py isn’t good enough to warrant your spam on every blog that mentions web technologies, dude.
http://static.alrond.com/results_2test.gif
Wow, I can’t believe Cake and Zend are so slow!
I’d like to see some ASP.NET and Python benches up there, as well as RoR and Merb with/without ActiveRecord or Datamapper. Good info though.
If you decide to update these, I hope you’ll add in Qcodo (http://www.qcodo.com) next time.
Pingback: PHP framework comparison benchmarks | AVNet Labs | Thats The New Thing!
You’ve really stirred something up here, Ekerete.
I’ve been seeing this reposted all over the PHP blogs this morning.
I can’t reasonably convince anyone that I’m unbiased, but I hope everyone reading this and drawing conclusions will consider the following.
This doesn’t mean that ZF or CakePHP are slow or are necessarily bad choices. If it did, then it appears we’ve all made a big mistake by using a framework in the first place; baseline PHP is *15 times* faster than the fastest of frameworks in this comparison. But I hope we all see the flaw in this conclusion.
Also, as thorough as Ekerete has documented his methods, this is actually a very small performance test that I would guess covers less than 10% of the code, albeit very critical code- in each framework. The application uses a few features in each framework to give an indication of performance, but it is also trivial and can only represent the performance of a real world application in a very rough way. Ekerete can test particular features that he will be using to get an idea of the performance characteristics that are important to him.
Also, performance is just one aspect of a framework. Give me a day and I’ll deliver an MVC framework that outperforms Code Igniter many times over. It won’t have any features, but it will be fast.
The fact is, requests/sec is a factor that many people do not consider at all in choosing a framework because they will never max out the computing resources they have available to them, and those for whom it does matter are talking serious business and should create tests as Ekerete has done to check the performance of the features that they will be using in their own apps.
What I see this as is a great beginning to a discussion about performance across the frameworks. Keep in mind that gauging performance accurately is *very* hard, and the best we can do is understand it in more detail and test it against our own, very different, application environments. Ekerete’s assertion that ‘Code Igniter is twice as fast as ZF’ is still an extremely broad generalization even with these tests to back it up; I would wager from the sound methods he uses here that he knows enough about performance testing to agree with me on this.
Pingback: links for 2008-07-02 « that dismal science
Also interesting in these sort of comparisons is a quick breakdown on what features each contender has that are relevant to the tests.
For example, if framework A is a slower with its ORM, but it has 200 useful features that framework B has, then it might be just fine and understandable that framework A performs fewer queries per second.
Its a little like comparing apples to oranges otherwise.
It would be interesting to see these benchmarks also include the Kohana PHP framework (which was derived from CodeIgniter). The SVN version has a very interesting cache method that makes for impressive performance.
@Wil,
I do agree with you there and considering we have ended up choosing Zend over CodeIgniter, the powers that be also agree.
I have run the same tests on a more production-like setup and updated the post. In this case all the results are definitely respectable and should not be an issue in the framework choice. And this is without caching enabled.
Wil – great points all round. Such benchmarks should be taken with a large pinch of salt as guides for your own needs. If framework X is your toolset of choice you may very well be able to get better performance results (on top of functionality) going with your tool of choice. However if this discussion prompts people to try out a new box of tricks and they learn something new then that’s all good. Be warned – if you only know how to hammer then all you’ll see is nails….
Hi,
I’m actually developing a new PHP Framework, and was wondering if their was any way to get it benchmarked… I’m betting it’s close to or faster than CodeIgniter
I would be intrested in symfony -> http://www.symfony-project.org/ being tested and akelos -> http://www.akelos.org/
Pingback: roScripts - Webmaster resources and websites
Pingback: PHP Framework Benchmark | romantika.name
Pingback: daily links for Jul 02 | Jackson Miller
Pingback: Benchmarking PHP frameworks
Pingback: Winn.ws » PHP framework comparison benchmarks
The problem with zend (I’ve never used cakephp), is that they got got everything wrong. It’s trying to re-invent the well, ezComponents developed the same components long time ago. Second, they are trying to do backend stuff, like web services, and for that, you don’t use zend framework, you use java. No big comp nay will every ever consider to consume or expose web services with zend, because like this benchmark shows, it’s slow.
Pingback: links for 2008-07-03 | JeremiahTolbert.com
Ekerete,
Thanks for providing such a nice performance comparison. I use Grails in my work, so I ported your web app to Grails and tested it as well. The full analysis is at http://www.siwiz.com/publications.html
The short of it is that Grails 1.0.3 would probably serve about 85 requests / second on your test machine. 85 requests / second places Grails behind the PHP environments with caching solutions but ahead of those without.
Regards,
Stephen
Ekerete & Stephen Kuenzli,
Great contributions, thanks!
We are using Symfony for all our front-end web development. CakePHP and CI, unfortunately, do not support PHP5. Zend’s API is not very consistent and very difficult to learn. There’s a lot of functionality missing, you end up coding your own file uploader and image resizer, just to mention a few, and that slows down the development process.
Zend, PEAR and ezComponents are not full-stack frameworks, like Symfony or Rails, they are libraries.
Anyway, it’s always good to know what the realistic top limit is.
Many thanks,
Tibor
Pingback: daily links for Jul 03 | Jackson Miller
Pingback: :: Kohaci Com :: » CI tetap lebih cepat
@Tibor: Please learn and then write. CakePHP and CI do support PHP5 and PHP4, unlike Symfony that supports only PHP5.
@Eugene
Sorry, correction: CakePHP and CI support a deprecated version of the language, reason why I don’t use them. And Zend because it’s slow.
Pingback: Comparación de Frameworks PHP
Pingback: daily links for Jul 04 | Jackson Miller
Pingback: daily links for Jul 05 | Jackson Miller
Pingback: Zend_Form Performance Issues « PHP::Impact ( [str Blog] )
Pingback: daily links for Jul 06 | Jackson Miller
I converted the CI benchmark app over to cognifty, and cognifty is about 10% faster than CI.
I get around 266 req/s in CI with apc cache on and 292 req/s with cognifty. Pretty neck and neck.
System is dual core AMD 2Ghz with 1 Gig of RAM.
I used “ab -n 100 -c 2″
Can we get the httperf and apache bench command line settings used to generate these?
Pingback: ?????????????? « eweibookmark
Pingback: daily links for Jul 07 | Jackson Miller
Thanks for benchmarking the frameworks. Just the article I was looking for on the net. Definately makes my decision about the framework more easier.
can you include kohana framework
cakephp sucks
Pingback: daily links for Jul 08 | Jackson Miller
Pingback: PHP framework comparison benchmarks | Hang Xach Tay
Pingback: daily links for Jul 09 | Jackson Miller
@Tibor: don’t know about CI, but you are wrong about cakephp. Go check yourself! They have separation for version 4 and for version 5 (model_php4 and model_php5 for example).
And this is a good benefit if you write scripts for sale. At least a year ago there were much more people with php4 hosting.
Pingback: daily links for Jul 10 | Jackson Miller
Pingback: ArsGeek » Benchmarking PHP frameworks
sounds like big deal. great work!
Kudos guys nice comparison.
Although the test does not include Symfony, a major player in PHP frameworks.
It would be great to run the test using Symfony so we can have a complete perspective of what is going on.
Thanks
Pingback: Club Troppo » A brief note on a statistic.
Ha, FrameWorks need pure CPU power!!!
The result aof static html is tremendous and I like your conclusion about the FrameWorks comparison: mysql is the bottleneck!
Pingback: Zend Framework Architecture « PHP::Impact ( [str Blog] )
Somebody said CI does not support PHP5… hahaha! where did you read this?? I’ve just developed my second commercial app in CI and it works on PHP5 (the first one also works with Oracle db, btw:)
Pingback: PHP framework comparison benchmarks « llbbl
I’ve started delving into CodeIgniter, and it is definitely better than CakePHP and Zend (having tried both). I definitely recommend CodeIgniter to people considering a framework.
You might also want to check out PAL, a framework I developed in the past few months, to be faster than CodeIgniter and more powerful (though not as powerful as Zend).
Hopefully you’ll find it a pleasure to use. It does stuff including ORM easily for you and still clocks in at under codeigniter.
http://pal-framework.googlecode.com
why use a framework?
Pingback: Best PHP Framework / PHP Frameworks Comparison | Mahmoud M. Abdel-Fattah
Pingback: Paul M. Jones » Blog Archive » Labor Day Benchmarks
I see the inflation of benchmarkings of PHP frameworks and framework pro and cons war. In my humble opinion software speed in general will rarely be the reason for one’s product or app failure. Also these number become irrelevant in the context of real application that fee the mouth. So no one should take the speed in benchmarking as the baseline for deciding what framework to use or not.
Pingback: Codeigniter : Un lugar en el mundo…
You all make me sick with your performances benchmark. What’s important is first productivity, maintenability, security and THEN performances.
Sick.
http://www.web2py.com Web2Py – Python Framework is the newest kid on the block for Python frameworks. It has a lot of features that simply are not there in other frameworks. Even Ruby!. You can design database models graphically online. The templating language is pure python and there are no problems with indenting. The models are auto-migrating which allows easy updating of your database schema’s. Parent-child and Super-Sub Set type models are completely supported. Uses an advanced MVC pattern. At least 10 free applications ready to download and use in your own developments. Can be run from a USB stick. ZERO installation just click on the exe and away you go with web server and complete graphical admin interface. PLUS it will work on Google App Engine (your still constrained by the Google Storage Limitations but you don’t have to change any database model code at all). AND Complete International Support (with a graphical admin interface to add different languages)…there is more but just take a visit…
To check out a comparison read this one:
http://mdp.cti.depaul.edu/examples/static/web2py_vs_others.pdf
@Mala,
you have no idea what you are talking about.
I meant @Chala
I second Vic’s comment… Chala has nary a friggin’ clue.
Pingback: {|ihower.idv.tw| blog } | MerbCamp 2008
Pingback: PHP???????? | ????@NET
Pingback: Recent Links Tagged With "php" - JabberTags
Pingback: Ant » Blog Archive » Web Framework ??: PHP vs Ruby
i am so pleased to see this great performance test between frameworks..
when i begun with developing by frameworks i used first Cakephp, but i feel its slowing in processing and rendering data. then i moved to CodeIgniter which i’m using right now to develop php projects. Ruby On Rails…oh God!!.. the best ever the fastest also!!!
Finally, I’m using CodeIgniter and ROR for my projects:)
Good job Ekerete!
you just killed me with this news , i liked cakePhp , and worked with it along.
now you come to tell me that it’s the slower one
,
and inside me i think that , should i learn CodeIgniter ???
again what is missed here (php frameWorks) is a simple framework without any additions just Model , Controller and a view connection with each other (OOP Baseline PHP)
and let us handle the rest .
thanks man .
Pingback: [zz]PHP framework comparison benchmarks « Xujia’s Weblog
Pingback: Web Development Essentials - Frameworks | JustSearch Labs