Benchmarking Frameworks

[hProj]

Hello everybody,

 

First of all, I'm new to PHP Frameworks, let alone PHP itself. I've been researching and comparing several PHP frameworks and came to a final comparison between Symfony 2 and Zend Framework 2, which seemed best for the context of my assignment (where performance is of the essence). Now I've been reading a lot of negative acticles and blogs about Zend Framework 2. It is supposed to be faster, but tests show the opposite:

 

http://www.enrise.co...-2-performance/

http://www.piprime.f...web-frameworks/

http://www.ruilog.co...f0e42cf705.html

 

So I started testing them myself. This came out:

 

 

Dafuq. And yet again with 2 requests and 1 concurrent call:

 

 

Zend Framework 2: ZF2, ZendSkeletonApplication and Doctrine 2 implementation

Zend Framework 1: ZF1, existing fully-operational application, with various extensions.

Symfony 2: Symfony 2 with Doctrine 2 implementation.

 

It's always responding after 5 seconds or more... Now I'm also new at benchmarking, but isn't this supposed to be around the same as loading the page in a browser? Because it seems to load in less then a minute on windows. The pages I'm benchmarking select a table at the size of 500 rows. But even if I simply make them 'hello world' pages, ZF2 still uses 5s+. So it can't be the ORM.

So I tried to find the source with xhprof, but that obviously gave me a regular response time, about less then a minute.

 

I have no idea what is causing my benchmarks to give this weird result, and I was kind of hoping anybody could help me with this matter. Does anybody have any idea what could've caused these results? It seems to me it's caused by some configuration fault, but I have no clue where to start.

 

Thanks in advance...

[trq]

ZF1 was slow, and ZF2 is no better.

 

I have executed benchmarks with similar results: http://thorpesystems.com/2012/09/benchmarking-proem-vs-symfony2-vs-zf2/

 

I think one of ZF2's biggest issues is simply the complexity which causes a lot of function calls: http://thorpesystems.com/wp-content/uploads/2012/09/number-of-function-calls3.png, this in turn causes high memory use: http://thorpesystems.com/wp-content/uploads/2012/09/memory-usage3.png

 

In the end however, these types of benchmarks should be weighed up against functionality and your real needs. Most sites won't likely hit an issue running an application built well with ZF2. It's always good to know what the baseline is, and from my experience benchmarking and optimising my own framework Proem, I know I wouldn't be using ZF2 for anything myself.

[hProj]

Indeed. The negative results just keep summing up, wherever I look. In the end Symfony just seems the best option for heavy-weighted and high performing applications. Although my testresults are not quite accurate and trustworthy, I've seen enough to conclude my decision.

 

In further comparison I used xhprof to profile the calls, and those concluded in 17k calls for Symfony and 35k for ZF2, both using Doctrine. That's somewhat more as in your results x)

 

With your permission, I'd like to use your results in my documentation, since they're the most recent ones I have seen so far.

 

Thanks a lot!

[trq]

Use the results for whatever you like. The code is also available at https://github.com/trq/phpBenches (requires php5.4 - and has little to no documentation)

[Dracony]

Here are some benchmarks I did of different frameworks when testing my PHPixie:

PHP Framework Benchmark

 

IT seems that if your server is configured properly, most of mainstream frameworks perform roughly the same, Zend being the slowest of course

Edited January 3, 2013 by Dracony

[hProj]

For those still looking for an answer on the subject, here is your answer:

 

http://zend-framewor...84p4657525.html

 

I have not attempted another benchmark yet, and I probably won't. I'm currently using Symfony and I'm pretty fond of it, it's rather late to change my mind

Edited January 9, 2013 by hProj

[paulinetaylor85]

All the three framework are good, but I would prefer Symphony 2 as it is one of the rapidly used frameworks in the world.

[Hall of Famer]

One problem with these benchmark articles is that a lot of them(cant say all, but definitely most) use simple script like 'hello world' or a small application to test the performance. Sure it is easy and less time-consuming to write performance tests, but this gives misleading results as large frameworks tend to be way too slow. In reality larger applications the difference becomes less and less significant, and using complex frameworks turns out to be more beneficial the bigger your application becomes. Truth be told, if your script only displays a sentence of hello world on the screen, you are better off just using HTML. Frameworks are for sites/applications that expect to grow large in future, they are not for extremely small fansites that do not even need OOP to survive.

Edited April 16, 2013 by Hall of Famer

[trq]

And where are your benchmarks to back up your opinion Hall of Famer?

 

You can't seriously expect anyone here to take your word for it can you?

[Hall of Famer]

Well I am not saying large applications will actually run faster with frameworks, there is always an overhead cost for frameworks especially ORM. I am merely suggesting that the difference in performance is exaggerated since most do not really write a complete object-oriented software to compare the speed. Large applications with proper optimization will still run slower with frameworks like Zend, but wont be 4-6 times slower. You can say its slow, but its not notoriously slow.

[trq]

Where is your proof though? Or are you just making this stuff up?

[Hall of Famer]

Where is your proof though? Or are you just making this stuff up?

 

So what are you trying to say? Pro-framework or pro-nonframework? I am kinda confused.

[trq]

It's a pretty simple question. "Where is the proof for your argument?"

 

You reply to a post stating that there are problems with these benchmarks, you then go on to explain your theory about why you think they are floored.

 

Where is your proof? Or is this just another theory?

[jsmall]

For the record, the graphs at the top of this thread were most likely due to an apparent bug in ApacheBench, not ZF2.

 

The links to http://thorpesystems.com do not resolve, as the whole website seems to be down.

 

ZF2 has long since been patched to avoid this AB bug. See this PR from 6 months ago https://github.com/zendframework/zf2/pull/2889

[trq]

The links to http://thorpesystems.com do not resolve, as the whole website seems to be down.

My blog is back online. Server had completely halted without me being aware. I really got to move that thing one day.

[Hall of Famer]

http://www.thedeveloperday.com/framework-battles-yii-vs-zend-framework/

 

Well heres an interesting article if you want to take a look. Just as I stated before, frameworks like Zend do tend to be slower, but the severity of such performance issues is overexaggerated. I have nothing against Yii, and in fact I quite like it. However, doing benchmark evaluation based on 'Hello World' is never a good idea. In fact, if all your script does is to output 'Hello World' on the screen, you are better off just using plain HTML. Why bother with server-side programming languages at all in this case? XD

 

 

Here is another benchmarking article I found, also with flaws:

http://www.techempower.com/blog/2013/03/28/framework-benchmarks/

 

Sure it seems to be using more realistic code this time, but because it is evaluating frameworks among many different platforms, the coding style becomes an issue here. You cant write java-like code with programming languages like Python to claim that the latter is much slower. Not saying it is not slower, but definitely not that much slower. With PHP, Cake is one of the slowest frameworks available, why not use Yii or Codeigniter instead? If I were to choose the slowest framework for java servlet and node.js, and compare them with the fastest PHP frameworks, the difference will be much smaller or even invisible.

[trq]

Neither of those links prove your point at all.

[Hall of Famer]

Again I was just trying to prove that many of the benchmarking test is overexaggering the performance issue in large frameworks like Zend. I never said Zend would be faster than other frameworks like Symfony or Yii, just not that much slower. Nor was I referring to the benchmarking test the OP brought up, its for all benchmarking artices I've seen in general.

 

But anyway if you cant be convinced or refuse to be convinced, I am fine with it. All I have to say then is that I respect your opinion, but I have to accept our differences and stop here. I do not live my life trying to please everyone, especially when there's no actual benefit for me to even do this.

Edited April 25, 2013 by Hall of Famer

[trq]

Again I was just trying to prove that many of the benchmarking test is overexaggering the performance issue in large frameworks like Zend.

I am aware of what your trying to "prove". Unfortunately though, your opinion doesn't mean proof. Do you understand? You haven't provided "any" proof, just opinion. Opinion (and yours in particular) carries very little weight.

[448191]

The huge gap between ZF1 and S2 in the OP is more or less in concord with my own experiences. I wasn't to impressed with S1 but S2 ticks a lot of boxes, performance isn't at the top of our list of priorities but S2 in production mode is as fast as any full featured framework can be. We are migrating from ZF1 to S2 and the interface is noticeably more responsive on the same server. Of course you *could* make ZF equally performant with a good chunk of custom development but SF2 makes it effortless.

 

But I feel that's a valid argument: ZF provides a less restrictive infrastructure, S2 pre-chews more for you but Potencier made performance a key concern from the first lines of code so it's not surprising it's more performant. It does introduce some inflexibilities, but having worked with it for a couple of months now I can honestly say the extensibility provided is sufficient and the infrastructure is so much more mature than the ZF MVC component, there's really no comparison.

[annaharris]

I am not much impressed with ZF1 and ZF2. So, It Will be very good if you can test the Fuelphp framework.

[TMN70]

I am aware of what your trying to "prove". Unfortunately though, your opinion doesn't mean proof. Do you understand? You haven't provided "any" proof, just opinion. Opinion (and yours in particular) carries very little weight.

Isn't what Hall of Famer said just plain common sense? Every singe framework has some almost constant overhead for any request. What you really want to know is how the framework handles real life situations, or rather how it would handle your particular application. It should be familiar from mathematics as Ax^B + C. C would be the overhead, and A and B would define how it really scales. For small stuff like "Hello World", you just want 'C' to be as small as possible. For real situations you should be more concerned about 'A' and especially 'B', and would probably design your framework to minimize these with tho probable result that 'C' will grow a bit. A suit of benchmarks should contain additional tests to try to find out how the framework scales, not just the size of the overhead.