Jump to content

Benchmarking Frameworks


hProj

Recommended Posts

Hello everybody,

 

First of all, I'm new to PHP Frameworks, let alone PHP itself. I've been researching and comparing several PHP frameworks and came to a final comparison between Symfony 2 and Zend Framework 2, which seemed best for the context of my assignment (where performance is of the essence). Now I've been reading a lot of negative acticles and blogs about Zend Framework 2. It is supposed to be faster, but tests show the opposite:

 

http://www.enrise.co...-2-performance/

http://www.piprime.f...web-frameworks/

http://www.ruilog.co...f0e42cf705.html

 

So I started testing them myself. This came out:

 

s28dft.png

 

Dafuq. And yet again with 2 requests and 1 concurrent call:

 

68dcmt.png

 

Zend Framework 2: ZF2, ZendSkeletonApplication and Doctrine 2 implementation

Zend Framework 1: ZF1, existing fully-operational application, with various extensions.

Symfony 2: Symfony 2 with Doctrine 2 implementation.

 

It's always responding after 5 seconds or more... Now I'm also new at benchmarking, but isn't this supposed to be around the same as loading the page in a browser? Because it seems to load in less then a minute on windows. The pages I'm benchmarking select a table at the size of 500 rows. But even if I simply make them 'hello world' pages, ZF2 still uses 5s+. So it can't be the ORM.

So I tried to find the source with xhprof, but that obviously gave me a regular response time, about less then a minute.

 

I have no idea what is causing my benchmarks to give this weird result, and I was kind of hoping anybody could help me with this matter. Does anybody have any idea what could've caused these results? It seems to me it's caused by some configuration fault, but I have no clue where to start.

 

Thanks in advance...

Link to comment
Share on other sites

ZF1 was slow, and ZF2 is no better.

 

I have executed benchmarks with similar results: http://thorpesystems.com/2012/09/benchmarking-proem-vs-symfony2-vs-zf2/

 

I think one of ZF2's biggest issues is simply the complexity which causes a lot of function calls: http://thorpesystems.com/wp-content/uploads/2012/09/number-of-function-calls3.png, this in turn causes high memory use: http://thorpesystems.com/wp-content/uploads/2012/09/memory-usage3.png

 

In the end however, these types of benchmarks should be weighed up against functionality and your real needs. Most sites won't likely hit an issue running an application built well with ZF2. It's always good to know what the baseline is, and from my experience benchmarking and optimising my own framework Proem, I know I wouldn't be using ZF2 for anything myself.

Link to comment
Share on other sites

Indeed. The negative results just keep summing up, wherever I look. In the end Symfony just seems the best option for heavy-weighted and high performing applications. Although my testresults are not quite accurate and trustworthy, I've seen enough to conclude my decision.

 

In further comparison I used xhprof to profile the calls, and those concluded in 17k calls for Symfony and 35k for ZF2, both using Doctrine. That's somewhat more as in your results x)

 

With your permission, I'd like to use your results in my documentation, since they're the most recent ones I have seen so far.

 

Thanks a lot!

Link to comment
Share on other sites

  • 3 months later...
  • 1 month later...
  • 1 month later...

One problem with these benchmark articles is that a lot of them(cant say all, but definitely most) use simple script like 'hello world' or a small application to test the performance. Sure it is easy and less time-consuming to write performance tests, but this gives misleading results as large frameworks tend to be way too slow. In reality larger applications the difference becomes less and less significant, and using complex frameworks turns out to be more beneficial the bigger your application becomes. Truth be told, if your script only displays a sentence of hello world on the screen, you are better off just using HTML. Frameworks are for sites/applications that expect to grow large in future, they are not for extremely small fansites that do not even need OOP to survive.

Edited by Hall of Famer
Link to comment
Share on other sites

Well I am not saying large applications will actually run faster with frameworks, there is always an overhead cost for frameworks especially ORM. I am merely suggesting that the difference in performance is exaggerated since most do not really write a complete object-oriented software to compare the speed. Large applications with proper optimization will still run slower with frameworks like Zend, but wont be 4-6 times slower. You can say its slow, but its not notoriously slow.

Link to comment
Share on other sites

It's a pretty simple question. "Where is the proof for your argument?"

 

You reply to a post stating that there are problems with these benchmarks, you then go on to explain your theory about why you think they are floored.

 

Where is your proof? Or is this just another theory?

Link to comment
Share on other sites

http://www.thedeveloperday.com/framework-battles-yii-vs-zend-framework/

 

Well heres an interesting article if you want to take a look. Just as I stated before, frameworks like Zend do tend to be slower, but the severity of such performance issues is overexaggerated. I have nothing against Yii, and in fact I quite like it. However, doing benchmark evaluation based on 'Hello World' is never a good idea. In fact, if all your script does is to output 'Hello World' on the screen, you are better off just using plain HTML. Why bother with server-side programming languages at all in this case? XD

 

 

Here is another benchmarking article I found, also with flaws:

http://www.techempower.com/blog/2013/03/28/framework-benchmarks/

 

Sure it seems to be using more realistic code this time, but because it is evaluating frameworks among many different platforms, the coding style becomes an issue here. You cant write java-like code with programming languages like Python to claim that the latter is much slower. Not saying it is not slower, but definitely not that much slower. With PHP, Cake is one of the slowest frameworks available, why not use Yii or Codeigniter instead? If I were to choose the slowest framework for java servlet and node.js, and compare them with the fastest PHP frameworks, the difference will be much smaller or even invisible.

Link to comment
Share on other sites

Again I was just trying to prove that many of the benchmarking test is overexaggering the performance issue in large frameworks like Zend. I never said Zend would be faster than other frameworks like Symfony or Yii, just not that much slower. Nor was I referring to the benchmarking test the OP brought up, its for all benchmarking artices I've seen in general.

 

But anyway if you cant be convinced or refuse to be convinced, I am fine with it. All I have to say then is that I respect your opinion, but I have to accept our differences and stop here. I do not live my life trying to please everyone, especially when there's no actual benefit for me to even do this.

Edited by Hall of Famer
Link to comment
Share on other sites

Again I was just trying to prove that many of the benchmarking test is overexaggering the performance issue in large frameworks like Zend.

I am aware of what your trying to "prove". Unfortunately though, your opinion doesn't mean proof. Do you understand? You haven't provided "any" proof, just opinion. Opinion (and yours in particular) carries very little weight.

Link to comment
Share on other sites

  • 4 weeks later...

The huge gap between ZF1 and S2 in the OP is more or less in concord with my own experiences. I wasn't to impressed with S1 but S2 ticks a lot of boxes, performance isn't at the top of our list of priorities but S2 in production mode is as fast as any full featured framework can be. We are migrating from ZF1 to S2 and the interface is noticeably more responsive on the same server. Of course you *could* make ZF equally performant with a good chunk of custom development but SF2 makes it effortless.

 

But I feel that's a valid argument: ZF provides a less restrictive infrastructure, S2 pre-chews more for you but Potencier made performance a key concern from the first lines of code so it's not surprising it's more performant. It does introduce some inflexibilities, but having worked with it for a couple of months now I can honestly say the extensibility provided is sufficient and the infrastructure is so much more mature than the ZF MVC component, there's really no comparison.

Link to comment
Share on other sites

  • 1 month later...

I am aware of what your trying to "prove". Unfortunately though, your opinion doesn't mean proof. Do you understand? You haven't provided "any" proof, just opinion. Opinion (and yours in particular) carries very little weight.

Isn't what Hall of Famer said just plain common sense? Every singe framework has some almost constant overhead for any request. What you really want to know is how the framework handles real life situations, or rather how it would handle your particular application. It should be familiar from mathematics as AxB + C. C would be the overhead, and A and B would define how it really scales. For small stuff like "Hello World", you just want 'C' to be as small as possible. For real situations you should be more concerned about 'A' and especially 'B', and would probably design your framework to minimize these with tho probable result that 'C' will grow a bit. A suit of benchmarks should contain additional tests to try to find out how the framework scales, not just the size of the overhead.

Link to comment
Share on other sites

This thread is more than a year old. Please don't revive it unless you have something important to add.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.