I'd suggest to get XDebug installed on your server and then take a look at where your scripts are taking the longest, and then seeing what can be improved to bring those times down.
If you're repeatedly generating (frying) content when serving a static (baked) file would suffice, then it's worth looking in to seeing if you can bake the currently-fried pages in an efficient manner, paying attention to knowing when the cache is stale, and re-generating it.
Of course all of this depends highly on the work your site is performing, which is why I'd suggest profiling the performance to see if it's worth caching, and how to go about that sensibly.