Jump to content

NEED HELP AVOIDING $75k bill for one night of loopin' in the cloud! Measuring CPU in PHP script?


sen5241b
 Share

Recommended Posts

 I have questions but I also have some good info to share about putting your software in the cloud.  

The situation:

Some of you may have read about the nightmare stories. A developer had a infinite loop in his code that ran all night. This code did things inefficiently that devoured CPU in each iteration. This developer, greatly skilled,  opened his email the next morning and saw a email bill from his cloud provider totaling for $75K. True story. 

Most cloud providers let you define CPU usage thresholds that, when breached, send you a warning but these thresholds, if I understand them, are per account. 

It would seem the claim that cloud resources are available in whatever amounts you need, CPU, disk space, enough RAM to never have to wait on a page faults, etc... the claim that the cloud provides you with infinitely elastic  resources in an "all you can eat for one price" contract smells just a like a little like 💩

I did cloud development for many years with ServiceNow starting when it was a help desk and I watched it evolve into one of the best cloud development platforms out there. At one customer site I installed and managed it it out of the cloud and saw its insides and I can tell you its core code is not so terribly efficient. IMHO the cloud DOES take away 90% of a developer's worries about app performance..

If you call ServiceNow tech support and your problem is diagnosed as a performance issue with your code the first thing they will ask you is "did you follow the developer best practices"? They will politely say "sorry. Here's a link to them implying "fix your code". 

Questions: 

PHP Functions that devour CPU and where there is a better way?

What PHP functions or code techniques waste CPU? I am using similar_text and it does the job but it is slow. Better way?

What is the best way to measure CPU used by a PHP script or by a particular code module, defined as a set of related functions that fulfill a common purpose or by a single line of code? The purpose being to identify inefficient modules of code and improve them and even if the code is damn near perfect then at least can know what code modules are the most expensive.

CPU killin users (and developers too), how can they be identified?

I need to store data on cumulative CPU usage for any of the above and compare it with the free amount they give you and warn CPU hogs before they breach a threshold and generate $75K bills that were not in the plan.

Any info you have on avoiding surprise $75K CPU bills from a cloud provider are welcome

 

 

 

 

It would seem the 

Link to comment
Share on other sites

20 minutes ago, sen5241b said:

Some of you may have read about the nightmare stories. A developer had a infinite loop in his code that ran all night. This code did things inefficiently that devoured CPU in each iteration. This developer, greatly skilled,  opened his email the next morning and saw a email bill from his cloud provider totaling for $75K. True story.

Citation needed.

If you're asking questions like "what PHP functions devour CPU?" then that means you don't understand the subject matter.

Measure resource usage with whatever tools the cloud provider gives you to do so.

Link to comment
Share on other sites

 

If you are asking for citations and you don't think efficient CPU usage has higher priority these days you don't understand the issue. A list of inefficient PHP functions is a reasonable request. PHP can't measure CPU accurately. 

 Google apparently has cumulative CPU usage at org and project level. . I've not looked at all cloud provider tools. My cloud provider doesn't provide tools to measure CPU at at a granular level.

Citations

https://blog.tomilkieway.com/72k-1/

https://www.electropages.com/blog/2021/01/how-start-received-75000-bill-2-hours-google-cloud-services

https://medium.com/milkie-way/we-burnt-72k-testing-firebase-cloud-run-and-almost-went-bankrupt-part-1-703bb3052fae

https://www.reddit.com/r/aws/comments/g1ve18/i_am_charged_60k_on_aws_without_using_anything/

Other on topic links:

https://feedback.azure.com/forums/170030-signup-and-billing/suggestions/3238796-spending-limit-or-maximum-cost-cap-for-azure

https://feedback.azure.com/forums/170030-signup-and-billing/suggestions/3238796-spending-limit-or-maximum-cost-cap-for-azure

https://cloud.google.com/billing/docs/how-to/view-linked

Link to comment
Share on other sites

2 hours ago, sen5241b said:

If you are asking for citations and you don't think efficient CPU usage has higher priority these days you don't understand the issue. A list of inefficient PHP functions is a reasonable request. PHP can't measure CPU accurately. 

 Google apparently has cumulative CPU usage at org and project level. . I've not looked at all cloud provider tools. My cloud provider doesn't provide tools to measure CPU at at a granular level.

An infinite loop and a runaway recursive algorithm spawning cloud resources to process its workload are two very different things. Much harder to believe that the former could create a $72K cloud bill than the latter.

I'm just going to side-step the whole issue of you not understanding how CPU usage works, and move on to the answer: if you have to worry about micro-optimizing your PHP code to minimize the number of CPU cycles then you do not have the budget to afford the cloud. Buy some traditional, dedicated site hosting package instead.

Link to comment
Share on other sites

I've only ever used VMs in a cloud service and those pretty much have a fixed cost.  Running an infinite loop isn't going rack up some gigantic bill.

Those citations to me read mostly as people just jumping into something they don't understand and getting burned by it.  Taking the time to learn how something works and use caution helps to avoid such costly mistakes.  For example what that guy in the first story learned about not using the high defaults when testing is something I would have done out of the gate.  Whenever I'm testing something that has the potential to run-away like that I always implement a hard stop (ie, die after $x loops) and start with a small concurrency factor (2-4 concurrent threads) until I'm sure things are behaving properly.  Such precautions probably would have limited their bill to less than $1000 it sounds like.  Still rough, but not destroys the company rough.

Secondly, test locally as much as possible.  It doesn't matter if you end up in big loop when your just testing on local hardware as you're not getting charged for usage.  Their issue of "If there’s a back link to the previous page, the Cloud Run service will be stuck in infinite recursion" never would have made it to the cloud service for me as I would have caught that during local testing.

 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.