Email faster, its really slow

[scootstah]

Because it is waiting on a response from the external HTTP server. That response time is going to vary slightly, based on a whole bunch of factors.

 

It might be helpful if you make a quick diagram of what your infrastructure looks like, just so that we're all on the same page. What it sounds like to me is that you have a server A which is reaching out to server's B and C to get some data, then sending emails from server A. Is that accurate?

[brentman]

Because it is waiting on a response from the external HTTP server. That response time is going to vary slightly, based on a whole bunch of factors.

 

It might be helpful if you make a quick diagram of what your infrastructure looks like, just so that we're all on the same page. What it sounds like to me is that you have a server A which is reaching out to server's B and C to get some data, then sending emails from server A. Is that accurate?

 

No. All the data is on Server A. It sends the data to other servers B, C and D who actually send the email out. Currently, it is all done on one machine but under different cpanel accounts, domains, ips. In the future I will be moving it/ adding more on other physical machines, or at least should have the option. The actual code that emails is pasted prior in the thread.

 

It cannot possibly be the http request because I can comment it out and there isn't a significant increase in performance. It's about 10%. I am running a duel hexacore 2.4ghz, it really should be able to handle running a php loop and some queries so that shouldn't be the problem. And without the http request that cannot be the problem either.

 

It does do about 6 mysql queries one of which is checking against a table of 700,000 records, the others are checking against other tables with about 100k records and one pulling the info and one writing the final statuses and stuff. All of which is in a loop. Each record does need to be checked just prior to emailing so I cannot significantly change the function that is taking place. Although I have tried commenting it out just to test and it had very little effect on performance.

 

The entire file is 484 lines with plenty of spacing and such, just to give you an idea of how much is going on. Most of it falls into two features, one is manipulating the message. ie Dear Joe to joe but Hey Friend to someone without a name etc. each email is quite personalized, more so than most people are doing. The second main feature is checking the email vs all my black lists I have acquired and other checks for proper formatting, likelyhood of being good address, etc. ie. brentman@gmail.com is ok to send to but brentman@northkorea.kp is not ok.

[scootstah]

How are you commenting out the HTTP request and the system still work?

 

Which server is the dual hexacore 2.4GHz - is that the mail server, or the one sending HTTP requests?

[brentman]

How are you commenting out the HTTP request and the system still work?

 

Which server is the dual hexacore 2.4GHz - is that the mail server, or the one sending HTTP requests?

 

1. Well it doesn't send emails but it runs through all the loops and things, just doesn't actually send it out.

 

2. They are the same server. It has multiple cpanel accounts with a different domain and ip for each. *The end goal is to have it on different servers at some point so it isn't helpful for me to make it a one server solution.

Edited July 17, 2015 by brentman

[scootstah]

Gotcha. So everything at the moment is on a single server, but different domains. Therefore there is no network latency, and you're not really going to get a sense of scalability.

 

Running a bunch of queries inside of a loop could certainly be another potential bottleneck, especially if there is inefficiencies in the query or database setup. You should look at ways to get rid of that looping. Have you posted that code already? If not, please do so.

[brentman]

Found the problem. My main table was not set on InnoDB *facepalm* so each instance I was making was waiting to complete updates and with each new instance it made it run slower and slower as it was having to wait for more likely collisions on the table. Quick change made it significantly faster but I will play around more with it later and see if that is the only problem.

[scootstah]

You're going to want to enable slow query logging, make sure you have proper indexing, have efficient schema, etc. It would also help to profile the PHP code to see which things are taking time.

 

If you have to loop queries you didn't design something properly.

[fastsol]

Found the problem. My main table was not set on InnoDB *facepalm* so each instance I was making was waiting to complete updates and with each new instance it made it run slower and slower as it was having to wait for more likely collisions on the table. Quick change made it significantly faster but I will play around more with it later and see if that is the only problem.

Once you verified the stats on the computers and where everything was getting info from, I was becoming very certain the issue lied in the query.

[brentman]

I have another table that I suspect is having collisions as well as I still am seeing decrease performance with more scripts running although now I am up to 5 before performance degrades. This one cannot be fixed simply with InnoDB though. Lesson from thread, pay more attention to query and DB design. At work but will get some code posted later or this weekend.