In need of a miracle. Some emails still not being received.

[njdubois]

Hello once again Php Freaks!  I've probably posted this problem a few times a year for the past 5 or so years.  I am a dev/admin for the site marcomtechnologies.com.  From that website, using php we send emails out.  Some users never get the email and it isn't in their spam folder.  The two users we can confirm never get the email have @att.net email address and, let me warn you.  Do not ever, EVER call att/yahoo/sbc for tech support.  It's an endless circle and when you do finally talk to someone, they have a really hard time grasping that it is about emails coming from a server they are not hosting.

 

I've talked to GoDaddy everytime, and everytime it is the same thing.  Everything is ok on their end.

 

I will post the code below.  First what I have done.

 

- The emails are coming "FROM" the correct domain.  The website is marcomtechnologies.com, and the emails are from admin@marcomtechnologies.com.  This was something that if not set this way, would be flagged as spam right off the top.  this is a valid email, on the email services provided by godaddy.  I can log in and send and receive emails from there no problem.

 

- I was using the standard PHP mail command, than switched to PHPMailer and we still have the same problem.

 

- I have checked to see if the domain is blacklisted.  As far as I can tell it is not.  If it was true, than my address, @gmail wouldn't get the emails and I do.  (quick note, a few weeks ago even I didn't get some?)

 

- GoDaddy tells me that we have 250 "relays" or emails that can go out.  On a daily basis, we may send 50 messages.  If that.

 

- I have tried to get bounce back messages stating the message was undelivered and have yet to ever get one.

 

- We have set up read receipts and they work, but not every sales person reads their emails, just kinda ignores them.  So read receipts are not really helping us, cause we don't know if they are ignoring the message, or not getting them.

 

- For one of our customers, we had to talk to an admin and have them put a rule on there server allowing messages from our domain.  From what we can tell it worked, but again are the sales staff ignoring, or not getting the message?  We don't know until we are there and they say so.  Why is att.net not getting them?

 

The emails are going out and leaving godaddy.  Somewhere between godaddy and the receiver, the message is blocked, lost, deleted or just decides to take a day off.  Which is really what seems to be happening.  I've reach a state of utter frustration with this.  Emails are a simple, all over the place, everyone uses, and has for a long long time technology.  What can I do to confirm they got the message without them having to actually read it?  What can I do to determine why the message isn't making it?  I'm never getting those bounce backs, so I have no details at all.

 

The clients we serve are not the most tech capable, and for all we know everything is ok.  the problem is, for my boss, this is unacceptable.  I need a solution, and a confirmation that the solution is working 110%. 

 

What am I missing?  Here is the chuck of code that sends the messages out :

require 'PHPMailer/PHPMailerAutoload.php';
require_once('PHPMailer/class.phpmailer.php');

$results_messages = array();
 
$mail = new PHPMailer(true);
$mail->CharSet = 'utf-8';
 
class phpmailerAppException extends phpmailerException {}

if($file_name[0]=='') {
		
	try {	
		if(!PHPMailer::validateAddress($to)) {
			throw new phpmailerAppException("Email address " . $to . " is invalid -- aborting!");
		}

		$body = $email_message;
		$body = eregi_replace("[\]",'',$body);

		$mail->AddReplyTo($_SESSION['member_email'],$_SESSION['WholeName']);
		$mail->SetFrom("admin@marcomtechnologies.com",$_SESSION['WholeName']);

		$mail->AddAddress($to,$to);


		$mail->Subject = $MSG_row['msg_subject'];
		$mail->MsgHTML($body);

		if(!$mail->Send()) {
			$results_messages[] = "Mailer Error: " . $mail->ErrorInfo;
		}
		else {
			$results_messages[] = "Message has been sent to " . $to . ', ';
		}

	}
	catch (phpmailerAppException $e) {
		$results_messages[] = $e->errorMessage();
	}

}
else {

	$path = '../audio_upload/' . $audio_filename;
	$path_name=$audio_filename;


	try {

		if(!PHPMailer::validateAddress($to)) {
			throw new phpmailerAppException("Email address " . $to . " is invalid -- aborting!");
		}

		$mail->isSendmail();
		$mail->addReplyTo($_SESSION['member_email'], $_SESSION['WholeName']);

		$mail->SetFrom("admin@marcomtechnologies.com",$_SESSION['WholeName']);

		$mail->addAddress($to, $to);
		$mail->Subject  = $MSG_row['msg_subject'];

		//$body = "<<<'EOT'";
		$body = $email_message;
		//$body .= "EOT";

		$mail->WordWrap = 80;
		$mail->msgHTML($body, dirname(__FILE__), true); //Create message bodies and embed images

		// preparing attachments
		$main_path='email_upload/';

		for($x=0;$x<count($file_name);$x++){
			$mail->addAttachment($main_path.$file_name[$x]);  // optional name
		} 

		try {
			$mail->send();
			$results_messages[] = "Message has been sent to " . $to . ', ';
		}

		catch (phpmailerException $e) {
			throw new phpmailerAppException('Unable to send to: ' . $to. ': '.$e->getMessage());
		}
	}

	catch (phpmailerAppException $e) {
		$results_messages[] = $e->errorMessage();
	}
}	

$mail->clearAddresses();
$mail->clearAttachments();

The above is just the part that sends the email, it is part of an AJAX file, and you can download a ZIP with the whole file here:

http://marcomtechnologies.com/email_code.zip

 

I have removed the database usernames/password stuff.  You will see what I mean, other than that it is 100% complete.

 

I have absolutely no idea what to do.  I'm in the dark blindfolded.

 

Thank you so much for taking the time to review my problem!

 

Nick

[njdubois]

UPDATE.  A more experienced programmer recommended opening a new gmail account and sending a message there.  That message went right to the spam box.  I am looking into ways of preventing that from happening without requiring the end user to make any changes.  It would be difficult and time consuming to go through each user we currently have, and everytime we add a user from here on out.

 

Any suggestions?

 

Thanks

 

Nick

[fastsol]

If you't using phpmailer there isn't much that can go wrong there.  I can tell you from first hand experience with lots and lots of testing and research to make my own site deliver emails effectively, that emails are NOT simple like you think they are.  There is a tremendous amount of things that go into delivering an email to an inbox, expecially at msn/outlook/hotmail account, they have the strictest spam filters of any of them. 

 

For an email to go to an inbox you need to have a number of very specific DNS records setup on your domain.  I can also tell ou first hand that GoDaddy cannot achieve these for you.  I used to have GoDaddy a couple years back and they are honestly terrible for more than just a simple non-dynamic website.  So in order for you to get the DNS records done properly, you must use a dedicated server that you can control every single aspect of.  Now some people may say that "oh you can just use a VPS or any cPanel setup", no you can't!  Those setups still have restrictions that will not allow you to effectively setup the correct DNS records that the email ISP's are now REQUIRING to deliver to the inbox. 

 

The other newly and VERY important aspect that the email ISP's use is email/domain reputation.  Even with all the proper DNS records setup perfectly, you will still not go into inboxes with out a good reputation.  Unfortunately reputation takes time to build unless you send thousands of emails a month.  At a rate of 50 emails a day, your reputation will take a couple months to get high enough to really do anything for you.  Now with the proper DNS records, you will generally get in to most inboxes, except the ones listed above, right off the bat simply cause the other ISP's are not as strict. 

 

If your company is truly set on delivering emails to inboxes EVERY time and can afford the expense to help guarantee that, then you should get certified with returnpath.com.  Simply by having their certification you will almost be guaranteed to hit inboxes immediately.  BUT they are very strict on requirements to become certified and will require all the things I already talked about and then some. 

 

You should start googling stuff about reverse DNS record, spf record, DMRAC record and DKIM.  Those are the main DNS records that you will need to setup for the domain and as I already said GoDaddy cannot support this.  You need to find a good host that provides fully dedicated servers.  I personally use
http://ixwebhosting.com and have a friend that uses bluehost.  If you call ix and talk to the sales staff and then ask to speak with Mel Vidal and tell him JD from remotelystartedmn.com sent you and he can help set things up once you sign up.

 

Now I will say that I am not always 100% correct in many things I know about coding but this is something that I do know what I'm talking about.  Hopefully that helps you get the results you want even if it's not the answer you were hoping for.

 

P.S. one last thing to try quickly is to NOT send the email as html and only as simple text.  HTML emails are also a flag to ISP's to be very careful in delivering to inboxes as they can disguise harmful things much easier than simple text.

[njdubois]

First, thank soooo much!  That is the most detailed, and clearest explanation as to what is going on here.  Thank you!

 

We are looking into text messages, and you can see some questions I have regarding that in the general forum:

http://forums.phpfreaks.com/topic/288968-ethics-question-pertaining-to-work-related-text-messages-to-non-work-related-devices/

 

I am going to forward this to my bosses as I think it will put front and center the complex nature of what we are doing.  I personally had no idea there was so much to it!

 

Thanks again!!

 

Nick

[mac_gyver]

dnsstuff has a number of test services. here's the main (free) one for your domain - http://www.dnsstuff.com/tools#dnsReport|type=domain&&value=marcomtechnologies.com

 

there are some warn and fail results that may be relevant to your email problems.

 

they also have paid services (which i have not use and cannot vouch for) that may be of help.

[fastsol]

There are a few other detail things that you would need/want to do after the new server it setup.  Mainly sign up for the aol, yahoo and msn feedback loops and microsoft has one other sign up thing that I can't remember at this time, but that is covered in the certification from returnpath.  Hopefully your boss listens and will help in fixing this issue for you.  You can tell him that the investment would be ~$1500/year for the server and ~$1000/year for the returnpath certification.

[njdubois]

Thanks mac_gyver for the site recommendation.  I have sent the results to those in charge and will see what they say.

 

Again, thanks fastsol for the help and recommendations as well.

 

So far it has been radio silence from those I need a reply to.  I think the 2 of them are on a plane.

 

Fingers crossed I get this resolved.  I'm not paid hourly after all =P

 

Thanks so much!

 

Nick

[Clarkey]

There's some great help in this thread but Fastsol, can you explain why a Linode or DigitalOcean VPS wouldn't be capable of this?

[fastsol]

There's some great help in this thread but Fastsol, can you explain why a Linode or DigitalOcean VPS wouldn't be capable of this?

The main reason you can't really use a VPS is even though you may be told you are getting a "dedicated" IP address, you most likley are not getting a truly dedicated AND whitelisted IP.  Both of those things are HIGHLY important to make all this stuff work.  What I mean by dedicated IP is exactly what it states, a 100% fully dedicated IP never before used and not shared in any manner by another domain. 

 

I have a normal shared hosting account with IX also that I used for previous website stuff, within there I have normal shared IPs and then I can also mark a couple as dedicated IPs.  The problem there is that even though it's marked as dedicated and I could actually use the ip in the url to navigate to the domain the IP isn't truly dedicated cause it is used by other aspects of IX to do other things behind the scenes.  Those "other things" can have a HUGE direct effect on your email reputation simply because whatever else happens on the IP may be hurting your reputation if another domain is using the IP in some way and sending out spam. 

 

All the stuff I just talked about is coming straight from the sales people at the few hosting companies I talked to about setting up a dedicated server for my business.  Plus if your company wants to have total control of it's business and how it's reputation is handled, then you can't really leave it up to chance that someone or something else is effecting it besides you the company.  After researching and understanding the amount of things that higher companies go through to ensure their emails are received, I found it was well worth the investment for my business to go the same route.

[njdubois]

Yeah, I've been dealing with these email issues for years.  Obviously it is WAY more complex than I had thought!  This thread has answered some questions, and not that we already didn't know but we obviously need to get away from gocrappy!

 

Thanks a lot!!

 

Nick

[gizmola]

The main reason you can't really use a VPS is even though you may be told you are getting a "dedicated" IP address, you most likley are not getting a truly dedicated AND whitelisted IP.  Both of those things are HIGHLY important to make all this stuff work.  What I mean by dedicated IP is exactly what it states, a 100% fully dedicated IP never before used and not shared in any manner by another domain. 

 

I have a normal shared hosting account with IX also that I used for previous website stuff, within there I have normal shared IPs and then I can also mark a couple as dedicated IPs.  The problem there is that even though it's marked as dedicated and I could actually use the ip in the url to navigate to the domain the IP isn't truly dedicated cause it is used by other aspects of IX to do other things behind the scenes.  Those "other things" can have a HUGE direct effect on your email reputation simply because whatever else happens on the IP may be hurting your reputation if another domain is using the IP in some way and sending out spam. 

 

All the stuff I just talked about is coming straight from the sales people at the few hosting companies I talked to about setting up a dedicated server for my business.  Plus if your company wants to have total control of it's business and how it's reputation is handled, then you can't really leave it up to chance that someone or something else is effecting it besides you the company.  After researching and understanding the amount of things that higher companies go through to ensure their emails are received, I found it was well worth the investment for my business to go the same route.

 

 

You're spouting a lot of hearsay -- sorry.  

 

Looking back you've provided some good information about the mechanics of email delivery, so I'm not trying to come down on you, but at the end of the day, there is no difference with having a VPS with a dedicated IP or a "dedicated" server with a "dedicated IP".  Everything you allude to in regards to IP addresses can happen just as easily with a dedicated server as it can with a VPS.  

 

The fact of the matter is that v4 IP addresses are a precious commodity.  ISP's have allocations and they reuse those.  Period, end of story.  The way internet routing works guarantees that people outside a network have no idea what specific machine(s) are using which IP's.

 

These days, you must have at very least, all the DNS glue to make email work.  These are things you've brought up like: SPF, DKIM, and a valid reverse DNS entry for the IP.  

 

There are RBOC's that are relied upon by some domains, but you can always query these and determine quickly whether or not your IP has been blacklisted.   It is not the problem here, as the big webmail providers have custom spam rules and don't trust RBOC's which can easily be gamed to screw up email delivery for valid domains.

 

GoDaddy does not provide reliable email delivery.  All you need do, is spend a bit of time researching to find complaints from customers where for entire blocks of time, email delivery was not occurring.  I also have experience with someone who has a business based on email accounts and delivery, who had fairly definitive proof that emails from GoDaddy were disappearing into various accounts in Yahoo mail.

 

When you utilize a package like phpmailer, you still have to deliver the email into an MTA which will then actually send it to the destination, and that MTA is the place where all these DNS issues come into play.  If you're dumping mail into GoDaddy, it's not coming from you -- it's coming from GoDaddy's email infrastructure.  So you really have no way of verifying or debugging delivery issues, or even knowing how long it might be before a specific mail transfer/SMTP conversation occurred, or what the result of that was, because you have no logs available to you, or MTA debugging.  All you know is that you dumped the mail into one of who knows how many GoDaddy servers and what happened from there is a question that you simply can't get anyone at GoDaddy to EVER help you with.  Email delivery, not to mention hosting, is not their core business, although I'm continually surprised as to how many people host with them.  Frequently, when you look into WHY they are hosting with GoDaddy, it's purely financial.  They are cheap, when compared to the higher echelon hosting companies.  And as the old saying goes... you get what you pay for.

 

You also have the issue of dealing with the fact that the email was relayed by GoDaddy, which some spam algorithms might penalize, because at the end of the day, GoDaddy is sending out a gazillion emails, much of which people consider spam.

 

In general, the way spam evaluation works, involves a point system.  When an email is received, spam points will be added for any number of issues that can occur even in completely normal emails.  The spammers are constantly battling with spam recognition software to get around these scores, and experimenting with ways to avoid getting spam points.  Ultimately, there are thresholds involved, and if a particular company like yahoo or gmail, decide that for certain scores, those emails will simply be deleted, you have absolutely no way of knowing.   The only thing you can do with these providers is setup test accounts and seed your system with these accounts.

 

Even then, a user can do something that blacklists your emails, and you'll never know the difference.  The person I alluded to previously could never determine why some emails to  Yahoo mail customers got through, while others didn't.  

 

Companies that send a lot of email (like one I worked for at one time, that sent millions of non-spam emails per day) have to constantly monitor what is going on with delivery.  We would have to pro-actively contact some of the big providers and ask for help with delivery to certain customers, and the only reason we were able to get support with that is because we were a mid-size highly recognized publicly traded company, and even then issues could drag on for days or weeks.

 

You have one of two options.  You can either own the email delivery problem end-to-end, or you can pay a "remailing" company to do that for you.  There are numerous options out there you can look at, with various pricing models, all of which give you access to logs to see when your emails were received and when they were delivered.  

 

As their entire business requires delivery of email, they have to stay on top of the obvious issues I've described, although even with those, you frequently have to add at the very least, SPF records into your DNS.

 

If email is essential to your business, you might want to look at one of those, and to just close out my prior comments, many huge name brand internet 2.x companies are entirely hosted on cloud servers (aka VPS's) and send more email than you can imagine, and have no problems doing so.

[fastsol]

Well I suppose that's more info on VPS's than I was given by the (most likely under-paid and don't know much) salespersons at the hosting companies I contacted when deciding on how to handle my situation.  All though I did talk to the lead tech at one place and he elluded to the same things I was already told, but maybe that means didly too

 

Anyway the real issue I would say in the server you choose is the ability to setup the proper DNS records and have a whitelisted (as in it's not in any blacklist currently and hopefully has no previous record associated with it.) dedicated IP from the start.

 

Maybe I am a little wrong on the server aspect but all the rest holds true.

Cheers!

[gizmola]

Well I suppose that's more info on VPS's than I was given by the (most likely under-paid and don't know much) salespersons at the hosting companies I contacted when deciding on how to handle my situation.  All though I did talk to the lead tech at one place and he elluded to the same things I was already told, but maybe that means didly too

 

Anyway the real issue I would say in the server you choose is the ability to setup the proper DNS records and have a whitelisted (as in it's not in any blacklist currently and hopefully has no previous record associated with it.) dedicated IP from the start.

 

Maybe I am a little wrong on the server aspect but all the rest holds true.

Cheers!

 

If an IP or IP block is blacklisted, then that's going to be a problem.  When you say "whitelisted" I'm not sure what you mean by that.  

 

Getting individual customers to whitelist your emails is hard to do in my experience, even though companies routinely try to send people instructions on how to do just that.  But that doesn't involve knowledge of an IP address.  

 

Otherwise, SPF, Reverse DNS and to a lesser extent, DKIM are all important, so I agree with you strongly that an understanding of those things is very helpful in figuring out how to maximize acceptance of your outbound email.

[njdubois]

An update for you all.

 

Another co worker and I spent a long period of time this morning testing stuff.  By looking in the headers, we were able to determine that godadys relay is using a range of ips, and we found them.  Tested to see if they were black listed, and sure enough 3 of the 5 we found were black listed.

 

Not all around black listed, but each one had 2 different spam filters where as the other 20+ where ok.  Each of the 2 ips being black listed, were blocked on different spam filters.

 

Another thing we found out is that SPF None, *domain here* does not designate permitted sender host, is shared server have proper DNS setup?

 

I may have missed some of that.

 

Called godaddy telling them of the black listing and they pretty much told me something like this.  I bought a ford explorer, went to fill it up, the line to the gas tank is full and ford told me it is shells fault.

 

Big boss called, pointed out the SPF issue and now I guess something wasn't configured right (even though for YEARS they have said it was) and is now been changed and with in 24 hours we are going to test again.

 

At this point I detest godaddy with all my heart and soul.  They have had the same damned hold music for 6 years, and I swear it feels like I'm in line for a rollercoaster, and the line never moves.  Calling godaddy feels like I have died, and am in hell.

 

I have to step away, but I see that there is a lot of other posts that have happened since I checked yesterday.  When I get back I am going to read all of it cause I feel this is something I need to know about if I am going to be a web developer.

 

Again, thanks for everything!

 

Nick