Network Load Balancer With Static IP Troubles

[DeX]

We currently have 2 separate ISP providing us internet at our office and I set up our firewall to load balance the traffic on both networks. We have a production server behind the firewall and it's set to a static IP given to us by ISP 1.

 

I set up a PHP script on the server to connect to our Google Drive through their API and it works great 50% of the time. The other 50% it times out so I tried taking ISP 2 out of the load balancer and having the company run completely on ISP 1, this caused the script to work 100% of the time and complete in under 1 second. When I remove ISP 1 and have the company running on just ISP 2, the script fails 100% of the time with the same timeout, saying it couldn't resolve the hostname of the Google Drive server.

 

While running on either network, our internet works just fine, no problems at all. I run speed tests and they're perfect, we're supposed to get 10 down from one ISP and 20 down from the other but the tests show around 28 down at each terminal.

 

How do I set the server to always make its outbound API requests on ISP 1? Can I add a second static IP for ISP 2 and will that work? I don't know what to do next, thanks.

 

API error:

 

 

 RuntimeException: Error creating resource: [message] fopen(https://www.googleapis.com/oauth2/v4/token): failed to open stream: Connection timed out [file] /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Handler/StreamHandler.php [line] 324 in /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Handler/StreamHandler.php:252 Stack trace: #0 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Handler/StreamHandler.php(335): GuzzleHttp\Handler\StreamHandler->createResource(Object(Closure)) #1 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Handler/StreamHandler.php(52): GuzzleHttp\Handler\StreamHandler->createStream(Object(GuzzleHttp\Psr7\Request), Array) #2 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(66): GuzzleHttp\Handler\StreamHandler->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #3 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Middleware.php(30): GuzzleHttp\PrepareBodyMiddleware->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #4 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(70): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Request), Array) #5 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Middleware.php(59): GuzzleHttp\RedirectMiddleware->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #6 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/HandlerStack.php(67): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Request), Array) #7 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Client.php(277): GuzzleHttp\HandlerStack->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #8 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Client.php(99): GuzzleHttp\Client->transfer(Object(GuzzleHttp\Psr7\Request), Array) #9 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Client.php(106): GuzzleHttp\Client->sendAsync(Object(GuzzleHttp\Psr7\Request), Array) #10 /var/www/versions/development/vendor/google/auth/src/HttpHandler/Guzzle6HttpHandler.php(34): GuzzleHttp\Client->send(Object(GuzzleHttp\Psr7\Request), Array) #11 /var/www/versions/development/vendor/google/auth/src/OAuth2.php(501): Google\Auth\HttpHandler\Guzzle6HttpHandler->__invoke(Object(GuzzleHttp\Psr7\Request)) #12 /var/www/versions/development/vendor/google/auth/src/Credentials/ServiceAccountCredentials.php(115): Google\Auth\OAuth2->fetchAuthToken(Object(Google\Auth\HttpHandler\Guzzle6HttpHandler)) #13 /var/www/versions/development/vendor/google/auth/src/FetchAuthTokenCache.php(84): Google\Auth\Credentials\ServiceAccountCredentials->fetchAuthToken(Object(Google\Auth\HttpHandler\Guzzle6HttpHandler)) #14 /var/www/versions/development/vendor/google/auth/src/Middleware/AuthTokenMiddleware.php(115): Google\Auth\FetchAuthTokenCache->fetchAuthToken(Object(Google\Auth\HttpHandler\Guzzle6HttpHandler)) #15 /var/www/versions/development/vendor/google/auth/src/Middleware/AuthTokenMiddleware.php(102): Google\Auth\Middleware\AuthTokenMiddleware->fetchToken() #16 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(37): Google\Auth\Middleware\AuthTokenMiddleware->Google\Auth\Middleware\{closure}(Object(GuzzleHttp\Psr7\Request), Array) #17 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Middleware.php(30): GuzzleHttp\PrepareBodyMiddleware->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #18 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(70): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Request), Array) #19 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Middleware.php(57): GuzzleHttp\RedirectMiddleware->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #20 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/HandlerStack.php(67): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Request), Array) #21 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Client.php(277): GuzzleHttp\HandlerStack->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #22 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Client.php(99): GuzzleHttp\Client->transfer(Object(GuzzleHttp\Psr7\Request), Array) #23 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Client.php(106): GuzzleHttp\Client->sendAsync(Object(GuzzleHttp\Psr7\Request), Array) #24 /var/www/versions/development/vendor/google/auth/src/HttpHandler/Guzzle6HttpHandler.php(34): GuzzleHttp\Client->send(Object(GuzzleHttp\Psr7\Request), Array) #25 /var/www/versions/development/vendor/google/apiclient/src/Google/Http/REST.php(74): Google\Auth\HttpHandler\Guzzle6HttpHandler->__invoke(Object(GuzzleHttp\Psr7\Request)) #26 /var/www/versions/development/vendor/google/apiclient/src/Google/Task/Runner.php(176): Google_Http_REST::doExecute(Object(GuzzleHttp\Client), Object(GuzzleHttp\Psr7\Request), 'Google_Service_...') #27 /var/www/versions/development/vendor/google/apiclient/src/Google/Http/REST.php(58): Google_Task_Runner->run() #28 /var/www/versions/development/vendor/google/apiclient/src/Google/Client.php(788): Google_Http_REST::execute(Object(GuzzleHttp\Client), Object(GuzzleHttp\Psr7\Request), 'Google_Service_...', Array) #29 /var/www/versions/development/vendor/google/apiclient/src/Google/Service/Resource.php(232): Google_Client->execute(Object(GuzzleHttp\Psr7\Request), 'Google_Service_...') #30 /var/www/versions/development/vendor/google/apiclient-services/src/Google/Service/Drive/Resource/Files.php(206): Google_Service_Resource->call('list', Array, 'Google_Service_...') #31 /var/www/versions/development/drive-test.php(74): Google_Service_Drive_Resource_Files->listFiles(Array) #32 {main} Next GuzzleHttp\Exception\RequestException: Error creating resource: [message] fopen(https://www.googleapis.com/oauth2/v4/token): failed to open stream: Connection timed out [file] /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Handler/StreamHandler.php [line] 324 in /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:52 Stack trace: #0 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Handler/StreamHandler.php(67): GuzzleHttp\Exception\RequestException::wrapException(Object(GuzzleHttp\Psr7\Request), Object(RuntimeException)) #1 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(66): GuzzleHttp\Handler\StreamHandler->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #2 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Middleware.php(30): GuzzleHttp\PrepareBodyMiddleware->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #3 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(70): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Request), Array) #4 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Middleware.php(59): GuzzleHttp\RedirectMiddleware->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #5 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/HandlerStack.php(67): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Request), Array) #6 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Client.php(277): GuzzleHttp\HandlerStack->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #7 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Client.php(99): GuzzleHttp\Client->transfer(Object(GuzzleHttp\Psr7\Request), Array) #8 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Client.php(106): GuzzleHttp\Client->sendAsync(Object(GuzzleHttp\Psr7\Request), Array) #9 /var/www/versions/development/vendor/google/auth/src/HttpHandler/Guzzle6HttpHandler.php(34): GuzzleHttp\Client->send(Object(GuzzleHttp\Psr7\Request), Array) #10 /var/www/versions/development/vendor/google/auth/src/OAuth2.php(501): Google\Auth\HttpHandler\Guzzle6HttpHandler->__invoke(Object(GuzzleHttp\Psr7\Request)) #11 /var/www/versions/development/vendor/google/auth/src/Credentials/ServiceAccountCredentials.php(115): Google\Auth\OAuth2->fetchAuthToken(Object(Google\Auth\HttpHandler\Guzzle6HttpHandler)) #12 /var/www/versions/development/vendor/google/auth/src/FetchAuthTokenCache.php(84): Google\Auth\Credentials\ServiceAccountCredentials->fetchAuthToken(Object(Google\Auth\HttpHandler\Guzzle6HttpHandler)) #13 /var/www/versions/development/vendor/google/auth/src/Middleware/AuthTokenMiddleware.php(115): Google\Auth\FetchAuthTokenCache->fetchAuthToken(Object(Google\Auth\HttpHandler\Guzzle6HttpHandler)) #14 /var/www/versions/development/vendor/google/auth/src/Middleware/AuthTokenMiddleware.php(102): Google\Auth\Middleware\AuthTokenMiddleware->fetchToken() #15 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(37): Google\Auth\Middleware\AuthTokenMiddleware->Google\Auth\Middleware\{closure}(Object(GuzzleHttp\Psr7\Request), Array) #16 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Middleware.php(30): GuzzleHttp\PrepareBodyMiddleware->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #17 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(70): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Request), Array) #18 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Middleware.php(57): GuzzleHttp\RedirectMiddleware->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #19 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/HandlerStack.php(67): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Request), Array) #20 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Client.php(277): GuzzleHttp\HandlerStack->__invoke(Object(GuzzleHttp\Psr7\Request), Array) #21 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Client.php(99): GuzzleHttp\Client->transfer(Object(GuzzleHttp\Psr7\Request), Array) #22 /var/www/versions/development/vendor/guzzlehttp/guzzle/src/Client.php(106): GuzzleHttp\Client->sendAsync(Object(GuzzleHttp\Psr7\Request), Array) #23 /var/www/versions/development/vendor/google/auth/src/HttpHandler/Guzzle6HttpHandler.php(34): GuzzleHttp\Client->send(Object(GuzzleHttp\Psr7\Request), Array) #24 /var/www/versions/development/vendor/google/apiclient/src/Google/Http/REST.php(74): Google\Auth\HttpHandler\Guzzle6HttpHandler->__invoke(Object(GuzzleHttp\Psr7\Request)) #25 /var/www/versions/development/vendor/google/apiclient/src/Google/Task/Runner.php(176): Google_Http_REST::doExecute(Object(GuzzleHttp\Client), Object(GuzzleHttp\Psr7\Request), 'Google_Service_...') #26 /var/www/versions/development/vendor/google/apiclient/src/Google/Http/REST.php(58): Google_Task_Runner->run() #27 /var/www/versions/development/vendor/google/apiclient/src/Google/Client.php(788): Google_Http_REST::execute(Object(GuzzleHttp\Client), Object(GuzzleHttp\Psr7\Request), 'Google_Service_...', Array) #28 /var/www/versions/development/vendor/google/apiclient/src/Google/Service/Resource.php(232): Google_Client->execute(Object(GuzzleHttp\Psr7\Request), 'Google_Service_...') #29 /var/www/versions/development/vendor/google/apiclient-services/src/Google/Service/Drive/Resource/Files.php(206): Google_Service_Resource->call('list', Array, 'Google_Service_...') #30 /var/www/versions/development/drive-test.php(74): Google_Service_Drive_Resource_Files->listFiles(Array) #31 {main}

 

[dalecosp]

How do I set the server to always make its outbound API requests on ISP 1? Can I add a second static IP for ISP 2 and will that work? I don't know what to do next, thanks.

If all the requests are to the same IP or IP range, I'd try adjusting the routing tables first (add a static route to the target). Of course, that might not work, because the problem itself is that your IP is dynamic, right?

 

Of course, what about the real error, that it can't resolve the hostname? What DNS server are you using? Is it accessible from the machine in question when it's connected to ISP2? I'm guessing no ...

 

In that case, I'd select a DNS source that's accessible from anywhere (like Google's public DNS) or run a local caching DNS box behind the firewall.

 

What's your OS? Is "/etc/resolv.conf" a thing on your server? If so, what does it contain?

 

And let's get even MORE basic ... if the server is on ISP2 ... can it connect to ANYTHING at all? For example, by IP address ... "ping 8.8.8.8" ...

Edited February 15, 2018 by dalecosp

[DeX]

If all the requests are to the same IP or IP range, I'd try adjusting the routing tables first (add a static route to the target). Of course, that might not work, because the problem itself is that your IP is dynamic, right?

 

Of course, what about the real error, that it can't resolve the hostname? What DNS server are you using? Is it accessible from the machine in question when it's connected to ISP2? I'm guessing no ...

 

In that case, I'd select a DNS source that's accessible from anywhere (like Google's public DNS) or run a local caching DNS box behind the firewall.

 

What's your OS? Is "/etc/resolv.conf" a thing on your server? If so, what does it contain?

 

And let's get even MORE basic ... if the server is on ISP2 ... can it connect to ANYTHING at all? For example, by IP address ... "ping 8.8.8.8" ...

 

Our IP is static, we have a static IP from ISP 1 and a different static IP from ISP 2. I only have the static IP from ISP 1 set on the server and also the ISP 1 specific nameservers as well.

 

The script connects 100% of the time when only on ISP 1 and 0% of the time when only on ISP 2.

 

Can I change the DNS and still get access to the internet? I thought I had to use the DNS provided by my ISP.

 

The server is running Ubuntu 16.04 and /etc/resolv.conf contains the following with numbers instead of *:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 64.*.135.*
nameserver 207.*.226.*
nameserver 64.*.128.*

I'll have to run this last test for you outside work hours to test.

[DeX]

New discoveries. The /etc/network/interfaces file is unmodified which makes me think I never set any sort of static IP for this server, I just set port forwarding for port 80 on the firewall to go to this machine. It is in the DMZ which led me to believe it may not have permissions to access the WAN but I ran the same ping test (ping google.ca) on a Windows machine on the DMZ and it returned all packets without problem. When I run the ping test on the web server (the one in question), it loses all packets. I can't ping anything from the web server.

 

This is weird. And just to clarify, this is our production web server, it's working inside and outside the network perfectly otherwise.

[kicken]

Sounds like a routing problem of some kind. I've not messed with anything on this level so I can't speak to how things should be configured.

 

What IP's does the server have available if you run ifconfig?

 

I imagine two possible ways one might configure load balancing.

One would be at the router level. That means your server should be setup similar to any other PC in your network, with an internal network IP rather than a public IP from your ISP. The router could direct new connections to which ever ISP has the most available bandwidth.

 

The other at the server level. You would configure the server with a static IP for each ISP then make sure your router knows to route those IP's to that server. The OS (or individual applications) could then control which ISP handles the traffic by choosing which IP to use as the originating IP.

 

 

It sounds like at the moment the router only knows what to do with ISP #1's address. Traffic it receives for ISP #2's address gets dropped/rejected because the router doesn't know about it.

[DeX]

Sounds like a routing problem of some kind. I've not messed with anything on this level so I can't speak to how things should be configured.

 

What IP's does the server have available if you run ifconfig?

 

I imagine two possible ways one might configure load balancing.

One would be at the router level. That means your server should be setup similar to any other PC in your network, with an internal network IP rather than a public IP from your ISP. The router could direct new connections to which ever ISP has the most available bandwidth.

 

The other at the server level. You would configure the server with a static IP for each ISP then make sure your router knows to route those IP's to that server. The OS (or individual applications) could then control which ISP handles the traffic by choosing which IP to use as the originating IP.

 

 

It sounds like at the moment the router only knows what to do with ISP #1's address. Traffic it receives for ISP #2's address gets dropped/rejected because the router doesn't know about it.

 

It sounds like you're referring to incoming connections, is that right? Everything incoming is perfect because the domain points to the ISP 1 static IP and the router (in the firewall) forwards all port 443 andn 80 requests to that specific server internally. Perfect.

 

The issue is with the outbound requests, I can't ping anything from this server and can only succeed on the Google API requests if the packet goes out through X1 on the firewall (ISP 1). I have the load balancing set to Round Robin which means packets get sent out randomly through whichever ISP line it happens to pick up and it seems to affect whether or not these API requests succeed for some reason. I don't know why.

 

I feel like setting up a whole new Ubuntu server to see if that can ping.

 

Very important, I just discovered something else. I copied the script to a separate EC2 server I have with Amazon which is completely unrelated to our network here at the office and the script works 100% of the time on that server. It works perfectly, I just need to figure out why it is not working from my office server.

[kicken]

It sounds like you're referring to incoming connections, is that right?

I was referring to outbound, specifically the responses. Say you ping 8.8.8.8, it sounds to me like what is happening is roughly:

• Outbound ICMP packet goes to 8.8.8.8 via ISP #2 (1.2.3.4)
• 8.8.8.8 replies to 1.2.3.4
• Your router receives an ICMP reply to 1.2.3.4
• Router doesn't know what to do with destination 1.2.3.4, so it drops the packet.

I've had issues like this with IPv6 traffic before and my home ISP. One way to check for such a problem is to ping an external machine you control and watch it's traffic via something like TCP dump.

 

When I was diagnosing my IPv6 problem I had 3 points I could monitor. 1) My desktop, 2) My home router, and 3) My external VPS. I'd send a ping from my desktop to my VPS and use tcpdump on both the router and vps to watch the packet go out. VPS would reply, my router would see the reply, but it never made it to my desktop.

[DeX]

I was referring to outbound, specifically the responses. Say you ping 8.8.8.8, it sounds to me like what is happening is roughly:

• Outbound ICMP packet goes to 8.8.8.8 via ISP #2 (1.2.3.4)
• 8.8.8.8 replies to 1.2.3.4
• Your router receives an ICMP reply to 1.2.3.4
• Router doesn't know what to do with destination 1.2.3.4, so it drops the packet.

I've had issues like this with IPv6 traffic before and my home ISP. One way to check for such a problem is to ping an external machine you control and watch it's traffic via something like TCP dump.

 

When I was diagnosing my IPv6 problem I had 3 points I could monitor. 1) My desktop, 2) My home router, and 3) My external VPS. I'd send a ping from my desktop to my VPS and use tcpdump on both the router and vps to watch the packet go out. VPS would reply, my router would see the reply, but it never made it to my desktop.

 

 

I get it, so I need to forward ports 80 and 443 from the ISP 2 static IP as well. Is that what you're saying? I thought a ping would always come through but I guess that makes sense.

 

Something else I'm exploring is the fact that I can't really ping anything from this server which leads me to think it's something with the DNS or gateway not being set properly. That way inbound traffic works fine but outbound not so much. I'm just trying to figure out what I would set the gateway, DNS and subnet mask to right now to see if that's the issue. I'm not sure if it's the settings I got from my ISP or if it's an internal default for the firewall.

[DeX]

More discoveries. I remembered I still have the old virtual machine we were previously using as our web server, I have since replaced that with a new virtual machine on the same VMWare server. Both virtual servers are almost identical, the only difference is the new one has more disk space. I spooled up the old server, copied the test script to it and it runs flawlessly every time. I'm also able to ping out of that server no problem so it's some setting in the new server which is preventing me from making outbound requests. I'm going to continue my search there and not mess with any firewall settings.

[dalecosp]

The reason for testing with "ping 8.8.8.8" (or any public IP address that responds to ping) is to determine whether DNS is the issue. If you can "ping by number" but you cannot "ping by hostname" then the issue is DNS. If you can't do either, it's network/routes/firewalls/protocols, etc.

 

I thought I had to use the DNS provided by my ISP.

That depends on whether your ISP (and perhaps your firewall in front of the LAN systems) allows traffic from other DNS servers (besides its own) on the network. I've not heard of MANY ISP's that do this, but it is fairly common for them to block some protocols (namely SMTP, to keep home users and compromised user machines from becoming spam-bots).

 

"netstat -nr" output might be helpful. What does that yield?

[DeX]

I actually figured it out after 3 weeks of frustration. I disabled a NAT policy on my firewall for outbound requests and it solved everything. The script works great now and I can ping without issue.