Script Not Timing Out Using curl_exec( )

[l008com]

(copied from my own post on another forum) 

I have a CLI (launchd) script that runs periodically and makes some HTTP requests. I find that under certain situations, where the server is offline (like during a TCP reset), outgoing HTTP requests (via curl_exec( ) ) are hanging. But instead of timing out, they just keep going for 5, 10, 15 minutes or more.

set_time_limit(120);

$ch = curl_init($url);
curl_setopt($ch,CURLOPT_USERAGENT,"Mozilla/5.0 etc");
curl_setopt($ch,CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,10);
curl_setopt($ch,CURLOPT_TIMEOUT,10);
curl_setopt($ch,CURLOPT_INTERFACE,OPERATING_IP);
$return = curl_exec($ch);

Apparently time spent in curl_exec( ) is excluded from set_time_limit( ) time limits. 

But shouldn't either CURLOPT_TIMEOUT or CURLOPT_CONNECTTIMEOUT (especially CURLOPT_CONNECTTIMEOUT) cause the curl call to fail after 10 seconds? 

If not, whats the best way to get curl_exec( ) to fail (quickly) when it can't connect, rather than waiting minutes?

I was reading the curl docs after posting this. CURLOPT_TIMEOUT is supposed to INCLUDE CURLOPT_CONNECTTIMEOUT

So just the `curl_setopt($ch,CURLOPT_TIMEOUT,10);` line should be causing curl_exec( ) to fail after 10 seconds, when the server's internet connection has gone down. So I'm totally lost as to why this script is able to hang on curl_exec( ) for 6 minutes or more? 

[requinix]

16 hours ago, l008com said:

Apparently time spent in curl_exec( ) is excluded from set_time_limit( ) time limits.

Correct: (on most platforms,) PHP's max_execution_time/set_time_limit only affects the time spent running PHP code - "low-level" things, like cURL calls and database queries and such, don't count.

 

16 hours ago, l008com said:

I was reading the curl docs after posting this. CURLOPT_TIMEOUT is supposed to INCLUDE CURLOPT_CONNECTTIMEOUT

Also correct: TIMEOUT covers the "entire transfer operation" (connect + send + receive) while CONNECTTIMEOUT is just the "connection phase".

If you find that things are taking longer than 10 seconds, then I would suggest your problem isn't in that one, single cURL request. Have any loops? Other requests happening?
Try setting timeouts incredibly low, low enough that things time out, so you can see where time is being lost.
You can also try running a debugger and actually step through the execution so you can see what's happening for yourself.

edit: Just remembered there's your other thread too, so I'll try not to repeat things you've heard already.

[gizmola]

The connecttimeout is included in the timeout. 

In other words, connecttimeout should be < timeout.  I would try making connecttimeout 5 and timeout 15 just to make sure there's no conflict in the library, not to mention, that timeout will kick in even when there is no connection issues, as might be the case with a congested target host, or sluggish network on the server running the code.  

You might want to try adding this parameter as an experiment:  curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false)

While I doubt any of these things are likely to resolve this issue, as I understand it, the network connection of the server running this code is not available, and most likely what is happening is that the curl library calls are stuck waiting for the operating system tcp/socket calls to timeout, and there is likely nothing you can do about this issue purely within curl.

You have to consider that there are a variety of calls being made internally, among them DNS resolution calls, which might be hanging for a long period of time on the get_addr_info.  You could test this by using an IP based url rather than a hostname.

One workaround you could add would be a function call that makes a system call to your OS ping utility.  The specific command parameters to use are OS dependent, and you haven't stated what OS the code is running on, but the basic idea is to send 1 ping with a 1 second wait time to google or cloudflare or other reliably available domain.  Using the returned status you could either retry a few time, if the hope is the server's LAN connection will wake up, or you can just exit the script (logging or displaying an error) and thus avoid attempting the curl connections when the network is hung. 

[l008com]

7 hours ago, requinix said:

If you find that things are taking longer than 10 seconds, then I would suggest your problem isn't in that one, single cURL request. Have any loops? Other requests happening?

...

edit: Just remembered there's your other thread too, so I'll try not to repeat things you've heard already.

As per the other thread, I added __LINE__ logging to every SINGLE line of my script, that outputs to a log file that i monitored via the `tail` command. It is definitely hanging on the curl_exec( ) call, definitely running for 5-10 minutes. This happens specifically when my TCPIP stack is being reset, which happens for unrelated reasons. While there are loops in the script, thats not where its getting hung up. This I am fully certain of. I'm not sure why setting extremely low timeouts would help me debut? Won't I just get the exact same results? When everything is good, the request will be done within the 1 second, and when things are not working right, it should hang for minutes with a 1 second timeout just as easily as it hangs for minutes with a 10 second timeout? 

[l008com]

7 hours ago, gizmola said:

The connecttimeout is included in the timeout. 

...and most likely what is happening is that the curl library calls are stuck waiting for the operating system tcp/socket calls to timeout, and there is likely nothing you can do about this issue purely within curl.

I agree, that is most likely what is happening. And the idea of the script pinging a 3rd party source to check if the server is up... that's actually exactly what this script is doing. Checking if the server is up, and recording that info into sql for other scripts to make use of. AND also reset tcp and ultimately reboot if we're not up. But rather than do pings, I'm making an http request so an 8 byte text file I have hosted at 3 different places online, and I go through and test each one and only consider myself offline if they all fail. I do prefer doing it this way vs doing it with pings to servers I don't really have any control over or connection to at all. 

Buuuuuuuuuuuuut back so what you said, that its probably internal calls causing curl to fail... is there no way at all to make PHP give up on a function call after a given amount of time if it doesn't return? Like a line specific/statement specific/function call specific way to time something out? It seems like something that shouldn't be that hard. 

[l008com]

It also just occurred to me that the worst case scenario here is that I can actually just use exec( ) to call CLI curl like I used to back in the day, and wrap that in a cli Timeout call. That should work. Not the preferred solution but not as bad as wrapping native php curl calls in a separate php script. 

[l008com]

After further pondering, I'm thinking I could just write my own curl wrapper function and put it in a shared file on my server. You would code it in a way very similar to how you use curl_exec() but mine would call `timeout  curl;` instead of using php curl, to force the timeout to apply. Then I could use it all over my server. So maybe ill just do that, we'll see. 

[gizmola]

Maybe it wasn't clear, but what I was suggesting was adding a ping check to a server that is essentially guaranteed to be up, purely to insure the network is working.  After that, your curl calls should work.  

Curl utilizes its library (lib_curl) which is what PHP uses.  There is no benefit to execing a local copy of curl, as it is not going to perform any differently than the php extension does.

 

[l008com]

The benefit of using cli curl is that i can put it after a timeout cli call so it should properly fail after the timeout no matter what. 

[l008com]

Ok so here is the update. I switched my script over to using cli `curl` with a cli `timeout` instead of built in phpcurl and on the first test, the script behaved much more normally. It behaved the way you would expect it to behave based on the way all of it's internal timing works. So I think this was a success. A bummer I needed to leave php for it to work properly but what can you do. Also I ended up not writing a whole elaborate wrapper for this, since the list of cli flags and phpcurl options is so long. I could just add the ones I use, and add more as I run in to them. But I just wrote this directly into this script for now and thats it (for now). 

While typing this message, it just had a little freakout that I don't understand, but it seems to be working properly now. So I dunno, it could have just been an actual ddos syn flood right when I was switching the script over. 

But anyway, I replaced the following code:

unset($return,$ch);
$ch = curl_init($url);
curl_setopt($ch,CURLOPT_USERAGENT,"Mozilla/5.0 (Ccccustom user-agent)");
curl_setopt($ch,CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,10);
curl_setopt($ch,CURLOPT_TIMEOUT,10);
curl_setopt($ch,CURLOPT_INTERFACE,OPERATING_IP);
$return = curl_exec($ch);

if ( ( curl_getinfo($ch,CURLINFO_HTTP_CODE) == "200" ) and ( $return == "redacted key" ) )
    { $online++; }

With this code instead: 

$clicurl = "curl -A 'Mozilla/5.0 (Ccccustom user-agent)' -m 10 --interface ".OPERATING_IP." $url";
$check_result = exec("timeout 10s $clicurl");

if ($check_result === "redacted key")
    { $online++; }

Under normal circumstances, it is working perfectly. We'll see how it reacts during abnormal circumstances. In fact, I can reset tcp right now I guess and give it another test. Ack ok something is going heywire but i'm pretty sure its a totally different problem. Like occasional errant characters in the result of curl. 

[l008com]

Well that was fun. I was getting crazy results where the script was working right every time I tested it, yet my server was still resetting it's connection and rebooting like clockwork. An hour or two of fighting it and in the end, the issue was that when launchd runs my script, it needs the full path to `timeout` and without it, it returns nothing. But when I sudo run the script myself, it was fine and returned exactly what i expected it to. 

I forgot the golden rule, don't touch the server before bed. 

[gizmola]

3 hours ago, l008com said:

Well that was fun. I was getting crazy results where the script was working right every time I tested it, yet my server was still resetting it's connection and rebooting like clockwork. An hour or two of fighting it and in the end, the issue was that when launchd runs my script, it needs the full path to `timeout` and without it, it returns nothing. But when I sudo run the script myself, it was fine and returned exactly what i expected it to. 

For future reference it would be good to know what operating system(s) you are using.  This is the first time you've mentioned launchd, or provided much insight into the design or purpose of the application.   It's hard to know if we ever had a good idea of what your actual issues are/were, or whether we were able to contribute anything of any value, but hopefully you made some headway.  

[l008com]

Its a MacOS server, this script IS my "uptime checker" script that protects against ddos by reseting TCP when the server is bumped offline. And after 3 attempts at a TCP reset, if its still offline, it reboots the whole server. The last thing that tripped me up was that running the script through the terminal via sudo, the script knew where `timeout` was. But running the script via launchd, it didn't know where it was and needed the full path. But it didn't need the full path for curl. I'm not really clear on the details of all that but also, doesn't really matter. Its all working now and that 10s timeout really works which is really good for getting the server back up fast when this script has to act. 

[gizmola]

When you mentioned Launchd, that was the first time I realized macos was involved.  

As you've found, with cron and launchd, and systemd, it tends to be the case that shell scripts will not have the full interactive environment your user has.  This includes environment variables like PATH, where Launchd will default to a minimal PATH that won't include common directories like /usr/local/bin or /usr/local/sbin, or /opt/homebrew, so you basically have to specify the path to scripts you invoke through launchd, either in the .plist file, or doing some things like sourcing .profile/.bashrc/.zshrc files or whatever shell you might be working with.  

As I understand it, you have described this script as one that runs on this mac "server" in order to attempt to detect if it has ben forced offline by DDOS?  Is the server under some sort of constant attack?  I don't see how resetting the network and perhaps later rebooting entirely is a solution for DDOS attacks, which can easily just start up again as soon as the server is once again reachable, putting it into a constant state of resetting/rebooting.  I'm not really sure what the server does for you, but I also would be curious if you are using a proxy service like cloudflare?