CURL Bad Request nginx

[imperium2335]

Hi, I'm trying to use CURL to scape pages.

 

But the odd few sites return a blank page with a big "Bad Request 400" at the top of the page and below it in small writting I see "nginx" (I know this because I've used echo to output the page). If I go to the page in my browser it exists and looks fine.

 

Anyone know what or how to fix this?

 

I'm using the following CURL stuff:

 

    $ch=curl_init();
curl_setopt($ch,CURLOPT_USERAGENT,$agents[rand(0, 4)]) ;
curl_setopt($ch,CURLOPT_FOLLOWLOCATION,TRUE) ;
curl_setopt($ch,CURLOPT_URL,$urltoscan);
curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,20); //Time out after 20 seconds.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //Return the page as a string.
$currentsite = curl_exec($ch); //Execute above commands.
curl_close($ch);
    echo $currentsite ;

 

What is nginx?

[bonaparte]

From the Wikipedia

<quote>

nginx (pronounced as "engine X") is a lightweight, high performance web server/reverse proxy

</quote>