Crawler (2)

[hackalive]

Okay so thorpe has locked my post and pointed out it is far too vague. So here I go again.

 

Firstly I know how to build a basic PHP cURL web page crawler. Now my question is more specific to this, how do I get Site Stats to recognise the crawler as other ones such as Google Bot and MLBot are listed (MLbot http://www.metadatalabs.com/mlbot, so mine would be HackAliveCrawler htpp://mysite.com/hackalivecrawler or similar...). And how do I get it to recognise ROBOTS meta tags, ie NOINDEX, NOFOLLOW, or INDEX, FOLLOW etc and for robots.txt (to do this I need to first part to work, the name set part (eg GoogleBot or MLBot etc). Hope this is less vague and can produce some answers.

 

Links to tutorials or other forums that will achieve these desired result as well as personal opinion and commernts are most welcome. Many thanks in advance.

[ignace]

Now my question is more specific to this, how do I get Site Stats to recognise the crawler as other ones such as Google Bot and MLBot are listed

 

The User-Agent directive.

 

And how do I get it to recognise ROBOTS meta tags, ie NOINDEX, NOFOLLOW, or INDEX, FOLLOW etc and for robots.txt (to do this I need to first part to work, the name set part (eg GoogleBot or MLBot etc).

 

Upon crawling you check if a robots.txt file exists and adhere to it. You crawl the pages (or pages within directories) which are not blacklisted in the robots.txt file and search for a ROBOTS meta tag. If the robots meta-tag contains you act accordingly.

[hackalive]

Okay thanks once again ignace, just unsure how to set the User-Agent directive, if you know a good link for this or can tell me it will be much apprreciated.

 

Thanks again ignace

[hackalive]

oh bye the way it will be running as part of a cron job and somethimes via direct browser. Thought I might just add this as it may affect the User-Agent directive mentioned

[ignace]

http://www.w3.org/Protocols/HTTP/HTRQ_Headers.html#user-agent

 

It's part of the request headers.

[hackalive]

Okay so I get it is part of the header request so how then can I set my own like MLBot and Google Crawler for when it is executed via a cron job?

[hackalive]

Becuase MLBot have

 

MLBot (http://www.metadatalabs.com/mlbot)

 

this is what I want to achieve

[ignace]

User-Agent: HackAliveCrawler htpp://mysite.com/hackalivecrawler

[Adam]

I'm guessing you mean with PHP? You can send it as a header:

 

header('User-Agent: HackAliveCrawler htpp://mysite.com/hackalivecrawler');

[hackalive]

Yes with PHP thanks a million once again to ignace (who many a time has saved me from trauling the entiere internet) and also to MrAdam thanks.

[Adam]

Sorry, if you're using cURL you'll need to set the cURL option:

 

curl_setopt($ch, CURLOPT_USERAGENT, 'HackAliveCrawler htpp://mysite.com/hackalivecrawler');

[hackalive]

thanks again

[hackalive]

okay slight problem, this code

<?php
header('User-Agent: HackAliveCrawler http://mysite.com/hackalivecrawler');
echo $_SERVER['HTTP_USER_AGENT'];
?>

 

returns

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C; .NET4.0E)

 

not

HackAliveCrawler http://mysite.com/hackalivecrawler

 

any reason why?

 

if I use USER_AGENT and not HTTP_USER_AGENT it returns nothing

 

thanks once again

 

[Adam]

I believe that'll be the browser over-writing whatever you send. If you're using cURL to crawl the pages as I mentioned before just send it as a cURL option.

 

Edit: Should add I'm only speculating about the browser, I'm not fully sure how that part works.

[hackalive]

yeah I am just doing it on a smaple page at the moment with no curl, just exactly what I have posted

[ignace]

header obviously doesn't work because it sends response headers not request headers you need cURL in order to do that.

[hackalive]

MrAdam

Warning: curl_setopt(): supplied argument is not a valid cURL handle resource

 

for this code

<?php
curl_setopt($ch, CURLOPT_USERAGENT, 'HackAliveCrawler htpp://mysite.com/hackalivecrawler');
echo $_SERVER['HTTP_USER_AGENT'];
echo "|";
echo $_SERVER['USER_AGENT'];
?>

[trq]

MrAdam

Warning: curl_setopt(): supplied argument is not a valid cURL handle resource

 

for this code

<?php
curl_setopt($ch, CURLOPT_USERAGENT, 'HackAliveCrawler htpp://mysite.com/hackalivecrawler');
echo $_SERVER['HTTP_USER_AGENT'];
echo "|";
echo $_SERVER['USER_AGENT'];
?>

 

You haven't defined $ch.

[hackalive]

okay thorpe so what do I set $ch as?

[trq]

Why not take a look at the manual for the curl extension?

[Adam]

MrAdam

Warning: curl_setopt(): supplied argument is not a valid cURL handle resource

 

for this code

<?php
curl_setopt($ch, CURLOPT_USERAGENT, 'HackAliveCrawler htpp://mysite.com/hackalivecrawler');
echo $_SERVER['HTTP_USER_AGENT'];
echo "|";
echo $_SERVER['USER_AGENT'];
?>

 

That's not how it works. As thorpe has just replied, you need to define $ch (look at the cURL examples on the manual), but even then it only sets the user agent for the request you're going to make with cURL.

[hackalive]

<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, 'HackAliveCrawler htpp://mysite.com/hackalivecrawler');
?>

 

Yeah so if I do the above then do the crawl using cURL the user Agent will be recorded as HackAliveCrawler yes?

[trq]

And this is the same guy going on about designing and implementing his own OS virtualization?

[hackalive]

Well thorpe my OS virtulization project has started and is going well, so umm yeah. Just trying to clean this cURL thing out of my inbox (so to speak)

[trq]

I'm sure it is.