Pro.Luv Posted December 6, 2008 Share Posted December 6, 2008 Hi, I need to build a spider like this: http://www.webconfs.com/search-engine-spider-simulator.php Can anyone show me where to start I just want to be able to get all the links the way this spider does not the meta details just the links, I would really appreciate it. Thanks Reply With Quote Quote Link to comment Share on other sites More sharing options...
gaza165 Posted December 6, 2008 Share Posted December 6, 2008 Im not 100% about this but try looking into cURL Garry Quote Link to comment Share on other sites More sharing options...
curtis Posted December 6, 2008 Share Posted December 6, 2008 You might try breaking the problem down into smaller parts to get a better idea of where to start. Here's an idea for a specific spec. [*]Get a URL to search from the user. [*]Search data at the URL. [*]Search for URI patterns. [*]Store each URI. A tip: at this point, you should check if an entry already exists. Quote Link to comment Share on other sites More sharing options...
curtis Posted December 6, 2008 Share Posted December 6, 2008 Im not 100% about this but try looking into cURL That shouldn't be necessary, unless the need arises to handle things like authentication, cookies, or SSL. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.