Pro.Luv Posted December 6, 2008 Share Posted December 6, 2008 Hi, I need to build a spider like this: http://www.webconfs.com/search-engine-spider-simulator.php Can anyone show me where to start I just want to be able to get all the links the way this spider does not the meta details just the links, I would really appreciate it. Thanks Reply With Quote Link to comment https://forums.phpfreaks.com/topic/135788-follow-links/ Share on other sites More sharing options...
gaza165 Posted December 6, 2008 Share Posted December 6, 2008 Im not 100% about this but try looking into cURL Garry Link to comment https://forums.phpfreaks.com/topic/135788-follow-links/#findComment-707575 Share on other sites More sharing options...
curtis Posted December 6, 2008 Share Posted December 6, 2008 You might try breaking the problem down into smaller parts to get a better idea of where to start. Here's an idea for a specific spec. [*]Get a URL to search from the user. [*]Search data at the URL. [*]Search for URI patterns. [*]Store each URI. A tip: at this point, you should check if an entry already exists. Link to comment https://forums.phpfreaks.com/topic/135788-follow-links/#findComment-707577 Share on other sites More sharing options...
curtis Posted December 6, 2008 Share Posted December 6, 2008 Im not 100% about this but try looking into cURL That shouldn't be necessary, unless the need arises to handle things like authentication, cookies, or SSL. Link to comment https://forums.phpfreaks.com/topic/135788-follow-links/#findComment-707582 Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.