taith Posted May 26, 2007 Share Posted May 26, 2007 about there trolling... do search engines use cURL to periodically scan pages? or do they do it some other way? Quote Link to comment https://forums.phpfreaks.com/topic/53048-how-do-search-engines-do-it/ Share on other sites More sharing options...
mlavwilson Posted May 26, 2007 Share Posted May 26, 2007 There is no way that cURL has anything to do with it. They spend milllions to build crawlers. There are some open source projects that are similar to how the big guys do things. Quote Link to comment https://forums.phpfreaks.com/topic/53048-how-do-search-engines-do-it/#findComment-262044 Share on other sites More sharing options...
taith Posted May 26, 2007 Author Share Posted May 26, 2007 sweet deal... cuz i found a few ways to block curl just making sure i wont be blocking the big guys... lol Quote Link to comment https://forums.phpfreaks.com/topic/53048-how-do-search-engines-do-it/#findComment-262046 Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.