xgals Posted March 7, 2013 Share Posted March 7, 2013 I am trying to parse a huge website. The scripts are OK, working fine, however, there is one problem... Somehow the server knows, that the site is being parsed and constantly keeps blocking my IP. I have to restart router every time in order to proceed. Is there any way to avoid or bypass that programmatically? Link to comment https://forums.phpfreaks.com/topic/275376-parsing-a-website/ Share on other sites More sharing options...
Manixat Posted March 7, 2013 Share Posted March 7, 2013 Yes, don't parse. If people have put work into not letting you parse their site, they obviously don't want you to. Link to comment https://forums.phpfreaks.com/topic/275376-parsing-a-website/#findComment-1417269 Share on other sites More sharing options...
xgals Posted March 7, 2013 Author Share Posted March 7, 2013 Yes, don't parse. If people have put work into not letting you parse their site, they obviously don't want you to. I did not ask for moral advice, only technical. Link to comment https://forums.phpfreaks.com/topic/275376-parsing-a-website/#findComment-1417273 Share on other sites More sharing options...
Manixat Posted March 7, 2013 Share Posted March 7, 2013 I suppose you won't get any unmoral technical advice in this forum. Link to comment https://forums.phpfreaks.com/topic/275376-parsing-a-website/#findComment-1417274 Share on other sites More sharing options...
AyKay47 Posted March 7, 2013 Share Posted March 7, 2013 No advice on this topic will be given on this forum. Your IP is being blocked because what you are doing is most likely against their TOS. Link to comment https://forums.phpfreaks.com/topic/275376-parsing-a-website/#findComment-1417285 Share on other sites More sharing options...
ignace Posted March 7, 2013 Share Posted March 7, 2013 It could be blocked for several reasons though, most likely you are making too many requests. Put a decent sleep (1 second or more) amount after X requests. Link to comment https://forums.phpfreaks.com/topic/275376-parsing-a-website/#findComment-1417294 Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.