anon Posted December 20, 2007 Share Posted December 20, 2007 Hi, I want to build a crawler out of PHP or PERL. I want to know which one on average is the least resource intensive? Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/ Share on other sites More sharing options...
trq Posted December 20, 2007 Share Posted December 20, 2007 Just way too many variables come into play, it really depends on what your doing and how your doing it. Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/#findComment-419854 Share on other sites More sharing options...
Ninjakreborn Posted December 20, 2007 Share Posted December 20, 2007 It sounds like your trying to decide between the 2. I saw this post similar before. What are you wanting a web crawler for, give the reason and we can better give you feedback on which way would be the best one to proceed. Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/#findComment-419870 Share on other sites More sharing options...
anon Posted December 20, 2007 Author Share Posted December 20, 2007 I want the crawler to download HTML files from sites specified in an already existing database ??? Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/#findComment-419886 Share on other sites More sharing options...
Ninjakreborn Posted December 20, 2007 Share Posted December 20, 2007 Download xhtml pages for archival. You mean grab all of the source content. For that you are going to need a crawler, and xhtml parser. PHP/Perl would both be pretty good at it. But I prefer PHP for anything I can, there won't be much of a difference. Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/#findComment-419890 Share on other sites More sharing options...
trq Posted December 20, 2007 Share Posted December 20, 2007 I want the crawler to download HTML files from sites specified in an already existing database Does it actually need to crawl or just go to specific addressses? Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/#findComment-419894 Share on other sites More sharing options...
anon Posted December 20, 2007 Author Share Posted December 20, 2007 The database tells the crawler what sites to index. So, it downloads the HTML page and stores it in another Database. Kinda like Google. Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/#findComment-419904 Share on other sites More sharing options...
trq Posted December 20, 2007 Share Posted December 20, 2007 I'll ask that again. Does it need to crawl links within each site, or just save a specific file? Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/#findComment-419908 Share on other sites More sharing options...
anon Posted December 20, 2007 Author Share Posted December 20, 2007 No. I want it to only download HTML files specified by the database. So it downloads all HTML files within that site. Sorry, wasn't sure what you meant. Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/#findComment-419927 Share on other sites More sharing options...
Jessica Posted December 20, 2007 Share Posted December 20, 2007 "No. I want it to only download HTML files specified by the database. So it downloads all HTML files within that site. " That is a contradicting statement. It cannot download only the files listed in the database and all of the files on the site, unless you specify all files on the site in the list. Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/#findComment-419931 Share on other sites More sharing options...
Ninjakreborn Posted December 20, 2007 Share Posted December 20, 2007 It sounds to me like you want to visit a site, and make a copy of the pages on the site and get the information out of them, then database the page.. It's going to be easier to just take a screen shot of the page. In that case your going to want to look at Perl, they have a really good class for carbon copying image's of specific web pages. Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/#findComment-419936 Share on other sites More sharing options...
anon Posted December 20, 2007 Author Share Posted December 20, 2007 Hang on. Let me think this through :-\ A database gives the crawler a list of sites to crawl. By crawling, i mean visiting, storing the sites HTML files, then repeating the process on the next site in the databases list. Does this make sense. @businessman.........1 I only want the text located in head and body. Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/#findComment-419944 Share on other sites More sharing options...
Jessica Posted December 20, 2007 Share Posted December 20, 2007 (I responded to the wrong post, sorry) Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/#findComment-419949 Share on other sites More sharing options...
Ninjakreborn Posted December 20, 2007 Share Posted December 20, 2007 Then you just create the basic crawler using file get contents on the url's. When you get the data from the xhtml files then your going to write a parser. Google "regex" and go through some tutorials on it. You'll want to use regex to get everything out of the head and body sections, leaving the rest. Then from there you can customize it to be more specific based on exactly what you are needing. Note this is against the copyright laws of most site's, as well as against personal site policies. Be ready to get the ip of your server banned from ever being able to visit the site again if you misuse what you are trying to create. Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/#findComment-419952 Share on other sites More sharing options...
anon Posted December 20, 2007 Author Share Posted December 20, 2007 Why is this illegal Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/#findComment-419957 Share on other sites More sharing options...
phpSensei Posted December 20, 2007 Share Posted December 20, 2007 Why is this illegal Some sites just don't allow it, as it is against policies. Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/#findComment-419972 Share on other sites More sharing options...
anon Posted December 20, 2007 Author Share Posted December 20, 2007 but google do it. Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/#findComment-419978 Share on other sites More sharing options...
trq Posted December 21, 2007 Share Posted December 21, 2007 but google do it. What? Link to comment https://forums.phpfreaks.com/topic/82563-how-resource-intensive-is-php-and-perl/#findComment-420017 Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.