mattm1712 Posted November 22, 2012 Share Posted November 22, 2012 Firstly I want to know the pros and cons of scraping a website for data, eg prices. And does anyone know the best website to show me how to do it? Cheers matt Quote Link to comment Share on other sites More sharing options...
jcbones Posted November 22, 2012 Share Posted November 22, 2012 Pro's = You get data that you didn't work for. Con's = Most websites ban it in their TOS. The easiest way to do it IMHO is Simple HTML DOM parser. Quote Link to comment Share on other sites More sharing options...
Muddy_Funster Posted November 22, 2012 Share Posted November 22, 2012 Most website "data" is protected in some form or other from other people scraping it, usualy in the end user agreement, or in the terms and conditions. Sites like google run feeds from the original website, set up by the owner, to list things like price information in a dynamic/near-dynamic way. Quote Link to comment Share on other sites More sharing options...
topcodedeveloper Posted October 2, 2013 Share Posted October 2, 2013 (edited) ok, I've been struggling with this topic for about 6 months now, this is something I just can't seem to grasp; I've been trying to scrap content from this site: http://www.trulos.com/Trulos/Post-Truck-Loads/free-truckload-board.aspx?STATE=AL&Equipment=Reefer to add to my own load board, I seem to be stuck, I've tried just about every task: but this site seem to not have class id's just simple <td> and tables, could someone help me out, and maybe provide an sample I'm trying to scrap the loads that they have posted, 7/24/2013 Montgomery AL Robert LA Full 53 Reefer 0.00 21 0 Dispatch (800) 580-3101 Map 7/24/2013 Montgomery AL Birmingham AL Full 48 Reefer 0.00 40 1 Dispatch (800) 580-3101 Map Then I want to add it to my sql database that would show in my site: http://www.loadsready.com/ any help would be greatly appreciated. Edited October 2, 2013 by topcodedeveloper Quote Link to comment Share on other sites More sharing options...
.josh Posted October 2, 2013 Share Posted October 2, 2013 so you want to steal someone else's content and you don't know how? Perhaps you should change your username.. Quote Link to comment Share on other sites More sharing options...
topcodedeveloper Posted October 2, 2013 Share Posted October 2, 2013 (edited) Wow, that's not quite the answer i was looking for, i would expect a more professional answer. Now I remember why I don't user these type site's oh and by the way you might want to change your profile pic, you resemble one one of inmates were I happen to work. Thanks for the help!!!! OH Just an FYI, don't matter to me if you decide to band my from this site, it would just show me how UN professional you are. Virgin Media IP (82.8.81.100) Bournemouth, United Kingdom Edited October 2, 2013 by topcodedeveloper Quote Link to comment Share on other sites More sharing options...
.josh Posted October 2, 2013 Share Posted October 2, 2013 Well I'm sorry, but there's no professional way to say you're a bad person for wanting to steal someone else's content. And even if there was, I wouldn't say it that way..because why should I act professional towards someone trying to rob someone else? Also, this is a free, public help site. If you are looking for someone "professional" then go hire someone. However, I would think if someone were professional, they wouldn't be trying to steal other peoples' content, nor agree to be a part in it on any level. So IMO you and I obviously have 2 different ideas of what "professional" means. I think you're looking for the "mob" type, not the "business" type. Quote Link to comment Share on other sites More sharing options...
Psycho Posted October 2, 2013 Share Posted October 2, 2013 (edited) Also, the site you liked to has Term of Service which state, in part: Except for public domain material, the content on this site is protected by intellectual property laws, including U.S. copyright laws. You are hereby granted a non-exclusive license to use the material contained within this site, including transmitting email of individual stories to friends directly from this site. You are also granted a limited license to print copies of any Content posted at the site, but only for your personal use. Except as expressly provided above, all rights are reserved. Among other things, except to the extent required for the limited purpose of reviewing material on our site, electronic reproduction, adaptation, distribution, performance or display is prohibited. Changes to or deletion of author attribution or copyright notices are prohibited. Commercial use of any of the Content is strictly prohibited. Use of any of our trademarks as metatags on other web sites also is strictly prohibited. You may not display this website in frames (or any of the Content via in-line links) without our express written permission, which may be requested by contacting us at Support@Trulos.com. You may, however, establish ordinary links to our site without our written permission. dsa Also ok, I've been struggling with this topic for about 6 months now, this is something I just can't seem to grasp; Six months? Edited October 2, 2013 by Psycho Quote Link to comment Share on other sites More sharing options...
.josh Posted October 2, 2013 Share Posted October 2, 2013 I think what he meant to say was "struggling to find someone shady enough to be willing to help" Quote Link to comment Share on other sites More sharing options...
Ch0cu3r Posted October 2, 2013 Share Posted October 2, 2013 OH Just an FYI, don't matter to me if you decide to band my from this site, it would just show me how UN professional you are. Virgin Media IP (82.8.81.100) Bournemouth, United Kingdom Yeah that was me. How is looking at your site unprofessional? Quote Link to comment Share on other sites More sharing options...
.josh Posted October 2, 2013 Share Posted October 2, 2013 Oh. See I thought he was posting his own info as a challenge for me to ban him. Now that I see that he's posting other people's info that visit his site, I see now that he is most certainly a hypocrite about the whole "being professional" thing. Quote Link to comment Share on other sites More sharing options...
QuickOldCar Posted October 2, 2013 Share Posted October 2, 2013 Calling someone a thief and not being able to code as a response wasn't too nice, I surely wouldn't like it, neither would you. They could have easily had special permission from the owners of http://www.trulos.com I doubt was the case or would have stated that fact, but should think of this the next time someone is asking for help on the same subject. Quote Link to comment Share on other sites More sharing options...
.josh Posted October 2, 2013 Share Posted October 2, 2013 Right. I get what you're saying, but when literally 99.999% of all cases of scraping another site involve people NOT getting "special permission" from the owner, I think my money is on thief. If someone were determined to get permission for that sort of thing, they should work with them to create a better method in the first place, like make an API to request the info from the server. Quote Link to comment Share on other sites More sharing options...
QuickOldCar Posted October 2, 2013 Share Posted October 2, 2013 I agree with you, but person running other site may know even less, or not willing to pay someone to get it done. Quote Link to comment Share on other sites More sharing options...
.josh Posted October 2, 2013 Share Posted October 2, 2013 Nonetheless.. you don't just show up somewhere asking how to scrape someone else's site and expect it not to be taken as "I'm trying to steal someone else's content." It ranks up there with people posting "hack my site" threads. It is common knowledge that these sorts of questions almost certainly come from people trying to do things they shouldn't be doing. The legit people are the exception, not the norm. So it is on them to prove that they are legit, and I do not feel bad for assuming they are not legit unless proven otherwise. I mean come on.. it's like if some random dude came up to you and asked you for help breaking into a house and taking things. Not even saying he totally knows the owner and it's totally okay. I do not think it is unfair of me to assume he's a thief asking for help to steal something. Quote Link to comment Share on other sites More sharing options...
topcodedeveloper Posted October 3, 2013 Share Posted October 3, 2013 I will admit I was a little butt hurt, with the remark made, so yes I may have been a little harsh my self in responding to the thread. Considering You may not know me, nor I know you, how can you judge one person just based on fact, with out all the evidence in front of you. I assure you that my attentions are good, in fact I've tried contacting the sites owner, finding out their online form has a SQL error, so I've tried contacting them via local email instead, and not heard from them yet. I was simple trying to reach out for some assistance, but it seems that I may have came to the wrong place. And as for capturing your IP address and location I'm able to pull this information from my server. So again, how quick one is to judge. And!! as for coding I'm sure you may not no every language there is in coding, When I stated I've been struggling with this issue for the past six months, I should of mentioned that I also have a full time job, and only do this on the side, in which doesn't give me much time. So with that said, I'll apologizes, for my action and with draw my account from the site and just move along. Thanks any way for all your help.. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.