Creating A News Feed From Other Sites

[Zeradin]

I want to make a feed that goes to other websites and updates when they post a story. Not immediately, but just around the same time. I guess a better way to describe it would be i want a news feed that populates from other news feeds. The only way I can imagine this working is ajax that reads data from the pages in intervals and knows where to start looking for information and a table that logs the last information so it doesn't repeat.

 

Is there a better way to do this?

Thanks.

[flyhoney]

Are you familiar with RSS feeds?  RSS feeds are a very good way to retrieve information from other sites.  There is also a lot of PHP code already written to handle RSS feeds.

[Zeradin]

Yeah, I made an RSS feed a couple weeks ago, but I updated it using the submit php when the site posted a news story, I do not know how to harvest data from websites when I have no access to their code.

[JonnoTheDev]

If you need to grab lets say a block of text from a page you could use CURL to get the page, then a series of regular expressions to extract the portion of the page that you require. You will need delimeters to specify where the blocks start and end (maybe a set of div tags within the HTML)

[flyhoney]

I'm not saying make an RSS feed, I'm saying use the RSS feeds available on other news sites.  Magpie is a really great RSS feed parser for PHP. 

[JonnoTheDev]

I do not know how to harvest data from websites when I have no access to their code

 

Why do you need access to code? If you can see information on-screen then you can grab it.

[Zeradin]

I do not know how to harvest data from websites when I have no access to their code

 

Why do you need access to code? If you can see information on-screen then you can grab it.

 

Because the way I'd optimally want them to do it is have their updates write to my xml file. I want my news feed to update when they update theirs.

 

If you need to grab lets say a block of text from a page you could use CURL to get the page, then a series of regular expressions to extract the portion of the page that you require. You will need delimeters to specify where the blocks start and end (maybe a set of div tags within the HTML)

 

I'd still need to ajax to their site in regular intervals, no?

[bluesoul]

You'd likely want to use Cron.  Most RSS feeds update once an hour.

[Zeradin]

You'd likely want to use Cron.  Most RSS feeds update once an hour.

 

I have no idea what cron is after looking at it's wiki page. Well I do, but I have no idea how I'd ever implement it. I mean, the sites I'm fetching data from don't all have rss feeds.

[flyhoney]

There are all sorts of ways to do this.  Are you using a database?  For the sites that have RSS feeds, that's simple, just parse the feeds and display on your site.  If you want to make it AJAXy that's your decision, but it doesn't have to, it could just update on pageload.  For site's that don't use RSS, you will need to use CURL (if your server supports opening remote files, you can use file_get_contents) to grab the HTML and parse it for the news articles using regular expressions.

[Zeradin]

Do you know a good place to learn about CURL?

[flyhoney]

I would just google 'php curl' and you will find tons of examples.  You will need to simply get the contents of a web page as a string.  So something like this should work:

 

<?php
$ch = curl_init();

// there are a lot of other CURL options you might want to check out
// http://us2.php.net/manual/en/function.curl-setopt.php
curl_setopt($ch, CURLOPT_URL, "http://www.myurl.com/");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

$result = curl_exec($ch);

curl_close($ch);

echo $result;
?>