Bobafart Posted December 15, 2006 Share Posted December 15, 2006 Hello, I have dumped a number of RSS feeds into a mySQL table. These feeds are news RSS feeds (AP, Reuters, ABC, CNN, CBS news, digg, slashdot, etc)I am having difficulty writing a script that displays the content of each of the feeds WITHOUT showing duplicate articles. Instead I want duplicate articles to be listed under the initial article as "Related Headlines" -- much like Google News does. The question is: How to do this?Example:[i]le: Rumsfeld Resigns from Iraq.Source: Yahoo! News Date: Dec 3, 2006Article Body: Today, Donald Rumsfeld has resigned from Iraq. Millions of Iraqis are partying in the streets.Related Headlines: "Bush Fires Rumsfeld" - ABC News "Rumsfeld is No More" - CNN News[/i]Get the idea? I am using similar_text() to compare the headlines of each article. If the similarity is >70% then the compared headline is to be removed from the array so that it isn't displayed as an independent article but will be displayed as a Related Headline for that article.So here is my code attempt:[code]<?php$query = "select id, headline, intro, body, author, date, source, vote, xmlsitetype from anews1 where xmlsitetype = 0 ORDER BY date DESC LIMIT 10";$result = mysql_query($query); // Go through each news item from the database tablewhile ($rownews = mysql_fetch_assoc($result)){ // find similar_text and flag duplicate stories to be displayed as Related Headlines $dupquery = "SELECT id, headline, intro, body, author, date, source, vote, xmlsitetype from anews1"; echo $dupquery; $dupresult = mysql_query($dupquery); // strip unneeded characters like quotes out of headlines to clean them up: $cleanheadlines = array("\"", "'"); while ($dupcheck = mysql_fetch_assoc($dupresult)){ foreach ($dupcheck as $key => $dupcheck[headline]){ $str1 = str_replace($cleanheadlines, "", addslashes($rownews[headline])); $str2 = str_replace($cleanheadlines, "", addslashes($dupcheck[headline])); echo "<p><b>str1</b> is: ".$str1."<br><b>str2</b> is: ". $str2."<p>"; if (similar_text( $str1, $str2, $p ) > .70){ echo '<b><u>phrases are similar</u></b>'; // flag str2 in the array so that it is not displayed as an independent article, but rather as a Related Headline } else { echo 'phrases not similar'; // if phrases are not similar then print this article as the next independent news article in the news list } echo "Percent: $p%"; } } ?><ol> <li> <strong><a href="<?php echo $rownews[source]; ?>"><?php echo $rownews[headline]; ?></a></strong><br /> <span style="font-size:0.8em; color:#999; height: 10px;">→ <a href="<?php echo $rownews[source]; ?>"><?php echo $rownews[date]; ?></a> | <?php echo $rownews[date]; ?></span><br /> <div style="font-size:1em; color:#000; height: 130px;"><?php echo $rownews[body]; ?></div> </li> <?php } ?> </ol></div>[/code]The problem with this code is that the foreach loop is printing and comparing EVERY element in the array... so that $str2 is assigned each element in the array.. I only want $str2 to be assigned the second array element (which is the headline element) so that I can then compare it with $str1. How do I do this? Quote Link to comment Share on other sites More sharing options...
complex05 Posted December 15, 2006 Share Posted December 15, 2006 [quote]The problem with this code is that the foreach loop is printing and comparing EVERY element in the array... so that $str2 is assigned each element in the array.. I only want $str2 to be assigned the second array element (which is the headline element) so that I can then compare it with $str1.[/quote]I don't quite understand what you're trying to say. Can you try to rephrase what you want this script to do Quote Link to comment Share on other sites More sharing options...
Bobafart Posted December 15, 2006 Author Share Posted December 15, 2006 Let me show you the output:[code]select id, headline, intro, body, author, date, source, vote, xmlsitetype from anews1 where xmlsitetype = 0 ORDER BY date DESC LIMIT 10Recently Posted News Items…SELECT id, headline, intro, body, author, date, source, vote, xmlsitetype from anews1str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 1phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: `Raymond' dad Peter Boyle dies in NYC (AP)phrases are similarPercent: 100%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: Yahoo! News: Top Storiesphrases are similarPercent: 23.684210526316%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: AP - Peter Boyle, the actor who played the hilariously grouchy father on \Everybody Loves Raymond\ as the final note of a distinguished career that also included a memorable role in Mel Brooks\ \Young Frankenstein,\ has died. He was 71.phrases are similarPercent: 10.047846889952%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 00000000000000phrases are similarPercent: 3.030303030303%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: http://us.rd.yahoo.com/dailynews/rss/topstories/*http://news.yahoo.com/s/ap/20061213/ap_on_en_tv/obit_boylephrases are similarPercent: 13.836477987421%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 1phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 0phrases are similarPercent: 3.7735849056604%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 2phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: Kevorkian to be paroled in June (AP)phrases are similarPercent: 49.462365591398%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: Yahoo! News: Top Storiesphrases are similarPercent: 23.684210526316%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: AP - After more than eight years behind bars for murder, an ailing Dr. Jack Kevorkian will be paroled in June on a promise not to help anyone else commit suicide, prison officials said Wednesday.phrases are similarPercent: 6.0998151571165%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 00000000000000phrases are similarPercent: 3.030303030303%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: http://us.rd.yahoo.com/dailynews/rss/topstories/*http://news.yahoo.com/s/ap/20061213/ap_on_re_us/kevorkian_parolephrases are similarPercent: 14.545454545455%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 1phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 0phrases are similarPercent: 3.7735849056604%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 3phrases are similarPercent: 3.7735849056604%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: \\\Girls Gone Wild\\\ Producer Sentencedphrases are similarPercent: 21.739130434783%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: ABC News: Top Storiesphrases are similarPercent: 19.178082191781%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: \Girls Gone Wild\ Producer Gets Community Service Over Use of Drunken Underage Girls in Videosphrases are similarPercent: 26.027397260274%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 00000000000000phrases are similarPercent: 3.030303030303%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: http://abcnews.go.com/Entertainment/LegalCenter/wireStory?id=2723150&CMP=OTC-RSSFeeds0312phrases are similarPercent: 13.793103448276%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 1phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 0phrases are similarPercent: 3.7735849056604%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 4phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: Officials: Lettuce Source of E. Coliphrases are similarPercent: 15.909090909091%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: ABC News: Top Storiesphrases are similarPercent: 19.178082191781%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: Health Officials Say Lettuce Suspected Source of E. Coli Outbreak Linked to Taco Bellphrases are similarPercent: 16.058394160584%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 00000000000000phrases are similarPercent: 3.030303030303%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: http://abcnews.go.com/Business/wireStory?id=2724098&CMP=OTC-RSSFeeds0312phrases are similarPercent: 6.25%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 1phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 0phrases are similarPercent: 3.7735849056604%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 5phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: Senate control hinges on \stroke\phrases are similarPercent: 23.529411764706%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: BBC News | News Front Page | World Editionphrases are similarPercent: 21.276595744681%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: A US senator suffers a possible stroke, raising the possibility that control of the Senate could return to the Republicans.phrases are similarPercent: 22.857142857143%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 00000000000000phrases are similarPercent: 3.030303030303%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: http://news.bbc.co.uk/go/rss/-/2/hi/americas/6178065.stmphrases are similarPercent: 11.111111111111%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 1phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 0phrases are similarPercent: 3.7735849056604%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 6phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: Lettuce Implicated in Taco Bell Outbreakphrases are similarPercent: 23.913043478261%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: ABC News: Top Storiesphrases are similarPercent: 19.178082191781%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: Lettuce Implicated As Probable Cause of E. Coli Outbreak Linked to Taco Bell, Officials Sayphrases are similarPercent: 18.181818181818%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 00000000000000phrases are similarPercent: 3.030303030303%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: http://abcnews.go.com/US/wireStory?id=2724086&CMP=OTC-RSSFeeds0312phrases are similarPercent: 6.5573770491803%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 1phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 0phrases are similarPercent: 3.7735849056604%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 7phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: Lettuce probable cause of Taco Bell e. coli: FDA (Reuters)phrases are similarPercent: 31.304347826087%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: Yahoo! News: Top Storiesphrases are similarPercent: 23.684210526316%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: Reuters - Lettuce at Yum Brands Inc. Taco Bell restaurants was the most likely cause of an E.coli outbreak that has sickened dozens of people in the northeastern United States, an official said on a Food and Drug Administration conference call on Wednesday.phrases are similarPercent: 4.9180327868852%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 00000000000000phrases are similarPercent: 3.030303030303%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: http://us.rd.yahoo.com/dailynews/rss/topstories/*http://news.yahoo.com/s/nm/20061213/ts_nm/tacobell_lettuce_dcphrases are similarPercent: 13.58024691358%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 1phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 0phrases are similarPercent: 3.7735849056604%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 8phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: Lettuce implicated in Taco Bell outbreak (AP)phrases are similarPercent: 37.254901960784%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: Yahoo! News: Top Storiesphrases are similarPercent: 23.684210526316%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: AP - Lettuce was the most likely source of an outbreak of E. coli linked to Taco Bell, federal health officials said Wednesday.phrases are similarPercent: 6.0732984293194%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 00000000000000phrases are similarPercent: 3.030303030303%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: http://us.rd.yahoo.com/dailynews/rss/topstories/*http://news.yahoo.com/s/ap/20061213/ap_on_re_us/e_coli_outbreak_taco_bellphrases are similarPercent: 12.64367816092%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 1phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 0phrases are similarPercent: 3.7735849056604%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 9phrases are similarPercent: 3.7735849056604%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: Sen. Johnson suffers possible stroke (AP)phrases are similarPercent: 42.857142857143%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: Yahoo! News: Top Storiesphrases are similarPercent: 23.684210526316%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: AP - Democratic Sen. Tim Johnson of South Dakota suffered a possible stroke Wednesday and was taken to a Washington hospital, his office said. If he should be unable to continue to serve, it could halt the scheduled Democratic takeover of the Senate.phrases are similarPercent: 4.1036717062635%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: phrases not similarPercent: 0%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: 00000000000000phrases are similarPercent: 3.030303030303%str1 is: `Raymond' dad Peter Boyle dies in NYC (AP)str2 is: http://us.rd.yahoo.com/dailynews/rss/topstories/*http://news.yahoo.com/s/ap/20061213/ap_on_go_co/johnsonphrases are similarPercent: 14.102564102564%[/code] Quote Link to comment Share on other sites More sharing options...
Bobafart Posted December 15, 2006 Author Share Posted December 15, 2006 I just want $str2 to display headlines so that the $str1 headline is compared to the $str2 headline Quote Link to comment Share on other sites More sharing options...
complex05 Posted December 15, 2006 Share Posted December 15, 2006 ah now i see. I would recommend doing two loops.$query1 = "SELECT headline FROM articles";$different_articles = array();while($data = mysql_fetch_assoc($query1)){ $headline = $data["headline"]; $newquery = "SELECT id,headline FROM articles WHERE headline != '$headline'"; while($arr = mysql_fetch_assoc($newquery)) { $checkheadline = $arr["headline"]; if (similar_text( $headline, $checkheadline, $p ) > .70) { continue; } else { $different_articles[] = $checkheadline; } }} Quote Link to comment Share on other sites More sharing options...
complex05 Posted December 15, 2006 Share Posted December 15, 2006 err sorry, put the article ID in the $different_articles[] array, then just display them later (since u have all the ids shouldn't be difficult) Quote Link to comment Share on other sites More sharing options...
Bobafart Posted December 16, 2006 Author Share Posted December 16, 2006 ok, first, thanks for the help! I took your advice and here is the new code:[code]<?php$query = "select id, headline, intro, body, author, date, source, vote, xmlsitetype from anews1 where xmlsitetype = 0 ORDER BY date DESC LIMIT 10";$result = mysql_query($query); echo $query . "<p>";$different_articles = array();// Go through each news item from the database tablewhile ($rownews = mysql_fetch_assoc($result)){ // find similar_text and remove duplicate stories $dupequery = "SELECT id, headline, intro, body, author, date, source, vote, xmlsitetype from anews1 where headline != '$rownews[headline]'"; echo $dupequery; $duperesult = mysql_query($dupequery); // strip unneeded characters out of headlines to clean them up: $cleanheadlines = array("\"", "'", "/", "the", "<", ">", "cause", "or", "and", "but", "are", "in", "of", "(AP)", "(Reuters)"); while ($dupecheck = mysql_fetch_assoc($duperesult)){ $str1 = str_replace($cleanheadlines, "", addslashes($rownews[headline])); $str2 = str_replace($cleanheadlines, "", addslashes($dupecheck[headline])); echo "<p><b>str1</b> is: ".$str1."<br><b>str2</b> is: ". $str2."<p>"; if (similar_text( $str1, $str2, $p ) > .40){ echo 'phrases are NOT similar'; echo "Percent: $p%"; continue; // remove str2 from the array so that it is not displayed below } else { $different_article[] = $dupecheck[id]; echo 'phrases are similar'; echo "Percent: $p%"; print_r($different_article); } }?> <li> <strong><a href="<?php echo $rownews[source]; ?>"><?php echo $rownews[headline]; ?></a></strong><br /> <span style="font-size:0.8em; color:#999; height: 10px;">→ <a href="<?php echo $rownews[source]; ?>"><?php echo $rownews[date]; ?></a> | <?php echo $rownews[date]; ?></span><br /> <div style="font-size:1em; color:#000; height: 130px;"><?php echo $rownews[body]; ?></div> </li><?php } ?>[/code]and here is the output of the same code:[code]select id, headline, intro, body, author, date, source, vote, xmlsitetype from anews1 where xmlsitetype = 0 ORDER BY date DESC LIMIT 10SELECT id, headline, intro, body, author, date, source, vote, xmlsitetype from anews1 where headline != '`Raymond' dad Peter Boyle dies in NYC (AP)'str1 is: `Raymond' dad Peter Boyle dies NYCstr2 is: Kevkian to be paroled Junephrases are NOT similarPercent: 32.911392405063%str1 is: `Raymond' dad Peter Boyle dies NYCstr2 is: \\\Girls Gone Wild\\\ Producer Sentencedphrases are NOT similarPercent: 23.255813953488%str1 is: `Raymond' dad Peter Boyle dies NYCstr2 is: Officials: Lettuce Source E. Coliphrases are NOT similarPercent: 25%str1 is: `Raymond' dad Peter Boyle dies NYCstr2 is: Senate control hges on \stroke\phrases are NOT similarPercent: 20.779220779221%str1 is: `Raymond' dad Peter Boyle dies NYCstr2 is: Lettuce Implicated Taco Bell Outbreakphrases are NOT similarPercent: 16.666666666667%str1 is: `Raymond' dad Peter Boyle dies NYCstr2 is: Lettuce probable Taco Bell e. coli: FDAphrases are NOT similarPercent: 34.408602150538%str1 is: `Raymond' dad Peter Boyle dies NYCstr2 is: Lettuce implicated Taco Bell outbreakphrases are NOT similarPercent: 28.888888888889%str1 is: `Raymond' dad Peter Boyle dies NYCstr2 is: Sen. Johnson suffers possible strokephrases are NOT similarPercent: 38.636363636364%str1 is: `Raymond' dad Peter Boyle dies NYCstr2 is: Stocks edge up after oil price spikephrases are NOT similarPercent: 38.636363636364%str1 is: `Raymond' dad Peter Boyle dies NYCstr2 is: Red Sox head to Boston with Matsuzakaphrases are NOT similarPercent: 42.696629213483%str1 is: `Raymond' dad Peter Boyle dies NYCstr2 is: Miss USA Dethroned?phrases are NOT similarPercent: 9.2307692307[/code]You will notice that I dropped the similar_text threshold to 40% for testing purposes.Take a look at the last two items in the output.. one is 42% and the other is 9%... yet the end result is the same.... there is something wrong with the IF statement logic. Whether the percent is above or below 40% the first part of the IF statement executes when it should only execute if the percent is above 40%. Shouldn't the 9% be enetered into the $different_articles array?? Strange. Is this a PHP bug? Quote Link to comment Share on other sites More sharing options...
taith Posted December 16, 2006 Share Posted December 16, 2006 here, you might have use for this... it'll give you a very accurate percentage...[code]function comparestrings($str1,$str2){ $str1 = ereg_replace("[^a-z]", ' ', strtolower($str1)); while(strstr($str1,' ')) $str1 = str_replace(' ', ' ', $str1); $str1 = explode(' ', $str1); $str2 = ereg_replace("[^a-z]", ' ', strtolower($str2)); while(strstr($str2,' ')) $str2 = str_replace(' ', ' ', $str2); $str2 = explode(' ', $str2); if(count($str1)<count($str2)){ $tmp = $str1; $str1 = $str2; $str2 = $tmp; unset($tmp); } for($i=0; $i<count($str1); $i++) if(in_array($str1[$i], $str2)) $count++; return $count/count($str2)*100;}[/code] Quote Link to comment Share on other sites More sharing options...
Bobafart Posted December 16, 2006 Author Share Posted December 16, 2006 I used that function.. I appreciate it.. thanks.. very nicewhat is the smartest performance method to select out the Related Headlines from mySQL now that I have stored their id's in the array?the array, called $different_articles()", has only the IDs in them of the related news articles..Array("11', "32", "443", etc) I would like to display them.. I have never used an array in a WHERE clause before so I am unsure how to do it. Can someone help me please?[code]$queryRelatedHeadlines = "SELECT id, headline, intro, body, author, date, source, vote, xmlsitetype from anews1 where id = '$different_article()'";[/code] Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.