Hi all, this is my first time on this forum!
I have a background with HTML and CSS but have recently started a Masters in Computer Science hoping to come out of it with the tools to get a job with PHP development.
Our first assignment has somewhat 'thrown me in the deep end' as we have to construct a search engine that indexes the words of a number of documents and rank them using the TF*IDF algorithm along with the log rule associated with Information retrieval.
I am completely new to PHP so the past week has been something of a crash course - This is the code I have so far:
<?php
$filename = 'airlines.txt';
$fp = fopen( $filename, 'r' );
$file_contents = fread( $fp, filesize( $filename ) );
fclose( $fp );
//$new_contents = ereg_replace("[^A-Za-z0-9]", "", $file_contents);
/*$file_contents = trim($file_contents);
$file_contents = preg_replace('/\h+/', ' ', $file_contents);
$file_contents = preg_replace('/\v{3,}/', PHP_EOL.PHP_EOL, $file_contents); */
$pat[0] = "/^\s+/";
$pat[1] = "/\s{2,}/";
$pat[2] = "/\s+\$/";
$rep[0] = "";
$rep[1] = " ";
$rep[2] = "";
$new_contents = preg_replace("/[^A-Za-z0-9\s\s+]/","",$file_contents);
$new_contents = preg_replace($pat,$rep,$new_contents);
//preg_replace('~\s{2,}~', ' ', $text);
$commonWords = array('a','able','about','above'........and another few hundreds cut out of this not to hurt your eyes!);
$lines = explode ( "\n", $new_contents);
$lines2 = implode (" ", $lines);
$words = explode ( " ", $lines2 );
$useful_words = array_diff( $words, $commonWords );
/*for($i = 0; $i < count($lines); $i++) {
echo "Piece $i = $lines[$i] <br />";
}*/
for($i = 0; $i < count($useful_words); $i++) {
echo "Words $i = $useful_words[$i] <br />";
}
//$arr=array("blah1","blah2","blah3");
file_put_contents("demo2.txt",implode(" ",$useful_words));
//$file_c = file_get_contents("demo.txt");
//$colms = explode(",",trim($file_c));
//print_r($colms);
//echo $lines[2];
?>
I've got to the stage where that strips out most of the stop words when the final array is printed, but they have been replaced with spaces or something that I have not come acoss because as you may see I had a bit of trouble originally stripping the punctuation marks.
I'm hoping someone can point me in the direction as to how to organise the words I have left after the stripping of stop words which are of no use during the search. I need to store those words into another array and index them which says how many times they appear in that document.
I've come across the function array array_count_values ( array $input ) on the manual but I'm not sure about the best way to use it.
I've attached the files I've used if that helps.
Any help would be greatly appreciated!
[attachment deleted by admin]
