Removing Duplicate reocrds in a text file

[jschofield]

Hello all,

What I am trying to do is remove duplicate records that I have in a text file. I am doing way more then this to the file using a php script but this is the only part I dont know. I did some research and tried a few different things but with no luck. Does anyone know how I can make this happen in my script. Below is my script. Thanks for the help!! Johnnie

 

<?php

 

 

//Store the file paths in variables. When the importlibrary

//is included, the variable names must be as shown below

//in order for import logging to work successfully.

$inputfile = "C:/SMDIM/Data/ELStaff.TXT";

$outputfile = "C:/SMDIM/Data/ARCHER_CITY_ES_STAFF.TXT";

$logfilename = "C:/SMDIM/Logs/ARCHER_CITY_ES_STAFF_LOG.TXT";

//Include the import library.

include'C:/SMDIM/Translators/importlibrary.inc.php';

 

 

//Open the input file and assign a newly created variable

//to act as a reference or handle to the opened file stream.

if(!$in = fopen($inputfile, "r")) {

wlogdie("Failed to open $inputfile for reading");

    }else{

wlog("Opened $inputfile for reading");

}

 

 

//Open the output file and assign a newly created variable

//to act as a reference or handle to the opened file stream.

if(!$out = fopen($outputfile, "w")) {

        wlogdie("Failed to open $outputfile for writing");

    }else{

wlog("Opened $outputfile for writing");

}

 

 

//Use the fgetcsv function to parse the file stream into an

//array one line at a time until the end of the file stream

//is reached.

while ($line = fgetcsv($in, 1000, "\t")) {

 

    //Replace tab delimiters with commas and double quotes.

    $row = str_replace("\t", ",", $line);

 

    //Get rid of spaces, parenthesis, and dashes in the phone

    //number field.

    $trimmedphone = str_replace(" ", "",$row[5]);

    $trimmedphone = str_replace("(", "",$trimmedphone);

    $trimmedphone = str_replace(")", "",$trimmedphone);

    $trimmedphone = str_replace("-", "", $trimmedphone);

 

    //Add the desired fields together into one long,

    //comma-separated string.

    $tmpline = '"' . $row[4];

    $tmpline .= '"' . ',' . '"' . $row[2];

    $tmpline .= '"' . ',' . '"' . $row[1];

    $tmpline .= '"' . ',' . '"' . $trimmedphone;

    $tmpline .= '"' . ',' . '"' . "English";

    $tmpline .= '"' . ',' . '"' . "Archer City Elementary School";

    $tmpline .= '"' . ',' . '"' . "Staff";

    $tmpline .= '"' . ',' . '"' . $row[3];

    $tmpline .= '"' . ',' . '"' . $row[0] . '"';

    $tmpline .= "\r\n";

 

    //Count each time the while loop is executed. This will

    //represent the total number records imported.

    $count++;

 

 

// write the output line

fwrite($out, $tmpline);

}

 

 

//Display the number of records imported. If the number is zero

//then display an error message.

if($count !== '0') {

wlog("Finished the import with a total of $count records");

}else{

wlog("Failed to import data");

}

 

 

//Close the input and output files.

fclose($in);

echo "Closed the input file." . "\r\n";

fclose($out);

echo "Closed the ouput file." . "\r\n";

 

 

?>

 

Please HELP! Thanks to all that do help.