Find duplicate lines within a file?

[dodgerfan]

I need code that will scan a fairly large text file (10MB+) and either display or export all of the duplicate lines.  I know how to remove the dupes using array_unique and filling it with the contents of the file but I want to know what those lines are rather than just removing them.

 

Any thoughts on how I might go about doing this?

[MadTechie]

May fail on a very large file.. depends on memory and timeouts but

<?php file_put_contents("SmallFile.txt",array_unique(split("\n", file_get_contents("LargeFile.txt")))); ?>

 

EDIT: LOL may fail on small files.. yeah mean very large

[ghostdog74]

store each line as associative arrays, increasing the count as dups are found. At the end, check for count of more than 1.

$contents=file("file");
foreach ($contents as $k=>$v){
$v=trim($v);
$array[$v]++;
}
print_r($array);

[xylex]

array_unique() maintains key values, so you can find the removed lines by comparing the original array with the new one using array_diff_key()

[dodgerfan]

xylex,

 

Just to be clear how would that work with this code...

 

<?php
// Load file into Array
$list = file('file.txt');

// Remove duplicates
$list = array_unique($list);

// Write back to file
file_put_contents('uniques.txt', implode('', $list));
?>

 

Thanks in advance.

[xylex]

Untested

<?php

// Load file into Array
$original = file('file.txt');

// Remove duplicates
$uniques = array_unique($original);
$removed = array_diff_key($orginal, $uniques);

// Write back to file
file_put_contents('uniques.txt', $uniques);
file_put_contents('removed.txt', $removed);

?>

 

[dodgerfan]

I'm getting an error that the #1 argument is not an array in line 8.

 

I'll keep trying.  Maybe I've missed something.

[.josh]

That's because there's a typo.  Come on man...

[xylex]

Sorry, can't spell.

<?php

// Load file into Array
$original = file('file.txt');

// Remove duplicates
$uniques = array_unique($original);
$removed = array_diff_key($original, $uniques);

// Write back to file
file_put_contents('uniques.txt', $uniques);
file_put_contents('removed.txt', $removed);

?>

 

 

[dodgerfan]

That works.  Thanks for your help and sorry for overlooking the typo.  I've been staring at code all day so it's starting to run together.