[SOLVED] Remove dublicated

[raimis100]

Hey!

 

I am thinking of way to remove duplicate lines in my txt file

each sentence is in the different line

 

So far I did is

 

<?php

  $list = file ('sentencelist.txt');
  For ($I = 0; $I <= count ($list) - 1; $I++)
  {
    $dub = false;
    $corrent = trim ($list [$I]);
    For ($X = $I + 1; $X <= count ($list) - 1; $X++)
    {
      If ($corrent == trim ($list [$X]))
      {
        $dub = true;
        break;
      }
    If ($dub == False) 
    {
      $output = $output . $corrent ."\n";
    }
    }
  }
  echo $output;
  
?>

 

basically what it does is compare each line but it crashed my apache because my txt files are huge [ around 100 k lines]

 

Can anybody suggest me better way to do it

 

 

edit:

 

 

actually I found a bug. Script should be

 

<?php

  $list = file ('sentencelist.txt');
  For ($I = 0; $I <= count ($list) - 1; $I++)
  {
    $dub = false;
    $corrent = trim ($list [$I]);
    For ($X = $I + 1; $X <= count ($list) - 1; $X++)
    {
      If ($corrent == trim ($list [$X]))
      {
        $dub = true;
        break;
      }
    }
    If ($dub == False) 
    {
      $output = $output . $corrent ."\n";
    }
  }
  echo $output;
  
?>

 

But there is still high cpu usage.

[lemmin]

See if this works:

$nodups = array_unique(explode("\n", file_get_contents("sentencelist.txt")));

[raimis100]

Hmm ... And you did with one line of code

Yea. This wil work for me. Thanks!