this works for smaller files but.....

[devWhiz]

<?php
$orig = file('list.txt');
$new = array();
$new[] = 'dmy';
$slice_size = 8000000;
for($i=0;$i<count($orig);$i += $slice_size) {
   $new[] = array_slice($orig,$i,$slice_size);
}
for ($i=1;$i<count($new);++$i) {
file_put_contents('WordList ['.$i.'].txt',implode('',$new[$i]));
}
?>

 

I have text files and wordlists that have up to 250 million lines, I need this script to split the main file which is upwards to 800 MB, and split it into any size I specify, this script works for smaller files but It wont even split a file that is 6 million lines, ~58 mb

 

any help with this is appreciated

[micah1701]

I'd guess that your server is timing out.

 

do you have access to tweak your config/ini settings for apache and php?

 

up your memory_limit

[ignace]

The problem is that file() loads all the lines in memory probably exceeding the assigned limit. To avoid this you should read line by line:

 

$fopen = fopen('list.txt', 'rb');
while ($fopen && ($line = fgets($fopen)) {
    // do something with $line
}

// all lines processed
if ($fopen) fclose($fopen);

[requinix]

Is this just a one-time thing? Because Unix has a command for that:

split -C 8m list.txt

[devWhiz]

Ive set the time limit up and it still isnt working for me, ignace, how would I use your suggestion? Thanks for the responses