unzipping and splitting while within memory limits

[dyluck]

Hi everyone

 

I am having nothing but trouble with my site due to memory limitations set by the host company.

 

I am trying to do two things.  Unzip a large file and then parse the very large extracted txt file to enter applicable data into MYSQL.

 

So my current scripts work like a charm for files that are within 40MBs extracted.

The trouble comes when the zip file, and of course the extracted file, is too large.  I get the

"Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 20075177 bytes)" blaa blaa blaa.

 

Does anyone have a code snippet or a diffinitive direction on how to unzip large files via php (cpanel does it, so I know that the code is there somewhere   )  then break appart the text file (hopefully by size/carriage returns ) so that I can parse each piece within the memory limitations?    I have been working on this for days upon days and keep hitting roadblocks once I get a little progress. 

 

 

Thanks a ton!!!

 

[jonsjava]

can you run cli commands? if so you can use shell_exec() to run the command, bypassing PHP memory limits. grep/fgrep/gawk will become your friends, but it will make it doable.

[dyluck]

can you run cli commands? if so you can use shell_exec() to run the command, bypassing PHP memory limits. grep/fgrep/gawk will become your friends, but it will make it doable.

 

Ok, apparently I can use shell_exec() commands.  Just tested with this code here:

 

<? 
function unzip($zip_file, $src_dir, $extract_dir) 
{ 
mkdir($extract_dir.$zip_file,0777); 
copy($src_dir.$zip_file, $extract_dir.$zip_file); 
chdir($extract_dir); 
shell_exec("unzip $zip_file"); 
} 

if(unzip('testlargfile.zip','/zips/','/zips/')){ 

  echo($zip_file .'unpacked successfully'); 

}else{ 

  echo($zip_file .'was not unpacked'); 

} 
?> 

 

It unpacked a 150MB file you rock jonsjava !

 

So, i guess the next part will be harder:

Ok is there a shell_exec() command that will logically split the TXT files based on filesize and ending at a carriage return ( "\r\n" ) for each of the split files?

 

Would something like the following work translated into PHP: http://www.computerhope.com/unix/usplit.htm

How would the code look? Is there something like:  shell_exec('split -li 2000 splitme.txt') 

 

Hopefully looking for smething like  splitme_0.txt, splitme_1.txt, splitme_2.txt and so on.

 

Thanks so much for your help

[jonsjava]

split should do the trick for you. Make sure to read up on it and test it out a few times before placing it in production.

[dyluck]

Does anyone know the code syntax?  I can't seem to find the right syntax.

when i tested: shell_exec("split -li 2000 8297.txt splitme"); 

There was no errors and no splitting occuring.

[jonsjava]

shell_exec("split -l 2000 8297.txt split_");

[jonsjava]

actually, I forgot something:

shell_exec("split -l 2000 8297.txt -d split_");

[dyluck]

johnsjava.  You are litterally my hero.

I finally see light at the end of this unforgiving brutal tunnel!