PHP Allowed memory size problem.

[joblafors]

Hello.

I`m writing web crawler script for one specific site, and i`m using simple_html_dom to find links and specific information in site.

But I stumbled on PHP memory problem

Fatal error: Allowed memory size of 10485760 bytes exhausted (tried to allocate 16 bytes) in *****\simple_html_dom.php on line 924

.

I changed memory limit to 30MB, 50MB .. but this doesn't resolve the problem because eventually memory will be full, that is if script crawls few pages deeper in site.

Are there any suggestions how to manage memory in PHP, or are there some better html dom scripts to use (with swap files or smth like that)

Thank you !!

[Daniel0]

Why it does that depends on how you've written your code.

 

Maybe you don't clean up unused resources. That's simple to fix. Just do manual garbage collection.

 

Maybe you wrote your script recursively and the call stack keeps growing. In that case, you could refactor it to be tail recursive.

 

Maybe you're just trying to keep all the generated data in memory. In that case, write it to permanent storage more frequently.