Finding Memory Leaks

[pepa]

Hi Forum,

 

I've written an import script for a bigger image database. The script iterates over 20000 database entries and as much

lines of csv file, creates some object and updates all database entries. All in one big for-loop.

 

My problem is that every loop takes about 20k extra memory eating up all memory I feed it.

 

I tried to debug it using "get_defined_vars()" but it returns every time the same amount of variables. (I think this is because

get_defined_vars only returns accessible variables.)

 

Now I'm a little out of options. What can I do to find the root of the problem? Is there a way to output _every_ memory eating bastard

or to access the symbol-tables directly and look for stuff which shouldn't be there? Or are there any php memory debuggers which could

help me?

 

Help would be very much appreciated.

 

Best regards

 

PePa

[rarebit]

What actually happens... a memory leak is where a persistent variable isn't destroyed... You problem could be timing out or just not enough memory to handle the request...

 

One way round could be to read the db in chunks and compile a sublist which can be rechecked after?

[pepa]

... not enough memory ... well ... thats exactly my problem. At the current rate of memory consumtion I would need 400M to process the complete request. And the database is growing. But I don't store the complete db in memory.

I process one row at a time (Chunks couldn't be smaller). But php doesn't free the 20k it takes to process this one row.

What I do need now is some tool/command/trick which can tell me why...

 

Regards

 

Pepa

[cooldude832]

why not do it in batches and then sequence each batch together.  You would get to flush the memory using a  temporary flat file, and then you have the solution.  I.E Microsoft's Page File at its finest.

[pepa]

hi cooldude,

 

I'm not sure I do understand what you're suggesting? Splitting my 20k lines csv in small ones and jumping from one php processing-page to the next? Seems a little bit ... aehm ... harsh just to get php to free mem (which it should do in one way or the other?).

 

Come on, there has to be a way to solve the problem other than ignoring that I coded some shit somewhere and now ignore it

 

Regards,

 

PePa

[cooldude832]

Not at all, Read the file in 4 bunches of 5000 lines.  (or X groups of 5000)  once you get that 5000 chunk store it in a separate file work it and store your position in the database once it gets done simply reload the page via a header this will suck the memory clean and will get the same result.  The trick is just breaking it apart.

[pepa]

ok. this is a solution. work around some shitty code (mine or phps) by not calling it to often. If everything else fails I would give it a try. But it doesn't sound like the solution I'm looking for.

What I'm looking for is some way of replace shitty code (which is most probably mine) by some better code which can be called 20k times whithout fucking up. Only thing I need for is some way to locate the piece that is eating up all the memory. Every decent programming language must have a way to detect memory leaks. Why shouldn't php?

 

Regards,

 

PePa

[rarebit]

Use a single variable as a buffer, use it, then store results to another variable, then 'unset' original variable and reuse it. Surely this prevents a leak cos your unsetting (freeing) it. Memory has always been a limited resource, you gotta use it wisely!

[rarebit]

Was having a look at this earlier and wonderd if it could help?

http://uk3.php.net/manual/en/function.mysql-unbuffered-query.php

See reply dated '01-Apr-2004 11:19' (***note date!)

This is what led me there... http://uk3.php.net/manual/en/function.mysql-free-result.php

[trq]

Every decent programming language must have a way to detect memory leaks. Why shouldn't php?

 

Because you can't create a memory leak in php. In fact you don't even control memory in php, it is all handled automatically for you.

 

It sounds to me like you need to increase the max execution time within your php.ini.

[pepa]

Hi folks,

 

What I did the last days:

1. Monitor memory consumption using "memory_get_usage" which reports every loop iteration about 40k more consumption.

2. Monitor creation and destruction of objects in constructur and destructor including print_r( $this ) in destructor to make sure the object is empty when I unset the object. Additionally I used a global counter to keep track of the number of created objects. (Increase on new, decrease on __destroy) This counter stayed at about 80 objects. (Didn't increase to x * 20000).

3. Manually unset all variables before they leave scope. Recursively if necessary.

4. Output all scoped variables with "get_defined_vars": No new variables there.

5. Try to find out something using xdebug (but xdebug doesn't have significantly more info then php5 has)

6. Manually free the few sql results I have. (By turning on Mysql profiling in php.ini I make sure I didn't forget someting.) Bought me some k indeed but not nearly enough.

 

Result of all this: Problem stays the same. 20k iterations which eat up 40k ram each results in about 800M total memory consumption.

 

so thorpe: If php isn't leaking, where do the 800M go to? Except rewriting the script either without framework or in another programming language I'm running out of options.

 

btw: Can't leak memory was a myth in java and is most definitely in php. You can create leaks whenever you want. e.g. php had some issues with circular references.

 

Regards,

 

Florian