the limits of PHP

[Shadowing]

I'm having a problem with removing duplicates values in an array.

 

seems like array_unique wont accept arrays bigger then 5084

 

also i tried

 

$stargate_address_array= array_keys(array_flip($stargate_address_array));

 

$stargate_address_array= array_keys(array_count_values($stargate_address_array));

 

if i count the array before array_unique i get an array account of 600,000 but soon as it hits unique array its count is only 5084. anyone know how to get around this?

function stargate_address_check() {

			$map_x = 200;
			$map_y = 200; 
			$stars_per_sector = 15;

			$star_amount = $map_x * $map_y * $stars_per_sector; 							
									$stargate_address_array = ''; 
$create_address_array = 0;

while ($create_address_array < $star_amount) { // creates a huge array of address's 				$new_address = stargate_address_generator();

			$stargate_address_array[] = $new_address;

			++$create_address_array;
}
							// removes any duplicate address
			$stargate_address_arrays = array_unique($stargate_address_array); 
			$count = count($stargate_address_arrays); 
return $count;

}


echo stargate_address_check(); 

 

 

the function im using to generate the values inside the array

		// creates a stargate address
function stargate_address_generator() {	

		$chevron1 = rand(1,38);	

		$chevron2 = rand(1,38);	
while ($chevron1 == $chevron2) { $chevron2 = rand(1,38); }	

		$chevron3 = rand(1,38);	
while ($chevron3 == $chevron1 || $chevron3 == $chevron2) { $chevron3 = rand(1,38); }	

		$chevron4 = rand(1,38);	
while ($chevron4 == $chevron1 || $chevron4 == $chevron2 || $chevron4 == $chevron3) {$chevron4 = rand(1,38);}	

		$chevron5 = rand(1,38);	
while ($chevron5 == $chevron1 || $chevron5 == $chevron2 || $chevron5 == $chevron3 || $chevron5 == $chevron4) {$chevron5 = rand(1,38);	}	

		$chevron6 = rand(1,38);	
while ($chevron6 == $chevron1 || $chevron6 == $chevron2 || $chevron6 == $chevron3 || $chevron6 == $chevron4 || $chevron6 == $chevron5) { $chevron6 = rand(1,38);	}	


		$stargate_address = "$chevron1-$chevron2-$chevron3-$chevron4-$chevron5-$chevron6";	

return $stargate_address;
}

 

 

[requinix]

You really shouldn't be sticking that many items in an array...

 

Use an insertion sort when you add to the array: keep it sorted and insert the new addresses into the right position. When you're doing that you can also check for a duplicate.

[samshel]

Agree with requinix...such big arrays will generally land into problems.

 

however you can try using a SORT_FLAG with array_unique. Not sure, but worth a try.

 

$stargate_address_arrays = array_unique($stargate_address_array, SORT_REGULAR); 

 

Also you can try your code for a lower values like 8000 to check if array_unique works as expected.

 

[Shadowing]

Hey requinix

 

looking through all the array functions. not sure which one you are talking about.

 

i could use array filter in the loop as its being created?

 

I figured it would be doing more work checking for duplicates every time it adds a new element to the array.

 

i'll probably only end up needing 150,000 in the array.

 

i really only have two choices, have mysql do the work or php. and so far mysql failed to create  6000 records for me.

so it didnt even come close.

 

 

[scootstah]

You need to break it up into smaller batches. Instead of trying to load 5000 at once load a couple hundred or something.

[samshel]

Did you try the SORT FLAG or running it with lower limit?

[Shadowing]

hey samshel i tried sort_flag same results

 

but yah i tried with lower amounts as well.

 

i can for example do 4860 and i cant go past 5084.

 

so i dont know real dissapointing lol

 

its not a memory issue so it doesnt make any sense.

 

i can try filtering it as the array is being created and see how intensive that is.

[salathe]

seems like array_unique wont accept arrays bigger then 5084

 

The function should not have a problem with the number of values in your array, or even orders of magnitude more given time and memory.  Your code works absolutely fine for me, is there anything else you have not shown?  Does all arrays get reduced to 5084 values (e.g. array_unique(range(1,6000)))?

 

As for "filtering" the array for unique addresses, why not just skip that step entirely?  $stargate_address_array[$new_address] = $new_address;

[Shadowing]

hey Salathe you ran the function "stargate_address_generator" with my code too?

 

what was your return?

 

You will notice this at the top of the function. increase these values will increse how many are in the array

if i try 20,20 for these values below and assuming there are no uniques i should get a count of 6000

18,18 works i get 4860

$map_x = 18;

$map_y = 18;

 

what does this do? is that a way of filtering for duplicates?

$stargate_address_array[$new_address] = $new_address;

[salathe]

Is this topic really solved?..

[Shadowing]

no its not solved. what the heck i never hit solved on it. wierd. hmm

[kicken]

I ran your code as well and didn't have any problems with a limit like your suggesting.  I'm not sure if your code is really what your wanting though.  Based on what I see in the code you want to generate 600,000 unique addresses, one for each 15 stars in each of the sectors.  What your code does currently is generate 600,000 addresses but then after filtering out the duplicates you'll end up with less at the end.  You won't have enough for one address per star.

 

 

What you need to do if you want 600,000 unique addresses is keep looping until you get your 600,000 address.  So you generate an address, check if it already exists and if not, add it to the array.  Repeat until your 600,000 addresses are generated.

 

what does this do? is that a way of filtering for duplicates?

$stargate_address_array[$new_address] = $new_address;

 

Yes, you cannot have duplicate key names so by using the address as the key name you ensure you do not ever have a duplicate of that address.  The value can be whatever you want it to be, doesn't necessarily have to be the address.

 

 

This is some re-factored code based on what it seems like you want:

<?php 
function stargate_address_check() {
$map_x = 200;
$map_y = 200; 
$stars_per_sector = 15;

$star_amount = $map_x * $map_y * $stars_per_sector; 							
$stargate_address_array = ''; 
$create_address_array = 0;

for ($i=0; $i<$star_amount; $i++){ // creates a huge array of address's 				
	do {
		$address = stargate_address_generator();
	} while (isset($stargate_address_array[$address]));

	$stargate_address_array[$address] = 1;
}

$count = count($stargate_address_array);
return $count;

}

function stargate_address_generator() {	
$chevron1 = rand(1,38);	

$chevron2 = rand(1,38);	
while ($chevron1 == $chevron2) { $chevron2 = rand(1,38); }	

$chevron3 = rand(1,38);	
while ($chevron3 == $chevron1 || $chevron3 == $chevron2) { $chevron3 = rand(1,38); }	

$chevron4 = rand(1,38);	
while ($chevron4 == $chevron1 || $chevron4 == $chevron2 || $chevron4 == $chevron3) {$chevron4 = rand(1,38);}	

$chevron5 = rand(1,38);	
while ($chevron5 == $chevron1 || $chevron5 == $chevron2 || $chevron5 == $chevron3 || $chevron5 == $chevron4) {$chevron5 = rand(1,38);	}	

$chevron6 = rand(1,38);	
while ($chevron6 == $chevron1 || $chevron6 == $chevron2 || $chevron6 == $chevron3 || $chevron6 == $chevron4 || $chevron6 == $chevron5) { $chevron6 = rand(1,38);	}	


$stargate_address = "$chevron1-$chevron2-$chevron3-$chevron4-$chevron5-$chevron6";	

return $stargate_address;
}


echo stargate_address_check(); 

 

[samshel]

@kicken

I think he is not expecting any duplicates. The function stargate_address_generator() gets 6 unique random numbers between 1-38 and forms a string.

There should be very thin of chances of getting a duplicate here.

 

I am assuming that the array_unique function is somehow changing data type for these elements while sorting and mixing them up.

 

@Shadowing

You can try avoiding use of array_unique by specifying the address as the key itself like salathe suggested. Did you try it?

[Shadowing]

wierd what the heck why am I only getting this limit. possible some setting i have set for php or something?

 

Yah the chances of a duplicate is very rare. I already wrote code to add back the address's if it found any duplicates.

Reason i was doing it like that is cause it woud be more light weight since i wasnt expecting duplicates like samshel said.

 

what is really wierd is that i used in_array to search for uniques while it was creating the array and soon as i tried to create more then 19 by 19 the screen goes white. i mean what the heck lol. Its like any kind of filtering i try to do i cant filter more then 5048 stars. This is really frustrating that you guys arnt having this issue.

 

I"m using apache for my server.

 

Anyways i'll try the key idea. That seems like an amazing idea really brilliant. Sounds like it should work and would be really light weight. Also i didnt know i could assign a key like that lol. So i just learn something will let you guys know

 

 

[Shadowing]

Fatal error: Maximum execution time of 30 seconds exceeded

 

 

I just dont understand why can you all run this code and I cant? I can increase or make the execution time unlimited of course, but it will make the script run soooo slow

 

soon as I set the x and y to 20 times out.

 

but if i do 19 by 19 i create the stars instantly no problem

 

I used kickens exact loop on this. and im only running these two functions

now i know something is really wrong if you guys can do this and I cant. my server must not be set up right. My php memory is set to max of 128mb. but i dont believe thats even the issue anyways because i lowered to 16 mb and i can still create 19 by 19 instantly "i know its not really instantly but its a fraction of a second".

 

I googled this sooooo hard the other day and nothing came up saying about this issue.

[kicken]

This is on my desktop.

600000
Script took 3.4052720069885 sec and used 59.5 megabytes of memory

 

This is on my VPS.

600000
Script took 4.3914389610291 sec and used 55.5 megabytes of memory

 

 

And this is on my home server I built back in '03

600000
Script took 13.989145994186 sec and used 73.75 megabytes of memory

 

With the code I posted above.  If your server is taking that much longer or using a lot more memory sounds like perhaps your server is either underpowered or overloaded (my guess).  Are you testing only this code or is there other stuff involved when your testing?  I just have the above code in a file and run it via php-cli to test.

 

[samshel]

Do you get any error or the script completes but does not give expected results. Print some test text at the end to see what step it reaches. Also try increasing the memory limit.

 

It must be something specific to your server.

[Shadowing]

really appreciate you guys helping me out with this cause this problem is really bothering me. specially now that you guys are not having this issue. I'm using a Desktop computer with windows xp pro. Even though i dont think browser should matter i tried it on IE and FF. This machine has a 3 ghz dual core processor with 3 gigs of 800mhz RAM. I dont see how it could be a computer hardware issue due to how its happening between 18 by18 and 20 by 20. I should mention if I do make the execution time unlimited it will eventually work but will take like 10 minutes lol. The max memory you can have apache on is 128mb and mine is set to max. but i dont think its that setting cause i can lower it to 8mb and doesnt effect testing it with 18 by 18.

 

Ive been running only this code nothing else

the exact code that kicken posted.

 

So now i took it a step further i opened up a fresh page with nothing on it but this code.

Same results if i use "$map_x = 18; $map_y = 18;" the page loads in a fraction of a second.

 

The I try "$map_x = 20; $map_y = 20;" and it times out.

its almost like its a setting or something because why would 18 by 18 be a fraction of a second and then 20 by 20 doesnt work.

 

The only error I get

Fatal error: Maximum execution time of 30 seconds exceeded in C:\Software\XAMPP\xampp\htdocs\System_Lords\raw.php on line 27

 

 

ini_set("display_errors", "1");error_reporting(-1);

		// creates a stargate address
function stargate_address_generator() {	

		$chevron1 = rand(1,38);	

		$chevron2 = rand(1,38);	
while ($chevron1 == $chevron2) { $chevron2 = rand(1,38); }	

		$chevron3 = rand(1,38);	
while ($chevron3 == $chevron1 || $chevron3 == $chevron2) { $chevron3 = rand(1,38); }	

		$chevron4 = rand(1,38);	
while ($chevron4 == $chevron1 || $chevron4 == $chevron2 || $chevron4 == $chevron3) {$chevron4 = rand(1,38);}	

		$chevron5 = rand(1,38);	
while ($chevron5 == $chevron1 || $chevron5 == $chevron2 || $chevron5 == $chevron3 || $chevron5 == $chevron4) {$chevron5 = rand(1,38);	}	

		$chevron6 = rand(1,38);	
while ($chevron6 == $chevron1 || $chevron6 == $chevron2 || $chevron6 == $chevron3 || $chevron6 == $chevron4 || $chevron6 == $chevron5) { $chevron6 = rand(1,38);	}	


		$stargate_address = "$chevron1-$chevron2-$chevron3-$chevron4-$chevron5-$chevron6";	

return $stargate_address;
}







function stargate_address_check() {


			$map_x = 20;
			$map_y = 20; 
			$stars_per_sector = 15;

			$star_amount = $map_x * $map_y * $stars_per_sector; // finds out how many address's needs to be created 

									$stargate_address_array = ''; // advoid undefine error notice
$create_address_array = 0;

for ($i=0; $i<$star_amount; $i++){ // creates a huge array of address's 						

	do {			

		$address = stargate_address_generator();		

} 		while (isset($stargate_address_array[$address])); 		

			$stargate_address_array[$address] = 1;	

}	
			$count = count($stargate_address_array); 
return $count;

}


echo stargate_address_check();

[kicken]

The only thing I can think of is that maybe your rand() implementation sucks and you end up in a long loop generating several duplicates.  You could try using mt_rand instead and see if that results in anything different.

 

[samshel]

also try increasing your rand(1, 38) to something with bigger range like (1,100). May be as Kicken suggested, rand is messing up and range is not enough.

[Shadowing]

Holy crap Kicken you are a genius hahaha

 

that solved the problem wow.

 

so why is my rand function worse then yours?

[DavidAM]

Just for fun, I decided to test this on a 64-bit Win7 machine. At 18x18 it is almost instantaneous. At 19x19, well ... it's still running. 

 

Consider changing your generator. It looks like you are looking for six numbers between 1 and 38, without any duplicates. I suspect it is getting hung up in all those loops looking for unique numbers. But try this.

 

... still going ...

 

function stargate_address_generator() {
$chevrons = range(1,38);
shuffle($chevrons);
$stargate_address = implode('-', array_slice($chevrons, 0, 6));
return $stargate_address;
}

 

The shuffle randomizes the list of numbers. Then we can just take six off the top and have six random numbers without any of them being the same.

 

... still going ...

 

I just tested that at 35 x 35 and it only paused for half-a-second.

 

... still going ...

 

I don't think the first one will ever finish.

 

... still going ...

 

This one actually pukes at around 40 x 40. Are we getting close to the limit here? How many unique  combinations are there?

Actually, I'm betting it is a problem with the Windows random number generator. I bet everyone else that reported testing it was using Linux (which I prefer myself).

 

... still going ... --- I'm not waiting anymore -- CTRL-C

 

Well, I see someone else offered a different solution. But, you might want to optimize your generator anyway.

[Shadowing]

ahh yah thats a interesting idea david. I think i may just switch to that idea. that be heck of alot light weight then the 5 loops im using ha.

[Shadowing]

oh wait i cant remember did you guys say that you was able to also use array_unique function and get 600k results too? just curious even though thats not how i will be doing it anymore.

 

so what is the deal then my server?

[Shadowing]

this just doesnt make any sense to me. array_unique function works fine now with using mt_rand.

 

I dont understand how rand() was effecting the results of array_unique()

 

reason why is when i was using rand() i did count() on the array before it hit array_unique() and I was getting 600,000.

 

then i did count() after array_unique() and got only 5048.

 

Get what im saying guys? that is crazy wierd.