create a loop of md5 until there are now md5's to make left

[Jnerocorp]

Hello,

 

I know its possible to make a script using md5 to generate md5 strings out of these characters:

"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890"

 

I need it to just keep generating md5 strings until there are no more to do

 

the strings that will be md5 encoded will only be 6 char long so it will look like this:

 

String: dFdr3k MD5: 2681deea7403b468389918af97ef7c38

 

This will generate "377,149,515,625" Billion md5 encrypted strings

 

any one able to do this I have tried and failed

 

-John

[cags]

Why would you want to? Have you considered the storage space and execution time required for calculating every possibility?

[mikesta707]

theoretically

function createAll($length = 6){
$used = array();
$chars = array('a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z','A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','R','S','T','U','V','W', 'X','Y','Z','1','2','3','4','5','6','7','8','9','0');
$x = 0;
while ($x < pow(count($chars), $length)){
	$word = "";
	for ($i = 0; $i < length; $i++){
		$word .= $chars[rand(0, count(chars))];
	}
	if (!in_array($word, $used)){
		$used[] = $word;
		$x++;
	}

}
print_r($used);
}

createAll();

 

that might work. But that exceeded the max execution time of 60 seconds on my local host.

 

but like cags said for just 6 character strings (and that is ONLY 6 character strings, not 5  character or below) that you are going to have over 300 million different words. And then you have to md5 them? the function I just wrote probably won't be able to execute unless you let your server run for a while.

 

are you trying to make a rainbow table or something?

[Mchl]

Just for your consideration:

377,149,515,625 * 32B = approx 10TB of data. You're prepared to store that?

[Jnerocorp]

I have a 20TB office based server with 10GB of Ram.

 

I am building a database of md5 encryptions and will execute a code that will check select a typed in md5 string and find the decoded version of it

 

-John

[Mchl]

I think that with a server like that, it would be more efficioent to run some collision finder on it.

[Daniel0]

This will generate "377,149,515,625" Billion md5 encrypted strings

 

How did you get that number?

 

You have 26*2+10=62 different characters.

 

Given that your string is always 6 characters long, the number of permutations with repeats is 62^6 = 56,800,235,584. Still a large number, but pretty much smaller than 377,149,515,625,000,000,000.

 

At 32 B per MD5 hash, that's about 1.6 GiB assuming you're storing them consecutively with no form of separation.

 

Am I missing something?

 

Edit: @mikesta707: Your algorithm has a worst case running time of O(∞).

[Mchl]

How did you get that number?

 

You have 26*2+10=62 different characters.

 

Given that your string is always 6 characters long, the number of permutations with repeats is 62^6 = 56,800,235,584. Still a large number, but pretty much smaller than 377,149,515,625,000,000,000.

 

At 32 B per MD5 hash, that's about 1.6 GiB assuming you're storing them consecutively with no form of separation.

 

Am I missing something?

 

Yeah. I did same calculations after I posted. It's 1.6TB if I did it right. And that's only hashes, you still need to store keys, which brings it to 2TB.

 

And then... how exactly are you planning to search this data for the result?

[Daniel0]

Yeah. I did same calculations after I posted. It's 1.6TB if I did it right.

 

Yeah, sorry, tera of course. Not giga.

 

Oh yeah, and if it actually was 377,149,515,625,000,000,000 you would need 327 EiB of storage. That would be one hell of a RAID setup.

[cags]

Bugger it. I was about to make a post very similar to Daniel0, but he's beat me too it so I've started again. Even if you considere any length of string upto and including 6 I still only work out 57731386986 possibilities. Though I have to say at 32 bytes per hash I'm not sure how Daniel0 came up with 1.6Gigabytes as I worked it out at closer to 53.8Gigabytes (which assumes storing no characters between the hashes). Perhaps my math is bad?  :-\

[Mchl]

MD5 hash is 32B long not 1B

And yeah, I only counted strings that are exactly 6B long.

[Jnerocorp]

thats right i was doing 85 beacause there was originally more characters i forgot that I had removed some lol but thats a big space saver thanks for noticing that.

 

also there is the storing of the original string to so it would be like this

 

(string,md5)

 

and millions of rows in a database all populated like that but im not even sure of what type of database I could use to store billions or even millions of rows

 

im not sure how im going to search it yet i dont even know if php would even be fast enough to search a database that large in under 30 seconds load time

 

 

-John

[Mchl]

Well, you will need some external DBMS to do this work for sure. Forget about loading it into PHP array and going through it with foreach().

[Jnerocorp]

is there a way to make it show each function run

 

like this:

 

String: stringhere MD5: md5here

 

using ajax or something and make it only generate 50,000 at a time and store it into mysql for now

 

but it cant ever create to of the same md5

[Daniel0]

im not sure how im going to search it yet i dont even know if php would even be fast enough to search a database that large in under 30 seconds load time

 

PHP doesn't care how large your database is. For all it cares, it could have 1 or Googolplex (10^(10^100)) records.

 

Well, you will need some external DBMS to do this work for sure. Forget about loading it into PHP array and going through it with foreach().

 

Are you sure it would matter? A DBMS would also be really slow without an index, and to be honest, creating an index for this would be stupid and pointless.

[Mchl]

Are you sure it would matter? A DBMS would also be really slow without an index, and to be honest, creating an index for this would be stupid and pointless.

 

Yeah... I was actually wondering how to index such data. Indexing on for example 8 starting Bytes of hash?

 

Frak. Now I wanna do this myself to do benchmarks!

[Daniel0]

If you want to generate all of them, you could do like this:

 

<?php
function permute($length, array $chars, array $acc = array(''), $n = 1)
{
    if ($n > $length) {
        return $acc;
    }    
    
    $a2 = array();
    foreach ($chars as $c) {
        foreach ($acc as $s) {
            $a2[] = $c.$s;
        }
    }
    
    return permute($length, $chars, $a2, ++$n);
}

$chars = array_merge(range('a','z'),array_merge(range('A','Z'),range('0','9')));

print_r(permute(6, $chars));

 

It can without doubt be improved, but it was quick to write

 

Edit: That will be all the strings, you would still have to MD5 them.

[Daniel0]

Just for fun, I decided to throw some functional programming into this:

 

local
  fun add chars xs = foldl (fn (y,b) => (map (fn x => x^y) xs) @ b) [] chars
  fun perm' chars length acc n = if n > length then acc
                                 else perm' chars length (add chars acc) (n+1)
in fun perm chars length = perm' (map str chars) length [""] 1 end

 

That's the permutation function implemented in SML. I like this one better. It's more beautiful.