Base64 encoded URLs

[freeloader]

Hi guys,

 

I'm parsing a bunch of html sources, some of them (a minority) contain base64 encoded URLs. At least 20-30 URLs per page. Some of the information I need is taken from the URLs.

 

I'm looking for a way to parse these URLs without losing too much time (20-30 URLs per page and I need to parse about 10k pages daily).

 

Should I regex each source individually for base64 encoded URLs and then decode them each time and isn't that going to take a lot of time/resources (especially considering it's only a minority that have the base64 URLs in them)? Or is there a better way to do it?

 

Code snippets are absolutely welcome!

 

Thank you in advance

[silkfire]

How do you know it's base64 url in the first place?

[freeloader]

Because these are pre-parsed pages and in some of them, the URLs have been parsed through a base64 encoder.

 

Giving an output like this:

<link href="Oigregregregrer/gergeggege==" media="screen" rel="stylesheet" type="text/css" />

 

(The above is not an actual link.)

[silkfire]

I was more wondering if you could quickly filter out the base64 url strings to separate them from the rest and process the non-base64 first then the base64.

[freeloader]

Fixed it using preg_replace_callback() on the URL's, regex'ing them and base64 decoding them.