delayedinsanity

It's much harder on the server, agreed. The worry is having one process running for 30-60 minutes; I think there's more potential for that to hang or hit a snag and spiral out of control eating memory. Then again, with 100 processes being fired off nearly simultaneously, each should be done in a fraction of the time, but there is potential for 100 processes to hang. Augh. Okay. I'm going to test the multiple processes version on a sandbox server to see if I can hang it.

PS, thanks for the responses so far!

That's what I'm doing, the program doesn't grab the information on the fly. An import is scheduled via cron, and fired off once or twice a day depending on the needs of the particular site. That import grabs the data from an XML feed, turns it into a page and then retrieves the images. It's this image retrieval that's causing the script to run for an extremely long period of time. With the image import disabled, it takes on average 30 seconds for the script to import all the data from the remote XML feed. That's about 5-700 pages of information, adding anywhere from 1-18 seconds per page to grab the images. If we average 4 seconds (low-balling) per page due to the image import, we're looking at a half hour execution time on the script. I'd rather not run a single process for that length of time. If I make the script sleep and restart every minute, I'm actually extending that time, but I'm starting a new process every minute so it can't eat a ridiculous amount of memory. If I fire off a separate request for each page to grab the images, the whole process may complete in far less time, but it would require 5-700 php scripts being fired in the space of 30 seconds... I'm leaning this direction because it's not much different than a site that has 1000-1400 visitors per minute, which can easily be handled by a properly configured server. If it's scheduled during the lowest traffic time of the day, it will have the least competition and there should be no visible affect to regular visitors on the front end. Unless of course there's a better solution! I googled for about an hour on this earlier, and while I'm quite positive I'm not the only person who's had to pull in a large amount of remote files, there doesn't seem to be too many people interested in writing about the methodology they employed to do so.

I'm hoping to get a little feedback on what you all believe is the best way to handle this efficiently in PHP. I am working on a script that imports a large amount of data from remote feeds; this facilitates the quick deployment of real estate web sites, but has to download a large number of images to each new site. Assuming for right now that the bottleneck isn't in the method (fsock vs curl vs...) and that for each imported listing we're spending between .89439 and 17.0601 seconds on the image import process alone... what would you suggest for handling this over the space of 100-1000 occurrences? As of right now I have two ideas in mind, both fairly rudimentary in nature. The first idea is to shut the script down every 30-45 seconds, sleep for a second and fire off another asynchronous request to start the script again. The second idea is to fire off a new asynchronous to run the image imports separate from the main script. This would let the efficient ones clear out rather quickly while the slower imports would have their own process to run in. The only thing that worries me about this is the fact that 100 of these could be fired off every second. Even assuming half of them complete before the next round are fired off, they would still pile up.

"It seems like I'm running over the two arrays far too many times, and that I should be able to do it in one pass somehow." It works; There are no errors that I'm trying to work out. I just thought I would pick everybody's brain to see if it was possible to do it more efficiently.

Bump from page 3... I fixed the formatting (tabs vs spaces, oops) and repasted: http://pastebin.com/yD7u8bZ4 I know it's something simple I'm missing. A change in the function itself, somewhere, is going to reduce the number of times I have to run it from 3 to 1.

"Cannot find search daemon". I tried. I've been wracking my brain for the last two hours trying to simplify this. It seems like I'm running over the two arrays far too many times, and that I should be able to do it in one pass somehow. I have two associative arrays, the first contains a set of older options (previous version), the second is a default set of new options. I want to compare the two arrays, and strip out any old options that aren't in the new set. I want to add options from the new set that aren't already in the old, and I don't want to overwrite and of the old options that already have values. Here's a copy of the sandbox I've been working it out in; it works, but it does four separate operations on the arrays to achieve the result: http://pastebin.com/erbFujkG

Thanks all! I could have sworn I wrote a reply to s0c0 yesterday but I must not have hit Post. The last three projects I've worked on have all required PayPal integration so I defaulted to their terminology, NVP or Name Value Pair, aka Key Value or however you wish to refer to it. While there's a high probability of some obvious exceptions, my mentality is that optimization will often go hand in hand with a clean looking query. In this case there's not a huge difference though I think your version is a touch more readable and have chosen to go with that, and the INNER JOIN. Again, thanks.

MySQL 5.1.37, this query works, I'm just wondering if it's the best way to do it? I'm not a huge fan of NVP style tables but I have to work on one in this case to pull the name of a customer in connection with their transaction ID from another table; SELECT t.transaction_id, f.meta_value, l.meta_value FROM `wp_purchase` AS t LEFT JOIN `wp_usermeta` AS f ON t.user_id = f.user_id LEFT JOIN `wp_usermeta` AS l ON f.user_id = l.user_id WHERE f.user_id = '1' AND f.meta_key = 'first_name' AND l.meta_key = 'last_name' Is there a more efficient query method for joining NVP tables to other properly built tables, or even just a better way to go about this particular one? It's not slowing me down by any means and as mentioned it gets the results I want but I just happen to have a strong feeling like there's a far better way to go about it.

That should work quite well actually, it slipped my mind to make use of the rel attribute for this. I'm working on other tickets right now, but once I wrap back around, I'll give that a go.

Only problem with that is that I'm using the value of the href to determine what element the toggle should occur on. I think what's wierd is as I mentioned, I've used both of these methods before with absolutely no problem (return false or .preventDefault), it's just in this one case that it's being a buzzard. The other option I'm considering is removing the A element altogether and just making a clickable div.. then I can use the ID of the div to attach the toggle to a span of the same class name.

e/event is the event handler for the function being called, such as .submit or .click in this case. .preventDefault() is supposed to prevent the default event from occurring.

Yeah no I tried that... due to my lack of experience with jQuery/JavaScript I also tried function(e) { e.preventDefault(); and function(event) { event.prevent... as I've seen it said both ways (by people who claim their way is the only way, but thats besides the point). I have a feeling something else is conflicting with it, because I've successfully done this before, but I have no idea what could do that, or how to track it down. I'm an avid Firebug fan but it doesn't show anything in the console about regular calls like this, and I don't know if I can figure out the DOM page.

You're fast. The outermost function is just the document ready though... I'm wanting to prevent the action as a result of the click?

jQuery(document).ready(function($) { $("a#help-button").click(function(e) { var spanid = $($(this).attr("href")); spanid.slideToggle('slow'); e.preventDefault(); return false; }); }); I'm sure there's probably a better way of doing what I'm doing here, so if you have any tips for that I'm all ears. The problem though is in preventDefault/return false... neither is working (on their own or together as illustrated) to stop the browser from trying to find the named anchor from the A element. So if I have <a href="#keyword" id="help-button">...</a> it appends #keyword to the url... I just want to use the href to find the element it should open, I don't want it to scroll anywhere.

Not sure if this makes a difference, but I've been watching FireBug while trying to figure this out. The first thing I noticed was that more headers were being sent in the form submit, probably due to some of the files I call in ajax-signup.php to process the post data. I unset them and now the headers sent by both scripts are exactly the same. Didn't solve the problem though. The other odd thing is that when the validation checks are posted, I get a 200 response from ajax-validate-signup.php, but when the form is submitted I get a 404 from ajax-signup.php... even though I know full well it's actually posting to the form considering I can see the json response and as I mentioned before there are new rows in my database to back it up. I'm baffled as to the 404. I double checked... both files are sending text/plain headers, similar json encoded responses, and then they exit. EDIT: That was the problem. I forced a 200 response header and suddenly everything works. Friggin nother 2 hours wasted!

This is an extension of my earlier post, now that I've gotten past the validation part I'm trying to seamlessly submit the form and have a success message displayed in place of the form. I think I'm doing something wrong though because the most I ever see is a flicker and the form resets itself. Here's the code: $("#signup_submit").click(function() { $("#signup_submit .inside img").attr("src", "http://images.assets.handmade.loc/images/ajax-loader.gif"); if ( user_field.val() == "Username..." || email_field.val() == "Email..." || user_msg == false || email_msg == false ) $("#signup_submit .inside img").attr("src", "http://images.assets.handmade.loc/images/menus/icon-delete.png"); if ( user_field.val() == "Username..." || email_field.val() == "Email..." ) { $("#user_msg").removeClass("success").addClass("error"); $("#email_msg").removeClass("success").addClass("error"); return false; } if ( user_msg == false ) { $("#user_msg").removeClass("success").addClass("error"); return false; } if ( email_msg == false ) { $("#email_msg").removeClass("success").addClass("error"); return false; } $.ajax({ url: "<?php echo site_url(); ?>/ajax-signup.php", data: "user=" + user_field.val() + "&email=" + email_field.val(), dataType: "json", type: "post", success: function(j) { if ( j.ok == true ) { $("#signup_form").replaceWith("<div id='smessage'></div>"); $("#smessage").html("<p>Success!</p>"); } else { $("#signup_form h3").html("Error!"); } } }); return false; }); I know it's working on the php side because I can see the entry in the database. It just doesn't... do... anything. else.

Ahahaha, I appreciate the help, I got it working! Only wasted half the day, but the results are what counts, I guess. jQuery(document).ready(function($) { var form = $("#setupform"); var user_field = $("#user_name"); var email_field = $("#user_email"); user_field.keyup( function() { validateSignup( this, 'user_msg' ); }); user_field.blur( function() { validateSignup( this, 'user_msg' ); }); email_field.blur( function() { validateSignup( this, 'email_msg' ); }); form.submit(function(){ if ( user_field.val() == "Username..." || email_field.val() == "Email..." || user_msg == false || email_msg == false ) return false; else return true; }); function validateSignup( t, vfield ) { var vmsg = $('#'+vfield); var action = vfield.substr(0,vfield.length-4); if ( t.value != t.lastValue ) { if (t.timer) clearTimeout(t.timer); vmsg.removeClass('error').removeClass('success').html('<img src="http://images.assets.handmade.loc/images/ajax-loader.gif" height="8" width="8"/> checking availability...'); this.timer = setTimeout(function() { $.ajax({ url: '<?php site_url(); ?>/ajax-signup.php', data: 'action=' + action + '&data=' + t.value, dataType: 'json', type: 'post', success: function (j) { vmsg.html(j.msg); if ( j.ok == false ) { vmsg.removeClass("success").addClass("error"); $('#setupform').attr("action", ""); window['' + vfield] = false; return false; } else if ( j.msg.indexOf("is available!") != -1 ) { vmsg.removeClass("error").addClass("success"); $('#setupform').attr("action", "/wp-signup.php"); window['' + vfield] = true; return true; } } }); }, 200); // timer t.lastValue = t.value; } // if } // validateSignup }); I had planned to wait until Pro PHP and jQuery was released to learn all of this, but holy hannah today was an eye opener.

I'm not sure where else I can declare it that makes sense though. The code I was using to try and call it looked like this: var user_type = 'user'; var email_type = 'email'; $('#user_name').bind("keyup", function() { validateSignup( user_type ); }); $('#user_email').bind("keyup", function() { validateSignup( email_type ); }); Unless I can make a chain out of it somehow? $().vt.val(user_type).keyup( validateSignup ) or something like that?

Just some quick background; I'm a PHP developer normally, I'm just learning how to use JS/AJAX (in particular jQuery, loves it), so while this problem has me completely baffled I'm sure it's ridiculously simple. I'm trying to validate a form before it's submitted. Here's a portion of my code that I've mangled together; jQuery(document).ready(function($) { var form = $('#setupform'); var user_msg = $('#user_msg'); var email_msg = $('#email_msg'); var user_type = 'user'; var email_type = 'email'; $('#user_name').keyup( validateName ); $("#user_email").blur( validateEmail ); form.submit(function(){ if ( $('#user_name').val() == "Username..." || $('#user_email').val() == "Email..." ) { return false; } else if( validateName() & validateEmail() ) { return true } else { return false; } }); function validateName() { var t = this; if ( this.value != this.lastValue ) { if (this.timer) clearTimeout(this.timer); user_msg.removeClass('error').removeClass('success').html('<img src="/images/ajax-loader.gif" height="8" width="8"/> checking availability...'); this.timer = setTimeout(function() { $.ajax({ url: '<?php site_url(); ?>/ajax-signup.php', data: 'action=user&data=' + t.value, dataType: 'json', type: 'post', success: function (j) { user_msg.html(j.msg); if ( j.ok == false ) { user_msg.removeClass("success").addClass("error"); $('#setupform').attr("action", ""); return false; } else if ( j.msg.indexOf("This username is available!") != -1 ) { user_msg.removeClass("error").addClass("success"); $('#setupform').attr("action", "/wp-signup.php"); return true; } } }); }, 200); // timer this.lastValue = this.value; } // if } // validateSignup // validateEmail is almost an exact duplicate, omitted for brevity Since these two functions, validateName and validateEmail are nearly identical, it makes no sense to me to have two seperate functions. In PHP this would be dirt simple... pass a string (user or email) to the function and change the field and data internally based on the value. So, with my menial knowledge set (and believe me, I googled the heck out of this and I'm still coming up short), I tried: function validateSignup( vt ) { var t = this; if ( vt == 'user' ) { var tmsg = user_msg; } else { var tmsg = email_msg; } if ( this.value != this.lastValue ) { if (this.timer) clearTimeout(this.timer); tmsg.removeClass('error').removeClass('success').html('<img src="http://images.assets.handmade.loc/images/ajax-loader.gif" height="8" width="8"/> checking availability...'); this.timer = setTimeout(function() { $.ajax({ url: '<?php site_url(); ?>/ajax-signup.php', data: 'action=user&data=' + t.value, dataType: 'json', type: 'post', success: function (j) { tmsg.html(j.msg); if ( j.ok == false ) { tmsg.removeClass("success").addClass("error"); $('#setupform').attr("action", ""); return false; } else if ( j.msg.indexOf("This username is available!") != -1 ) { tmsg.removeClass("error").addClass("success"); $('#setupform').attr("action", "/wp-signup.php"); return true; } } }); }, 200); // timer this.lastValue = this.value; } // if } // validateSignup Amongst a plethora of variations. None of which work. I can get the value of vt just fine, but the rest of the function fails the instant I add it as a parameter, and I'm clueless as to why. Help!

http://dev.mysql.com/doc/refman/5.1/en/date-and-time-functions.html#function_date-add SELECT name, email, joined FROM `users` WHERE joined = DATE_SUB( NOW(), INTERVAL -51 week ) For an example. Of course you'll have to figure out the formatting, and you'll probably want to SUBSTR your date field, because this by itself will look for an entry with the exact same time of day. You'll want to search by just the day, I'm sure. It might be easier in the future if when the member is added to the database you included a column with the expiration date. Then you could avoid the SQL-FU.

I should have supplied an example, sorry -- however that's exactly what I was referring to. Good to know, that means the fault earlier today must have been between keyboard and chair, as it's been running smooth ever since.

I maintain an application that requires an aggregate method to run once daily. As of right now it's running under a sudo-cron application and as such is tripped when the site is accessed at or after a certain time (no, it can't be done directly via cron). Since the script needs to be allowed to finish, the primary method it activates sets ignore_user_abort() to true and set_time_limit() to 0. I may have been the cause since I've been checking in and out of svn all day, but despite this I had some data truncated during one of the test runs which is being fired off hourly for debugging. The faulty data is generated during the running of a seperate method which is called by this parent method (the allegedly non-abortable one). Therefore the question is raised, does ignore_user_abort() roll downhill? Any private methods run within the primary method should automatically be covered under the original ignore_user_abort() call should they not, or does the call have to reside in each and every function that's being run? I'm attempting to avoid this fiasco by using an asynchronous http request in the next version of the app, however given the chance that fsockopen could fail I've written it in such a way as to fall back on the original method of running the aggregate. To that end, any information about ignore_user_abort() would be greatly appreciated!

Hey all, just dropping in and checking things out - I'm a regular on another board (which I won't post here so I don't seem like I'm advertising), which is a lot smaller, but I likes it. However I'm in the market for an experienced programmer to work on an open source project with me, one that I've been developing for the past four months. Before I post details and contact information, I was curious if you guys were cool with that kind of thing over here, and where would be the best place to post it if so? Thanks, and hope to hear from y'all soon. -m