gnznroses

I'm trying to figure out how to keep a script executing after it reaches the timeout limit. It has always worked fine on PHP 5.2, but I just updated to PHP 5.4 and now scripts immediately quit executing after the allotted time. Brief summary of my problem: I have two pages in my site's admin panel that usually take some time to run. The first crunches some data and displays the results. It sometimes will time out before completing, but if you wait a moment to let it finish in the background and then refresh your browser, it'll display almost immediately, since MySQL has cached the results. A second page has to update a lot of records and sometimes will time out. I need the final output of the script, so after the rows update I log the output to a txt file which I can FTP in and read if need be. This is how things used to work, but now after the upgrade, when the page times out, script execution stops, and for the latter script that means that rows get updated but no log is ever created, which is a problem. I have tried: set_time_limit(0); ignore_user_abort(true); No effect. I think set_time_limit(0) fails to work because Apache itself times out? Although when the PHP timeout is set to 50, and Apache timeout set to 60, and it seems like execution stops at 50. (We're actually using Litespeed instead of Apache, but it reads the Apache ini file and everything. And we didn't update/change Litespeed when we updated PHP.) I don't want to raise the Apache timeout beyond 60 for sake of DDoS issues (we've been attacked before). From my understanding there's no way to raise the limit for just one page. I found some things that I thought might do that, for Litespeed, but they had no effect: In my htaccess file: RewriteEngine on RewriteRule ^$ - [E=noconntimeout:1] RewriteRule ^$ - [E=noabort:1] Before upgrading PHP I poured over the list of changes on php.net, and never found anything related to this. I've been relying on execution after the timeout for years (for these same two pages), so the sudden change and lack of a solution so far is strange to me. I could attempt to separate some code into chunks and on a cron, but I'd rather not. Any ideas? Thanks!

I would like to get others' opinion on whether or not to use Joomla for a project that I and a team are starting. I have no experience with Joomla, and have always developed my projects purely in my own PHP code. To me this seems like the best way forward for our new project as well, but one of our top developers believes fully in using Joomla. The project involves a fairly complex website, with lots of functionality -- two types of user accounts, administrator accounts/panel to manage/monitor users, transactions between users (eg purchases), and use of third-party APIs (one of our own + Facebook sign-in, etc). I typically think of a CMS as being used for blog/article/content sites, but I understand that any needed functionality could be coded into extensions/plugins/modules/components. The website will be responsive and adapt to mobile screens. There will also be native mobile apps. The core functionality of everything will be written as PHP web services, which both the website and app will tap into. If I had to give an example of what the project is similar to I'd have to say it's akin to the Google Play Store / etc, where you have developers, users, media being searched/browsed/purchased, and all of the related account functions for those two types of users. We would like people on our team who aren't familiar with coding to be able to make some content changes to the site. Although any changes would ideally be commitable to our subversion repository. I worry about a few things: - ability to maintain local/subversion copies of everything, when content changes are made via the Joomla template editing system - spreading both our PHP and HTML code into lots of modules/templates/etc, making it confusing to edit - variable scope for when different website elements need to share variables/data - security/bugs and having to apply patches which may break something - whether we truly have as much flexibility as a custom-coded project. And if we have to step outside the extensions and edit actual joomla files for any reason then applying updates is a major headache - having to learn a new way of doing things for questionable benefits I'm told the benefits are: - easier to design/edit/manage the front-end - easier (automatic?) to have the site be responsive - some built-in functionality that we might use, for accounts, comments, searching, etc? I don't personally see the benefits of using a Content Management System for this kind of project, where managing the content is a very small aspect of the bigger picture. Using the CMS as a base for what is otherwise a custom-coded project, and having to adapt to fit the CMS's architecture. It seems any benefits would be outweighed by the downsides. Am I wrong here?

ah, yeah and i just blindly copied it lol. but result is the same since the "If" line doesn't have the error. i guess my problem isn't a real problem, it's just that referrers aren't being passed in situations that i would expect them to. for example on pastehtml, i don't see why they're not. if you create a page with just a link to google.com for example, and use LiveHTTPHeaders to look at the headers when you click it, no Referer is being passed. not sure why. so i set up a test page on a domain of mine and tested with both a link and an iframe from there and it gets passed both times. and works as it should in my script. so it's not a mod_rewrite or apache problem appreciate the help man

oh i actually do need to accept any kind of character and not just an integer $SERVER['HTTP_REFERER'] is not set at all and hmm, oddly enough, if i follow a link to the "real" url, mydomain.com/links.php?id=123 it's still not set even if i go to pastehtml.com and enter a link to the page, then click that. it should have a referer either from the click or just the fact that it's i-framed (which i believe always sets the referer as the parent page)

I have a rewrite to make mydomain.com/123 load the script at mydomain.com/links.php?id=123 links.php generates html with some links on the page. the links are supposed to have the referring url appended to them in the query string ( eg http://whatever.com/s.php?ref=<?=$_SERVER['HTTP_REFERER'];?> ), but it's always blank no matter what. would the rewrite cause HTTP_REFERER to be blanked out? what else might cause this? RewriteEngine on Options +FollowSymlinks RewriteCond %{HTTP_HOST} ^www\.mydomain.com [NC] RewriteRule ^(.*) http://mydomain.com/$1 [L,R=301] RewriteCond %{REQUEST_URI} !^/$ RewriteCond %{REQUEST_URI} !^/links\.php RewriteCond %{REQUEST_URI} !^/robots\.txt RewriteRule ^(.*)$ links.php?id=$1 [L]

yep. well i ran it when things were about 80% done. i was showing about 500MB of overhead at the time, and it eliminated that. when i ran it it performed a Repair By Sorting. when i changed the data type it also did a Repair By Sorting. i thought that would completely rebuild the table but maybe not. there has to be more than 500-700MB freed up though... I was expecting about a 40% reduction in total size. I have 12 columns in that table. Most are Ints, one is DateTime, one is VarChar with a limit of 60, and then one other Text column (useragents) which i was going to replace via the same method as the URLs. ... actually 40% would be optimistic, since the table is 11GB of data and 5GB of index, and i'd only be eliminating space from the Data portion, but still... fyi also, the URLs column wasn't, and still isn't indexed. I plan on adding an index now that they're IDs.

k well, i had the URLs stored as Text, which was bad. so i replaced that with Integers that hold ID numbers. the URLs are now stored in another table (using VarChar and a Unique index). This was because of the amount of duplicated data. Instead of storing the same URL over and over in my main table, I store it just once in the new table and then have it's ID number in the rows of the main table. When I made the change, I did so by using a cronjob which went through every row of the main table, attempted to insert the URL into the new table, and then whether it inserted or failed due to the unique index, the URL gets replaced by it's ID. Because I was simply over-writing the URL itself with it's ID, the column type stayed as Text during this, until I finished with updating every single row. Once all the rows were updated to contain IDs, I changed the column type to Integer. But, even though I eliminated what was surely gigs and gigs of textual data, the total size of my table barely dropped. Hope this makes sense.

"were" that's why i was making the changes that i made so yeah that's not my question lol

Huh? I'm not

I have a table with over 50 Million rows. Two of the columns were Text fields, storing URLs and UserAgents. I just completed a change wherein I moved the URLs to a separate table and changed the field to Int, to store ID numbers pointing to the associated URLs in the other table. In doing this I expected the filesize of the table to plummet, but it barely changed. The table was 16 GB, and I assumed that storing a URL for each row (with an average length of 40-50 characters, but sometimes 300+ characters) was about 40% of the total size. After changing over to IDs the table is still over 15GB. What kind of sense does that make? The table is MyISAM and I understand that Text data is stored outside of the row itself, but it's still obviously part of the table and contained in the MYD file, so it should factor into the total size just the same. But it doesn't?

Yeah we rarely use data that is more than six months old, but we don't want to trash the old data either. I suppose what I could do is create the two new tables and apply the idea only to old data. Store the data in the main table as usual but then once it becomes outdated, replace the referring URLs and useragents with IDs (although it'd be ID numbers stored as strings). Then I could just add an index to the existing referrer and useragent columns in the main table, for viewing real-time stats on the live data. Does that sound like a good plan? I have indexes on 8 different columns (out of 14). I need to optimize some of it and create multi-column indexes, I just hate that any change to an index, a data type, etc, takes forever and brings our site offline. I'm expecting an hour or two of downtime with each index change, but I'm just guessing.

Thanks, Hmm so you think pulling those out into separate tables is a bad idea? There's a couple reasons I want to do that: 1. So i can easily run queries on those new IDs, for example a Group By on the referring URL ID, with a join to get the actual URL associated with the ID. I want to be able to see the number of hits per URL and currently that's pretty slow (no index on it now, mostly because of the next reason below) 2. Our table is currently 16 GB with a 5 GB index. It'd be 3x bigger but we've had two previous database crashes that corrupted the table. When this happened I was basically helpless to try and repair the table because of the amount of time (hours) that that was taking, and the whole time the website is down. I'm sure filesize, and not just the number of rows, is a factor in things like that (?). Restoring from a backup is a lengthy process as well (remote backup), and I also can't easily download the data to my home PC. We're adding hundreds of thousands of rows daily, so the filesize is just going to keep growing.

I'm handling a large amount of traffic and need to record details of each visit. I record things such as the referring URL and the user agent. I've been recording these things as Text columns in each row, and consequently the table is becoming very large. I would like to make two new tables, one for referrer and one for user agent, and then record just ID numbers in the main table. I'm worried about the time needed to query these two tables and fetch the ID numbers. A useragent can be 80-200 characters long, and a referring URL can be 20-200+ characters. I don't know how many rows would be in each of these two tables but I'm recording *lots* of duplicates -- in one day we might 100,000 hits from the same referrer, so it makes sense to normalize the info, in theory. I don't want there to be any lag caused by this though. If it took 1/4 second to do a query that'd be an issue. What kind of performance can I expect on indexed text fields (or VarChar i suppose) with millions of rows of this kind of data? Bonus question: once I launch the new method of recording data I'm going to need to change the existing table. I imagine using a cron to process all the rows, starting from the oldest, and substituting in the ID numbers in places of the actual text. But keeping the columns as Text fields until all rows are processed and I can change the column types to Int. When I change the column types it's going to suck basically. I expect the table to lock for at least 2 hours for each of the two column changes? It's 53 Million rows (MyISAM btw).

One of our database tables became corrupted during an apparent server crash. The table gets written to almost 24/7 so I'm sure it was in the midst of a write when the server crashed. I spent several hours trying to restore the table from our daily backups, but ran into a number of different issues and had no luck. It kept saying the table was corrupt even when it was a restored backup version. I eventually had to delete the table and start it over from scratch, but I'm hoping to be able to recover and re-import the data. I'm hoping someone can explain a bit about the issues I had, because I don't want the table to be corrupted again if I start importing data back in. I'll try and keep it short. Things went about like this: 1. I discovered that any new inserts to the table were failing. It would always return an error saying that the new record was a duplicate on ID 55097424. Should be impossible since it's auto-increment (the only unique column). The auto-increment value was at 55097423. 2. I did a Check Table and it told me: 3. I began a Repair Table but then realized that it would take ages (17gb table) so I stopped it. 4. I renamed the table's three files (.frm, .myd, .myi) as a backup (table_bak) and started a new repair while I retrieve yesterday's copy of the table from our backup server. 5. Retrieved yesterday's backup, when everything was working OK. Copied those three files into the mysql directory. The database (and the corrupted "bak" copies) both show up in phpmyadmin just fine. 6. Inserts are still failing, even with the restored table, with the same error as before (except with a lower auto-increment ID, since the backup had fewer rows). I run a Check Table and it tells me it's corrupt also. 7. At this point the table becomes inaccessible from phpmyadmin, since it's now marked as crashed, but now when I click on it I receive an error "Got error 28 from storage engine" (instead of "Table xxx is marked as crashed and should be repaired"). This apparently means "out of storage space" but our mysql drive is 48% free. Our /tmp drive however is 100% full. Killing the repair on the bak copy brings our /tmp usage back to 4% and now I get "Table xxx is marked as crashed and should be repaired" again. 8. I tried restoring a backup from 2 days priot and it says it's corrupt also. 9. I gave up and deleted the table, re-created it, and set the auto-increment a little above what it was before the crash, so I can fill in the old data later, if possible. Everything works fine. I'm stumped as to why it kept saying my table was corrupt, even on the backup copies. Almost like it is remembering that it's corrupt rather than actually checking if it is, but it shouldn't be doing that. I read that the "flagged as crashed" flag is in the index file (.myi) that I restored/over-wrote. So now I'm wondering if I am able to repair the old table and get my data back, if it's safe to import it back into my new table.

i found the solution, sorta. making a composite index on the two columns. the problem is, the table actually has about a dozen columns, not just the ones i included in my code. and there's several different combinations of columns that we use in various queries (campaign+user, campaign+date, user+date, etc etc). so i would almost need multiple composite indexes... so anytime you're selecting from one column (and using either distinct or group by) and filtering by another, it has to use a temporary table unless you have a composite index on both columns?