fenway

Well, substring will help you grab each field -- but you'll have to store the output of that somehow, somewhere.

Merged? I thought if I higher priority row was found, you update everything.

I can use find_in_set, but when i search for '2', it will find me all the numbers containing 2 (210, 12..) It is similar to LIKE, i think No, it won't.

I don't know what any of that means. How do each of those PHP scripts differ?

It's not crazy -- you're not escaping your string.

Never. But I doubt you removed single quotes when you cut & pasted.

But you said the duplicates are based on ID alone -- that's a single column, not 200.

If you call mysql_fetch_row() twice, you'll be skipping every other result.

39 fields that shouldn't be in the table to begin with?

I'm not sure what you mean -- have the priority as one of the columns, and just check if it's greater or not -- that's not time consuming at all.

To be fair, the number of JOINs won't be a problem -- it's a single index-key lookup, should be instant -- but the HAVING can be a problem, triggers a filesort, etc.

That means the UTF-8 isn't "set okay".

I didn't say not to do it in MySQL -- there is "INSERT ON DUPLICATE KEY UPDATE".

If they're really just integers separated by commas, you can "cheat" for now with FIND_IN_SET().

Well, short of crash recovery, you'll get no benefit from InnoDB here -- even with table-level locks, you have concurrent insert with MyISAM. And INSERT DELAYED can help you here, too. No, nothing you can do about indexes with ARCHIVE. Other options include using compressed MyISAM and MERGE all of the tables -- keep the most recent one uncompressed so it's not read-only. Sort of the best of both worlds. Partitioning is technically an option, e.g. on user_id, but that's not pretty.

Then something is stepping on something else.

Ignoring obvious SQL quoting problems, prove that what you're inserting works.

Then post what you figured out -- other people will benefit from your experience.

Sorting 70K rows isn't great -- but yes, removing the subqueries will help.

Post the query you have so far.

Why not just have a unique key on ID, and decide based on priority to UPDATE or not?

Yes, but not noticeably so. Also, don't mix * and GROUP BY -- you'll get garbage data. Selecting "more" with one of each payment type doesn't make any sense.

Duplicates how?

Are you sure your connections work?

I'm guessing you don't have any indexes. Post EXPLAIN output.