Resque jobs

[Alex_]

Hey..

 

 

So lately I've been hooking up large requests to resque jobs in my application, for the purpose of not keeping the user(s) waiting for a longer period of time before they can continue using the application. At the same time, I have a frontend library & eventlistener alerting the users when any specific job was actually completed by the workers. I have realized however that in some cases, this doesn't pan out.

 

I love the idea of resque jobs, but my gut feeling is that I somehow lose control of the request I'm processing. The reason I'm posting this is because there's one part of my application that processes an insane amount of data, in a loop(!). Obviously this can take a long time, and that's to be expected. But when an error occurs along the way, I have no good way of logging it nor alerting the user about it.

 

Picture a scenario where a Worker is doing a Job that would take 2 minutes, and is 80% done, at which point it fails for whatever reason. How would you recommend something like this to be handled?

 

Obviously I get internal logs from the jobs the workers perform, but more specifically how would you catch the errors right from the script? Currently this is my setup...

 

 

try {
...//code that takes > 1min
} catch(AppException $error) {
...//Internal code error, catch
} catch(SeqDbException $dbError) {
..//DbError or Query
} catch(SystemException $exception) {
..//More severe exceptions
}

 

 

Either I'm missing a catch, or I'm missing something else. The errors rarely occur as well, so it's not easy for me to debug it or re-create them. It's very dependent on the data that is being processed.

[QuickOldCar]

Ever think about processing data background a cron and cache results?

 

It seems to me from your other posts no matter what you do live for a client is gonna take too long.

 

Maybe even break the process up more, does this data infinitely grow over time?

 

If it's something that struggles the server or takes too long it probably shouldn't be done that method.

 

You may want to rethink your application and grab particular data only when need it.

 

How about to process all old information and cache results, then only merge new information as needed.

 

I still don't know your entire application.....but you can create a que system that could track smaller tasks and if all competed. When entire jobs are completed it removes from the que.

 

I imagine a que system for both jobs and tasks

 

control when jobs are started, either by cron or on demand

 

define a tasks list per job, save number of tasks to be completed

create a cron to cycle all jobs in job_que,  check for all tasks completed

if all tasks that job completed remove from job_que

if all tasks that job not completed and a task not currently running... start lowest sequential task not completed

 

add job to job_que

start job

mark job as running

mark which task started

 

 

add task to tasks_que

start lowest sequential task number to be processed

mark task as running

process...

task ends and marks task as completed

if task completed continue next task if is one

if no more tasks mark job completed and remove from job_que

[kicken]

Basically in whatever script you have that polls for the status of the background job you need to ensure it will detect if the job failed and if so, report any available error information to the user. Within your background jobs, if there is a failure you need to ensure that you log the error messages somewhere that the polling script will be able to later locate them and report them if necessary.

 

The script that kicks off the background job is not the right place to check for processing errors. The only errors you'd get at that point is if the job failed to queue for some reason. Once queued that script will be done, and any further reporting is up to your status polling script.

 

I'm not familiar with the system you are using so I can't really give any specific advice on how to actually implement any error reporting. A quick look at the README file for the project suggests it will report failures so you can probably at least detect if the job failed and let the user know. I'm not sure if it provides a method of providing the details of the failure (in case you want to display that) or if you'll need to find another way to do that.