PHP Background Tasks?

[dontknowphp]

Hello,

I am trying to find the best possible way to run php functions/files in the background without affecting the browser/user experience.

 

Problem:

Some tasks take 10-20 seconds to execute, when the php code is being executed and it has to process a lot of data, the browser hangs and I can not use the admin dashboard (even if i open a new tab and open a different page - it does not load unless the previous tab has completed executing)

 

Maybe if I run the php functions as a cron job (and show a progress bar to the user) will solve that issue?

e.g Plesk Hosting Panel, when you execute a cron job it runs on the background and you can use their admin dashboard without issues.

[ginerjm]

I would wonder about what I was doing if I had a task that took 20 seconds of execution time....

Are you by any chance running queries that process only one record at a time and doing a lot of them?  That's not good.  Or are you processing a data collection in a loop that executes multiple times to do something for you?

20 seconds is a huge amount of time in the computer processing world.  Unless you have a couple hundred thousand records that you go thru I think there is something wrong.

And no you cannot run a cron job that converses with a client/user.  Cron jobs are run as a completely isolated task by the server with no concurrent contact with anything else.  As for a 'hosting panel' and a cron job - again - the two do not converse.  The cron job runs when it is scheduled to do so and you working thru that panel or any other such as phpadmin will not interfere with the cron job not communicate with it.

[Barand]

It could be the sheer quantity of data, in which case a cron job would be a good move. On the other hand you may be using a really inefficient method for processing the data, in which case optimization could be the answer.

We have no way of knowing.

[ginerjm]

.... without seeing your code.  Hint, hint...

[kicken]

43 minutes ago, dontknowphp said:

even if i open a new tab and open a different page - it does not load unless the previous tab has completed executing

That's a symptom of session locking.  You can work around that by having your long-running process call session_write_close before it starts.

Assuming your long delay is an actual problem and any a symptom of poor code, there are a variety of ways to handle moving the task to a background process.  Cron is a common and generally easy way to handle it.  Another way is to use a job queue / message queue and separate worker services.  Examples would be gearman, beanstalkd, rabbitmq, zeromq, and others.

[dontknowphp]

hello,

 

thank you for the replies, the task that is taking long enough is not due to the code but its the amount of data that is processing.

Use Case:

process a product feed from an ecommerce platform and import them to another platform. If the products are >10000 it will obviously start to get slow, especially if the images are downloaded as well.

 

I will check gearman and get back if i have any questions.

is the cron job a good approach tho? do cron jobs still get limited by server limits (max execution time etc?)

[kicken]

4 minutes ago, dontknowphp said:

is the cron job a good approach tho

Cron is fine, you just have to use some care when implementing it. 

• You have to devise some way of passing information to your cron job, for example by writing things out to a DB table which the cron job then checks when it runs.
• There will be some amount of delay between when you add the task to the queue and when then cron job next runs to process it.  Running the job every minute to check for would would minimize that delay.
• If you're running your job every minute, you need to make sure that while one instance is processing the task, later instances do not attempt to pick up and process the same task.
• You'll probably want some way to re-try the task in case the instance processing it crashes and does not complete the task.

10 minutes ago, dontknowphp said:

do cron jobs still get limited by server limits

Make sure you use the php cli interface to run your job and you can configure different limits.  Max execution time is disabled by default for CLI.

[maxxd]

I worked with a system for a while that used AWS SQS to feed tasks and data into smaller php functions (microservice-style) that were long-polling for changes in the queue. Offloads all the data processing to a completely different thread (or server in some instances). However, one major drawback is that it can be difficult to reason about and document once you get more than two or three "microservices" handing the processing.

[dontknowphp]

Hi,

 

I am using this library to manage crons.

https://github.com/TiBeN/CrontabManager

I am successfully creating cron jobs in the crontab however (i can see them), the crontab seems never to be running. The admin_task.php is not being executed

<?php

	require_once("content/classes/class.crontab.php");
	require_once("content/classes/class.crontabcli.php");
    require_once("content/classes/class.cronjob.php");
    $crontabRepository = new CrontabRepository(new CrontabAdapter("user", false));


	$ar = $crontabRepository->getJobs();
	print_r($ar, true);
    $crontabJob = new CrontabJob(false, true);
    $crontabJob
        ->setMinutes(1)
        ->setHours('*')
        ->setDayOfMonth('*')
        ->setMonths('*')
        ->setDayOfWeek('*')
        ->setTaskCommandLine('php -f content/admin/admin_task.php')
        ->setComments('Logging disk usage'); // Comments are persisted in the crontab

    $crontabRepository->addJob($crontabJob);
    $crontabRepository->persist();



?>

 

[kicken]

You should specify the full path to your php file in the cron task.  Relative paths are not useful as you don't know what the base directory will be when the task is executed.

Rather than adding/removing tasks from your cron file via code, manually configure a task to run at a specific interval and then have that task's code check for work either in a DB or in a file somewhere.  Then have your code add/remove tasks by updating that DB/file.

 

[dontknowphp]

17 hours ago, kicken said:

You should specify the full path to your php file in the cron task.  Relative paths are not useful as you don't know what the base directory will be when the task is executed.

Rather than adding/removing tasks from your cron file via code, manually configure a task to run at a specific interval and then have that task's code check for work either in a DB or in a file somewhere.  Then have your code add/remove tasks by updating that DB/file.

 

thank you, full path worked thank you.

Yes, i will make a "task manager" class later.

 

Is there any way to pass parameters in the cronjob? I want to execute the following code as a cron but idk how to pass the parameters for the processProducts function ($vendor_box, $selected_language, etc.)

The value of the parameters do not contain simple string values, but arrays instead.

<?php

require_once("content/classes/class.feed_import.php");
$feed_import_helper = new FeedImport();
$feed_import_helper->processProducts($vendor_box, $selected_language, $selected_options, $selected_cid, $parsed_data, $selected_products);

?>

 

[kicken]

If you're going to create separate cron tab entries for your jobs, you can pass them as part of the command like just like you would any other app.

>setTaskCommandLine('php -f content/admin/admin_task.php "vendor_box" "selected_language" "selected_options" "selected_cid" "parsed_data" "selected_products"')

You'd read them using $argv in your script.

$vendor_box=$argv[1];
$selected_language=$argv[2];
$selected_options=$argv[3];
$selected_cid=$argv[4];
$parsed_data=$argv[5];
$selected_products=$argv[6];
$feed_import_helper->processProducts($vendor_box, $selected_language, $selected_options, $selected_cid, $parsed_data, $selected_products);

You have to be very careful doing something like this though as if you do not properly escape the parameters you open up the potential for a user to run any arbitrary command as part of the cron job.

This is one of the reasons why I suggest you do not create dynamic crontab entries and just manually create one that checks for work elsewhere.  Doing it that way, passing parameters needs to be part of whatever you code for your job management setup.  A common way of handling this would be to simply json_encode an array of data and save it to your file/db then have your crontab worker json_decode that data back into the necessary parameters.