Cooperative multitasking, promises, and future ticks

[NotionCommotion]

I was trying to prevent bottlenecks when sending emails, and the below post was the solution.

Similar predicament now, but I also need to generate some images first so they may be attached to the emails and the images are generated by a fairly slow command line program that emulates a browser.  Multiple emails might or might need the same image so I plan on creating all the images before queuing the emails.

I believe the futureTick approach executes each tick in series as blocking, but doesn't attempt to execute all of them at once so it doesn't impact the rest of the application.  Promises, however, appear to be more about a structured process for managing callback functions.  As such, the futureTick approach would likely be best for tasks requiring much PHP processing, and promises are more suitable for tasks which PHP is just waiting for a response from some other service.  True?

Would one ever want to attempt to use both approaches?  Seems like it might make sense if the code within a promise was PHP intensive to break it down into multiple futureTick tasks, but not not feed promises into the futureTick as promises are "kind of" non-blocking, true?

To execute the CLI image program, considering https://github.com/clue/reactphp-shell.  Or do you think plan old exec() would be applicable?

For using promises, would it just look something like the following?

$deferred = new React\Promise\Deferred();

$deferred->promise()
->then(function ($rs) {
    // create the first image
})->then(function ($rs) {
    // create the second image
})->then(function ($rs) {
    // create the last image
})->then(function ($rs) {
    // send the first email
})->then(function ($rs) {
    // send the second email
})->then(function ($rs) {
    // send the last email
});

$deferred->resolve($nothing?); 

Or maybe something like the following?  I would be amazed if it works as is, but hopefully is on the right track.

$deferred = new React\Promise\Deferred();
$deferred->promise()
->then(function ($rs) {
    // Create images
    // See https://github.com/clue/reactphp-shell
    $launcher = new ProcessLauncher($loop);
    $shell = $launcher->createDeferredShell('bash');

    $shell
    ->execute('create the first image')
    ->execute('create the second image')
    ->execute('create the lst image')
    
    ->then(function ($rs) use($loop) {
        // Send emails
        // Just showing this way but will actually use kicken's EmailQueue solution
        $loop->futureTick(function () {
            // send the first email
        });
        $loop->futureTick(function () {
            // send the second email
        });
        $loop->futureTick(function () {
            // send the last email
        });
    });
});

$deferred->resolve($someQueueOfImages?); 

 

[NotionCommotion]

Actually, worked better than I expected.  Still would appreciate any feedback. Thanks
 

$time=microtime(true);

logger('start');

$loop = \React\EventLoop\Factory::create();
$launcher = new ProcessLauncher($loop);
$shell = $launcher->createDeferredShell('bash');

$promises = [];
$shell = $launcher->createDeferredShell('bash');
logger('create bash promises');
foreach($cmds as $name=>$cmd) {
    $promises[] = $shell->execute($cmd)->then(
        function ($value) use($name) {
            logger($name.' image complete.  Do anything here?  $value is the output of the script');
            //throw new \Exception('error in items promise');
        },
        function (\Throwable $e) {
            logger('image error handler.  What should be done here?: '.$e->getMessage());
        }
    );
}

logger('bash promises complete');
$allImagePromises = \React\Promise\all($promises);

logger('all promise complete');

$filenames = array_keys($cmds);

$allImagePromises->then(
    function($value) use($filenames){
        logger('then all image promises are resolved so deal with emails.  Do anything with $value which is [0=>null, 1=>null, 2=>null]');
        //throw new \Exception('error in email promise');
        $files = [];
        foreach($filenames as $file) {
            $files[] = sprintf('name: %s type: %s size: %d', $file, filetype($file), filesize($file));
        }
        logger(implode(', ',$files));
        // Instead of using futureTick, maybe just add emails as promises?
    },
    function (\Throwable $e) {
        logger('images error handler.  What should be done here?: '.$e->getMessage());
    }
)->done(
    function($value){
        logger('done. what should be performed here?');
    }
);

// Should I be using Deferred::resolve()???

logger('before shell end');
$shell->end();

logger('after shell end');
$loop->run();

Quote

time: 0.000002 message: start
time: 0.005833 message: create bash promises
time: 0.006871 message: bash promises complete
time: 0.007278 message: all promise complete
time: 0.007335 message: before shell end
time: 0.007353 message: after shell end
time: 0.698389 message: item1.png image complete.  Do anything here?  $value is the output of the script
time: 1.407365 message: item2.png image complete.  Do anything here?  $value is the output of the script
time: 2.099762 message: item3.png image complete.  Do anything here?  $value is the output of the script
time: 2.099877 message: then all image promises are resolved so deal with emails.  Do anything with $value which is [0=>null, 1=>null, 2=>null]
time: 2.099937 message: name: item1.png type: file size: 38986, name: item2.png type: file size: 38986, name: item3.png type: file size: 38986
time: 2.099970 message: done. what should be performed here?

 

[kicken]

2 hours ago, NotionCommotion said:

I believe the futureTick approach executes each tick in series as blocking

PHP is single-threaded*, so yea each tick callback is run in series and must finish before the next tick callback can be run.  Using event-based processing lets you make things that seem like they can multi-task but they don't really.

2 hours ago, NotionCommotion said:

As such, the futureTick approach would likely be best for tasks requiring much PHP processing, and promises are more suitable for tasks which PHP is just waiting for a response from some other service.

A promise can be used to do work as well, it doesn't have to just wait for a result.  Conceptually they are something that is used to just wait for something to finish, but that doesn't mean it has to be some external process.  You could create a promise that just does work within PHP, such as that emailer queue in the previous thread.  That could be implemented as a promise if one desired.

From what I've seen, one of the biggest things people generally don't understand about promises is that they are not something that can just magically wait until another thing is done.  They need to periodically check if that other thing is done.  Since PHP is single-threaded, they can't just start up a background-thread to do that.  Even if they could, the main thread would still need to know not to exit while the promise is pending.  What is needed then is a way to hook into the main loop so the promise can do it's work from time to time and this is what functions like futureTick and addPeriodicTimer are for.

These functions provide a way for the promise to register a callback with the main loop so that they will occasionally gain control of the main thread and be able to do whatever needs done.  For promises that are just waiting on something, that likely is just a simple check to see if the thing is ready yet and nothing more.  For something like the email queue, that would be sending out a small batch of emails.

If you don't need the extra functionallity of the promise interface, then futureTick / addPeriodicTimer can still be used to just hook into the event loop for whatever reason.  That's what the original email queue does, it hooks the loop to allow processing of emails but doesn't go so far as to provide a full promise implementation.

As for your task, the react project has a package for running commands that you can use: reactphp/child-process.  I think using that would be your best bet to handle your image generation, the shell thing you linked seems unnecessary and I'm not sure if it'd handle running commands in parallel or not.  With the child-process package you'd just have to create a small function to execute your process and return a promise.  That promise would be resolved when the process emits a exit event.  For example:

function executeCommand($loop, $command){
	$deferred = new React\Promise\Deferred();
	$process = new React\ChildProcess\Process($command);
	$process->on('exit', function ($code) use ($deferred){
		if ($code === 0){
			$deferred->resolve();
		} else {
			$deferred->reject();
		}
	});
	$process->start($loop);

	return $deferred->promise();
}

With that, you can generate all your images in parallel using something like:

$promiseList = [];
foreach ($imageCommand as $cmd){
	$promiseList[] = executeCommand($loop, $cmd);
}

React\Promise\all($promiseList)->done(function(){
	//All images generated, send your emails.
});

 

* pthreads and pht provide some threading capability, but I've never tried using them.

[NotionCommotion]

Thanks kicken!  I updated my original post before even knowing you commented, however, your advise is 100% relevant.

29 minutes ago, kicken said:

From what I've seen, one of the biggest things people generally don't understand about promises is that they are not something that can just magically wait until another thing is done.

While I wish promises were magical, I unfortunately don't believe so but instead boringly somehow execute a quick loop which constantly checks if anything is ready to perform work.

29 minutes ago, kicken said:

As for your task, the react project has a package for running commands that you can use: reactphp/child-process.

I have some limited experience with it and will focus more on it.  Thank you for your example of use.

29 minutes ago, kicken said:

* pthreads and pht provide some threading capability, but I've never tried using them.

Almost didn't see this, and never knew they were options.  Another maybe option I know nothing about is https://www.php.net/manual/en/intro.parallel.php.