Queue – Deferred execution in CakePHP

At the beginning of 2.x I ported the Queue plugin from 1.x to 2.0 and started to integrate it into a few apps of mine.
Since then it was running OK, but neither with fixed tests, nor with any real documentation.
I fixed all that up – the plugin is available at github.

Background

Since this whole topic isn’t really covered by the book at all, we might start at the very beginning.

What is deferred execution – and why do we need it?

Most of us have at least a few typical examples in our applications where this would be either handy or a really good idea.
Let’s say your website is sending notification emails after a comment has been posted to one of your blog posts. You are probably doing this as a afterSave()
callback and trigger maybe 4-9 emails to you (the author) and maybe some previous commenters who wish to be notified on updates, as well.
Now, with basic CakePHP coding, you would probably also need a sleep(1) throttling timeout between the email sending processes to avoid
hitting a flood timeout. All in all you make the commenting user wait at least a few seconds before the page will refresh and he will finally see that his comment went though.
Imagine the email sending would need a little bit longer due to network issues. And the user hits F5 several times as nothing happens for him.

An annoying situation that could easily be avoided if unrelated tasks like notifications are not directly processed by the same thread (in this case the browser request of this commenting user). We delegate this task to another thread in the background which then processes this and many other tasks in the order they come in. This is called deferred execution. In our example above the main thread just puts those 9 emails to be sent out into the "queue" and immediately responds to the user within less than a second.
This can be combined with a priority flag to make tasks around email sending being processed before cleanup jobs for example.

There are many other scenarios where you should use a queue to handle background-tasks. Examples are:

  • Image processing/resizing
  • PDF (or other larger/demanding file) generation/extraction
  • Notification or larger update jobs
  • (Regular) cleanup or data monitoring

The Queue plugin is a good way to get to know background-processing and how to reduce response-time for heavy-processing tasks.
It also doesn’t have any other dependencies – except for basic CLI access.

Setup and configuration

As always, we need to load the plugin using CakePlugin::load('Queue') or CakePlugin::loadAll().

You can set any configuration values using your APP config file:

$config['Queue'] = array(
    ...
);

The available options are explained in the README file.

The most basic setup – for local testing for example – would be to use the default "workermaxruntime" with 0 (running forever) and manually
starting a single worker cake Queue.Queue runworker. But once a worker dies in case of an error or server restart it won’t restart without another manual
trigger. Bear that in mind. I recommend the following approach using scheduled restarts for productive systems.

If you plan on having multiple workers or a fail-safe restart every few minutes you need a cronjob scheduler.
For linux the tool "crontab" is usually already available and used for other cronjob activities in CakePHP anyway.
With this you can simply create a new worker every x seconds – note that the lifetime of a worker should also be around that time.
Open the crontab editor for the www-data user (assuming you are using apache): crontab -u www-data -e

*/30 * * * * cd /srv/path/to/app && ../lib/Cake/Console/cake Queue.Queue runworker

If you have a Console folder inside your app, you can simplify the cake command ../lib/Cake/Console/cake to ./Console/cake.

Example usage

As explained above you should have always at least one worker up and running – either via crontab every x seconds or manually started.

Then you can start adding tasks to the queue. To play around you can use the command line. This might also be useful for some tasks you only
want to trigger once in a while – manually. cake Queue.Queue will list you all available queue tasks.

cake Queue.Queue add Example

This will add the Example task to the queue and should be executed and finished immediately.

If you want to see how the queue processes multiple tasks in order, try the LongExample task and while it is running add some Example tasks.
You can also play with more than a single worker at once.

Real usage

Enough of playing around, let’s get down to business.

Image generation

Let’s say we have an upload form and after save we want to trigger the rendering in the afterSave() callback of the model.

I created a task QueueImageTask inside my APP/Console/Command/Task folder. This way it will be recognized by the Queue plugin as "Image".
Then I could schedule this rendering task after each new upload using the "QueuedTask" model:

// In the afterSave() callback of the model, for example
$QueuedTask = ClassRegistry::init('Queue.QueuedTask');
$QueuedTask->createJob('Image', array('id' => $id));

The queue when idle will run in the background and check for scheduled tasks. It will see the new Image task and trigger its execution.
So when the worker processes it we can retrieve the record via the passed id and do all the magic we need to.
We can also can put a callback at the end of the task method before returning true. This way we can for example then update the status of the image from "inactive (not rendered)" to "active" (fully rendered and ready to be used).
This is an example of a two way communication task, reporting back to the app on success.
If it is a long running task (minutes to hours), one could also leverage the "progress" feature of the plugin.

Sending emails

For some projects I just send out SMTP mails without the queue. But if I want to switch, I simply have to modify the email settings.
Using my EmailLib, I can just switch transport from Smtp to Queue.Queue with 4 lines of adjustment:

public $default = array(
    ...
    'transport' => 'Queue.Queue',
    'logTrace' => false, // Detailed trace log
    'logReport' => true, // Report log entry
    'queue' => array('transport' => 'Smtp', 'logTrace' => true), // Overwrite for inside the queue

This is the code to create a new email:

App::uses('EmailLib', 'Tools.Lib');

$Email = new EmailLib();
$Email->to(...);
$Email->subject('Testing Message');
$Email->domain(...); // If generated in a shell/task
$result = $Email->send('Foo');

This would now send the email directly into the queue – where it then will be processed and actually send through a worker.
With logReport => true we can see a log entry for passing the task on to the queue.

Using logTrace => true we get a detailed trace report in the log file after the worker processed the task. It would not make sense for the initial creation (as there is no trace from the SmtpTransport yet).

Emails are usually just sent out. They are one-way communication.

Debugging

With background tasks debugging is a little bit more difficult than with synchronous execution.
That’s why, by default, the Queue plugin will store the last execution time of all currently running workers in the TMP/queue/ folder (last modified date).
This gives you a basic idea of how many workers currently are running and how they can be identified.

You can, accessing the CLI, also identify and display all currently running workers using top. They have the command "php" in the last column.
If you have to kill all at once (especially when you are experimenting with lifetime 0), use killall php then.

Sugar

As noted above, I do have a little admin backend of its own for a quick look on how many workers are currently running and a glance at the stats/settings.
Just navigate to /admin/queue and it should display all of that. Note that it might need the Tools plugin as dependency here, though.

Vista

The above approach is a "CakePHP only" minimalistic tool for small and medium sized applications.
If you need a more robost approach, you can also look at complete tools like php-resque, djjob, celery or gearman which also usually have a CakePHP plugin these days.
So I will most likely look at Gearman and CO in the near future. For resque there is a nice integration article available.

Additionally, this article is more about raising awareness of background jobs in every day apps than actually proclaiming the one perfect solution for this.
But with more and more in the community actually going down that path, the existing tools around queuing in CakePHP sure will become more sophisticated. Check the awesome list for Queue plugins.

4.00 avg. rating (81% score) - 4 votes

13 Comments

  1. Thank you for forking this and updating it to 2.x. It saved me a lot of time trying to figure out how to accomplish deferred execution.

  2. Hey Mark.

    What do you mean by:

    "We can also can put a callback at the end of the task method before returning true. This way we can for example then update the status of the image from “inactive (not rendered)” to “active” (fully rendered and ready to be used)."

    I thought ->createJob returned right away, how would this work if we would have to wait until the task has completed to call back?

  3. A callback in a way that this task itself in its last lines then calls some code to update a flag in the DB or sth.

    $this->ModelName->setProcessed($id);

    The original script that called it cannot directly process this of course. But after refreshing the page you would then notice it maybe. Or you can listen on the change or poll any change of data on this model.

  4. Hi Mark,

    Thanks for updating this plugin. I’m looking into using it for a project.

    Is there a reason why it wants tasks to be named after the plugin (eg. QueueSomethingTask)? I have some existing Tasks which I run on a schedule or sometimes manually. I’d like to reuse these for the Queue. Is there anything against implementing the run() action to point to execute() action?

    Example:

    public function run($data) {
        $result = $this->execute($data);
        return $result
    }

    That would enable me to continue executing this Task manually if I wanted and make it more reusable. It just seems weird that the plugin wants to "own" these tasks with the special naming scheme.

  5. It is not weird, it is just a convention to make sure the Tasks you are running with this plugin are "compatible" with it, thus the prefix.
    If they are not compatible, things might go wrong.
    The quick fix would be to just rename your tasks, or extend them to be used with the plugin.
    But feel free to PR any different approach on github.

  6. Hey Mark! I hope you are doing great. I’m currently using your plugin for Queued Tasks in Cakephp 2.7 and it’s working great. I’m facing another problem right now where I need to Queue orders and those orders need to be executed one after the other in order to check for the allotment of each product in that order, however, I’m running several workers in my environment which kinda defeat the whole point of queueing these tasks. Is there a way to run workers for specific tasks with the current plugin, I want to create a worker only for these kinds of tasks so they will won’t be processed at the same time. I read the docs but I couldn’t find anything about this. Thank you!

  7. Hi Mark, thanks for the plugin. It is awesome.
    I´ve been looking in the CronTask and it seems to be an incomplete (or initial) version of QueueTask. Is there a way to use QueueTask like CronTask? Some jobs must be run from time to time (like daily jobs)

  8. Actually, it was the other way around, CronTask was kind of the idea to add cronjob stuff.
    But I removed it in the newer version already again, because it was not a good idea.
    You can just directly set up cronjobs this way using crontab or alike. No need for the abstraction here.

  9. Hello Mark, my name is Roger. I come from Uganda. please forgive my English. I have been using cakephp 2.5 for about two years now. My application currently uses a plugin for generating excel files. The plugin is based on PhpExcel library. I’m using it as a component in a few of my controllers. The problem is some of the excel files are large and take forever to download because they are being served from the controller logic through an action. How could I possibly make your queue plugin work with this PhpExcel component such that the file downloads happen in the background and are provided for download when complete. I want the user to continue using my application while the download continues in the background.

  10. Mark . plz help me chang status queue when queue run fail or retries xx times fail

  11. Hello Mark,

    I’m trying to install your Queue plugin for CakePHP 2.X. But I’m confused as to what I can do exactly with it.
    What I am trying to do is to have each user’s validation email be sent from a queue instead of slowing down the rendering of the success register page on my website. At the moment, after correctly completing the register form, users have to wait 5-10 seconds for the successfull register page to display.

    How can I move the send email on register process in a queue so as to reduce the rendering time for the next page (success register page) ?

    I’ve searched for ressources on the web concerning cakephp and cron jobs or deffered execution but I didn’t manage much. Maybe if you have any documentation or ressources on how to work with taks/cronjobs with CakePHP 2.x woulb be extremely appreciated.

    PS: thank you for your "UrlCache" plugin, it speeded up my old website while being easy to install 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.