RSS
 

Posts Tagged ‘CakePHP2’

Use 3.x Migrations for your 2.x CakePHP app

03 Oct

In this post I reveal one of my tricks on how to leverage 3.x power in a legacy 2.x project.

You might have already read on how to use some of the splits, like the ORM, in 2.x projects.
Today I want to talk about migration as a topic.

Status Quo

I do have to maintain two remaining CakePHP 2.x apps that have been too large to just upgrade yet.
And time and budget was not on my side so far.
In 2.x there was also not a real powerful database migration tool available so far.

Let’s use the 3.x Migrations plugin

We can use the Migrations plugin quite easily in all 2.x apps do all database modification this way.

First we create a subfolder in your 2.x root folder, let’s call it /upgrade.
This will contain a standalone 3.x app including the Migrations plugin as dependency.
In my case the composer.json looks like this:

	"require": {
		"cakephp/cakephp": "~3.3",
		"cakephp/migrations": "~1.6",
		"cakephp/bake": "~1.2",
		"dereuromark/cakephp-setup": "dev-master",
		"dereuromark/cakephp-tools": "~1.1"
	},
	"require-dev": {
		"cakephp/debug_kit": "~3.2"
	},

Since I include cakephp/bake, I can also leverage the Bake plugin to generate the necessary migration file.

Any time I need a new migration file I simply go to the subfolder and use the 3.x shell:

cd upgrade
bin/cake bake migration CreateArticles ...

Of course you can now modify it further and once complete commit this file into version control.

On the server your deployment script just also needs to contain the following lines then to fully automate it:

cd upgrade
chmod +x bin/cake
bin/cake Migrations migrate
cd ..

Once you upgrade to 3.x you can move all migration files to the actual place in your app, remove the subfolder and simplify the deployment lines to just the single command 🙂

At least I now have to only remember one way to do migrations, for all 3.x and the old 2.x apps. And I can benefit from all recent improvements in those plugins even in those old apps.

Upgrading from existing 2.x migration tool?

You might be using a 2.x tool like this already, but upgrading to the state of the art 3.x Migrations plugin hotness is not a problem here, either.

Just create a dump of the current schema, put it into an SQL file and include that in your first Migrations file:

public function change() {
	$sql = file_get_contents('dump.sql');
	$this->query($sql);
}

Make sure you mark it as migrated, so it is not accidentally executed again.
As alternative you could build in a switch to auto-detect if one of the tables already exists:

$exists = $this->hasTable('users');
if ($exists) {
	return;
}
...

Tips

Deploy admin user along with the schema

In some cases it can make sense to provide a basic admin login along with the first initial migration:

$data = [
	[
		'username' => 'admin',
		'email' => 'admin@example.com',
		'password' => '...' // Must be changed right afterwards
	]
];
$table = $this->table('users');
$table->insert($data)->save();

For more, it is advised to leverage the seed functionality the Migrations plugin ships with.

Further goodies

As you can see, I also included my Setup and Tools plugin, which in v3 also contain very useful and powerful tools for database maintenance and alike. I can now also leverage them and do not have to backport everything to 2.x anymore.
The same would be true for any such plugin and will help you save time for other things if you can focus on development for the 3.x branch only.

Bottom line

Every pre-3.x project should definitly have a subfolder which runs an up-to-date CakePHP 3.x shell including all useful and required plugin shells.

Feel free to share your ideas and experiences on advancing slowly towards 3.x as comments.
See my old post about it for some more details on how to share the credentials, so you can keep them DRY in your main app config.

 
No Comments

Posted in CakePHP

 

CakePHP 2.6 – and the end of the beginning

15 Mar

I was first thinking about the title "and the beginning of the end" – but that sounded a little bit too Armageddon. In fact, 2.x will probably still be around for years – and at least 2.7 will still be released some day (it is not impossible that there might even be a 2.8 …).
The title "and the end of the beginning" fits much better as it allows fresh projects and early migrations to already leverage the new 3.x milestone while the rest just sticks to the 2.x one a while longer.

So what does 2.6 and 2.7 mainly bring?
First of all they benefit a lot from 3.x backports.
Many of the new 3.x functionality has been and will continue to be backported to 2.x.

They also allow the chance to further cleanup the code-base and make the code itself more "3.x-ish". Some of that can be done by looking at how the new core does things, some of it can be achieved using Shims (Code bridges between two versions).
Some of those things became already clear from the dev-preview versions of 3.x – and my article around it.

A few basic things that are very useful in light of the above:

  • Make your 2.x code (app, plugins) PHP5.4+ (maybe also use short array syntax). It will make the upgrade process smoother
  • Remove deprecations and outdated ways of doing things
  • Stay up to date with the 3.x developments and how to best use that information for future proof 2.x development

And most importantly in general

If you must still use 2.x at this point when 3.x is long released, then you must always be up to date with the latest 2.x minor release. Everything else is shooting yourself in the foot.

My recent doings

First I made sure, every app is now running on latest 2.6 stable, and added a few more tests along the road.
Further I made sure relevant changes or new features in 3.x core are backported to the 2.x core and that I plan to migrate to those in my apps ASAP.

Plugin cleanup

I started to extract my super-fat Tools plugin into smaller chunks. Most recent split off is the Shim plugin as I had to acknowledge the fact that I mixed too many fixes/shims and new functionality, which is usually not the best thing to do.
So there it is: A Shim plugin to contain all the bridge code towards 3.x and a few fixes along with it. And a Tools plugin that builds on top of it and adds the actual functionality.
This was the logical thing to do. Most of the shims are not needed beyond 3.x, and as such they shouldn’t be in a more persistent plugin.

Shimming

That brings me right to the next point: I looked into how to get 2.x apps closer to 3.x. Especially for lager code bases this really helps the migration to the next major version. Less necessary changes mean less change for breaking and faster upgrading results.

I ported flash messages to my 2.x version of FlashComponent and FlashHelper – including the syntactic sugar of $this->Flash->success($message).
This code will not have to be modified again at all when upgrading then.

Instead of the "mocking the hell out of it"-ControllerTestCase class I backported the IntegrationTestCase to 2.x which has a more sane approach on actual controller testing.
Using the syntax of 3.x in 2.x allows me to add a lot of new integration tests that will flawlessly work after the major version jump some day.

Password hashing

I migrated my 2.x apps from sha1 to state of the art PHP5.5+ password hashing (and the default in 3.x) – which can already work in 5.4, as well, thanks to shims. But I also needed to support existing passwords to provide BC.
So basically, I use the Shim.Fallback password hasher along with Shim.Modern and Simple ones to have a graceful fallback on old accounts and an auto-hash migration upon login. Each time a user logs in the new hash replaces the old sha1 one. Over time all users will be fully migrated and I can switch back to just Shim.Modern hasher directly (which is the Default hasher in 3.x by the way).

So after migrating to 3.x it will be:

  • Shim.Modern => Default
  • Simple => Weak

The latter is only relevant in case not all passwords have yet been migrated.

See my other article for details on how to implement them (via Passwordable behavior for example) or directly visit the Shim plugin documentation.

 
2 Comments

Posted in CakePHP

 

UTF8 can be tricky – especially with PHP

15 Aug

Everybody uses (or should!) UTF8 these days. An easy and fully supporting PHP version I did not come across yet, though.
It seems there is sometimes more to it. This article is supposed to guideline the basic setup of a CakePHP app using UTF8 and will go beyond that to the really tricky parts regarding the de facto standard encoding these days.

Note: this post is really long overdue and was in my draft folder for 2+ years. So here it is, quickly published before it got even more dusty^^
And dusty sure is the right word with (hopefully) no one using ANSI/ISO-8859-1 anymore these days.

UTF8 and PHP

Use the mb_ functions if you know that you real with strings than can contain UTF8 chars. So if you want to count the length of such a string:

$length = mb_strlen($string);

If you are simply manipulating strings, you do not always have to use those slower and UTF8-aware fnctions, though. But in doubt always do so.

UTF8 and preg_match()

Now this is a tricky one – especially if you don’t want to recompile PHP with the PCRE UTF-8 flag enabled or if you don’t know about it at all. IMO that should be the default, but it usually isn’t it seems.

Most times, when dealing with UTF8 strings, the /u modifier and p{L} helps:

preg_match('/^\p{L}[\p{L} _.-]+$/u', $username, $matches)

In other cases you might have to add (*UTF8) in your pattern.

UTF8 and CakePHP

CakePHP setup

The main parts are handled in the book, especially in the getting-started section.
But the main part that sometimes people get wrong is that the APP encoding is "utf-8" while in the database.php its spelled utf8.

Make sure you save all files as "UTF8 without BOM" via your IDE as soon as they start to contain UTF8 chars. Failing to do so will cause output issues.
I usually try to avoid this and use Locale translation and mainly English chars in all files as much as possible.

Note: Before adding any UTF8 chars to files, those files are always ANSI (there is no way without the BOM to distinguish those two encoding formats as they are one and the same here). So no matter how often you try to save them as UTF8, they will always still be ANSI. In case you wondered why it falls back to it again in most IDEs.

Correcting PHP functions

Some PHP functionality has been wrapped in CakePHP to overcome deficiencies regarding Unicode.
String::wordWrap() for example replaces the faulty wordwrap() function.

I also added a few fixes to my Tools plugin as Utility/Utility class:

  • pregMatch(): Unicode aware replacement for preg_match()
  • pregMatchAll(): Unicode aware replacement for preg_match_all()
  • strSplit(): Unicode aware replacement for str_split()
  • pregMatchAll(): Unicode aware replacement for preg_match_all()

Probably more to come..

Proper validation

Make sure your validation is unicode aware – that’s probably one of the most made mistakes from mainly English speaking devs/people.
They maybe assume that it works to simply use strlen() or a [a-z] regex or alike – not taking into account that for example many normal first/last names contain a lot of special chars.
Validation here should never be too strict. Otherwise a lot of people will be very upset.

So in the above example we do NOT want to use

preg_match('/^\[a-z][a-z .-]+$/i', $firstName)

but something more like

preg_match('/^\p{L}[\p{L} .-]+$/u', $firstName)

to validate a first name.
IF we actually have to validate this further than a simple "not empty" check is a different topic (I don’t think so). But if you really must, PLEASE do not shut people out because their parents gave them non-English names 😉

A similar thing I had to fix in the core a while back, regarding domains/urls.
And this is CakePHP2.5 – the current master – so that topic sure is still quite current for some cases. More and more so with further internationalization.

Checklist for your CakePHP app

  • Ideally, use utf8_unicode_ci as collasion for your DB
  • Your layout should contain <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  • The apache/nginx should serve files as UTF8 via header Content-Type text/html; charset=UTF-8

Outview

Only in PHP7 (as PHP6 got skipped) there will be a more built-in approach then for UTF8. Until then (and maybe even then) we will have to fight quite a lot here for the next years.

There are even a few popular projects in GitHub around the UTF8 issues, e.g:

  • https://github.com/nicolas-grekas/Patchwork-UTF8
  • https://github.com/heartsentwined/php-utf8
  • https://github.com/voku/portable-utf8
  • https://github.com/gatsu/UTF8

Might be worth checking out.

Anything missing? Please let me know.

 
4 Comments

Posted in CakePHP

 

CakePHP 2.5 arrived + CakePHP Tips 2014

13 May

You probably read my last tip sections.
And I started to move some of them to my sandbox app.

But once in a while it might also be nice to publish a few selected tips, here as well.

Oh, and CakePHP 2.5 is out! Get it – now 🙂
See the last chapter of this post about why you really should upgrade ASAP. And it doesn’t matter if you
are on 1.x or 2.x.

URLs in CLI

In CLI and your shells there is no HTTP HOST and therefore usually no absolute URLs.
But in order to properly work with links/urls (e.g. sending batch emails), we need that to work.
So I use Configure::read(‘Config.fullPageUrl’) here.
My configs.php file then contains:

$config['Config'] = array(
	'fullPageUrl' =>  http://www.myapp.de // The primary URL
)

In case you have a different domain for local development like http://myapp.local and you want that to be the fullPageUrl, make sure you overwrite the default in your configs_private.php file:

Configure::write('Config.fullPageUrl', 'http://myapp.local'); // Usually defaults to the (live) primary URL

And in the end of my bootstrap file (after including all config files), I simply do:

if (php_sapi_name() === 'cli') {
	Configure::write('App.fullBaseUrl', Configure::read('Config.fullPageUrl'));
}

To test the current domain/fullBaseUrl, you can use my Setup plugin command cake Setup.TestCli router.
It will output both relative and absolute URL examples generated by the Router class with your current settings.

So on the live server then it will output http://myapp.local instead of http://localhost when generating Router::url()s in your shells.

Careful with running shells in CLI

Most are probably not aware, but running shells in CLI needs to have a proper user management around them in most cases.
Imagine yourself running your apache as www-data (default) and log in as root or any other user not affiliated with that www-data user/role (bad idea).
Once you execute a shell and tmp cache data are (re)built, your www-data user cannot access them anymore, losing the ability to cache and triggering a lot of errors.
So make sure you only log in with a user that shares the role of www-data at least, so that both can access each others’ generated files.

A popular example is the ClearCache shell which re-builds your cache dirs in debug 0 (when changing files or db schema makes this necessary).

PS: Of course you could also switch to another cache system than the default File engine. But most probably didn’t do that yet, either.

Merging arrays

Ever wondered what Hash::merge(), array_merge and array_merge_recursive have in common – or don’t have in common?
Check out these merge comparison examples.

See what the requirements are – and use the appropriate merge method then.

There is also the + operator, which is quite useful when merging flat arrays and string key based options. This is quite commonly used in the core to merge
options and defaults:

$defaults = array(
	'escape' => true
):
$options += $defaults;

In this case the $defaults are merged on top of $options, but only if that key has not been specified yet.
This kind of merge is really efficient and fast (4-5x faster than array_merge() itself) – but should really only be used if all keys are definitely strings.

Paginating and merging

A propos merging: When setting up paginate settings in your controllers, try to prevent

public function index() {
	$this->paginate = array(...);
}

This way you kill all defaults you might have set via public $paginate or in your extending controllers (AppController’s beforeFilter() callback for example).

So it is better to use:

$this->paginate = array_merge($this->paginate, array(...));
// or
$this->paginate = array(...) + $this->paginate;

In my 2.x code for example I have this snippet in all my AppControllers to have query strings for paginations:

public function beforeFilter() {
	parent::beforeFilter();
	$this->paginate['paramType'] = 'querystring';
}

This will only work with proper merging of defaults and custom settings.
I prefer the latter because the settings are string based and here the + operator is the fastest and easiest way of doing things.
Once the key is already set in your method, the default will be ignored right away (with array_merge() and nullish values this can be different/unexpected).

And remember to not mix controller and component pagination.

Pagination and sort default order

Adjust your bake templates so that some fields like created/modified are ordered DESC instead of ASC per default in the index actions.
For those fields the first click on the header row should display them DESC right away as one would then most likely be interested in the
latest changes. Same goes for most date fields like "published" as well as fields like "priority", "rating", …

That’s how the baked code (or manually adjusted one if done afterwards) could then look like (index.ctp):

<th><?php echo $this->Paginator->sort('name');?></th>
<th><?php echo $this->Paginator->sort('amount');?></th>
<th><?php echo $this->Paginator->sort('priority', null, array('direction' => 'desc'));?></th>
<th><?php echo $this->Paginator->sort('status', null, array('direction' => 'desc'));?></th>
<th><?php echo $this->Paginator->sort('publish_date', null, array('direction' => 'desc'));?></th>
<th><?php echo $this->Paginator->sort('modified', null, array('direction' => 'desc'));?></th>
<th><?php echo $this->Paginator->sort('created', null, array('direction' => 'desc'));?></th>

Using modified model data in the form again

Some of you might have had the wish of posted data that was modified in the model due to beforeValidate/beforeSave to appear modified in the view again (so
that the reason for validation errors might be more clear etc).
So let’s say you have a beforeValidate callback to clean the input of a textarea:

public function beforeValidate($options = array() {
	if (isset($this->data[$this->alias]['comment']) {
		$this->data[$this->alias]['comment'] = $this->_clean($this->data[$this->alias]['comment']);
	}
	return true;
}

So in this case it could easily be that _clean() removes some invalid content and thus the minLength rule is suddenly triggered.
Which is weird, since we posted at least twice the length of text.
To clarify to the user what is going on, one could adjust the error message – but one could additionally return the modified (ready to save) data instead of the
actually posted data.

public function add() {
	if ($this->request->is('post')) {
		$this->{$this->modelClass}->create();
		if ($this->{$this->modelClass}->save($this->request->data)) {
			$this->Session->setFlash(...);
			return $this->redirect(array('action' => 'index'));
		} else {
			// Here we assign the modified model data back to the request
			// object and therefore to the view/form
			$this->request->data = $this->{$this->modelClass}->data;
			$this->Session->setFlash(...);
		}
	} else {
		// Default values for the form
		$this->request->data[$this->modelClass]['status'] = true;
	}
}

The input field will now contain the content that was served to beforeValidate(). And combined with a good error message this will probably clear things up.

PS: You can also directly use the model’s name instead of {$this->modelClass}, e.g. ‘Comment’.

MySQL – MyISAM vs InnoDB

InnoDB is a little bit more robust as it allows transactions. Especially with CakePHP and "multiple queries" most of the time (per default) this can be quite helpful in keeping the DB in a valid state.
Also read myisam-versus-innodb on pros/cons for each.

One additional problem with InnoDB, though: Per default it creates a file that always increases in size, never decreases again. This can soon be a nightmare with backuping when it becomes >> xx GB of size.
See how-to-shrink-purge-ibdata1-file-in-mysql how to avoid that by not using that file, and instead using innodb_file_per_table.

Testing

Testing Controllers

I stumbled upon a few issues with testing controllers – especially plugin controllers.
For plugin controllers to be testable unfortunately you always need to use $uses, even if it wasn’t necessary due to conventions.
So for your Tools.TinyUrls Controller you would need

public $uses = array('Tools.TinyUrl');

Otherwise it would always try to look for the model/fixture in your app dir, which eventually always fails.

Do not forget to use --stderr when testing in CLI. Otherwise all tests that use the session (and which work fine in webtest runner) will fail:

cake test app AllController --stderr

Test Coverage

If you want to generate a HTML overview of all your locale test coverage:

cake test app AllApp --stderr --log-junit tmp/coverage/unitreport.xml --coverage-html tmp/coverage --coverage-clover tmp/coverage/coverage.xml

The report index.html will be in your /tmp/coverage folder.

More on testing – especially controller testing – can be found on the corresponding Tools plugin Wiki page.

Upgrade (deprecated) code

It is always good practice to upgrade to the current master. Not only for CakePHP, but there especially. It will save you a lot of time in the long run, as migration will be easier and faster in small steps instead of one huge step. It will also make it easier to use the new features and more importantly will also come with a lot of fixes and method corrections that the older versions didn’t get anymore. Those outdated versions usually only get security-bugs fixed. So if you look hours for an error that is already fixed in the current master, it was just a huge waste of time. I have seen that a thousands times – on IRC and elsewhere.

So in case you are not using the current master (2.5), do it now. Internally, upgrading 2.x is a "piece of cake".
Upgrading from 1.x is also not that big of a deal – just needs a little bit more manual adjustments. For most things you can use the core UpgradeShell as well as my Upgrade plugin.

In case you are already upgraded to 2.5, you can and should also remove deprecated functionality in favor of the right one.
Those deprecated things will only add to the file of upgrades for the next 2.x release or 3.x. And using the upgrade shell it is usually just one single command to execute.
So for 2.5, you should have removed all the "easy" stuff that will clearly be switched with a different way of doing things as it is mentioned in the migration guide, e.g.

  • loggedIn() in favor of Auth::user(‘id’)
  • CakeRequest::onlyAllow() in favor of CakeRequest::allowMethod()
  • Use first argument (string|array) instead of var args for CakeRequest::allowMethod(), AuthComponent::allow(), etc
  • $title_for_layout in favor of $this->fetch(‘title’) and all other deprecated view vars

From 2.4 and below there are also a few left overs that could easily be corrected:

  • FULL_BASE_URL, DEFAULT_LANGUAGE, IMAGES_URL, JS_URL, CSS_URL to Config variables
  • Remove CAKEPHP_SHELL, IMAGES, JS, CSS usage
  • Simplify HtmlHelper::css()
  • Remove Sanitize class usage in favor of a more sane approach
  • Simplify CakeRequest and PaginatorHelper usage with param() if applicable
  • Don’t use request shortcuts anymore – like $this->action in favor of $this->request->action etc
  • Get rid of Model::read() calls – in 3.x this will be get() – I use my Tools plugin MyModel::get() method here already for years
  • Use the new Model::save() syntax for options
  • Completey get rid of named params in favor query strings (Use my Tools plugin CommonComponent and Configure value App.warnAboutNamedParams to detect left-overs)
  • Replace all Set calls with Hash calls, as Set is deprecated. Make sure it is tested thoroughly (as the functionality of Hash might be slightly different).
  • Prevent using Validation::range() in favor of a custom validation rule for inclusive ranges to easier migrate to 3.x – or simply use my Tools plugin MyModel::rangeInclusive() method.
  • Further deprecations in favor of the 2.5+ way to do things

and so on.
For some details see Tips-Upgrading-to-CakePHP-2.x.

My Tools plugin also contains a few further tweaks that can help ease migration from 2.x to 3.x See the full list on the corresponding Wiki page.

This will help making sure any future upgrade is less painful.
Think about it: When you do that now the remaining TODO list will only be half the size and look a lot less intimidating. When the time comes to upgrade to 3.x it will look quite more doable.

My codez is now officially all 2.5 – and as 3.0 ready as it can get 😛

And what always helps a lot is to code clean and structured. A code mess will always be difficult to maintain. So use coding standards and enforce them. Use best practice approaches. Refactor once in a while to prevent a mess from happening.
Happy coding 🙂

 
1 Comment

Posted in CakePHP

 

CakePHP flash messages 2.0

21 Apr

This is a topic often discussed. What is the best approach?
While I might not have the best, I sure have a fully working one since 2008 and CakePHP1.3. I still use it in all 2.x apps.
It is easy and does not require a lot of overhead.

Basically, it enhances the core one with the following features

  • Different types possible (now in 2.x kind of in the core, as well, using elements)
  • Multiple flash messages per type possible (with a max limit to avoid session flushing)
  • Transient messages (via Configure) and real ones (via Session)
  • Transient ones can also be fired from the views (to display some hint/info for this page) – although one should try to stick to the controller

Demo

sandbox/examples/messages

How does it work?

We attach the Common Component and the Common Helper to the AppController:

public $components = array(..., 'Tools.Common');
public $helpers = array(..., 'Tools.Common');

In our layouts we need to switch the default output to ours:

echo $this->Common->flash();

And we can use `em:

public function add() {
	...
	if ($this->Entry->save()) {
		$this->Common->flashMessage('The Entry has been saved.', 'success');
	} else {
		$this->Common->flashMessage('The Entry could not be saved. Please check the form.', 'error');
	}
}

Styling

We style our flashmessages via CSS

div.flash-messages {
	width: 90%;
}
div.flash-messages div {
	padding: 10px;
	padding-left:40px;
}
div.flash-messages div.error {
	width:100%; border-style: solid; border-width:1px; border-color:#B84D17; 
	margin-right:2px; color:#000000; margin-bottom:8px;
	background: url(/img/css/layout/icon_error.gif) left center no-repeat;
	background-color:#F7C6A5;
}
div.flash-messages div.warning {
	width:100%; border-style: solid; border-width:1px; border-color:#D0C130; 
	margin-right:2px; color:#000000; margin-bottom:8px;
	background: url(/img/css/layout/icon_warning.gif) left center no-repeat;
	background-color:#F6F3A4;
}
div.flash-messages div.success {
	width:100%; border-style: solid; border-width:1px; border-color:#009900; 
	margin-right:2px; color:#000000; margin-bottom:8px;
	background: url(/img/css/layout/icon_success.gif) left center no-repeat;
	background-color:#A5F7A8;
}
div.flash-messages div.info {
	width:100%; border-style: solid; border-width:1px; border-color:#cccccc; 
	margin-right:2px; color:#000000; margin-bottom:8px;
	background: url(/img/css/layout/icon_info.gif) left center no-repeat;
	background-color:#ffffff;
}

Or any other layout for that matter. The images are small icons and also optional, of course 🙂

Transient flash messages

You can also put flash messages on top that are not stored in session but Configure (for this request only).
This can be useful if you don’t intend to redirect and don’t want them to show up if that happens.

From your controller:

$this->Common->transientFlashMessage('This page is currently being redesigned', 'info');

Or from your view ctp (even elements, blocks or the layout if it happens prior to the flash message output):

$this->Common->addFlashMessage('This page is under maintenance. It may be broken!', 'warning');

Output in a specific order or only specific types

You can filter the output, both in order and types:

// Using Common helper
echo $this->Common->flash(array('warning', 'error'));

In this case it would only output the warning and error messages, in this order (Usually the order is "error, warning, success, info").

Details

For details please the Wiki page.

Outview

It would probably be nice to add element support at some point. This would allow an easier approach to customization of those messages.

There are also quite a few CakePHP core tickets open regarding flash message enhancements – see this or this which might even lead to an own component for it some day.
But until then my approach will still be used in all my xx apps 🙂
So in 3.x there will be a Flash component and Flash helper to provide a clean way to produce flash messages. If they provide the same features my implementation currently does, will have to be investigated. But it will use templating which will sure be nice.

Update 2014-12

A FlashComponent and FlashHelper now replace the current way of handling flash messages. This helps to ease migration to 3.x at some point, as there this is also extracted from the Session classes.
As syntactic sugar you can directly use the types as methods now:

// Normal
$this->Flash->message('Yeah', 'success');
// Possible now
$this->Flash->success('Yeah');
// Same with
$this->Flash->warning('Careful');
$this->Flash->error('o_O');
$this->Flash->info('Some information');

2015-01 CakePHP 3

The documentation for 3.x can be found in the repo itself now: Docs. Check it out!

2016-10 CakePHP 3.3+

If you don’t want to have an own type "auth" for the AuthComponent flash error message, simply load your AuthComponent like this:

$this->loadComponent('Auth', 
    ['flash' => ['key' => 'flash', 'element' => 'error']]);
 
3 Comments

Posted in CakePHP

 

Generating PDFs with CakePHP

08 Apr

The recommended approach is to use the CakePHP plugin CakePdf for this.
With that plugin it is "a piece of cake" to output HTML views as PDF or render them to files for download/emailing etc.

You can obviously just use an approach just like this article describes.
But this makes the code quite dependent on the actual rendering library used isn’t very DRY when there is more than a single page that needs PDF rendering.

Note: This article is about generating PDF files from HTML templates. If you already have PDF files you just want to output or serve as download, then this article is not for you. See serving-views-as-files-in-cake2 then.

Demo

A live demo can be found in my sandbox app. It also helps to compare the different engines.

Installation

First you need to decide on what engine you want to go with.

WkHtmlToPdf is the fastest, but it also depends on CLI and the binaries being installed properly. So it might work differently or not at all on different OS. But using binary files it should work out of the both on both UNIX and Windows. The latter just needs a single exe file drag-and-dropped on your system. wkhtmltopdf.org/downloads.html contains such a ready-to-use binary.
You can then link it in your Configure set-up for CakePdf:

Configure::write('CakePdf.binary', YOUR_PATH_TO . 'wkhtmltopdf\wkhtmltopdf.exe');

For UNIX those precombiled binaries on that download page might not work and a simple apt-get install wkhtmltopdf usually won’t do the trick. You need to manually compile it in this case. But on most systems it should be fine. Don’t forget to adjust the path to the binary here, as well, if it is not the default one:

Configure::write('CakePdf.binary', YOUR_PATH_TO . 'wkhtmltopdf/bin/wkhtmltopdf');

I usually go with DomPdf as it is quite reliable, and (as PHP code) even though slower than a native approach will work flawlessly with all OS and setups without additional dependencies. If speed really is an issue you might want to reconsider using the server internal libraries. But for me generating invoices and such never is time-critical and usually done asynchronous using background tasks via shells 🙂

For DomPdf one important setting when using images in your PDFs is to "enable remote":

define('DOMPDF_ENABLE_REMOTE', true);

Either way don’t forget the rest of the settings for CakePdf – it could look like this:

$config['CakePdf'] = array(
	'engine' => 'CakePdf.DomPdf',
	'options' => ...,
	'margin' => ...,
	'orientation' => 'portrait',
);

The official documentation on the installation part tells you all you need to know. Check it out.

Advanced Setup

The recommended way is to use CakePlugin::load('CakePdf', array('bootstrap' => true, 'routes' => true)); which is supposed to take care of automatic View class switching.

If you don’t want to load the plugin bootstrap/routes and rather do it manually you need to automatically map all .pdf extension URLs to the CakePdf plugin PdfView ourselves:

public $components = array(
	'RequestHandler' => array(
		'viewClassMap' => array('pdf' => 'CakePdf.Pdf')
	)
);

This can also be useful if you want to overwite/extend the PdfView class for some reason (MyCakePdf etc).

Skipping the plugin routes file is also useful, if you already have extension settings in your routes. You can then simply update your APP/Config/routes.php:

Router::parseExtensions();
Router::setExtensions(array('json', 'xml', 'rss', 'pdf')); // We just add pdf to the already defined ones

With both routes and bootstrap defined manually, CakePlugin::load('CakePdf'); suffices.

Usage

The official documentation already states that, so I won’t go into much detail. For the sake of completeness a short example:

public function view($id) {
	$this->pdfConfig = array(
		'filename' => 'invoice',
		'download' => (bool)$this->request->query('download')
	);
	$invoice = $this->Invoice->find('first', array('conditions' => array('id' => $id)));
	$this->set(compact('invoice');
}

If you use my Tools plugin, you can just use $invoice = $this->Invoice->get($id) here.

Make sure you have a basic pdf layout in /View/Layouts/pdf/default.ctp and a view template for the PDF in /View/Invoices/pdf/view.ctp

Now if you allow access without extension (/invoices/view/1) the action can render a normal HTML page with the invoice in a table like form.

If you link it as /invoices/view/1.pdf it will automatically switch the view class to PdfView and render it as PDF file:

$this->Html->link('View PDF', array(
		'action' => 'view', $id, 'ext' => 'pdf'
));

Note the trick with download using the query string for it. If you link to it with ?download=1 it will trigger the download instead of just displaying it:

$this->Html->link('Download PDF', array(
		'action' => 'view', $id, 'ext' => 'pdf', '?' => array('download' => 1)
));

This way the action can do both automatically.

Tips

URLs and Paths

For all engines to work with your URLs and images outputted by helpers and therefore with schema-less absolute URLs (/controller/action/ and /img/foo.jpg) this little modification in your AppHelper is quite useful:

/**
 * Overwrite to make URLs absolute for PDF content.
 *
 * @param mixed $url
 * @param bool $full
 * @return string
 */
public function url($url = null, $full = false) {
	if (!empty($this->request->params['ext']) && $this->request->params['ext'] === 'pdf') {
		$full = true;
	}
	return parent::url($url, $full);
}
/**
 * Overwrite to make paths for assets absolute so they can be found by the PDF engine.
 *
 * @param string $path
 * @param array $options
 * @return string
 */
public function assetUrl($path, $options = array()) {
	if (!empty($this->request->params['ext']) && $this->request->params['ext'] === 'pdf') {
			$options['fullBase'] = true;
	}
	return parent::assetUrl($path, $options);
}

This will make them absolute including the full base (schema + domain) in PDF context.

Filename and downloading

If you don’t force downloading and display the PDF, make sure you read the "eastereggs" part of serving-views-as-files-in-cake2/ regarding an additional passed param here to actually make the downloaded file name what you want it to be.

 
10 Comments

Posted in CakePHP

 

CakePHP and NamedScope for DRY conditions

15 Feb

A behavior for CakePHP 2.x

Background

I stumbled upon this fork and SimpleScope.
The latter has the disadvantage of redundancy in the scope conditions when used in multiple find configs. The first was pretty much how Rails’ scopes work.
But amongst other small issues it lacked the possibility of using model attributes for configuration.
And both didn’t have test cases.
So I decided to combine both, test the hell out of them and get the best out of the basic implementation ideas.

Basic Usage

For the behavior there is a more detailed documentation in the wiki.
But here it goes.

First install/download and load the Tools plugin as documented in the cookbook or its readme file.

Attach the behavior to your AppModel:

App::uses('Model', 'Model');
class AppModel extends Model {
    public $actsAs = array('Tools.NamedScope');
}

Then define some scopes in your model:

App::uses('AppModel', 'Model');
class User extends AppModel {
    public $scopes = array(
        'active' => array('User.active' => 1),
        'admin' => array('User.role LIKE' => '%admin%'),
    );
}

Then you can use those scopes in any of your find queries:

$activeUsers = $this->User->find('all', array('scope' => array('active')));
$activeAdmins = $this->User->find('all', array('scope' => array('active', 'admin')));
$activeAdminList = $this->User->find('list', array('scope' => array('active', 'admin')));

Advanced Usage

If you also want to use scopedFind(), you will also get rid of all the many find wrappers around those scopes that will often be placed inside the models.

An example:

public function getActiveAdmins() {
    $this->virtualFields['fullname'] = "CONCAT(User.firstname, ' ', User.lastname)";
    $options = array(
        'fields' => array('User.id', 'User.fullname'),
        'conditions' => array('User.role LIKE' => '%admin%'),
        'order' => array('User.fullname'),
    );
    return $this->find('all', $options);
}

Now there will maybe also be a getActiveUsers() method and maybe a few dozen more, which all
contain the same condition – which is not really DRY and might be quite error-prone if the conditions have to be adjusted (easy to miss one of the many occurrences in and out of the model).

So what would be a smarter way to approach this?
Let’s try to use the above scopes here – and also use the single wrapper method.
Besides the above scopes, you also need to define some scopedFinds in your model:

App::uses('AppModel', 'Model');
class User extends AppModel {
    public $scopes = array(
        'activeAdmins' => array(
            'name' => 'Active admin users',
            'find' => array(
                'type' => 'all',
                'virtualFields' => array(
                    'fullname' => "CONCAT(User.firstname, ' ', User.lastname)"
                ),
                'options' => array(
                    'fields' => array('User.id', 'User.fullname'),
                    'scope' => array('active', 'admin'),
                    'order' => array('User.fullname'),
                ),
            ),
        ),
        'activeUsers' => array(
             ...
        )
    );
}

The scope itself will both contain active, and the config about this scope key will be stored in a single place. So if you have some very complex condition around published (> a && < b && != c && …) this will take the overhead from multiple definitions and reduce it to a single location.

Let’s execute it:

$activeAdmins = $this->User->scopedFind('activeAdmins');

In case we need to only get a list or the count, we can adjust the scopedFind:

$activeAdminList = $this->User->scopedFind('activeAdmins', array('type' => 'list'));
$activeAdminCount = $this->User->scopedFind('activeAdmins', array('type' => 'count'));

We can also overwrite the default options:

$config = array(
    'options' => array(
        'limit' => 2, 
        'order' => array('User.created' => 'DESC'))
);
$twoNewestActiveAdmins = $this->User->scopedFind('activeAdmins', $config);

You can also get a list of available scoped finds:

$scopedFinds = $this->User->scopedFinds();

Scoped finds:

  • require a name string
  • optionally use a find array

The find arrays:

  • optionally use a type string (defaults to all)
  • optionally use an options array
  • optionally use virtualFields

The options arrays:

  • can use the behaviors’ scope property
  • support all other find options (including contain, order, group, limit, …)

Tip: See the test cases for more complex examples.

Testing

You should test your scopes, even if it’s just something like this:

public function testScopes() {
    $scopes = $this->User->scopes;
    // Each on its own
    foreach ($scopes as $scope) {
        $this->User->find('first', array('scope' => $scope));
    }
    // All together
    $this->User->find('first', array('scope' => $scopes));
}

In case there was invalid SQL, missing fields, wrong contain statements, it would be noticeable right away.

If you use scopedFinds, don’t forget to also unit test them (regarding valid SQL).
This can easily be forgotten now as you don’t have the find wrapper methods anymore.
In case you are lazy, add this test case to any model test that uses custom scopedFinds:

public function testScopedFinds() {
    $scopedFinds = $this->User->scopedFinds();
    foreach ($scopedFinds as $key) {
        $this->User->scopedFind($key);
    }
}

This will at least execute each find and throw an error if the SQL is invalid.
It is advisable to have a more thorough test case for each find key, though, that includes the assert of the return value.

Outlook

I bet when this behavior is thoroughly used, there will be quite a few adjustments necessary. But all in all this already seems to cover most of the use cases.

With Cake3 and stackable custom finders much of the functionality we need here will be part of core functionality. Which will be awesome.
Never-the-less, until then this can be a solid solution to keep the scopes/conditions DRY.

 
No Comments

Posted in CakePHP

 

AJAX and CakePHP

09 Jan

I don’t know why I didn’t post about AJAX earlier. It is probably one of the topics that most don’t really know how to approach.
The docs don’t say much about it, and so many try to rely on (outdated) blog posts out there.

Why AJAX

We shouldn’t use stuff just because it sounds fun. So sometimes AJAX is abused or used where it shouldn’t.
I use it mainly where the user cannot or should not reload the whole page just to get more information or an updated piece of content which will be loaded
after the page completed rendering (asynchronous).
This can be part of a (sub) form, a search result or detailed information on some piece of content in a popover like window.

Basic concept

I always recommend to use the proper extension when dealing with AJAX. So if we deal with basic HTML, we can simply go for /controller/action/.
See the AJAX Pagination example on how to use AJAX here as sugar on top of the basic pagination (that then serves as a fallback for non-JS-browsing or search engines).
In this case it is fine to serve AJAX extension-less.

In most cases (and in all where fallbacks are not necessary) it is better to use JSON as a return type, though. It allows you to deal with the returned content in a more flexible way. That also means that is is not extension-less anymore now.
Thus, you should target /controller/action.json and use the RequestHandler to get all the CakePHP magic going here (setting the correct response header application/json and switching/disabling the layout/views.

This is especially important when throwing exceptions (404 = NotFoundException, 403 = NotAllowedException). All those would then automatically be a proper JSON response message right away.

So for a basic JSON response via AJAX, all you then need is:

/**
 * AJAX action to get favorites
 */
public function favorites() {
	$this->autoRender = false; // We don't render a view in this example
	$this->request->onlyAllow('ajax'); // No direct access via browser URL
	
	$data = array(
		'content' => ...,
		'error' => '',
	);
	return json_encode($data));
}

This will not use a view or layout file and simply return a json encoded string.

But to me that looks kind of wrong. Manually using json_encode() in the controller is not something you should aim for. The view classes are responsible for the actual translation into the target representation format. So such a view class should contain that call. We only make sure that this view class gets the data it needs.

That said you could use the JsonView to make it cleaner and cut down the code by a few lines:

public function favorites() {
	$this->request->onlyAllow('ajax'); // No direct access via browser URL - Note for Cake2.5: allowMethod()
	
	$data = array(
		'content' => ...,
		'error' => '',
	);
	$this->set(compact('data')); // Pass $data to the view
	$this->set('_serialize', 'data'); // Let the JsonView class know what variable to use
}

Note that by using _serialize here we tell the JsonView class that we don’t need a view template to be rendered here. It will skip all the rendering and just use what
you set() to the view class.
Make sure to check out the JsonView docs on how to use a view template if you really need it. though.
It will then automatically use a subfolder "json" and its View/ControllerName/json/favorites.ctp template here.

In the view we can then use this action via JS and the URL /controller_name/favorites.json:

$.ajax({
	type: 'get',
	url: '<?php echo $this->Html->url(array('action' => 'favorites', 'ext' => 'json')); ?>',
	beforeSend: function(xhr) {
		xhr.setRequestHeader('Content-type', 'application/x-www-form-urlencoded');
	},
	success: function(response) {
		if (response.error) {
			alert(response.error);
			console.log(response.error);
		}
		if (response.content) {
			$('#target').html(response.content);
		}
	},
	error: function(e) {
		alert("An error occurred: " + e.responseText.message);
		console.log(e);
	}
});

Note that we can first check on the error string to abort modification of the DOM if there is some error (similar to validation errors or flash message errors) occuring.
And if there is some content we can then easily output that.

AjaxView

Now that is all nice, but for me that is not yet it. In this particular case we don’t just pass down model data. We render the view just as we would for normal GET access. Thus I would like it to be as close to it as possible. I want to avoid the serialize call and manual json_encode part (both in controller and view layer), just as in JsonView, but with all the basic View rendering still intact.
So I created the AjaxView.

It takes care of mapping the content into the above JSON structure, and automatically tries to render via "ajax" subfolder (View/ControllerName/ajax/favorites.ctp).
It will also not use any layout, just the simple HTML template code in the ctp.

A typical use case:

public function favorites() {
	$this->request->onlyAllow('ajax');
	$this->viewClass = 'Tools.Ajax';
}

The result can be of the following structure:

{
    "content": [Result of our rendered favorites.ctp],
    "error": ''
}

See the test cases for it on more cool stuff – e.g. adding some more data to the response object via _serialize.

Full blown example: Chained dropdowns

This has been one of the examples around the usage of AJAX. When you have two dropdowns and select something from the first, the second dropdown should contain updated content relevant for that selection.
In our example, we select countries, and upon selection you can then chose from country provinces (if applicable).

The example can be seen live in the sandbox, as well.

Our controller fetches the lists of countries and country provinces:

public function chained_dropdowns() {
		$countries = $this->Controller->Country->getList();
		$countryProvinces = array();
		foreach ($countries as $key => $value) {
			$countryProvinces = $this->Controller->CountryProvince->getListByCountry($key);
			break;
		}
		$this->set(compact('countries', 'countryProvinces'));
	}

The CountryProvinceHelper component (from my Data plugin) which I usually use here basically does some more, but for a simple example this suffices. It will simply pass down all countries
and the country provinces for the first country (which will be auto selected right away).

The view then contains the form:

<?php echo $this->Form->create('User');?>
	<fieldset>
 		<legend><?php echo __('Countries and Country Provinces');?></legend>
	<?php
		$url = $this->Html->url(array('plugin' => 'sandbox', 'controller' => 'ajax_examples', 'action' => 'country_provinces_ajax', 'ext' => 'json'));
		$empty = count($countryProvinces) > 0 ? __('pleaseSelect') : array('0' => __('noOptionAvailable'));
		echo $this->Form->input('country_id', array('id' => 'countries', 'rel' => $url));
		echo $this->Form->input('country_province_id', array('id' => 'provinces', 'empty' => $empty));
	?>
	</fieldset>
	The province list is updated each time the country is switched. It also has a basic fallback for POST data (will auto-remember the previous selection).
	<br /><br />
<?php echo $this->Form->end(__('Submit'));?>
</div>

The jQuery code can either be added to the page or stored in a separate js file:

$(function() {
	$('#countries').change(function() {
		var selectedValue = $(this).val();
		var targeturl = $(this).attr('rel') + '?id=' + selectedValue;
		$.ajax({
			type: 'get',
			url: targeturl,
			beforeSend: function(xhr) {
				xhr.setRequestHeader('Content-type', 'application/x-www-form-urlencoded');
			},
			success: function(response) {
				if (response.content) {
					$('#provinces').html(response.content);
				}
			},
			error: function(e) {
				alert("An error occurred: " + e.responseText.message);
				console.log(e);
			}
		});
	});
});

Then all we need is the AJAX action

public function country_provinces_ajax() {
		$this->request->onlyAllow('ajax');
		$id = $this->request->query('id');
		if (!$id) {
			throw new NotFoundException();
		}
		$this->viewClass = 'Tools.Ajax';
		$this->loadModel('Data.CountryProvince');
		$countryProvinces = $this->CountryProvince->getListByCountry($id);
		$this->set(compact('countryProvinces'));
	}

and the AJAX view for this action:

<?php
if (!empty($countryProvinces)) {
	echo '<option value="">' . __('pleaseSelect') . '</option>';
	foreach ($countryProvinces as $k => $v) {
		echo '<option value="' . $k . '">' . h($v) . '</option>';
	}
} else {
	echo '<option value="0">' . __('noOptionAvailable') . '</option>';
}

To see the full code in action take a look at the live example in my Sandbox or the source code of it.

Note that an alternative to this could be a ("_serialize") JSON response containing the list of country provinces and let the JS render the HTML. But I prefer to have as much of HTML generation in (Cake)PHP itself and let JS only handle displaying the response data.

Error out

If some error occurs in your action you could either throw an exception and handle that in the frontend or pass down an error code/string to the AjaxView, disabling rendering and returning only the error to the frontend (in response.error):

$this->set('error', 'No record found');

For simple stuff it’s probably easier to use the exception free approach (using HTTP code 200). The resulting JSON would be something like:

{
    "error": "No Record found",
    "content": "",
}

But in general try to stick to exception based error handling.
That would return a response with an error code of 4xx or 5xx usually (depending on the type of exception) and the following JSON object:

{
    "code": 404,
    "message": "Not Found", // Note that in Cake < 2.5 this might be "name"
    "url": "...",
}

In this example it’s a NotFoundException.

Your JS code needs to handle this gracefully. There could always be some database exception or alike thrown. So better account for that scenario.

Further improvement

You could write some generic JS code or even a generic jquery plugin to handle the AJAX response. This would cut down the lines for the query code drastically and keep it DRY.

If you want to automatically make all ajax requests respond with this view class, you could use something like this in your AppController’s beforeFilter callback:

if ($this->request->is('ajax') {
	$this->viewClass = 'Tools.Ajax'
}

It would cut down on that one line your actions.

I would not recommend that, though. as the JSON view is usually a good default. In my experience you usually need that more often, e.g. when just returning data (multiple records in particular) from the model without rendering any view.

But in general you could try to leverage a component to take care of all that and on top add even more cool things like redirect handling (output it along with the rest of the JSON data instead of actually redirecting) or flash message handling.
That together could make it a real useful bundle and cut down your controller code a lot.
No need to add a lot of switchs and if statements, if the code is exactly the same for both AJAX and normal requests.
See my AjaxComponent which could be the answer to this.

Last tips

Don’t forget to include the RequestHandler component and set up Router::parseExtensions in your routes:

Router::parseExtensions();
Router::setExtensions(array('json', ...));

This will tell CakePHP that all .json URLS are automatically served as JSON (which the appropriate headers).

Remember to not cache ajax actions to avoid old content being delivered:

if ($this->request->is('ajax')) {
    $this->disableCache();
}

I actually use that statement for all my actions/content that contain dynamically build content.

If you plan on updating the DB, never use "get", always use type: 'post' and also assert that your action only accepts POST. "get" is only appropriate if the DB is not changed by the AJAX request, so if you only use find().

Note the difference between <option value=""> and <option value="0">. Using validation and the numeric rule we assert that the latter passes validation (when there is nothing to select) whereas the first would trigger a "You need to select a province" error of some sort when trying to save.

Using var targeturl = $(this).attr('rel') + '?id=' + selectedValue; or tricks like data('url') and data-url etc keeps the URL handling and other dynamic PHP based content out of the JS files and makes it easier to store them separately and without direct dependencies. It also avoids shortcut solutions like hardcoding URLs and such.

The xhr.setRequestHeader('Content-type', 'application/x-www-form-urlencoded'); part is quite useful as some browsers might not properly
work without it (at least for me it used to).

Final note

I should mention that the AjaxView is not necessarily THE way to go. It was more an idea over night to overcome those manual render and "echo json_encode" calls and to unify some of the redundant code in my app.
For your app it might be a different AjaxView. Go for it. This post, besides the AJAX topic itself, can also just show how easy it is in 2.x to extend the View class to something you can leverage to make your code DRY and consistent throughout your app.

CakeFest Talk 2014 Madrid

I had a talk about this topic at the CakePHP Conference this year, check out the slides @ slideshare.net or speakerdeck.com.

CakePHP 3.x

With 3.x being beta and all, it might be worth noting that not much will change here for the new major framework version.
The parseExtensions() method is gone, so this simplifies it a bit:

Router::extensions(['json']);

And in case you need to build the URL for this action, it is now _ext instead of ext as key:

$this->Html->link('Title', 
	['controller' => 'MyController', 'action' => 'myAction', '_ext' => 'json']);

Mind your casing: The above example assumes you are using the recommended DashedRoute class for URL generation.

All the rest of the code above is the same 🙂

2015-01 CakePHP 3

The code has been extracted into its own Ajax plugin. Check it out!

The live examples are to be found in the upgraded sandbox.

 
22 Comments

Posted in CakePHP

 

Queue – Deferred execution in CakePHP

22 Dec

At the beginning of 2.x I ported the Queue plugin from 1.x to 2.0 and started to integrate it into a few apps of mine.
Since then it was running OK, but neither with fixed tests, nor with any real documentation.
I fixed all that up – the plugin is available at github.

Background

Since this whole topic isn’t really covered by the book at all, we might start at the very beginning.

What is deferred execution – and why do we need it?

Most of us have at least a few typical examples in our applications where this would be either handy or a really good idea.
Let’s say your website is sending notification emails after a comment has been posted to one of your blog posts. You are probably doing this as a afterSave()
callback and trigger maybe 4-9 emails to you (the author) and maybe some previous commenters who wish to be notified on updates, as well.
Now, with basic CakePHP coding, you would probably also need a sleep(1) throttling timeout between the email sending processes to avoid
hitting a flood timeout. All in all you make the commenting user wait at least a few seconds before the page will refresh and he will finally see that his comment went though.
Imagine the email sending would need a little bit longer due to network issues. And the user hits F5 several times as nothing happens for him.

An annoying situation that could easily be avoided if unrelated tasks like notifications are not directly processed by the same thread (in this case the browser request of this commenting user). We delegate this task to another thread in the background which then processes this and many other tasks in the order they come in. This is called deferred execution. In our example above the main thread just puts those 9 emails to be sent out into the "queue" and immediately responds to the user within less than a second.
This can be combined with a priority flag to make tasks around email sending being processed before cleanup jobs for example.

There are many other scenarios where you should use a queue to handle background-tasks. Examples are:

  • Image processing/resizing
  • PDF (or other larger/demanding file) generation/extraction
  • Notification or larger update jobs
  • (Regular) cleanup or data monitoring

The Queue plugin is a good way to get to know background-processing and how to reduce response-time for heavy-processing tasks.
It also doesn’t have any other dependencies – except for basic CLI access.

Setup and configuration

As always, we need to load the plugin using CakePlugin::load('Queue') or CakePlugin::loadAll().

You can set any configuration values using your APP config file:

$config['Queue'] = array(
	...
);

The available options are explained in the README file.

The most basic setup – for local testing for example – would be to use the default "workermaxruntime" with 0 (running forever) and manually
starting a single worker cake Queue.Queue runworker. But once a worker dies in case of an error or server restart it won’t restart without another manual
trigger. Bear that in mind. I recommend the following approach using scheduled restarts for productive systems.

If you plan on having multiple workers or a fail-safe restart every few minutes you need a cronjob scheduler.
For linux the tool "crontab" is usually already available and used for other cronjob activities in CakePHP anyway.
With this you can simply create a new worker every x seconds – note that the lifetime of a worker should also be around that time.
Open the crontab editor for the www-data user (assuming you are using apache): crontab -u www-data -e

*/30 * * * * cd /srv/path/to/app && ../lib/Cake/Console/cake Queue.Queue runworker

If you have a Console folder inside your app, you can simplify the cake command ../lib/Cake/Console/cake to ./Console/cake.

Example usage

As explained above you should have always at least one worker up and running – either via crontab every x seconds or manually started.

Then you can start adding tasks to the queue. To play around you can use the command line. This might also be useful for some tasks you only
want to trigger once in a while – manually. cake Queue.Queue will list you all available queue tasks.

cake Queue.Queue add Example

This will add the Example task to the queue and should be executed and finished immediately.

If you want to see how the queue processes multiple tasks in order, try the LongExample task and while it is running add some Example tasks.
You can also play with more than a single worker at once.

Real usage

Enough of playing around, let’s get down to business.

Image generation

Let’s say we have an upload form and after save we want to trigger the rendering in the afterSave() callback of the model.

I created a task QueueImageTask inside my APP/Console/Command/Task folder. This way it will be recognized by the Queue plugin as "Image".
Then I could schedule this rendering task after each new upload using the "QueuedTask" model:

// In the afterSave() callback of the model, for example
$QueuedTask = ClassRegistry::init('Queue.QueuedTask');
$QueuedTask->createJob('Image', array('id' => $id));

The queue when idle will run in the background and check for scheduled tasks. It will see the new Image task and trigger its execution.
So when the worker processes it we can retrieve the record via the passed id and do all the magic we need to.
We can also can put a callback at the end of the task method before returning true. This way we can for example then update the status of the image from "inactive (not rendered)" to "active" (fully rendered and ready to be used).
This is an example of a two way communication task, reporting back to the app on success.
If it is a long running task (minutes to hours), one could also leverage the "progress" feature of the plugin.

Sending emails

For some projects I just send out SMTP mails without the queue. But if I want to switch, I simply have to modify the email settings.
Using my EmailLib, I can just switch transport from Smtp to Queue.Queue with 4 lines of adjustment:

public $default = array(
	...
	'transport' => 'Queue.Queue',
	'logTrace' => false, // Detailed trace log
	'logReport' => true, // Report log entry
	'queue' => array('transport' => 'Smtp', 'logTrace' => true), // Overwrite for inside the queue

This is the code to create a new email:

App::uses('EmailLib', 'Tools.Lib');
$Email = new EmailLib();
$Email->to(...);
$Email->subject('Testing Message');
$Email->domain(...); // If generated in a shell/task
$result = $Email->send('Foo');

This would now send the email directly into the queue – where it then will be processed and actually send through a worker.
With logReport => true we can see a log entry for passing the task on to the queue.

Using logTrace => true we get a detailed trace report in the log file after the worker processed the task. It would not make sense for the initial creation (as there is no trace from the SmtpTransport yet).

Emails are usually just sent out. They are one-way communication.

Debugging

With background tasks debugging is a little bit more difficult than with synchronous execution.
That’s why, by default, the Queue plugin will store the last execution time of all currently running workers in the TMP/queue/ folder (last modified date).
This gives you a basic idea of how many workers currently are running and how they can be identified.

You can, accessing the CLI, also identify and display all currently running workers using top. They have the command "php" in the last column.
If you have to kill all at once (especially when you are experimenting with lifetime 0), use killall php then.

Sugar

As noted above, I do have a little admin backend of its own for a quick look on how many workers are currently running and a glance at the stats/settings.
Just navigate to /admin/queue and it should display all of that. Note that it might need the Tools plugin as dependency here, though.

Vista

The above approach is a "CakePHP only" minimalistic tool for small and medium sized applications.
If you need a more robost approach, you can also look at complete tools like php-resque, djjob, celery or gearman which also usually have a CakePHP plugin these days.
So I will most likely look at Gearman and CO in the near future. For resque there is a nice integration article available.

Additionally, this article is more about raising awareness of background jobs in every day apps than actually proclaiming the one perfect solution for this.
But with more and more in the community actually going down that path, the existing tools around queuing in CakePHP sure will become more sophisticated. Check the awesome list for Queue plugins.

 
9 Comments

Posted in CakePHP

 

ResetBehavior and HazardableBehavior

29 Oct

ResetBehavior

Reset is a new behavior I recently had to write to update some geocoded entries as well as records with processed titles/slugs – via beforeSave() – that had to be re-saved with a corrected processing method. Basically, it is a batch update that works even with huge tables as it processes the data in steps.

Example: Resetting slugs

Either via shell or via controller action you can trigger an update of all slugs that have been generated via Tools plugin Slugged behavior:

// First we need to re-load the Slugged behavior to enable "overwrite" mode
$this->Post->Behaviors->load('Tools.Slugged', array('label' => 'title', 'overwite' => true));
// Load the Reset behavior with only the title and slug field to read and modify.
$this->Post->Behaviors->load('Tools.Reset', array('fields' => array('title', 'slug')));
$res = $this->Post->resetRecords();
// flash message with number of records modified in $res

You should make a shell command for this and execute this migration code once on deploy of the modified code or SQL schema.
If you are not using CLI, make sure you set the time limit in your controller action accordingly (HOUR for example).

Example: Retrigger/Init geocoding

If you got records that now need to be geocoded, you probably added a lat and lng field (decimal 6,2 etc) to your table.
You probably also attached the Tools plugin Geocoder behavior to this model.
In order to quickly geocode you just have to use Reset:

$this->Post->Behaviors->load('Tools.Reset', array('fields' => array('address', 'lat', 'lng'), 'timeout' => 3));
$res = $this->Post->resetRecords();

Since all lat/lng fields are still null it will geocode the records and populate those fields. It will skip already geocoded ones. If you
want to skip those completely (not even read them), just set the scope to 'NOT' => array('lat' => null) etc.

Note that in this case we also use a timeout to avoid getting a penalty by Google for geocoding too many records per minute.

Advanced example: Resetting composite cache field

In this case we added a new cache field to our messages in order to make the search faster with >> 100000 records. The data was containing
all the info we needed – in serialized format.
We needed a callback here as there was some logic involved. So we simply made a shell containing both callback method and shell command:

$this->Message->Behaviors->load('Tools.Reset', array(
	'fields' => array('data'), 'updateFields' => array('guest_name'),
	'scope' => array('data LIKE' => '{%'), 'callback' => 'UpdateShell::prepMessage'));
$res = $this->Message->resetRecords();
$this->out('Done: ' . $res);

The callback method (in this case just statically, as we didnt want to mess with the model itself):

public static function prepMessage(array $row) {
	if (empty($row['Message']['data_array']['GUEST_FIRST_NAME'])) {
		return array();
	}
	$row['Message']['guest_name'] = $row['Message']['data_array']['GUEST_FIRST_NAME'] . ' ' . $row['Message']['data_array']['GUEST_LAST_NAME'];
	return $row;
}

See the test cases for more ways to use callbacks – including adjusting the updateFields list.

So as you can see, everything that involves a complete "re-save" including triggering of important callbacks (in model and behaviors) of all or most records can leverage this behavior in a DRY, quick and reusable way.

HazardableBehavior

This behavior is a very useful tool to test vulnerability against XSS or unescaped html output (especially accidental one).
The basic idea is to test all views that output data from varchar or text fields for proper escaping. Even if it not user input, it is still vital to properly escape.
Admin input can have chars like <, > etc in there, as well. Without the use of h() it can destroy the layout or worse. So it is always a good idea to cover all
views.

Just attach it temporarily (!) to any of your models and quickly fill your table with hazardous strings. Those strings can potentially end up there via Form input, of course.
This just automates it.
Then you can browse your site and see if an alert or other strange behavior occurs. This tells you that you forgot to use h() or other measures to secure your output properly.

You can also apply this behavior globally to overwrite all strings in the find result.
This way you don’t need to modify the database. On output it will just inject the hazardous strings and you can browse your website just as if they were actually stored in your db.
Just add it to some models or even the AppModel (temporarily!) as $actsAs = array('Tools.Hazardable' => array('replaceFind' => true)).
A known limitation of Cake behaviors in 2.x, though, is, that this would only apply for first-level records (not related data). So it is usually better to insert some hazardous strings into all your tables and make your tests then as closely to the reality as possible.

You can use skipFields to blacklist certain stringish fields from being overwritten and populated with hazardous strings.

Note: In 3.x the behavior callback issue regarding non-primary records will be solved 🙂 I am really looking forward to that.

CakePHP 3

This article is 2.x only.
For CakePHP 3 please see the 3.0 Tools Plugin documentation on the Reset behavior.

 
No Comments

Posted in CakePHP