RSS
 

Archive for the ‘WebDevelopment’ Category

Update your Server to PHP5.4+

13 Apr

With many new features and more speed it makes sense to upgrade to PHP5.4. It is also a requirement when you start using CakePHP3.0 (the dev releases) or code that contains short array syntax. So the sooner you upgrade the better.

The following examples are for Debian Squeeze.

Backup

Make sure you have a backup of your server/files/db and your php.ini and other config files.

Updating

Debian doesn’t offer this natively. We need to add some special packages for this.

Open the sources list using nano, vi or the edit tool of your choice:

vi /etc/apt/sources.list

And add those two lines at the end:

deb http://packages.dotdeb.org stable all
deb-src http://packages.dotdeb.org stable all
 
deb http://packages.dotdeb.org squeeze-php54 all
deb-src http://packages.dotdeb.org squeeze-php54 all

We need a public key to avoid those warnings showing up:

wget http://www.dotdeb.org/dotdeb.gpg
cat dotdeb.gpg | apt-key add -

Use apt-get update and apt-get install php5.

That should take care of it. PHP5.4.x should now be installed and ready to work with. And have fun playing around with CakePHP3.0 or some new features of PHP5.4 :)

Keep other packages up to date

Also make sure other packages are up to date using apt-get upgrade.

If it doesn’t install right away, one might also need apt-get dist-upgrade. Careful with this, though. It might install/overwrite more than it should.

Dev Libraries

If php5-dev libraries are needed, they can be installed via apt-get install php5-dev.

Key Benefits of 5.4

  • Speed/Stability
  • Traits
  • Closures and $this
  • callable as typehint
  • Short array syntax (no mixin of method brackets and array brackets anymore)

The only real benefit coming in 5.5 is “Generators” which will allow handling of huge arrays and larger amount of data more gracefully. But other than that 5.4 is just fine for the moment.

Features of 5.3 you can now also use in Cake3

  • Namespaces
  • Closures
  • Late Static Bindings
  • Ternary shorthand form
  • Magic method __callStatic()
  • Nested exceptions
  • These features weren’t used in Cake2 core (and probably most apps) so far due to the BC for PHP5.2. If you already updated your minimum requirement for your server and application, those are also now of great use.

Further tips

The encoding even in more current verions of apache usually defaults to the old “ISO-8859-1″ one. You should make it “utf-8″ per default in your php.ini:

; PHP's default character set is set to empty.
; http://php.net/default-charset
default_charset = "utf-8"

Then even error pages or non-normal responses will be properly displayed. It is sad that even 2014 and in PHP5.4 the default is still not a proper one – probably due to BC or something.

 
No Comments

Posted in PHP

 

Generating PDFs with CakePHP

08 Apr

The recommended approach is to use the CakePHP plugin CakePdf for this. With that plugin it is “a piece of cake” to output HTML views as PDF or render them to files for download/emailing etc.

You can obviously just use an approach just like this article describes. But this makes the code quite dependent on the actual rendering library used isn’t very DRY when there is more than a single page that needs PDF rendering.

Note: This article is about generating PDF files from HTML templates. If you already have PDF files you just want to output or serve as download, then this article is not for you. See serving-views-as-files-in-cake2 then.

Installation

First you need to decide on what engine you want to go with.

WkHtmlToPdf is the fastest, but it also depends on CLI and the binaries being installed properly. So it might work differently or not at all on different OS. But using binary files it should work out of the both on both UNIX and Windows. The latter just needs a single exe file drag-and-dropped on your system. wkhtmltopdf.org/downloads.html contains such a ready-to-use binary. You can then link it in your Configure set-up for CakePdf:

Configure::write('CakePdf.binary', YOUR_PATH_TO . 'wkhtmltopdf\wkhtmltopdf.exe');

For UNIX those precombiled binaries on that download page might not work and a simple apt-get install wkhtmltopdf usually won’t do the trick. You need to manually compile it in this case. But on most systems it should be fine. Don’t forget to adjust the path to the binary here, as well, if it is not the default one.

I usually go with DomPdf as it is quite reliable, and (as PHP code) even though slower than a native approach will work flawlessly with all OS and setups without additional dependencies. For DomPdf one important setting when using images in your PDFs is to “enable remote”:

define('DOMPDF_ENABLE_REMOTE', true);

Either way don’t forget the rest of the settings for CakePdf – it could look like this:

$config['CakePdf'] = array(
    'engine' => 'CakePdf.DomPdf',
    'options' => ...,
    'margin' => ...,
    'orientation' => 'portrait',
);

The official documentation on the installation part tells you all you need to know. Check it out.

Advanced Setup

The recommended way is to use CakePlugin::load('CakePdf', array('bootstrap' => true, 'routes' => true)); which is supposed to take care of automatic View class switching.

If you don’t want to load the plugin bootstrap/routes and rather do it manually you need to automatically map all .pdf extension URLs to the CakePdf plugin PdfView ourselves:

public $components = array(
    'RequestHandler' => array(
        'viewClassMap' => array('pdf' => 'CakePdf.Pdf')
    )
);

This can also be useful if you want to overwite/extend the PdfView class for some reason (MyCakePdf etc).

Skipping the plugin routes file is also useful, if you already have extension settings in your routes. You can then simply update your APP/Config/routes.php:

Router::parseExtensions();
Router::setExtensions(array('json', 'xml', 'rss', 'pdf')); // We just add pdf to the already defined ones

With both routes and bootstrap defined manually, CakePlugin::load('CakePdf'); suffices.

Usage

The official documentation already states that, so I won’t go into much detail. For the sake of completeness a short example:

public function view($id) {
    $this->pdfConfig = array(
        'filename' => 'invoice',
        'download' => (bool)$this->request->query('download')
    );
    $invoice = $this->Invoice->find('first', array('conditions' => array('id' => $id)));
    $this->set(compact('invoice');
}

If you use my Tools plugin, you can just use $invoice = $this->Invoice->get($id) here.

Make sure you have a basic pdf layout in /View/Layouts/pdf/default.ctp and a view template for the PDF in /View/Invoices/pdf/view.ctp

Now if you allow access without extension (/invoices/view/1) the action can render a normal HTML page with the invoice in a table like form.

If you link it as /invoices/view/1.pdf it will automatically switch the view class to PdfView and render it as PDF file:

$this->Html->link('View PDF', array(
        'action' => 'view', $id, 'ext' => 'pdf'
));

Note the trick with download using the query string for it. If you link to it with ?download=1 it will trigger the download instead of just displaying it:

$this->Html->link('Download PDF', array(
        'action' => 'view', $id, 'ext' => 'pdf', '?' => array('download' => 1)
));

This way the action can do both automatically.

Tips

URLs and Paths

For all engines to work with your URLs and images outputted by helpers and therefore with schema-less absolute URLs (/controller/action/ and /img/foo.jpg) this little modification in your AppHelper is quite useful:

/**
 * Overwrite to make URLs absolute for PDF content.
 *
 * @param mixed $url
 * @param bool $full
 * @return string
 */
public function url($url = null, $full = false) {
    if (!empty($this->request->params['ext']) && $this->request->params['ext'] === 'pdf') {
        $full = true;
    }
    return parent::url($url, $full);
}
 
/**
 * Overwrite to make paths for assets absolute so they can be found by the PDF engine.
 *
 * @param string $path
 * @param array $options
 * @return string
 */
public function assetUrl($path, $options = array()) {
    if (!empty($this->request->params['ext']) && $this->request->params['ext'] === 'pdf') {
            $options['fullBase'] = true;
    }
    return parent::assetUrl($path, $options);
}

This will make them absolute including the full base (schema + domain) in PDF context.

Filename and downloading

If you don’t force downloading and display the PDF, make sure you read the “eastereggs” part of serving-views-as-files-in-cake2/ regarding an additional passed param here to actually make the downloaded file name what you want it to be.

 
No Comments

Posted in CakePHP

 

Using Composer

30 Mar

There are quite a few good articles on this topic, already. Especially on how to leverage it for CakePHP projects.

See

So I won’t go into detail too much. It is worth noting that composer really made a few things easier for us in the long run. And I was really against composer and packagist at the beginning as I didn’t want to use more dependencies than necessary. But that one sure is worth it if you use it right. And as every project pretty much uses it these days it is fairly stable and future proof to rely upon.

Installation

Easiest way: Download the phar and run php composer.phar [command]. You can just place this file into your app dir. You can also install it more globally, of course to just have to run composer [command] directly. The above articles or the composer docs explain that in more detail.

Usage

In general you can either use the composer commands to add dependencies or you can directly modify the composer.json file via editor (note that some IDEs support that natively with auto-completion). You also need to distinguish between “install” and “update” command. The first respects the current composer.lock file. The latter should only be used for development/staging, not for productive systems. See the basic usage docs for more details.

Composer and CakePHP

Can I switch my 2.x git project from submodules to composer?

Most definitly, yes. And you probably should. Using submodules really is a great annoyance, especially with the number of plugins increasing in a project. Unfortunately, getting rid of them is also not always easy – but doable.

You need to

  • git submodule deinit <path/to/plugin>
  • git rm <path/to/plugin>

Which, under windows, for example can proof a little bit difficult sometimes.

Then re-add them to the composer file and don’t forget to exlude the folder in the .gitignore file.

Composer and CakePHP3.x

In 3.x composer will pretty much be the de facto standard for CakePHP. So it is worth getting used to it now.

A few tips and tricks

I can add least talk about a few things I came across working with composer now for a few months.

VCS

Version control system repositories can easily be included if they don’t support composer or if you need to run a modified version of a 3rd party plugin/dependency. We then include the main source in the “require” section and add our specific (tweaked) fork as VCS in “repositories”:

"require": {
    "josegonzalez/cakephp-wysiwyg" : "dev-master",
},
"repositories" : [{
        "type" : "vcs",
        "url" : "https://github.com/MyOwnNameSpace/cakephp-wysiwyg" 
}],

Our version then contains some bug-fixes or additional stuff that needed it to branch off here.

dev prefix/suffix?

It kind of confused me when to use the suffix and when to use the prefix for “dev”. I found quite a look explanation from someone I added to this PR:

Basically dev releases are now simply branch names with a dev suffix for numeric branches which are comparable (3.0.*-dev) – or a dev prefix for textual names that are not comparable, like feature branches and master (dev-master)

You should authenticate the server/locally to avoid hitting the API limit

This page describes on how to generate an OAuth token and use this to avoid hitting the limit of requests per minute for GitHub.

Use proper user management to run composer commands

If you log in as root or any user that is not affialited with the www-data user, the files that are updated/created will most likely not be readable/usable. And using chmor -r 777 * sure is a really bad idea, as well. It is better to create a user that shares the same rights/group as www-data and use this one to deploy or do any changes on the code.

If something aint working

Try it again first – sometimes there are failures in the communication between GitHub, Packagist and your local connection.

Also try the self-update command to make sure you are running a recent version of composer. diagnose can also tell you of there is something wrong. Maybe you forgot to enable some required PHP extensions, like “curl” (which is necessary for it). But the error message usually tells you what exactly what the issue is.

 

Continuous Integration with Jenkins

04 Mar

CI with Jenkins and GitHub is especially interesting for private repositories, as CI with Travis is mainly for free GitHub repos (unless you have Travis Pro, of course). But since Jenkins is OpenSource and free, it might make sense to set up this on your server and run CI directly in-house. It integrates nicely with GitHub. So the following tutorial will focus on that example.

So for the beginning we have our example app in GitHub, e.g. a CakePHP app (optionally including composer dependencies).

The main goals are:

  • Continuous test results on each push (especially to master).
  • Automatic test results for each PR (very important when working in teams) – linked inside the PR just as Travis would.
  • Coverage being analysed to see where the weak points can be (classes with < 50% test case coverage).

Set Up Jenkins

This is probably the most difficult part of all.

We first set up Jenkins.

wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add -
 
nano /etc/apt/sources.list
------------------------------
deb http://pkg.jenkins-ci.org/debian binary/
------------------------------
 
apt-get update
apt-get install jenkins

Check if Jenkins runs properly

http://hostname.domain.local:8080

Install plugins and all their dependencies

Let’s then include the plugins we need to integrate GitHub: “GitHub Pull Request Builder” and “GitHub plugin”. The first is especially interesting as it makes it possible to have Jenkins “build/pass” statuses directly in each PR (similar to Travis).

Access Data -> Global Access Data -> Add Access Data

Create access data for the repository you want to access. Provide username and password or keyfiles.

Manage Jenkins -> Configure System

  • Add your repositories under “Git”
  • Set “Manually manage hook URLs” under “GitHub Web Hook”
  • Generate token (https://github.com/settings/applications) with Github user which has access to the private repository and put the token under “GitHub pull requests builder”

Add Job

  1. Select “Free Style” project and provide a jenkins working space name
  2. Put your repository url (e.g. https://github.com/Name/repo) to “GitHub-Project” input field
  3. Select Git under “Source-Code-Management” Put your repository url again and select the access data from step 5 of this tutorial. Click on advanced settings and set “+refs/pull/:refs/remotes/origin/pr/” for Refspec and “origin” for name. Set also the branch specifier to “${sha1}”
  4. Select “Build when a change is pushed to GitHub” and “GitHub pull requests builder” under “Build trigger” Add persons or organizations to “GitHub pull requests builder” which are whitelisted for trigger an PR build request
  5. Add shell execution script for your test cases under build proceed (see below)
  6. Add “Set build status on GitHub commit” as post build action.

My shell execution script for the CakePHP apps looks like this:

cd /var/lib/jenkins/jobs/projectname/workspace/app
cp /var/www/projectname/app/Config/database.php Config/
mkdir -p tmp && chmod -R 777 tmp && rm -R tmp/*
cp /var/www/projectname/app/composer.phar ./ && php composer.phar update
Console/cake test app AllApp --stderr

It first navigates into the freshly pulled repo code and into the APP dir. Then it copies over the database.php from the current staging website (could also be some other place where you store them). It then creates tmp dirs and sets the correct permissions. The composer.phar file is also copied over and executed to pull all composer dependencies. At finally the AllApp (group) tests are executed – with –stderr in order for the session to work properly. One can then also apply -q for silent (=none) output.

Setup GitHub hooks

To trigger the build process on Jenkins, we need to configure service hook on GitHub. In order to achieve this, navigate to GitHub repository settings and configure Jenkins Hook URL for GitHub plugin. The URL format http://<jenkins-username>:<jenkins-password>@<Elastic-IP-Address>:8080/github-webhook/

Running tests

Run first test manually

Click on “Build with Parameters” and hit “Build” and you will see the first manually triggered build in progress. If all goes well the icon should be blue (pass) – not red (fail). In case you need to see what is going on, check the “Console Output” of a specific build.

Check automated tests

Commit to master and see if a build is triggered. Also open a PR and check if there is a badge displayed with the current build status.

Sugar

Badge

To have a small “build status: passing” badge in our Readme, we can install the plugin “Embeddable Build Status”. It will create a new menu entry from where you can select your badges including ready-to-use Markdown syntax if needed.

And if it is a private repo you might also want to htaccess the Jenkins web interface or protect it some other way from the public.

Code Coverage

You can install the plugin “Clover PHP Plugin”. Create a folder “coverage” in your workspace.

Add this to the above command:

--log-junit '../coverage/unitreport.xml' --coverage-html '../coverage' --coverage-clover '../coverage/coverage.xml'

And set up a “Publish Clover PHP Coverage Report” Post Build Action with the absolute path to this coverage.xml file in the workspace. In my case just coverage/coverage.xml as it can be relative to the workspace, as well. If you also want to publish the HTML coverage report (which is pretty neat), you can simply put coverage in there then.

Make sure you got Xdebug up and running (apt-get install php5-xdebug). It is a direct dependency here. Note that using code coverage reports will slow the whole process down by “x3″ at least. Usually that shouldn’t really make a difference, though.

In most cases you want to exclude few dirs. In case you are already using a phpunit.xml file which resides in your APP dir, you can easily add

<phpunit>
    <filter>
        <blacklist>
            <directory suffix=".php">../lib</directory>
            <directory suffix=".ctp">../lib</directory>
            <directory suffix=".php">*/Vendor</directory>
            <directory suffix=".php">*/tmp</directory>
            <directory suffix=".php">*/Config/Migration</directory>
            <directory suffix=".php">*/Config/Schema</directory>
            <directory suffix=".php">*/Console/Templates</directory>
            <directory suffix=".php">*/Test</directory>            
            <directory suffix=".php">*/Plugin</directory>
        </blacklist>
    </filter>
</phpunit>

You can also just create this file and also copy it over to the jenkins workspace.

Memory Limit

If you have a project of medium to large size you will soon see the memory usage for the test coverage of 50%+ way above 200MB. With more coverage and the project getter bigger, or some coverage report generated you will run out of memory with 256MB (which one would think should suffice). Raise your limit to at least 512 for the PHPUnit testing. You can use the phpunit.xml file:

<phpunit>
    <php>
        <ini name="memory_limit" value="512M" />
    </php>
    ...
</phpunit>

Cleanup

Use the plugin “Discard Old Build plugin” to keep your diskspace reasonable. Without it (especially with CodeCoverage) the disk space will soon exceed several GB. We usually keep the last 40 builds.

More

This article goes more into detail about the installation of Jenkins and also how to use additional tools like CodeSniffer, MessDetector and CO using a build file and Ant. But I didn’t check that out yet. There is also jenkins-php.org if you want to go all in :)

 
No Comments

Posted in Testing

 

CakePHP and NamedScope for DRY conditions

15 Feb

A behavior for CakePHP 2.x

Background

I stumbled upon this fork and SimpleScope. The latter has the disadvantage of redundancy in the scope conditions when used in multiple find configs. The first was pretty much how Rails’ scopes work. But amongst other small issues it lacked the possibility of using model attributes for configuration. And both didn’t have test cases. So I decided to combine both, test the hell out of them and get the best out of the basic implementation ideas.

Basic Usage

For the behavior there is a more detailed documentation in the wiki. But here it goes.

First install/download and load the Tools plugin as documented in the cookbook or its readme file.

Attach the behavior to your AppModel:

App::uses('Model', 'Model');
 
class AppModel extends Model {
    public $actsAs = array('Tools.NamedScope');
}

Then define some scopes in your model:

App::uses('AppModel', 'Model');
 
class User extends AppModel {
    public $scopes = array(
        'active' => array('User.active' => 1),
        'admin' => array('User.role LIKE' => '%admin%'),
    );
}

Then you can use those scopes in any of your find queries:

$activeUsers = $this->User->find('all', array('scope' => array('active')));
$activeAdmins = $this->User->find('all', array('scope' => array('active', 'admin')));
$activeAdminList = $this->User->find('list', array('scope' => array('active', 'admin')));

Advanced Usage

If you also want to use scopedFind(), you will also get rid of all the many find wrappers around those scopes that will often be placed inside the models.

An example:

public function getActiveAdmins() {
    $this->virtualFields['fullname'] = "CONCAT(User.firstname, ' ', User.lastname)";
    $options = array(
        'fields' => array('User.id', 'User.fullname'),
        'conditions' => array('User.role LIKE' => '%admin%'),
        'order' => array('User.fullname'),
    );
    return $this->find('all', $options);
}

Now there will maybe also be a getActiveUsers() method and maybe a few dozen more, which all contain the same condition – which is not really DRY and might be quite error-prone if the conditions have to be adjusted (easy to miss one of the many occurrences in and out of the model).

So what would be a smarter way to approach this? Let’s try to use the above scopes here – and also use the single wrapper method. Besides the above scopes, you also need to define some scopedFinds in your model:

App::uses('AppModel', 'Model');
 
class User extends AppModel {
    public $scopes = array(
        'activeAdmins' => array(
            'name' => 'Active admin users',
            'find' => array(
                'type' => 'all',
                'virtualFields' => array(
                    'fullname' => "CONCAT(User.firstname, ' ', User.lastname)"
                ),
                'options' => array(
                    'fields' => array('User.id', 'User.fullname'),
                    'scope' => array('active', 'admin'),
                    'order' => array('User.fullname'),
                ),
            ),
        ),
        'activeUsers' => array(
             ...
        )
    );
}

The scope itself will both contain active, and the config about this scope key will be stored in a single place. So if you have some very complex condition around published (> a && < b && != c && …) this will take the overhead from multiple definitions and reduce it to a single location.

Let’s execute it:

$activeAdmins = $this->User->scopedFind('activeAdmins');

In case we need to only get a list or the count, we can adjust the scopedFind:

$activeAdminList = $this->User->scopedFind('activeAdmins', array('type' => 'list'));
$activeAdminCount = $this->User->scopedFind('activeAdmins', array('type' => 'count'));

We can also overwrite the default options:

$config = array(
    'options' => array(
        'limit' => 2, 
        'order' => array('User.created' => 'DESC'))
);
$twoNewestActiveAdmins = $this->User->scopedFind('activeAdmins', $config);

You can also get a list of available scoped finds:

$scopedFinds = $this->User->scopedFinds();

Scoped finds:

  • require a name string
  • optionally use a find array

The find arrays:

  • optionally use a type string (defaults to all)
  • optionally use an options array
  • optionally use virtualFields

The options arrays:

  • can use the behaviors’ scope property
  • support all other find options (including contain, order, group, limit, …)

Tip: See the test cases for more complex examples.

Testing

You should test your scopes, even if it’s just something like this:

public function testScopes() {
    $scopes = $this->User->scopes;
    // Each on its own
    foreach ($scopes as $scope) {
        $this->User->find('first', array('scope' => $scope));
    }
    // All together
    $this->User->find('first', array('scope' => $scopes));
}

In case there was invalid SQL, missing fields, wrong contain statements, it would be noticeable right away.

If you use scopedFinds, don’t forget to also unit test them (regarding valid SQL). This can easily be forgotten now as you don’t have the find wrapper methods anymore. In case you are lazy, add this test case to any model test that uses custom scopedFinds:

public function testScopedFinds() {
    $scopedFinds = $this->User->scopedFinds();
    foreach ($scopedFinds as $key) {
        $this->User->scopedFind($key);
    }
}

This will at least execute each find and throw an error if the SQL is invalid. It is advisable to have a more thorough test case for each find key, though, that includes the assert of the return value.

Outlook

I bet when this behavior is thoroughly used, there will be quite a few adjustments necessary. But all in all this already seems to cover most of the use cases.

With Cake3 and stackable custom finders much of the functionality we need here will be part of core functionality. Which will be awesome. Never-the-less, until then this can be a solid solution to keep the scopes/conditions DRY.

 
No Comments

Posted in CakePHP

 

AJAX and CakePHP

09 Jan

I don’t know why I didn’t post about AJAX earlier. It is probably one of the topics that most don’t really know how to approach. The docs don’t say much about it, and so many try to rely on (outdated) blog posts out there.

Why AJAX

We shouldn’t use stuff just because it sounds fun. So sometimes AJAX is abused or used where it shouldn’t. I use it mainly where the user cannot or should not reload the whole page just to get more information or an updated piece of content which will be loaded after the page completed rendering (asynchronous). This can be part of a (sub) form, a search result or detailed information on some piece of content in a popover like window.

Basic concept

I always recommend to use the proper extension when dealing with AJAX. So if we deal with basic HTML, we can simply go for /controller/action/. See the AJAX Pagination example on how to use AJAX here as sugar on top of the basic pagination (that then serves as a fallback for non-JS-browsing or search engines). In this case it is fine to serve AJAX extension-less.

In most cases (and in all where fallbacks are not necessary) it is better to use JSON as a return type, though. this allows you to deal with the returned content in a more flexible way. That also means that is is not extension-less anymore now. Thus, you should target /controller/action.json and use the RequestHandler to get all the CakePHP magic going here (setting the correct response header application/json and switching/disabling the layout/views.

This is especially important when throwing exceptions (404 = NotFoundException, 403 = NotAllowedException). All those would then automatically be a proper JSON response message right away.

So for a basic JSON response via AJAX, all you then need is:

/**
 * Ajax action to get favorites
 */
public function favorites() {
    $this->autoRender = false; // We don't render a view in this example
    $this->request->onlyAllow('ajax'); // No direct access via browser URL
 
    $data = array(
        'content' => ...,
        'error' => null,
    );
    return json_encode($data));
}

This will not use a view or layout file and simply return a json encoded string.

But to me that looks kind of wrong. Manually using json_encode() in the controller is not something you should aim for. The view classes are responsible for the actual translation into the target representation format. So such a view class should contain that call. We only make sure that this view class gets the data it needs.

That said you could use the JsonView to make it cleaner and cut down the code by a few lines:

public function favorites() {
    $this->request->onlyAllow('ajax'); // No direct access via browser URL - Note for Cake2.5: allowMethod()
 
    $data = array(
        'content' => ...,
        'error' => null,
    );
    $this->set('_serialize', 'data');
}

Also make sure to check out the JsonView docs on how to use a view (if needed). It will then automatically use a subfolder “json” and its View/ControllerName/json/favorites.ctp template here.

In the view we can then use this action via JS and the URL /controller_name/favorites.json:

$.ajax({
    type: 'get',
    url: '<?php echo $this->Html->url(array('action' => 'favorites', 'ext' => 'json')); ?>',
    beforeSend: function(xhr) {
        xhr.setRequestHeader('Content-type', 'application/x-www-form-urlencoded');
    },
    success: function(response) {
        if (response.error) {
            alert(response.error);
            console.log(response.error);
        }
        if (response.content) {
            $('#target').html(response.content);
        }
    },
    error: function(e) {
        alert("An error occurred: " + e.responseText.message);
        console.log(e);
    }
});

Note that we can first check on the error string to abort modification of the DOM if there is some error (similar to validation errors or flash message errors) occuring. And if there is some content we can then easily output that.

AjaxView

Now that is all nice, but for me that is not yet it. In this particular case we don’t just pass down model data. We render the view just as we would for normal GET access. Thus I would like it to be as close to it as possible. I want to avoid the serialize call and manual json_encode part (both in controller and view layer), just as in JsonView, but with all the basic View rendering still intact. So I created the AjaxView.

It takes care of mapping the content into the above JSON structure, and automatically tries to render via “ajax” subfolder (View/ControllerName/ajax/favorites.ctp). It will also not use any layout, just the simple HTML template code in the ctp.

A typical use case:

public function favorites() {
    $this->request->onlyAllow('ajax');
    $this->viewClass = 'Tools.Ajax';
}

The result can be of the following structure:

{
    "content": [Result of our rendered favorites.ctp],
    "error": ''
}

See the test cases for it on more cool stuff – e.g. adding some more data to the response object via _serialize.

Full blown example: Chained dropdowns

This has been one of the examples around the usage of AJAX. When you have two dropdowns and select something from the first, the second dropdown should contain updated content relevant for that selection. In our example, we select countries, and upon selection you can then chose from country provinces (if applicable).

Our controller fetches the lists of countries and country provinces:

public function chained_dropdowns() {
        $countries = $this->Controller->Country->getList();
        $countryProvinces = array();
        foreach ($countries as $key => $value) {
            $countryProvinces = $this->Controller->CountryProvince->getListByCountry($key);
            break;
        }
        $this->set(compact('countries', 'countryProvinces'));
    }

The CountryProvinceHelper component (from my Data plugin) which I usually use here basically does some more, but for a simple example this suffices. It will simply pass down all countries and the country provinces for the first country (which will be auto selected right away).

The view then contains the form:

<?php echo $this->Form->create('User');?>
    <fieldset>
        <legend><?php echo __('Countries and Country Provinces');?></legend>
    <?php
        $url = $this->Html->url(array('plugin' => 'sandbox', 'controller' => 'ajax_examples', 'action' => 'country_provinces_ajax', 'ext' => 'json'));
        $empty = count($countryProvinces) > 0 ? __('pleaseSelect') : array('0' => __('noOptionAvailable'));
 
        echo $this->Form->input('country_id', array('id' => 'countries', 'rel' => $url));
        echo $this->Form->input('country_province_id', array('id' => 'provinces', 'empty' => $empty));
    ?>
    </fieldset>
 
    The province list is updated each time the country is switched. It also has a basic fallback for POST data (will auto-remember the previous selection).
    <br /><br />
 
<?php echo $this->Form->end(__('Submit'));?>
</div>

The jQuery code can either be added to the page or stored in a separate js file:

$(function() {
 
    $('#countries').change(function() {
        var selectedValue = $(this).val();
 
        var targeturl = $(this).attr('rel') + '?id=' + selectedValue;
        $.ajax({
            type: 'get',
            url: targeturl,
            beforeSend: function(xhr) {
                xhr.setRequestHeader('Content-type', 'application/x-www-form-urlencoded');
            },
            success: function(response) {
                if (response.content) {
                    $('#provinces').html(response.content);
                }
            },
            error: function(e) {
                alert("An error occurred: " + e.responseText.message);
                console.log(e);
            }
        });
    });
 
});

Then all we need is the AJAX action

public function country_provinces_ajax() {
        $this->request->onlyAllow('ajax');
        $id = $this->request->query('id');
        if (!$id) {
            throw new NotFoundException();
        }
 
        $this->viewClass = 'Tools.Ajax';
 
        $this->loadModel('Data.CountryProvince');
        $countryProvinces = $this->CountryProvince->getListByCountry($id);
 
        $this->set(compact('countryProvinces'));
    }

and the AJAX view for this action:

<?php
if (!empty($countryProvinces)) {
    echo '<option value="">' . __('pleaseSelect') . '</option>';
    foreach ($countryProvinces as $k => $v) {
        echo '<option value="' . $k . '">' . h($v) . '</option>';
    }
} else {
    echo '<option value="0">' . __('noOptionAvailable') . '</option>';
}

To see the full code in action take a look at the live example in my Sandbox or the source code of it.

Error out

If some error occurs in your action you could either throw an exception and handle that in the frontend or pass down an error code/string to the AjaxView, disabling rendering and returning only the error to the frontend (in response.error):

$this->set('error', 'No record found');

For simple stuff it’s probably easier to use the exception free approach (using HTTP code 200). The resulting JSON would be something like:

{
    "error": "No Record found",
    "content": "",
}

But in general try to stick to exception based error handling. That would return a response with an error code of 4xx or 5xx usually (depending on the type of exception) and the following JSON object:

{
    "code": 404,
    "message": "Not Found", // Note that in Cake < 2.5 this might be "name"
    "url": "...",
}

In this example it’s a NotFoundException.

Your JS code needs to handle this gracefully. There could always be some database exception or alike thrown. So better account for that scenario.

Further improvement

You could write some generic JS code or even a generic jquery plugin to handle the AJAX response. This would cut down the lines for the query code drastically and keep it DRY.

If you want to automatically make all ajax requests respond with this view class, you could use something like this in your AppController’s beforeFilter callback:

if ($this->request->is('ajax') {
    $this->viewClass = 'Tools.Ajax'
}

It would cut down on that one line your actions.

I would not recommend that, though. as the JSON view is usually a good default. In my experience you usually need that more often, e.g. when just returning data (multiple records in particular) from the model without rendering any view.

But in general you could try to leverage a component to take care of all that and on top add even more cool things like redirect handling (output it along with the rest of the JSON data instead of actually redirecting) or flash message handling. That together could make it a real useful bundle and cut down your controller code a lot. No need to add a lot of switchs and if statements, if the code is exactly the same for both AJAX and normal requests. See my AjaxComponent which is still “beta” and could be the answer to this.

Last tips

Don’t forget to include the RequestHandler component and set up Router::parseExtensions in your routes:

Router::parseExtensions();
Router::setExtensions(array('json', ...));

This will tell CakePHP that all .json URLS are automatically served as JSON (which the appropriate headers).

Remember to not cache ajax actions to avoid old content being delivered:

if ($this->request->is('ajax')) {
    $this->disableCache();
}

I actually use that statement for all my actions/content that contain dynamically build content.

If you plan on updating the DB, never use “get”, always use type: 'post' and also assert that your action only accepts POST. “get” is only appropriate if the DB is not changed by the AJAX request, so if you only use find().

Note the difference between <option value=""> and <option value="0">. Using validation and the numeric rule we assert that the latter passes validation (when there is nothing to select) whereas the first would trigger a “You need to select a province” error of some sort when trying to save.

Using var targeturl = $(this).attr('rel') + '?id=' + selectedValue; or tricks like data('url') and data-url etc keeps the URL handling and other dynamic PHP based content out of the JS files and makes it easier to store them separately and without direct dependencies. It also avoids shortcut solutions like hardcoding URLs and such.

The xhr.setRequestHeader('Content-type', 'application/x-www-form-urlencoded'); part is quite useful as some browsers might not properly work without it (at least for me it used to).

Final note

I should mention that the AjaxView is not THE way to go. It was more an idea over night to overcome those manual render and “echo json_encode” calls and to unify some of the redundant code in my app. For your app it might be a different AjaxView. Go for it. This post, besides the AJAX topic itself, can also just show how easy it is in 2.x to extend the View class to something you can leverage to make your code DRY and consistent throughout your app.

 
3 Comments

Posted in CakePHP

 

Queue – Deferred execution in CakePHP

22 Dec

At the beginning of 2.x I ported the Queue plugin from 1.x to 2.0 and started to integrate it into a few apps of mine. Since then it was running OK, but neither with fixed tests, nor with any real documentation. I fixed all that up – the plugin is available at github.

Background

Since this whole topic isn’t really covered by the book at all, we might start at the very beginning.

What is deferred execution – and why do we need it?

Most of us have at least a few typical examples in our applications where this would be either handy or a really good idea. Let’s say your website is sending notification emails after a comment has been posted to one of your blog posts. You are probably doing this as a afterSave() callback and trigger maybe 4-9 emails to you (the author) and maybe some previous commenters who wish to be notified on updates, as well. Now, with basic CakePHP coding, you would probably also need a sleep(1) throttling timeout between the email sending processes to avoid hitting a flood timeout. All in all you make the commenting user wait at least a few seconds before the page will refresh and he will finally see that his comment went though. Imagine the email sending would need a little bit longer due to network issues. And the user hits F5 several times as nothing happens for him.

An annoying situation that could easily be avoided if unrelated tasks like notifications are not directly processed by the same thread (in this case the browser request of this commenting user). We delegate this task to another thread in the background which then processes this and many other tasks in the order they come in. This is called deferred execution. In our example above the main thread just puts those 9 emails to be sent out into the “queue” and immediately responds to the user within less than a second. This can be combined with a priority flag to make tasks around email sending being processed before cleanup jobs for example.

There are many other scenarios where you should use a queue to handle background-tasks. Examples are:

  • Image processing/resizing
  • PDF (or other larger/demanding file) generation/extraction
  • Notification or larger update jobs
  • (Regular) cleanup or data monitoring

The Queue plugin is a good way to get to know background-processing and how to reduce response-time for heavy-processing tasks. It also doesn’t have any other dependencies – except for basic CLI access.

Setup and configuration

As always, we need to load the plugin using CakePlugin::load('Queue') or CakePlugin::loadAll().

You can set any configuration values using your APP config file:

$config['Queue'] = array(
    ...
);

The available options are explained in the README file.

The most basic setup – for local testing for example – would be to use the default “workermaxruntime” with 0 (running forever) and manually starting a single worker cake Queue.Queue runworker. But once a worker dies in case of an error or server restart it won’t restart without another manual trigger. Bear that in mind. I recommend the following approach using scheduled restarts for productive systems.

If you plan on having multiple workers or a fail-safe restart every few minutes you need a cronjob scheduler. For linux the tool “crontab” is usually already available and used for other cronjob activities in CakePHP anyway. With this you can simply create a new worker every x seconds – note that the lifetime of a worker should also be around that time. Open the crontab editor for the www-data user (assuming you are using apache): crontab -u www-data -e

*/30 * * * * cd /srv/path/to/app && ../lib/Cake/Console/cake Queue.Queue runworker

If you have a Console folder inside your app, you can simplify the cake command ../lib/Cake/Console/cake to ./Console/cake.

Example usage

As explained above you should have always at least one worker up and running – either via crontab every x seconds or manually started.

Then you can start adding tasks to the queue. To play around you can use the command line. This might also be useful for some tasks you only want to trigger once in a while – manually. cake Queue.Queue will list you all available queue tasks.

cake Queue.Queue add Example

This will add the Example task to the queue and should be executed and finished immediately.

If you want to see how the queue processes multiple tasks in order, try the LongExample task and while it is running add some Example tasks. You can also play with more than a single worker at once.

Real usage

Enough of playing around, let’s get down to business.

Image generation

Let’s say we have an upload form and after save we want to trigger the rendering in the afterSave() callback of the model.

I created a task QueueImageTask inside my APP/Console/Command/Task folder. This way it will be recognized by the Queue plugin as “Image”. Then I could schedule this rendering task after each new upload using the “QueuedTask” model:

// In the afterSave() callback of the model, for example
$QueuedTask = ClassRegistry::init('Queue.QueuedTask');
$QueuedTask->createJob('Image', array('id' => $id));

The queue when idle will run in the background and check for scheduled tasks. It will see the new Image task and trigger its execution. So when the worker processes it we can retrieve the record via the passed id and do all the magic we need to. We can also can put a callback at the end of the task method before returning true. This way we can for example then update the status of the image from “inactive (not rendered)” to “active” (fully rendered and ready to be used). This is an example of a two way communication task, reporting back to the app on success. If it is a long running task (minutes to hours), one could also leverage the “progress” feature of the plugin.

Sending emails

For some projects I just send out SMTP mails without the queue. But if I want to switch, I simply have to modify the email settings. Using my EmailLib, I can just switch transport from Smtp to Queue.Queue with 4 lines of adjustment:

public $default = array(
    ...
    'transport' => 'Queue.Queue',
    'logTrace' => false, // Detailed trace log
    'logReport' => true, // Report log entry
    'queue' => array('transport' => 'Smtp', 'logTrace' => true), // Overwrite for inside the queue

This is the code to create a new email:

App::uses('EmailLib', 'Tools.Lib');
 
$Email = new EmailLib();
$Email->to(...);
$Email->subject('Testing Message');
$Email->domain(...); // If generated in a shell/task
$result = $Email->send('Foo');

This would now send the email directly into the queue – where it then will be processed and actually send through a worker. With logReport => true we can see a log entry for passing the task on to the queue.

Using logTrace => true we get a detailed trace report in the log file after the worker processed the task. It would not make sense for the initial creation (as there is no trace from the SmtpTransport yet).

Emails are usually just sent out. They are one-way communication.

Debugging

With background tasks debugging is a little bit more difficult than with synchronous execution. That’s why, by default, the Queue plugin will store the last execution time of all currently running workers in the TMP/queue/ folder (last modified date). This gives you a basic idea of how many workers currently are running and how they can be identified.

You can, accessing the CLI, also identify and display all currently running workers using top. They have the command “php” in the last column. If you have to kill all at once (especially when you are experimenting with lifetime 0), use killall php then.

Sugar

As noted above, I do have a little admin backend of its own for a quick look on how many workers are currently running and a glance at the stats/settings. Just navigate to /admin/queue and it should display all of that. Note that it might need the Tools plugin as dependency here, though.

Vista

The above approach is a “CakePHP only” minimalistic tool for small and medium sized applications. If you need a more robost approach, you can also look at complete tools like php-resque, djjob, celery or gearman which also usually have a CakePHP plugin these days. So I will most likely look at Gearman and CO in the near future. For resque there is a nice integration article available.

Additionally, this article is more about raising awareness of background jobs in every day apps than actually proclaiming the one perfect solution for this. But with more and more in the community actually going down that path, the existing tools around queuing in CakePHP sure will become more sophisticated.

 
1 Comment

Posted in CakePHP

 

ResetBehavior and HazardableBehavior

29 Oct

ResetBehavior

Reset is a new behavior I recently had to write to update some geocoded entries as well as records with processed titles/slugs – via beforeSave() – that had to be re-saved with a corrected processing method. Basically, it is a batch update that works even with huge tables as it processes the data in steps.

Example: Resetting slugs

Either via shell or via controller action you can trigger an update of all slugs that have been generated via Tools plugin Slugged behavior:

// First we need to re-load the Slugged behavior to enable "overwrite" mode
$this->Post->Behaviors->load('Tools.Slugged', array('label' => 'title', 'overwite' => true));
// Load the Reset behavior with only the title and slug field to read and modify.
$this->Post->Behaviors->load('Tools.Reset', array('fields' => array('title', 'slug')));
$res = $this->Post->resetRecords();
// flash message with number of records modified in $res

You should make a shell command for this and execute this migration code once on deploy of the modified code or SQL schema. If you are not using CLI, make sure you set the time limit in your controller action accordingly (HOUR for example).

Example: Retrigger/Init geocoding

If you got records that now need to be geocoded, you probably added a lat and lng field (decimal 6,2 etc) to your table. You probably also attached the Tools plugin Geocoder behavior to this model. In order to quickly geocode you just have to use Reset:

$this->Post->Behaviors->load('Tools.Reset', array('fields' => array('address', 'lat', 'lng'), 'timeout' => 3));
$res = $this->Post->resetRecords();

Since all lat/lng fields are still null it will geocode the records and populate those fields. It will skip already geocoded ones. If you want to skip those completely (not even read them), just set the scope to 'NOT' => array('lat' => null) etc.

Note that in this case we also use a timeout to avoid getting a penalty by Google for geocoding too many records per minute.

Advanced example: Resetting composite cache field

In this case we added a new cache field to our messages in order to make the search faster with >> 100000 records. The data was containing all the info we needed – in serialized format. We needed a callback here as there was some logic involved. So we simply made a shell containing both callback method and shell command:

$this->Message->Behaviors->load('Tools.Reset', array(
    'fields' => array('data'), 'updateFields' => array('guest_name'),
    'scope' => array('data LIKE' => '{%'), 'callback' => 'UpdateShell::prepMessage'));
$res = $this->Message->resetRecords();
$this->out('Done: ' . $res);

The callback method (in this case just statically, as we didnt want to mess with the model itself):

public static function prepMessage(array $row) {
    if (empty($row['Message']['data_array']['GUEST_FIRST_NAME'])) {
        return array();
    }
 
    $row['Message']['guest_name'] = $row['Message']['data_array']['GUEST_FIRST_NAME'] . ' ' . $row['Message']['data_array']['GUEST_LAST_NAME'];
    return $row;
}

See the test cases for more ways to use callbacks – including adjusting the updateFields list.

So as you can see, everything that involves a complete “re-save” including triggering of important callbacks (in model and behaviors) of all or most records can leverage this behavior in a DRY, quick and reusable way.

HazardableBehavior

This behavior is a very useful tool to test vulnerability against XSS or unescaped html output (especially accidental one). The basic idea is to test all views that output data from varchar or text fields for proper escaping. Even if it not user input, it is still vital to properly escape. Admin input can have chars like <, > etc in there, as well. Without the use of h() it can destroy the layout or worse. So it is always a good idea to cover all views.

Just attach it temporarily (!) to any of your models and quickly fill your table with hazardous strings. Those strings can potentially end up there via Form input, of course. This just automates it. Then you can browse your site and see if an alert or other strange behavior occurs. This tells you that you forgot to use h() or other measures to secure your output properly.

You can also apply this behavior globally to overwrite all strings in the find result. This way you don’t need to modify the database. On output it will just inject the hazardous strings and you can browse your website just as if they were actually stored in your db. Just add it to some models or even the AppModel (temporarily!) as $actsAs = array('Tools.Hazardable' => array('replaceFind' => true)). A known limitation of Cake behaviors in 2.x, though, is, that this would only apply for first-level records (not related data). So it is usually better to insert some hazardous strings into all your tables and make your tests then as closely to the reality as possible.

You can use skipFields to blacklist certain stringish fields from being overwritten and populated with hazardous strings.

Note: In 3.x the behavior callback issue regarding non-primary records will be solved :) I am really looking forward to that.

 
No Comments

Posted in CakePHP

 

RSS feeds in CakePHP

03 Oct

Recently I noticed that Thunderbird doesn’t import and update RSS feeds anymore that are (partially) invalid. Well, mine always were I guess. Back in the days (4 years ago) I wasn’t aware that the author tag was only for email addresses, not for usernames. One has to say that it also wasn’t possible in an easy way to handle “dc:creator” instead of author. So my feed from back then had this and a few other “errors” in them. But they worked, so I never bothered. Until recently.

Upgrading more apps to 2.x made me realize that there now are such cool View classes for other types of data (Json, Csv, Xml). And since RSS is pretty much XML, I thought, why the hell are we even using a really old and partially working helper + view + layout? It should have been upgraded to a View class, as well, IMO. So I opened a ticket for discussion and started to do that.

Goals:

  • Use view-less actions via serialize.
  • Get rid of the ridiculously verbose “inline” namespace declarations.
  • Simplify the use of namespaces and their prefixes (auto-add only those that are actually used).
  • Support CDATA (unescaped content).

First, I just dropped all the helper code into the class. That didn’t really work out as the rendering of the existing code renders each item by itself, thus not having any information about the global namespaces. I had to drop the whole thing and start from scratch, orienting myself on how the XmlClass does it. So I used my old feed content to slowly get it working again using a complete array structure and only a single Xml::fromArray() call at the very end. I dropped the class into my Tools plugin and added a few basic tests.

If you include the plugin, don’t forget the CakePlugin::load('Tools') or CakePlugin::loadAll() call.

Usage

We first need an action to output our feed using Router::parseExtensions(array('rss')) (in your routes or bootstrap file) and access it via:

/controller/action.rss

We also need to add $this->viewClass = 'Tools.Rss' to every action that outputs RSS as the RequestHandler component only switches yet for the core Xml and Json View classes.

A basic feed contains at least a title, description and a link for both channel and items. It is also advised to add the atom:link to the location of the feed itself.

$this->viewClass = 'Tools.Rss'; // Important if you do not have an auto-switch for the rss extension
$atomLink = array('controller' => 'topics', 'action' => 'feed', 'ext' => 'rss'); // example controller and action
$data = array(
    'channel' => array(
        'title' => 'Channel title',
        'link' => 'http://channel.example.org',
        'description' => 'Channel description',
        'atom:link' => array('@href' => $atomLink),
    ),
    'items' => array(
        array('title' => 'Title One', 'link' => 'http://example.org/one', 
            'author' => 'one@example.org', 'description' => 'Content one'),
        array('title' => 'Title Two', 'link' => 'http://example.org/two', 
            'author' => 'two@example.org', 'description' => 'Content two'),
    ));
);
$this->set(array('data' => $data, '_serialize' => 'data'));

It is also possible to use one of the already built in namespaces – e.g. if you want to display a post’s username instead of email (which you should^^). You can also add the content itself as CDATA. The description needs to be plain text, so if you have HTML markup, make sure to strip that out for the description but pass it unescaped to the content namespace tag for it.

$data = array(
    'channel' => array(
        'title' => 'Channel title',
        'link' => 'http://channel.example.org',
        'description' => 'Channel description'
    ),
    'items' => array(
        array('title' => 'Title One', 'link' => 'http://example.org/one', 
            'dc:creator' => 'Mr Bean', 'description' => 'Content one',
            'content:encoded' => 'Some <b>HTML</b> content'),
        array('title' => 'Title Two', 'link' => 'http://example.org/two', 
            'dc:creator' => 'Luke Skywalker', 'description' => 'Content two',
            'content:encoded' => 'Some <b>more HTML</b> content'),
    )
);
$this->set(array('data' => $data, '_serialize' => 'data'));

Extendability

You can easily register new namespaces, e.g. to support the google data feeds (xmlns:g="http://base.google.com/ns/1.0"):

$data = array(
    'document' => array(
        'namespace' => array(
            'g' => 'http://base.google.com/ns/1.0'
        )
    )
    'channel' => array(
        ...
    ),
    'items' => array(
        array('g:price' => 25, ...),
    )
);
$this->set(array('data' => $data, '_serialize' => 'data'));

So forget that there was a helper and a complicated way to set up RSS feeds via view files. This is the way to go.

View class mapping

See the documentation on how to use view class mapping to automatically respond with the RssView for each request to the rss extension:

'rss' => 'Tools.Rss'

With the help of parseExtensions() and RequestHandler this will save you the extra view config line in your actions.

Passing params.

If you need to pass params to this view, use query strings:

.../action.rss?key1=value1&key2=value2

Demo

sandbox.dereuromark.de/sandbox/rss_examples (including feed validation check)

Source code: RssExamplesController.php

Vista

There are still lots of things that could be implemented. It still does not handle all the use cases possible, for example.

It also stands to discussion if one could further generalize the class to not only support RSS feeds, but other type of feeds, as well.

 
No Comments

Posted in CakePHP

 

Moving a CakePHP app

29 Sep

This weekend I had to move some small to medium sized CakePHPs from one server to another. Basically, I had to switch hosting for them (web space and database).

I haven’t had the time to write a lot lately. But once in a while – even if only for later reference – doesn’t hurt. So here a small real-life progress report about an app migration the Cake way :)

Key goals

  • Moving the app including non-version-controlled files and database
  • Keeping the downtime to a minimum
  • Once migration has started, source must not be modified (database, files).
  • Target should only allow modification after migration is complete and verified.

Requirements

All shells used are part of my Setup plugin. For the MaintenanceMode to work you need the following snippet added to the end of your bootstrap file:

if (php_sapi_name() !== 'cli') {
    App::uses('MaintenanceLib', 'Setup.Lib');
    $Maintenance = new MaintenanceLib();
    $Maintenance->checkMaintenance();
}

I don’t have to add that the plugin needs to be loaded first, of course :)

Initial Steps

The following needs to be done, already, of course:

  • The target server needs to be set up properly, including PHP, MySQL including the proper Apache/environment settings.
  • The target htdocs folder is already prepared using “svn checkout repo-url” or “git checkout repo-url” depending on the type of version control in use. It also has already basic “writable” TMP dirs and other things that can already be applied beforehand.

Since I already did that I just copied over the vhost setup and reloaded Apache:

/etc/init.d/apache2 force-reload

Up until this point the source is still up and running.

Moving on

Now it’s time to put the source into “maintenance mode”. Once we move files we don’t want the source to be modified again afterwards and to compromise a clean switch.

I used my MaintenanceShell to quickly enable it:

cake Setup.Maintenance activate

Optionally, one can also whitelist the own IP first:

cake Setup.Maintenance whitelist 123.123.123.123

This way you will still be able to navigate normally (remember: read only!), while all others will see the 503 maintenance error message.

Now, no-one can change the source anymore. Let’s start to backup the DB.

Using my Setup plugin shell “DbDump”, I could back-up the whole database in seconds.

cake Setup.DbDump create

The idea is not use DBO access and single queries but the MySQL internal command line tools. This works even with larger DB. It will store a complete dump file in the APP/files/backup/ folder of the application. We can copy that over to the same folder on the target server using the CLI command scp:

scp -r -C /path/to/backup/file user@otherserver:/path/to/backup/file

Final steps

After the migration seemed to be complete, I checked if the shells in CLI work using the CurrentConfig shell:

cake Setup.CurrentConfig

This would tell me if the database is found, valid and accessible. Also if cache is setup and running properly.

I could (thanks to my whitelisted IP) also already browse the new site even in maintenance mode. I also tested upload forms and other potentially broken functionality due to missing/wrong permissions.

Then it’s time that we imported the database content using the DbDump shell:

cake Setup.DbDump restore

It will find this copied file in the backup folder and upon confirmation will import the whole dump into the new DB.

Tip: If the shell on the new server doesn’t work, make sure you got the execution rights (-x) set for the cake file.

You can also make sure your code is up to date by using “git pull”, “svn update” etc. Don’t forget to clear the persistent and model cache after such changes:

cake Setup.Clear cache

You should then be able to browse the website again and see both content and images/filesystem data.

When everything looked fine, I just had to disable the maintenance mode again:

cake Setup.Maintenance deactivate

Some typical problems you might run into

Depending on whether you also have the tmp dirs in version control, you might have to create them (or raise debug level for a sec to automatically trigger that). It is also a good idea to flush the tmp again before going live again to avoid any accidentally copied files to cause trouble. So basically I raise debug to 2 then (still in maintenance mode) and quickly afterwards set it back to default 0.

Result

With almost zero preparation (OK, I had to get one of the shells working again) I had a total downtime of 4 hours in the middle of the night – which is acceptable. Especially with a proper 503 header for search engines and a small note for visitors.

Note: Ideally, you would also notify users in advance about the maintenance window and its scheduled length.

Vista

One could have reduced the downtime further by packing and the files instead of using rsync and multi-file-transfer. There are also a lot of server tools and other helpers available that will easy migration for larger applications. But for smaller apps that is usually out of scope or simply unnecessary. Depending on source and target OS that can also be simplified by moving the complete image or something. Hope this still helps some of you as a guideline or checklist.

 
No Comments

Posted in CakePHP