In the non-App Engine world, things are a lot trickier and frankly, less fun. In this post I will detail how I have pulled together something almost as fun and easy as App Engine deployments for Flask apps. The secret sauce is Fabric, a python library/command-line tool for deployments and systems administration. Essentially, Fabric acts as a wrapper around the shell and, more importantly, ssh. I started out using it just for the deployment portion of my pet projects but have since used it to completely replace the ANT scripts I had been using previously. It is simpler, more powerful and doesn’t require you to use anything besides python to manage your targets. While I am using this with Flask apps specifically, there is nothing in this example which couldn’t apply to any other framework you happen to prefer using.

Let’s jump right into things with the actual deployment. Listing 1 contains two functions, one to pack up the files of your app (pack) and one to upload the files to the server (upload). Have a look at it and then I will step through the various pieces.

Listing 1: Pack and upload methods complete.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

# python imports
from __future__ import with_statement
import os, re
 
# fabric imports
from fabric.api import local, run, cd, put, env
 
env.host_string = "user@someserver.org"
env.password = 's3kr3tz'
 
def pack(target, version, revision, release_dir, tmp_dir='/tmp'):
    """
    Method to package a release for deployment
    """
    archive_name = '.'.join([target, version, revision, 'tgz'])
    full_archive_path = os.path.join(tmp_dir, archive_name)
    local('tar czf %s -C %s .' % (full_archive_path, release_dir))
    return archive_name
 
def upload(target, version, revision, release_dir):
    """
    Method to upload the released code to a server
    """
 
    # package the release for deployment
    tmp_dir = '/tmp'
    archive_name = pack(target, version, revision, release_dir, tmp_dir)
 
    with cd('public_html'):
        dir = run('ls')
        dir_contents = [d.strip() for d in re.split(r"\s*",dir) if d.strip()]
        if not target in dir_contents:
            run('mkdir %s' % target)
 
        with cd(target):
            dir = run('ls')
            dir_contents = [d.strip() for d in re.split(r"\s*",dir) if d.strip()]
 
            if not version in dir_contents:
                run('mkdir %s' % version)
 
            with cd(version):
                run('rm -rf *')
                put(os.path.join(tmp_dir, archive_name), '.')
                run('tar xzf %s' % archive_name)
                run('rm -f %s' % archive_name)
                run('mv fcgi.htaccess .htaccess')
                run('chmod 755 index.fcgi')
 
            run('rm -f current')
            run('ln -s %s current' % version)

First, let’s look at the pack(target, version, revision, release_dir, tmp_dir='/tmp') method, since it is relatively straight-forward. The arguments themselves are really a matter of personal preference here. I have chosen to build the archive name of the code I am releasing from the target I am deploying to (typically this would be ‘test’, ‘demo’ or ‘prod’ as needed), the version I am deploying (for test deployments I usually deploy to a version of ‘daily’ which gets overwritten with each subsequent release and for demo and prod I use a loose interpretation of Semantic Versioning with a <major>.<minor>.<patch> scheme) and the revision of the latest commit in the release. The release_dir argument specifies where the code to be packaged up resides and tmp_dir is simply the location of a temporary directory. Lines 15 and 16 of Listing 1 show the first three arguments being assembled, with the temp directory, into an archive filename. There are more sophisticated ways to package things up for distribution (such as Distribute which is given some attention on the Flask site) but I just prefer the simplicity of gzipping everything up and then sshing the file onto the server.

Line 17 of listing one utilizes the Fabric method local() which makes it easy to run shell commands from within your python code. Here, I am simply calling tar to archive the specified release directory into an archive with the filename I just put together. Finally, the pack method returns the filename of the newly created archive.

The pack method is all well and good, but all it has really done is saved you from having to write a little ANT code. There is nothing in there that couldn’t have been done very easily from a traditional build script. The upload method is where the real power of Fabric starts to shine. The arguments to this method are identical to those of pack with the exception of the temp directory which is defined in this method. More properly, this should be defined in a properties file (the subject of another post for another day) but for this example it is fine here. Secondly, upload just calls pack to package up the deliverables and return the name of the archive. Listing 2 shows the first piece of upload that is actually interesting.

Listing 2: Preparing the target directory

1
2
3
4
5

    with cd('public_html'):
        dir = run('ls')
        dir_contents = [d.strip() for d in re.split(r"\s*",dir) if d.strip()]
        if not target in dir_contents:
            run('mkdir %s' % target)

Line 1 of Listing 2 introduces the Fabric method cd which does exactly what its counterpart in your shell does. This version, however, does the change directory over an ssh connection. Two important lines in Listing 1 are repeated here in Listing 3:

Listing 3: SSH authentication

1
2

env.host_string = "user@someserver.org"
env.password = 's3kr3tz'

These two lines specify the user and server to make an SSH connection to and the password to use for that user’s login. For a production environment using key-based authentication is recommended if not required in order to keep everything properly secured. These lines set values in Fabric’s environment that it then uses to connect to the remote server for the remainder of the operations we have ahead.

Jumping back to Listing 2, we can see that the script changes into the directory public_html. This will be the root of your deployments, whatever that path might be. This is another value that should more properly be placed in a properties file. Utilizing the python with statement, lines 2, 3, 4 and 5 will all execute within the context of the public_html directory. Line 2 performs an ls in the public_html directory and line 3 parses the result into a list of names. Line 4 then checks if we have a directory in public_html which matches the name of the target we are deploying and if not, line 5 creates it. As an example, if this was our first time deploying a demo build, this would create a demo directory within public_html.

Listing 4 shows the next block, which does a similar operation only this time within the target directory, checking for and creating a version subdirectory.

Listing 4: Creating a directory for the version being deployed

1
2
3
4
5
6

        with cd(target):
            dir = run('ls')
            dir_contents = [d.strip() for d in re.split(r"\s*",dir) if d.strip()]
 
            if not version in dir_contents:
                run('mkdir %s' % version)

So, to continue our example, if we are deploying version 1.0.1 to demo, then we would now have the path public_html/demo/1.0.1/ created. Finally, listing 5 contains the actual deployment of code to the server!

Listing 5: Deploying the new version of the code

1
2
3
4
5
6
7

            with cd(version):
                run('rm -rf *')
                put(os.path.join(tmp_dir, archive_name), '.')
                run('tar xzf %s' % archive_name)
                run('rm -f %s' % archive_name)
                run('mv fcgi.htaccess .htaccess')
                run('chmod 755 index.fcgi')

Line 1 of Listing 5 should be familiar now, it is a with statement to perform the following operations within our version directory. Line 2 removes any files and directories that might already exist there. This allows for deploying the same version over and over for daily builds from a Continuous Integration platform. Line 3 does the actual transfer of files utilizing the Fabric put() method to send our archive file to the server. Line 4 extracts the files from the archive and line 5 removes the archive afterwards. Lines 6 and 7 are needed for my webhost (HostGator) with 6 renaming the deployed .htaccess file needed to have FastCGI function properly and 7 sets the permissions as needed for the .fcgi entry script.

Last but not least we have the two simple lines shown in Listing 6.

Listing 6: Serving up the correct version

1
2

            run('rm -f current')
            run('ln -s %s current' % version)

Now that the script is ready, you can deploy your code to the server using the fab shell command. For a real project you’d likely have a deploy method which does all of your prep work and then calls the upload method above. For now, however, you could call upload directly, adding the target, version, revision and release_dir arguments. Arguments to fab can be done just like calling the python methods themselves, either with position arguments, named arguments or a combination. Listing 7 shows some example calls to deploy your code.

Listing 7: Deploying with fab

1
2
3
4
5
6
7
8

# Positional arguments
fab upload:demo,1.0.1,0264103894,/home/jread/project
 
# Named arguments
fab upload:target=demo,version=1.0.1,revision=0264103894,release_dir=/home/jread/project
 
# combination
fab upload:demo,1.0.1,revision=0264103894,release_dir=/home/jread/project

It is all well and good that we now have the new version of our code deployed to the server, but it still needs to be available via your webserver. You can’t very well have your users change all of their URLs to point to the new version subdirectory, so you need to add a little magic. In this case, I have pointed my webserver to use public_html/demo/current as the webroot for the demo version of my app. The two lines in Listing 6 remove an existing current and then create a new one which is a symlink to the newly deployed version directory. No server configurations need to be changed and certainly no DNS needs changing. If you discover a nasty bug in a new deployment, you can simply shell to your server and remake the symlink to the previous version’s directory and you have an instant rollback of the release.

The possibilities of this technique are actually quite limitless. Deployments could be kicked off from certain commits to your source repo. It would be fairly trivial to implement a small dashboard which gives you manual control over the symlink current allowing for a web interface to set the currently active version, just like in the Google App Engine admin.

It is worth noting that the entire script as presented in Listing 1 is only a small fraction of the full deployment script that I am using for my real projects. In fact, I have everything split into several different files for easier organization. I will be adding future posts on this subject covering things like automated versioning, resource versioning, compilation of Coffeescript or LESS files, preparing files for release and the far more complicated problem of data schema deployment.

Update 2012-01-29: Added example of actually calling Fabric to deploy code.

Tonight I was planning on upgrading to the latest Jira release when I realized that I had never hooked the existing install up to MySQL. In light of that I decided to just do a fresh installation. During the course of this installation I ran into an error I had yet to encounter before:

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

After a good deal of poking around and Googling, I found and fixed the problem. You see, the default setup for MySQL and Jira is (if you follow Atlassian’s excellent how-tos) to connect to the MySQL server running on localhost. This works great most of the time. However, if you are like me and want your MySQL to be remote manageable from another machine on your network, you may have configured MySQL to listen on an IP (e.g. 192.168.0.40) instead of just localhost. The solution is actually quite simple.

Where the Atlassian docs say to put

url=”jdbc:mysql://localhost/jiradb?useUnicode=true&characterEncoding=UTF8″

into your server.xml file, you’ll actually want to specify the IP address you have MySQL listening on. So, if MySQL is listening on 192.168.0.40, you’d put

url=”jdbc:mysql://192.168.0.40/jiradb?useUnicode=true&characterEncoding=UTF8″

into the server.xml file.

Hopefully this will help anyone out there who comes across this error. As an aside, if you are interested in trying out Jira (and the rest of Atlassian’s excellent tools), I highly recommend taking on their Dragon Quest. You’ll get a tonne of excellent experience setting up their suite of tools, a working setup of said tools, and if you send them a screenshot when done, you’ll also get a free T-shirt!

The only real difficulty you are likely to encounter in creating a JSLint target for your build scripts is that JSLint is, itself, written in JavaScript. Other helpful libraries such as JSMin or the YUICompressor from Yahoo are distributed as jar files and can be executed easily with an exec block with the executable set to java. With JSLint, however, you are going to need a JavaScript engine which you can execute from the command line in order to run the JSLint script to validate your JavaScript code. THis actually sounds harder than it is.

For our implementation, I chose to use the Rhino engine since there is documented support for JSLint on it, and even a handy helper file to include in your system: rhino.js. Grab a copy of Rhino (the actual file you’ll want to include is js.jar), the Rhino helper script I just mentioned and JSLint itself. Place them all together into some sort of tools directory in your project. Ours is located in /tools/ant.

Now comes the fun part, writing the actual target in ANT. For our purposes, we want to process all .js files contained in a particular location, as defined in a build property called jsdir. We also want to ignore any files that are contained in a vendor subdirectory of jsdir, as we should not need to worry about third-party scripts that don’t pass JSLint. Let’s get things started with our basic, empty target:

1
2
3
4
5
6

<!--
jslint - Runs lint checks on js files
-->
<target name="jslint">
 
</target>

Listing 1: The basic, empty jslint target

Within the target there are two things we are going to need. First a way to reference all of the files we want to operate over and second, the actual call to run JSLint on those files. For the first requirement we’ll use a pathconvert tag with a nested fileset. The pathconvert will allow us to generate a space-separated list of files and stuff that into a property which we’ll call jsfiles. The fileset block will do the actual work of finding the files in question. It will operate over ${jsdir} including all .js files and excluding all .js files in the vendor subdirectory. Listing 2 shows the pathconvert block added to our empty target.

1
2
3
4
5
6
7
8
9
10
11

<!--
jslint - Runs lint checks on js files
-->
<target name="jslint">
    <pathconvert pathsep=" " property="jsfiles">
        <fileset dir="${jsdir}">
            <include name="**/*.js"/>
            <exclude name="vendor/**/*.js"/>
        </fileset>
    </pathconvert>
</target>

Listing 2: The pathconvert block added to our target.

Finally, it is time to add the actual call to JSLint. To allow our build scripts to work for multiple projects, we have pushed as much information into properties that get defined in project-specific files as possible. The location of the Rhino jar (js.jar) is placed into the property js.jar and the location of the JSLint script (jslint.js) is placed into the property jslint.js. With those properties suitably defined elsewhere, we have the final piece of our puzzle: the exec block. We have the executable set to java and failonerror set to true, so that our target will appropriately fail when JSLint fails to give our code a passing grade. The final target, including the exec block can be seen in listing 3.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<!--
jslint - Runs lint checks on js files
-->
<target name="jslint">
    <pathconvert pathsep=" " property="jsfiles">
        <fileset dir="${jsdir}">
            <include name="**/*.js"/>
            <exclude name="vendor/**/*.js"/>
        </fileset>
    </pathconvert>
    <exec dir="${jsdir}" executable="java" failonerror="true">
        <arg line="-jar ${js.jar} ${jslint.js} ${jsfiles}"/>
    </exec>
</target>

Listing 3: The final target.

Put all together, the pathconvert block stuffs the list of files to work on into the property ${jsfiles} which is then passed as an argument to the jslint.js file. The jslint.js file is itself the first argument passed to the Rhino JavaScript engine (js.jar) which is run directly by java. Running your ant jslint target should give you some nice feedback on the syntax of your JavaScript code.

I first started using Django when I started at VendAsta and this issue didn’t creep up there because we are use Google App Engine and its DataStore on the backend instead of a traditional database. With no database there was no issue with creating/dropping anything. It was only when I started using Django on my own that I came across that problem. There was one more issue with Django that I came across only when I started doing things outside of work: Django expects all of your models to be in a single file called models.py. I hate that. A lot.

I did some googling about the problem and found that there were a bunch of other people who didn’t like it much either. There were a few solutions but they all involved manually mucking about with the __all__ attribute in the __init__.py file of your models directory. Sure, it works, but I don’t like having to define a model in two places to use it. Oddly enough it was in some test code at work that I found the answer.

For our project at VendAsta we are using the Django App Engine Helper to provide a nice easy interface to the DataStore on the backend. A part of this is a small __init__.py file which allows you to spread your tests out in multiple files. Django likes its tests to be in a single tests.py file just like it likes its models to be in models.py. The wee bit of Google code is shown in Listing 1.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

#!/usr/bin/python2.4
#
# Copyright 2008 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
 
 
 
 
"""Loads all the _test.py files into the top level of the package.
 
This file is a hack around the fact that Django expects the tests "module" to
be a single tests.py file and cannot handle a tests package inside an
application.
 
All _test.py files inside this package are imported and any classes derived
from unittest.TestCase are then referenced from this file itself so that they
appear at the top level of the tests "module" that Django will import.
"""
 
import os
import re
import types
import unittest
 
PACKAGE = 'appname.tests'
TEST_RE = r"^.*_test.py$"
 
# Search through every file inside this package.
test_names = []
test_dir = os.path.dirname( __file__)
for filename in os.listdir(test_dir):
  if not re.match(TEST_RE, filename):
    continue
  # Import the test file and find all TestClass clases inside it.
  test_module = __import__('%s.%s' % (PACKAGE, filename[:-3]),
                           {}, {},
                           filename[:-3])
  for name in dir(test_module):
    item = getattr(test_module, name)
    if not (isinstance(item, (type, types.ClassType)) and
            issubclass(item, unittest.TestCase)):
      continue
    # Found a test, bring into the module namespace.
    exec "%s = item" % name
    test_names.append(name)
 
# Hide everything other than the test cases from other modules.
__all__ = test_names

Listing 1: __init__.py file from Google for a tests module

Essentially what this file does is loop through all files in the same directory and for each on that matches the pattern listed on line 40 (in this case, files that end in ‘_test.py’) and import all of the Classes from those files that are subclasses of the TestCase class. Finally, the code on line 62 does the magic with __all__ that I wanted to avoid doing manually for my models. It was this line that caught my attention and started me wondering if this code could be used for spreading models across multiple files.

In order for this to work for models there are a few things that need to be changed. First of all, on line 40, the pattern should be changed to match on *.py to grab all of the python files in the directory. Then, on line 46, add

or filename == “__init__.py”

to ensure that we don’t try to import the file we are in. Finally, remove the subclass requirement from line 55, as we are just concerned with importing classes here not what they inherit from. Putting it all together results in Listing 2. Place that in the __init__.py of a directory called models and you’ll not have to manually manage the __all__ list for your models in separate files.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

#!/usr/bin/python2.4
 
import os
import re
import types
import unittest
 
PACKAGE = 'my_app.models'
MODEL_RE = r"^.*.py$"
 
# Search through every file inside this package.
model_names = []
model_dir = os.path.dirname( __file__)
for filename in os.listdir(model_dir):
  if not re.match(MODEL_RE, filename) or filename == "__init__.py":
    continue
  # Import the model file and find all clases inside it.
  model_module = __import__('%s.%s' % (PACKAGE, filename[:-3]),
                           {}, {},
                           filename[:-3])
  for name in dir(model_module):
    item = getattr(model_module, name)
    if not isinstance(item, (type, types.ClassType)):
      continue
    # Found a model, bring into the module namespace.
    exec "%s = item" % name
    model_names.append(name)
 
# Hide everything other than the classes from other modules.
__all__ = model_names

Listing 2: __init__.py file modified for a models module

Now, there is one last thing that you are going to need to do in order for everything to run properly in Django. It is a very simple addition to the end of each of your classes. I have seem some people complaining about the need to do this in various forums, but honestly, if you are using any form of modern text editing application, it should only take you about 2 minutes to create a new template for a model that has the little piece of code shown in Listing 3 at the bottom of the class (replace your_app_name with your actual app name).

1
2
3

class Meta:
        """Meta Class for your model."""
        app_label = 'your_app_name'

Listing 3: Meta class required at the end of each model for proper functioning.

The first hurdle is actually getting a Django site running on Dreamhost. Luckily, there is this blog post which details exactly how to get your site up and running on Dreamhost. I have used this method a couple of times (once modified to get a Satchmo install running) and have met with success each time.

Once the site is running, setting up tests poses another problem. The default test behaviour for a Django site creates and then destroys a whole test database on each test run. Normally this is not a problem, but the way Dreamhost has their MySQL setup running, the commands to create and drop databases are not available outside of their administration panel. What follows is the solution I have come up with and am currently using to run Django unittests with my Dreamhost MySQL databases.

First of all, it is important to understand a little about the default testing setup that Django uses. When you run the command python manage.py test, Django looks at the TEST_RUNNER setting to determine what to do. By default this setting points to django.test.simple.run_tests. A quick look and we can see that this is the functionality we want to alter. Specifically, lines 2 and 4 shown in listing #1 are what we need to replace.

1
2
3
4

from django.db import connection
connection.creation.create_test_db(verbosity, autoclobber=not interactive)
result = unittest.TextTestRunner(verbosity=verbosity).run(suite)
connection.creation.destroy_test_db(old_name, verbosity)

Listing 1: Excerpt from django.test.simple.run_tests

It is these lines which make the calls to create and then drop the test database. The functionality needed, however, in order to work with Dreamhost’s setup, is to “flush” and existing test database both before and after the test run is complete. I have accomplished this through the creation of a method called flush_test_db, shown in Listing #2.

1
2
3
4
5
6
7
8
9
10
11

def flush_test_db(test_db):
    """Method to flush out the test database, removing tables, before
    and after test runs"""
 
    from django.db import connection
    cursor = connection.cursor()
 
    current_tables =  connection.introspection.table_names()
    for table in current_tables:
        cursor.execute("use %s" % test_db)
        cursor.execute("drop table %s" % table)

Listing 2: New method to remove all tables from the test db

With this method, we can drop all of the tables in the test db both before and after a test run. It is necessary to do both just in case something goes horribly awry during a test run and we end up with leftover data from a previous test run. With this new method, we can then create a new version of the run_tests method and place both methods into a new file. I called my file test_runner.py and placed it in a tests directory within my app. The complete test_runner.py is shown in Listing 3.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70

"""Override of the simple tests module from django in order to skip
database creation and use an existing db"""
import unittest
from django.conf import settings
from django.db.models import get_app, get_apps
from django.db.backends.creation import TEST_DATABASE_PREFIX
from django.test import simple
from django.test.utils import setup_test_environment, teardown_test_environment
from django.core import management
 
def flush_test_db(test_db):
    """Method to flush out the test database, removing tables, before
    and after test runs"""
 
    from django.db import connection
    cursor = connection.cursor()
 
    current_tables =  connection.introspection.table_names()
    for table in current_tables:
        cursor.execute("use %s" % test_db)
        cursor.execute("drop table %s" % table)
 
def run_tests(test_labels, verbosity=1, interactive=True, extra_tests=[]):
    """
    Replacement for the run_tests in Django.
    """
    setup_test_environment()
 
    settings.DEBUG = False
    suite = unittest.TestSuite()
 
    if test_labels:
        for label in test_labels:
            if '.' in label:
                suite.addTest(simple.build_test(label))
            else:
                app = get_app(label)
                suite.addTest(simple.build_suite(app))
    else:
        for app in get_apps():
            suite.addTest(simple.build_suite(app))
 
    for test in extra_tests:
        suite.addTest(test)
 
    old_name = settings.DATABASE_NAME
 
    if settings.TEST_DATABASE_NAME:
        test_database_name = settings.TEST_DATABASE_NAME
    else:
        test_database_name = TEST_DATABASE_PREFIX + settings.DATABASE_NAME
 
    #replace the normal DATABASE_NAME with the name of the test db
    settings.DATABASE_NAME = test_database_name
 
    #flush the test db
    flush_test_db(test_database_name)
    #load the test db with models
    management.call_command('syncdb')
 
    result = unittest.TextTestRunner(verbosity=verbosity).run(suite)
 
    #flush the test db again
    flush_test_db(test_database_name)
 
    #reset the value of DATABASE_NAME in settings
    settings.DATABASE_NAME = old_name
    teardown_test_environment()
 
    return len(result.failures) + len(result.errors)

Listing 3: The complete test_runner.py

The final step is to point the test framework to your new test runner by placing a new line in your settings.py. The exact value will vary based on where in your project you have placed the new test runner. In this example I have placed the test_runner.py file, containing the run_tests method in the tests directory of an app called ‘your_app’. This is shown in listing 4.

1

TEST_RUNNER = 'your_app.tests.test_runner.run_tests'

Listing 4: Settings.py setting pointing the test framework towards your new test_runner

Finally a few words of caveat. I have omitted any comments I am using to tell pylint to ignore certain things and I have done little in this sample code to properly handle Exceptions that might arise. Nevertheless, this should get you a good start on running unit tests in Django if you host your site with Dreamhost.

For the past three and a half years, I have been working in a very heavy Microsoft shop. As a result, it was always difficult to get acceptance for open source projects at Cover-All. The more work I started doing on web projects the easier it became to bring some open source into the fold but there certainly wasn’t any interest in giving anything back to those projects we benefited from and, even if there was interest, there wouldn’t have been time to do so anyway.

At VendAsta, things are a lot different. Here is a company that not only embraces the use of open source projects but actually encourages everyone to give back to those projects if they make any improvements / changes to them. I absolutely love that! This has given me the opportunity to give back to my first open source project!

I have been doing some work on developing Facebook applications over the past few weeks. Specifically, I have been using the awesome pyFacebook library. This is a Python library which gives the same functionality as the Facebook-supplied PHP client library, except with the goodness of Python. A total bonus for us is the fact that the library includes functionality to work with Google App Engine, another new technology we are using. There was a small gap in this functionality and that is where I found my opportunity. While everything was setup to work with retrieving information from Facebook in GAE (Google has disabled socket access in the version of Python available through GAE, so you need to use their urlfetch method instead of calls to httplib), the Photos.upload() method was not setup in the same manner. I needed to upload files to Facebook, so I modified the library to work the way I needed.

Once my mods were complete I sent them to the maintainer of the pyFacebook project and, after a few helpful suggestions from him, I had a version that he was willing to commit to the project’s svn repository: http://code.google.com/p/pyfacebook/source/detail?r=166. After that I was able to track down and fix an actual bug involving some infinite redirects: http://code.google.com/p/pyfacebook/source/detail?r=168.

Two very small contributions to a single project, but I am hopeful that this is only the beginning!

1. How old were you when you started programming?

   Honestly, I think I got a later start at programming than a lot of the programmers I know. I guess technically, the first time I made a computer bend to my will would have been in grade 10 with some HyperCard scripting. Honestly, I don’t really consider that programming though, so I’ll have to say I first started programming in grade 11 when I was 16.

2. How did you get started in programming?

   I got started by taking a class in high school. Not very exciting at all. I was already heavily into computers at that time, having been a huge computer gaming fan since my first computer – an 8086 with two double-density floppy drives and no hard drive at all. It was a great computer for playing Sierra adventure games and got me my first taste of the Wing Commander (even if I had to send away the high-density floppies for twice as many double-density ones and had to get used to swapping disks during dogfights). It was the games that got me interested in the computers and it was that interest that got me to take the classes in high school.

   I’m sort of ashamed to admit that, aside from some web stuff (the only portion thereof that could remotely qualify as “programming” was a JavaScript routine to display a random “fun thing to do with monkies” on a Geocities page), within six months of finishing the last programming class in high school I didn’t do any more programming until my thrid year of University. It was then that I took a 100-level CompSci class as an elective. I was introduced to Java. I went to maybe half of the classes. It was all very easy for me and I ended up with a 94% in the class for almost zero effort. It happened at just the right time too: my Bio classes were all boring me because I had taken more advanced versions of a lot of them in BioChem. So, I made the leap, changed my major to CompSci and haven’t looked back.

3. What was your first language?

   Ignoring the aforementioned HyperCard experience, my first real language was Pascal. Think Pascal on the Macs at school and Turbo Pascal on my computer at home.

4. What was the first real program that you wrote?

   Well, I won’t include the first “real” program I wrote specifically for a class (a D&D character sheet generator). Instead, I’ll say that the first real program I wrote was a bowling league / tournament management application, in Turbo Pascal. My parents were playing in a league at the time and one of their friends wanted a program that could track teams’ progress.

   I was even ambitious enough to make it such that you could run one season / tournament for free but if you wanted to start over it would show you a message saying that you needed to buy the program.

5. What languages have you used since you started programming?

   Let’s see. I started with Pascal in high school and then learned Java at University. I also had to deal with C,C++,Eiffel, TCL/Tk, Prolog and Lisp before finishing my degree. I’ve worked with Perl, Python, PHP, VB6, VB.NET, and C# professionally since then. Plus a whole schwack of web technologies such as JavaScript, Flex/Actionscript and the many markups that make up the web. Oh, and a bunch of scripting languages on *nix.

6. What was your first programming gig?

   My first real job that involved a significant amount of programming wasn’t a programming position at all. I was working for a web hosting company, doing all of the duties of a Systems/Network Administrator with little more than an intern’s pay. When my boss realized that I was actually more of a programmer than a systems guy, he got me working on rebuilding their online ordering system which had been shutdown for the better part of a year when they changed their domain name. The remainder of my time there was spent wrestling to learn Perl and re-implement the ordering system and online payment processing systems – interfacing with their domain provider’s horrible Perl API and Moneris’ horrible payment processing Perl API.

7. If you knew then what you know now, would you have started programming?

   Definitely. Hell, if I knew then what I know now I could have avoided 3 years of Biology classes and I’d have two more years’ experience under my belt.

8. If there is one thing you learned along the way that you would tell new developers, what would it be?

   Trust yourself, and your code. One of the more annoying things I have seen lately with newer developers is the tendency to ask more experienced coders to look at their code / query and tell them if it will work the way they think it will. I honestly don’t remember ever being like this, as I’ve always been the type to just try things out – but it is possible I was one of those guys. Seriously, if you don’t have the confidence in your abilities to try running code without someone looking it over first, this might not be the career for you.

Taking Philip’s word for it that this is a meme (but Milhouse still is not), I’ll tag a few others I’d like to see answer these questions as well: Ralph Marchildon,James Sapara, Dustin Wasyliw. I’d add Andrew Munro to the list but he thinks he’s too good for blogging.

I can’t help but wonder about this. With drive sizes ever-increasing and drive prices ever-plummeting, why would a so-called modern OS have such a low limit? Is it something that slipped through the cracks with the decision to not include WinFS in the initial release of Vista? I am also not sure about the number 16,384. I have been copying a lot of files today and I swear that one of those operations involved either 19K or 20K files…it is only when I try for 23K+ that it seems to misbehave. Very odd.

No matter how securely you code your site, someone with a valid password can access that which you mean to keep private. The best solution is, of course, to keep any and all pages that require the input of a password protected using an SSL certificate. In reality, not all sites can afford the luxury of their own SSL certificate. Another, entirely free, method of keeping passwords from being transmitted across the web in clear text is to use a challenge-response system for logins.

The basic idea behind a challenge-response system is that the page requesting a password will have some extra information embedded in it – information that is unique to that particular login attempt – which is used to encode the password before it is sent across the network. In this tutorial, we’ll see how to generate this extra information, called the salt and how to incorporate it into a secure hash using SHA1

The work upfront

First of all, we’ll take a look at things on the client end of things. We will need a javascript implementation of the SHA1 algorithm. I found one as done by a guy named Chris Vaness. I am not sure where I found it, but I did maintain his copyright as requested. Grab it for your own use here: sha1.js. For CakePHP, place this file in the app/webroot/js directory. All you really need to know about this file is that it defines a function sha1Hash that takes one argument – the string to hash – and returns the hashed value.

Now we need a form for logging in to a site. The html code for a simple login is shown in snippet #1.

Snippet #1: HTML code for our simplified login.

<p class="error_message">The login credentials you supplied could not be recognized. Please try again.</p>
 
<form method="post" action="/admin/login">
    <label class="label" for="username">Username:</label>
    <?php input('User/username', array('size' => 20, 'class' => 'TextField', 'id'=>'username')); ?>
    <?php tagErrorMsg('User/username', 'Username is required.')?>
    <label class="label" for="password">Password:</label>
    <?php password('User/password', array('size' => 20, 'class' => 'TextField', 'id'=>'password')); ?>
    <?php tagErrorMsg('User/password', 'Password is required.')?>
    <?php submit('Login', array('class'=>'Button'));?>
</form>

Of course, the code in Snippet #1 is just the login bits of the page, you’ll want to embed that in a full page (and hopefully pretty it up with some CSS) for production use. In our controller, we’ll need to generate the salt to embed in the page. Code for doing this is shown in Snippet #2.

Snippet #2: Adding the salt in the controller.

function login()
{
    if(!empty($this->data))
    {
         //handle the login - we'll get to this later
    }
    else
    {
        //create the salt by creating an MD5 Hash of the current time
        $salt = md5(time());
        //make the salt available to the view
        $this->set('special_sauce',$salt);
        //store the salt in the session
        $this->Session->write('salt', $salt);
    }
}

As you can see, generating the salt and making it available in a variable which we can access in the view is really a trivial matter. A couple of points bear explanation. First of all, why am I using MD5 here when using SHA1 for the hashing itself? Well, this particular use of a hash is simply to generate a random assortment of characters to use as salt, therefore we can use the less secure MD5 algorithm which will run a little faster than SHA1. It really makes no difference, and is more a matter of preference than anything else. For the actual hashes, however, you should use SHA1 as it is more secure. Secondly, you may wonder why I am saving the salt in the session when I have made it available to the view (yes, I am going to be placing it in a hidden form element). The reason for this is to ensure that we use the salt generated here when we do our comparisons later on. If we were to place it in the form element and use that return to check the stored password, an attacker could simply supply the salt he wanted to use in that form element and he could bypass our security. This way, we are protected against clever people like that.

With our form defined and our salt available we need to write some javascript to process the user input before sending it to the server. Specifically, we are going to take the entered password, run the SHA1 algorithm on it, concatenate the resulting hash with the salt and run the algorithm on it again. Following that process we write our new password value into the password field and let the form submit. Javascript to accomplish this is shown in Snippet #3.

Snippet #3: Javascript transform_login function

function transform_login()
{
  var password = document.getElementById('password').value;
  var salt = document.getElementById('special_sauce').value;
 
  var hashed_pass = sha1Hash(password);
  var hashed_and_salted_pass = hashed_pass + salt;
  var salted_hash_pass_hash = sha1Hash(hashed_and_salted_pass);
 
  document.getElementById('password').value = salted_hash_pass_hash;
 
}

This new javascript function requires that we make some changes to our login code, as shown in Snippet #4.

Snippet #4: Modfied login form.

<script src="/js/sha1.js" language="javascript"></script>
<script src="/js/login.js" language="javascript"></script>
<?php if ($error): ?>
    <p class="error_message">The login credentials you supplied could not be recognized. Please try again.</p>
<?php endif; ?>
<form action="/admin/login" method="post")>
        <label for="username" class="label">Username:</label>
        <?php echo $html->input('User/username', array('size' => 20, 'class' => 'TextField', 'id'=>'username')); ?>
        <?php echo $html->tagErrorMsg('User/username', 'Username is required.')?>
        <label for="password" class="label">Password:</label>
        <?php echo $html->password('User/password', array('size' => 20, 'class' => 'TextField', 'id'=>"password")); ?>
        <?php echo $html->tagErrorMsg('User/password', 'Password is required.')?>
        <?php echo $html->submit("Login", array('class'=>'Button', 'onclick'=>"Javascript:return transform_login();")); ?>
        <input type="hidden" name="special_sauce" id="special_sauce" value='<?php echo $special_sauce; ?>'>
</form>
<?php if ($error): ?>
    <script language="javascript">
        document.getElementById('password').value = "";
        document.getElementById('username').value = "";
    </script>
<?php endif; ?>

As you can see, we have added code to include the sha1.js and login.js files which contain our sha1 functions and login function transform_login. We have also inserted our hidden value with the salt in it, called special_sauce here. Finally, we have added an onclick to the submit button to call the javascript function and transform our password. This should be everything we need on the client end of things.

The login process in the controller

Earlier, in Snippet #2, we passed over the code in the controller that actually handles the processing of the login. Now it is time to revisit the controller and finish off our secure login. The goal here is to replicate the actions taken in the javascript function transform_login and compare the results of the server-side transform and that done on the client end.

Since we embedded the salt in the login form using a hidden form value, that value will be available to us here in the controller. We never want to use that value, however, because we can never be sure of the source of any information coming to us from a web form. It is very easy to construct a duplicate form which submits faulty information in an attempt to gain access to a secured system. What we will do is take the password from the user’s database record and the salt stored in the session and put that information through the same transformations as we did on the client side. If the values match then we know the user entered the right password and can be granted access. Snippet #5 contains the full code of the final controller method.

Snippet #5: Completed controller login code.

function login()
{
     // If a has submitted form data:
     if (!empty($this->data))
     {
         $someone = $this->User->findByUsername($this->params['data']['User']['username']);
 
         //the password from the database
         $password = $someone['User']['password'];
         //the salt value from the session
         $salt = $this->Session->read('salt');
         //hash the password
         $hashed_password = sha1($password);
         //add the salt to the hashed password
         $salted_and_hashed_password = $hashed_password . $salt;
         //salt the salted and hashed password
         $salted_hash_pass_hash = sha1($salted_and_hashed_password);
 
         //do our comparison
         if($salted_hash_pass_hash == $this->params['data']['User']['password'])
         {
              // This means they were the same. We can now build some basic
              // session information to remember this user as 'logged-in'.
              $this->Session->write('User', $someone['User']);
 
              // Now that we have them stored in a session, forward them on
              // to a landing page for the application.
              $this->redirect('/logged_in/');
         }
         // Else, they supplied incorrect data:
         else
         {
             //Remember the $error var in the view? Let's set that to true:
             $this->set('error', true);
         }
     }
     else
     {
         $salt = md5(time());
         $this->set('special_sauce',$salt);
         $this->Session->write('salt', $salt);
     }
}

Summary and Caveats

That’s really all there is to securing your logins with a challenge-response system. Of course, if you have an SSL certificate available to you, using it would be in your best interests. There is, however, nothing stopping you from using both at the same time. You can never have too much security.

A few words of warning, however. This is not intended to be a complete solution for your login needs. There are several aspects of a login system that have been overlooked in the name of brevity. Not the least of these are validating required fields and protecting all input fields from SQL Injection vulnerablilities. This only provides you with the means of obscuring password data in transit over public networks. I recommend that you do your research up front on a complete solution to prevent getting burned by something simple you may have overlooked.

An Example Scenario

To best illustrate this method we will need an example scenario to work with. As I mentioned in my blog entry on Single Table Inheritance, it is particularly helpful when creating a site like Digg. For our example, let’s consider building a site called Excavate. Our requirements are rather simple: a site for people to input links with descriptions. Once a link has been entered it will be considered an "upcoming" entry. Once enough members (only people who have signed up can view and excavate upcoming items) excavate an item it gets green-lighted and becomes a "current" entry. We will store both types of entry in one database table.

The Database Bits

The basic information we need for both kinds of entry is the same: a title, a description, a url, an author name, an author email address, and a created date. A current posting will also require the date which it was green-lighted to order the postings chronologically. To these required fields we will be adding a type field which will hold a reference to which object the record references, upcoming or current. Snippet 1 shows the SQL code for creating our table in MySQL.

Snippet #1: SQL Code for our entries table.

CREATE TABLE `entries` (
    `id` INT(11) NOT NULL AUTO_INCREMENT,
    `title` VARCHAR(255) NOT NULL,
    `description` text NOT NULL,
    `url` var CHAR(255) NOT NULL,
    `author_name` VARCHAR(255) NOT NULL,
    `author_email` VARCHAR(255) NOT NULL,
    `created` datetime DEFAULT NULL,
    `greenlit` datetime DEFAULT NULL,
    `type` VARCHAR(255) NOT NULL DEFAULT 'BlogEntry',
    PRIMARY KEY  (`id`), )
    ENGINE=MyISAM DEFAULT CHARSET=latin1;

There are a few things to check with your database code to ensure that everything will function smoothly. First of all, ensure that your `created` field is called created. CakePHP will automagically handle setting this value for you when you are saving your model. Secondly, ensure that you do not set any model-specific fields to NOT NULL. If, for example, we had set the `greenlit` field to be NOT NULL we would run into troubles when saving an upcoming object, as that object has not been greenlit yet.

The Models

Now that we have the database table in place to hold our data it is time to write the Model classes for our two objects. We will need to add several variable to the models and override several methods from the parent class AppModel. Let’s start by looking at the Current model.

Snippet #2: Class definition and variables
for the Current Model.

class Current extends AppModel
{
    var $name = 'Current';
 
    //single table inheritance vars
    var $__table = 'entries';
    var $__fields = array('Current.id', 'Current.title','Current.description',
'Current.author_name','Current.author_email','Current.created',
'Current.greenlit');
    var $__type_field = 'type';

The first variable we need to define is $name. This is required in CakePHP if you are using PHP4 but not in PHP5. Nevertheless, we will be using it as it will make many things we are going to do much easier. Ensure that the value of this variable matches exactly the name of your class. The next variable $__table contains the name of the database table in which we are storing our objects. $__fields is an array containing a list of the fields that belong to this model. Note that each field is prefixed with the name of the class it belongs to – even though the name of the table is not Current this will work because the queries created by CakePHP will have FROM `entries` AS `Current` to make everything play nicely. Also of note is the fact that we are not specifying that the type field is a part of the model – this is because it is not. The type field determines which model a record is but we will be handling that automatically and so do not need to manipulate the field directly. Finally, we place the name of the field in which we are storing our type into the variable $__type_field.

With the above variables in place, we can turn our attention to the methods we need to override. Any methods which involve reading or writing records need to be modified to take our changes into account. For my particular needs, I have found that I needed to override __construct, findAll, findCount, findAllThreaded and save. __construct is modified to supply the tablename to the __construct method of AppModel and, as such, is the simplest modification we will be making. The modifications to the save method are a little more significant by equally easy to accomplish. All that needs doing is to add a reference to the type field and its value to the $data variable before passing it to the parent.

The data retrieval methods findAll, findCount and findAllThreaded require a little more work than the fields detailed above. There are two things that need to be done here: modify the conditions to include a reference to the value of the type field and check for the fields being queried. The latter requires a little more explanation. Since we are storing the information for both objects in one table there are certain fields which belong to one model but not to the other. We need to make sure that we are only grabbing the fields that belong to the model in question. In our example, we need to check that we are not querying the greenlit field if we are querying for an Upcoming model. If the fields are not being queried specifically and $fields is null, then we will substitute our $__fields array for $fields to only return that which belongs to this model. Snippet #3 contains the completed code for our Current model.

Snippet #3: Complete definition for the Current Model.

class Current extends AppModel
{
     var $name = 'Current';
 
     //single table inheritance vars
    var $__table = 'entries';
    var $__fields = array('Current.id', 'Current.title', 'Current.description', 'Current.author_name', 'Current.author_email', 'Current.created', 'Current.greenlit');
    var $__type_field = 'type';
 
    function __construct($id = false,$ignore_table = null, $ds = null)
    {
        $table = $this->__table;
        parent::__construct($id,$table,$ds);
    }
 
    function findAll($conditions = null, $fields = null, $order = null, $limit = null, $page = 1, $recursive = null)
    {
        if($conditions == null)
        {
            $conditions = $this->name. "." . $this->__type_field . " = '" . $this->name . "'";
        }
        else
        {
            $conditions = $conditions . " AND " . $this->name . "." . $this->__type_field . " = '" . $this->name . "'";
        }
 
        if($fields == null)
        {
            $fields = $this->__fields;
        }
        else
        {
            foreach($fields AS $field)
            {
                if(!array_search($field,$this->__fields))
                {
                    throw new Exception("Field " . $field . " does not exist in Model " . $this->name);
                }
            }
         }
 
         return parent::findAll($conditions, $fields, $order, $limit, $page, $recursive);
    }
 
    function findCount($conditions = null, $recursive = 0)
    {
        if($conditions == null)
        {
            $conditions = $this->name . "." . $this->__type_field . " = '" . $this->name . "'";
        }
        else
        {
            $conditions = $conditions . " AND " . $this->name . "." . $this->__type_field . " = '" . $this->name . "'";
        }
        return parent::findCount($conditions, $recursive);
     }
 
    function findAllThreaded($conditions = null, $fields = null, $sort = null)
    {
        if($conditions == null)
        {
            $conditions = $this->__table . "." . $this->__type_field . " = '" . $this->name . "'";
        }
        else
        {
            $conditions = $conditions . " AND " . $this->name . "." . $this->__type_field . " = '" . $this->name . "'";
        }
        if($fields == null)
        {
            $fields = $this->__fields;
        }
        else
        {
            foreach($fields AS $field)
            {
                if(!array_search($field,$this->__fields))
                {
                    throw new Exception("Field " . $field . " does not exist in Model " . $this->name);
                }
            }
        }
        return parent::findAllThreaded($conditions, $fields, $sort);
    }
 
    function save($data = null, $validate = true, $fieldList = array())
    {
        $data[$this->name][$this->__type_field] = $this->name;
        return parent::save($data, $validate, $fieldList);
    }
}

Snippet #4 contains the complete code for our Upcoming Model. Notice that it does not reference the greenlit field whereas the Current Model did.

Snippet #4: Complete definition for the Upcoming Model.

class Upcoming extends AppModel
{
    var $name = 'Upcoming';
 
    //single table inheritance vars
    var $__table = 'entries';
    var $__fields = array('Upcoming.id', 'Upcoming.title','Upcoming.description', 'Upcoming.author_name','Upcoming.author_email', 'Upcoming.created');
    var $__type_field = 'type';
 
    function __construct($id = false,$ignore_table = null,$ds = null)
    {
        $table = $this->__table;
        parent::__construct($id,$table,$ds);
    }
 
    function findAll($conditions = null, $fields = null, $order = null, $limit = null, $page = 1, $recursive = null)
    {
        if($conditions == null)
        {
            $conditions = $this->name. "." . $this->__type_field . " = '" . $this->name . "'";
        }
        else
        {
            $conditions = $conditions . " AND " . $this->name . "." . $this->__type_field . " = '" . $this->name . "'";
        }
 
        if($fields == null)
        {
            $fields = $this->__fields;
        }
        else
        {
            foreach($fields AS $field)
            {
                if(!array_search($field,$this->__fields))
                {
                    throw new Exception("Field " . $field . " does not exist in Model " . $this->name);
                }
            }
        }
 
        return parent::findAll($conditions, $fields, $order, $limit, $page, $recursive);
    }
 
    function findCount($conditions = null, $recursive = 0)
    {
        if($conditions == null)
        {
            $conditions = $this->name . "." . $this->__type_field . " = '" . $this->name . "'";
        }
        else
        {
            $conditions = $conditions . " AND " . $this->name . "." . $this->__type_field . " = '" . $this->name . "'";
        }
        return parent::findCount($conditions, $recursive);
    }
 
    function findAllThreaded($conditions = null, $fields = null, $sort = null)
    {
        if($conditions == null)
        {
            $conditions = $this->__table . "." . $this->__type_field . " = '" . $this->name . "'";
        }
        else
        {
            $conditions = $conditions . " AND " . $this->name . "." . $this->__type_field . " = '" . $this->name . "'";
        }
        if($fields == null)
        {
            $fields = $this->__fields;
        }
        else
        {
            foreach($fields AS $field)
            {
                if(!array_search($field,$this->__fields))
                {
                    throw new Exception("Field " . $field . " does not exist in Model " . $this->name);
                }
            }
        }
        return parent::findAllThreaded($conditions, $fields, $sort);
    }
 
    function save($data = null, $validate = true, $fieldList = array())
    {
        $data[$this->name][$this->__type_field] = $this->name;
        return parent::save($data, $validate, $fieldList);
    }
}

As you can see, aside from the values of the variables at the top, these two models are nearly identical for our simple example. In a real system you would likely add other methods to your models to handle more than just the simplest of situations. An example here would be for the green-lighting process. A method in the Upcoming model which sets the value of the greenlit field and changes the type field to ‘Current’ would effectively change the model of the record and have it show up where needed. I’ll leave that code as an exercise.

Given the fact that the modified methods are the same in both models, this method lends itself to scaffolding quite easily. One of these days I am going to look into modifying / extending the CakePHP script bake.php to allow for scaffolding Single Table Inheritance models. If someone has a chance to do this before I do, please drop me a line!

Controllers and Views

With the database table and Model classes taken care of it is time to look at the Controllers and Views. What needs to be done specially to handle the Single Table Inheritance? Nothing! This is the realy beauty of a setup like this, you make your changes at the Database and Model levels but your controllers and views are handled like they are for any other models. As far as you are concerned when writing the controllers and views, there are two database tables – one for Upcoming and one for Current – and you can handle them as you would normally.

Caveats

As I mentioned at the beginning of this, not every scenario will be covered by this method. Depending on your needs you may need to override more or less methods from AppModel. I am currently using this on two projects in addition to on this site. If anyone else manages to get some use out of it or if you have any questions/comments/error reports, please feel free to leave a comment below.