For downloading, the direct link is here.

This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.

For downloading, the direct link is here.

This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.

For downloading, the direct link is here.

This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.

Let’s look at the SQL that our create_order() view function generated, with the transaction middleware turned on, and no transaction decorators on the function:

LOG:  statement: SET DATESTYLE TO 'ISO'
LOG:  statement: SHOW client_encoding
LOG:  statement: SHOW default_transaction_isolation
LOG:  statement: BEGIN; SET TRANSACTION ISOLATION LEVEL READ COMMITTED
LOG:  statement: SET TIME ZONE E'America/Los_Angeles'
LOG:  statement: SELECT version()
LOG:  statement: INSERT INTO "example_address" ("street_address", "city", "state", "zip") VALUES (E'1313 Mockingbird Lane', E'Mockingbird Heights', E'CA', E'90026')
LOG:  statement: SELECT CURRVAL('"example_address_id_seq"')
LOG:  statement: INSERT INTO "example_order" ("customer_name", "shipping_address_id") VALUES (E'Herman Munster', 5)
LOG:  statement: SELECT CURRVAL('"example_order_id_seq"')
LOG:  statement: COMMIT

Setting aside the overhead statements (and, good grief, there are a lot of those), what’s wrong with this? Well, not much, except that as of 8.2, PostgreSQL has a much better way of getting the primary key (or any other field, for that matter) back from an INSERT statement, which is the INSERT … RETURNING construct. It’s perfect for just this situation, when you’re inserting a record and creating the primary key at the same time, and want to et the primary key back for further operations.

Now, let’s look at the SQL we get from our show_order() function:

LOG:  statement: SET DATESTYLE TO 'ISO'
LOG:  statement: SHOW client_encoding
LOG:  statement: SHOW default_transaction_isolation
LOG:  statement: BEGIN; SET TRANSACTION ISOLATION LEVEL READ COMMITTED
LOG:  statement: SET TIME ZONE E'America/Los_Angeles'
LOG:  statement: SELECT version()
LOG:  statement: SELECT "example_order"."id", "example_order"."customer_name", "example_order"."shipping_address_id" FROM "example_order" WHERE "example_order"."id" = 1 
LOG:  statement: ROLLBACK

In this case, we don’t need that transaction at all.

At it happens, there’s a Django feature that can address both these problems at once, as long as you are running PostgreSQL 8.2 with the postgresql_psycopg2 backend…

Enter the autocommit

This functionality is kind of buried in the Django documentation, almost apologically stuck in the Notes about supported databases section. It’s an option you add to your settings.py file:

DATABASE_OPTIONS = {
    "autocommit": True,
}

What does this do? The documentation, I’m afraid, is pretty obscure about the actual effect. Groveling around in the code, we discover that what it does is prevent Django from automatically starting a new transaction on the first database operation. If we switch it on, our SQL traffic for show_order() now looks like:

LOG:  statement: SET DATESTYLE TO 'ISO'
LOG:  statement: SHOW client_encoding
LOG:  statement: SHOW default_transaction_isolation
LOG:  statement: SET TIME ZONE E'America/Los_Angeles'
LOG:  statement: SELECT version()
LOG:  statement: SELECT "example_order"."id", "example_order"."customer_name", "example_order"."shipping_address_id" FROM "example_order" WHERE "example_order"."id" = 1 

That’s much better! Well, how about for create_order?

Environment:

Request Method: GET
Request URL: http://127.0.0.1:8000/createorder/
Django Version: 1.1
Python Version: 2.5.2
Installed Applications:
['djangotrans.example']
Installed Middleware:
('django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware')


Traceback:
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/core/handlers/base.py" in get_response
  92.                 response = callback(request, *callback_args, **callback_kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/db/transaction.py" in _commit_on_success
  240.                 res = func(*args, **kw)
File "/Users/xof/Documents/Dev/examples/djangotrans/../djangotrans/example/views.py" in create_order
  9.     address.save()
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/db/models/base.py" in save
  410.         self.save_base(force_insert=force_insert, force_update=force_update)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/db/models/base.py" in save_base
  495.                     result = manager._insert(values, return_id=update_pk)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/db/models/manager.py" in _insert
  177.         return insert_query(self.model, values, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/db/models/query.py" in insert_query
  1087.     return query.execute_sql(return_id)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/db/models/sql/subqueries.py" in execute_sql
  324.             return self.connection.ops.fetch_returned_insert_id(cursor)
File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/db/backends/__init__.py" in fetch_returned_insert_id
  171.         return cursor.fetchone()[0]

Exception Type: ProgrammingError at /createorder/
Exception Value: no results to fetch

Hm. That’s not what we expected, at all.

As it happens, there’s a bug in Django 1.1 when using autocommit; it blows up if the very first thing that you do on a new database connection is an INSERT that expects a return value back. As a workaround, we can add a line to our create_order() function to force it to do some traffic before the first insert:

def create_order(request):
    cur = connection.cursor()
        # Work around Django 1.1 autocommit bug

    address = Address(street_address="1112 E Broad St", city="Westfield", state="NJ", zip="07090")
    address.save()

    order = Order(customer_name="Gomez Addams", shipping_address=address)
    order.save()

    return HttpResponse("Created: " + unicode(order))

And now, we get…

LOG:  statement: SET DATESTYLE TO 'ISO'
LOG:  statement: SHOW client_encoding
LOG:  statement: SHOW default_transaction_isolation
LOG:  statement: SET TIME ZONE E'America/Los_Angeles'
LOG:  statement: SELECT version()
LOG:  statement: INSERT INTO "example_address" ("street_address", "city", "state", "zip") VALUES (E'1112 E Broad St', E'Westfield', E'NJ', E'07090') RETURNING "example_address"."id"
LOG:  statement: INSERT INTO "example_order" ("customer_name", "shipping_address_id") VALUES (E'Gomez Addams', 82) RETURNING "example_order"."id"

Wow, that’s great! Notice the nice, efficient RETURNING clause, and notice… uh, the complete lack of a transaction block around the INSERTs. This is why this particular function is called “autocommit”: since we’re not running inside an enclosing transaction, each statement is “autocommitted” upon completion. (Calling a feature “autocommit” when the practical effect is that the framework does not automatically generate COMMIT statements is perhaps not the best choice of nomenclature.)

So, what are our options? Let’s add the @transaction.commit_on_success decorator, and see if that improves matters:

LOG:  statement: SET DATESTYLE TO 'ISO'
LOG:  statement: SHOW client_encoding
LOG:  statement: SHOW default_transaction_isolation
LOG:  statement: BEGIN; SET TRANSACTION ISOLATION LEVEL READ COMMITTED
LOG:  statement: SET TIME ZONE E'America/Los_Angeles'
LOG:  statement: SELECT version()
LOG:  statement: INSERT INTO "example_address" ("street_address", "city", "state", "zip") VALUES (E'1112 E Broad St', E'Westfield', E'NJ', E'07090') RETURNING "example_address"."id"
LOG:  statement: INSERT INTO "example_order" ("customer_name", "shipping_address_id") VALUES (E'Gomez Addams', 84) RETURNING "example_order"."id"
LOG:  statement: COMMIT

Perfect.

Wait. What’s going on?

The following is a digression, and isn’t really required to use autocommit… but you might find it interesting.

You might well be thinking, “That’s all a very nifty trick, but what in the world does INSERT ... RETURNING have to do with autocommit? And why is autocommit limited to PostgreSQL 8.2, when that behavior has been standard with PostgreSQL since ever?”

First, note how Django gets the result of an INSERT if it does have the INSERT … RETURNING syntax available:

LOG:  statement: INSERT INTO "example_address" ("street_address", "city", "state", "zip") VALUES (E'1313 Mockingbird Lane', E'Mockingbird Heights', E'CA', E'90026')
LOG:  statement: SELECT CURRVAL('"example_address_id_seq"')

That is, it calls the currval function. This works great… as long as you are assured that you are the same connection as you were when you issued the insert. But why wouldn’t you be?

Good question, and the answer is: Pooling.

Pretty much any site with any degree of load will, at some point, need pooling software like pgpool-II or PgBouncer to multiplex connections between the application and database server.

The base mode on both of these fine products simulates as closely as possible being directly connected to the database: Each time the client connects to the intermediate connection pooler, it appears that a new, isolated connection has opened up.

However, more aggressive forms of connection pooling are possible. For example, PgBouncer has transaction pooling, in which the connection to the database is only assigned to a client while it has a transaction open. Between transactions, the connection between PgBouncer and the database could switch, which would cause (really strange and hard-to-reproduce) errors.

“But,” you say, having read the PgBouncer docs, “it’s inside a transaction! That keep it from switching to a new connection, even when you’re using transaction pooling!”

That’s true… unless you are running with autocommit turned on, and take no special precautions to make sure that your INSERT is inside of a transation block (remember, it’s not by default, as we saw above).

Note that it is never wrong to use INSERT … RETURNING; it’s just required if you are using transaction pooling (Django doesn’t know if you are or not, so it assumes that you are), with autocommit turned on, and if you do not take special pains to put each INSERT or group thereof into its own transaction.

Back to Reality.

I have to say that it appears that Django’s autocommit support isn’t quite fully baked, but it’s still a very useful feature for getting rid of the redundant BEGIN/ROLLBACK blocks around read-only view functions (which will tend to be the majority of them), while still giving the programmer control over the commit model for those view functions that do write to the database. And, as a nice little bonus, you get the more efficient INSERT ... RETURNING operation as well, as long as you are using PostgreSQL 8.2 or higher. And, of course, you should be!

For all of this, I’m using Django 1.1.1 and PostgreSQL 8.4.1, through the psycopg2 interface.

For the application, we have a models.py:

from django.db import models

class Address(models.Model):
    street_address = models.CharField(max_length=80)
    city = models.CharField(max_length=80)
    state = models.CharField(max_length=2)
    zip = models.CharField(max_length=5)

    def __unicode__(self):
        return self.street_address + " " + self.city + " " + self.state + " " + self.zip

class Order(models.Model):
    customer_name = models.CharField(max_length=80)
    shipping_address = models.ForeignKey(Address)

    def __unicode__(self):
        return "Order " + unicode(self.id) + " going to " + \
            self.customer_name + ", " + unicode(self.shipping_address)

and a views.py:

from django.http import HttpResponse
from djangotrans.example.models import Order, Address
from django.db import transaction, connection

def create_order(request):

    address = Address(street_address="1112 E Broad St", city="Westfield", state="NJ", zip="07090")
    address.save()

    order = Order(customer_name="Gomez Addams", shipping_address=address)
    order.save()

    return HttpResponse("Created: " + unicode(order))


def show_order(request, order_id):
    order = Order.objects.get(pk=order_id)

    return HttpResponse(unicode(order))

and a urls.py:

from django.conf.urls.defaults import *

urlpatterns = patterns('djangotrans.example.views',
    (r'^createorder/$', 'create_order'),
    (r'^showorder/(?P<order_id>\d+)$', 'show_order'),

)

The Basic Rule

In absence of any mention of transaction management at all, Django opens a new transaction with each database operation, and closes it when the operation is complete. If all you do is read operations on the model, it does a ROLLBACK to close the transaction; if you do a data-changing operation, it does a COMMIT. For example, in create_order, the pattern looks something like:

# BEGIN
address = Address(street_address="1112 E Broad St", city="Westfield", state="NJ", zip="07090")
address.save()
# COMMIT

# BEGIN
order = Order(customer_name="Gomez Addams", shipping_address=address)
order.save()
# COMMIT

… and in show_order:

# BEGIN
order = Order.objects.get(pk=order_id)
# ROLLBACK

This is simple, straight-forward… and wrong. The two operations in save_order should be wrapped in a single transaction. If we saved the address, but didn’t then save the order, we’d have an “orphan” address, and that would be just tragic. So, what to do? Fortunately, Django provides a rather large toolbox full of other ways of managing transactions.

Transaction Middleware

First, let’s take a look at the transaction middleware. This is a middleware component that you add to settings.py:

MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.middleware.transaction.TransactionMiddleware',
)

This component sets a flag in Django turning off the automatic COMMIT/ROLLBACK behavior, and instead does an automatic BEGIN when each request starts, and then a COMMIT (if the view function returns normally and did a data-changing operation) or ROLLBACK (when the view function throws an exception, or returns normally having not modified data.)

Doing this to our extremely sophisticated application above, we get a much more pleasing pattern of transaction management:

# BEGIN
address = Address(street_address="1112 E Broad St", city="Westfield", state="NJ", zip="07090")
address.save()

order = Order(customer_name="Gomez Addams", shipping_address=address)
order.save()
# COMMIT

… and in show_order:

# BEGIN
order = Order.objects.get(pk=order_id)
# ROLLBACK

Much better.

Transaction Decorators

Django provides a set of transaction decorators that can be added to view functions to explicitly control how transactions work for that particular function. They work the same way whether or not the transaction middleware is present. They are:

• @transaction.autocommit — This sets the default Django behavior: A COMMIT after each save(), and a ROLLBACK after each database read. If the transaction middleware isn’t present, this doesn’t do anything; if the transaction middleware is present, this reverts this particular function back to the default behavior.
• @transaction.commit_on_success — This sets the transaction middleware behavior of commiting all of the work done in a single view method (if data modification occurs and the view returns successfully), or doing a single ROLLBACK at the end (if data modification does not occur, or the method throws an exception). It’s redunant if the transaction middleware is enabled.
• @transaction.commit_manually — This overrides the current transaction behavior, and requires that the view function call either transaction.commit() or transaction.rollback() at the appropriate place or places (they can be called multiple times within a single view function). If a view function exits with data-modifying work done after the last commit() or rollback() call, the decorator will throw a TransactionManagementError exception.

So, What’s Not to Like?

That seems to be a pretty comprehensive set of transaction control mechanisms. So, what else could one want? Let’s take a look at what is actually sent to the database when we show an order:

LOG:  statement: SET DATESTYLE TO 'ISO'
LOG:  statement: SHOW client_encoding
LOG:  statement: SHOW default_transaction_isolation
LOG:  statement: BEGIN; SET TRANSACTION ISOLATION LEVEL READ COMMITTED
LOG:  statement: SET TIME ZONE E'America/Los_Angeles'
LOG:  statement: SELECT "example_order"."id", "example_order"."customer_name", "example_order"."shipping_address_id" FROM "example_order" WHERE "example_order"."id" = 47 
LOG:  statement: SELECT "example_address"."id", "example_address"."street_address", "example_address"."city", "example_address"."state", "example_address"."zip" FROM "example_address" WHERE "example_address"."id" = 79 
LOG:  statement: ROLLBACK

The transaction here isn’t actually required at all, setting aside the other overhead. Can we get rid of it? In part two, we’ll see how we can.

So, I think object-oriented programming is the bee’s knees. And, in this post, I’m going to tell you to not use it.

I was recently working on a Django application, using PostgreSQL as the database. (This application has been heavily anonymized.) Among many other things, it has a table that is, in essence, a work flow queue. The queue is a list of orders, based on their (integer) primary key, order_pk. Periodically, a task comes through, reads the queue into a Python list, and runs down it, calling a method on each object:

for queue_entry in order_queue
    order = Orders.objects.get(pk=queue_entry.order_pk)
    order.allocate_inventory()

What does allocate_inventory() do? It calls a stored procedure, and returns:

def allocate_inventory(self):
    cursor = connection.cursor()
    cursor.execute("""
        SELECT allocate_inventory(%s)
    """, [self.pk])

Lovely! Let’s just look at the database traffic it creates and… oh, gross.

The program is reading in and instantiating an entire order object, so it can get the primary key out of it. But the way it was able to instantiate the object was because it had the primary key. In this application, order objects have lots and lots of fields, so it’s dragging all of those fields across the wire to build the object, only to use the one field we already knew before we even creted the object.

If everything was in memory, no problem. Instatiating a Python object isn’t free, but it’s no big deal, and this keeps everything nicely encapsulated. But this is one of the dangers of ORMs: Things that would be cheap in an in-memory world suddenly become much pricier when you have to go out to the database, and it can be somewhat non-obvious when you have to go out to the database, how many times, and in what particular way.

(In one particularly pathological example, in this same application, the Django ORM was doing seven calls to the database in order to populate one pop-up menu … which had exactly one entry in it.)

The fix, of course, is to pass the primary key around rather than object:

for queue_entry in order_queue
    allocate_inventory(queue_entry.order_pk)

@staticmethod
def allocate_inventory(order_pk):
    cursor = connection.cursor()
    cursor.execute("""
        SELECT orders.allocate_inventory(%s)
    """, [order_pk])

… even though this isn’t nearly as clean from an OO perspective as creating the object.

Whenever you’re developing against kind of ORM (or, really, any database application programming at all), it’s very wise to set all of the logging on the development PostgreSQL server to “log every statement,” and keep a tail -f of the log file going in another window. That way, if an operation that looks like the proverbial “just a few lines of code” generates an unexpectedly large number of database calls, you can address that before it falls over in production.

ORMs can be very powerful, but the typical usage patterns in a nicely structured OO program can interact with them very badly, and generate a lot more database traffic than you’d expect.

In strictly my opinion, for personal use, I like Mercurial. For projects that use Git, I use Git.

My two sentence summary is: Mercurial is your smart friend who likes to explain things to you. Git is your genius coworker who sighs and rolls his eyes every time you ask him a question.

Installing Django

We have choices here: We can install Django directly from the current Subversion repository, or from an official release. We also have the choice of installing Django as local to a particular user, or global for the whole system. For this install, we’ll do an official release (1.1.1 is current), and install it for the whole system.

[xof@blog ~]$ cd builds
[xof@blog builds]$ wget http://www.djangoproject.com/download/1.1.1/tarball/
[xof@blog builds]$ tar xvfz Django-1.1.1.tar.gz
[xof@blog builds]$ cd Django-1.1.1
[xof@blog Django-1.1.1]$ sudo python2.6 setup.py install
[xof@blog Django-1.1.1]$ python2.6
Python 2.6.3 (r263:75183, Oct 17 2009, 18:29:55) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
>>> 

Where is the project going to live?

Before we configure Apache and mod_wsgi to serve up our new Django project, we should decide where the project is going to live. I like to keep my builds and maintenance account separate from the user that contains the actual stuff being served up by the web server… so let’s create a new user just to hold the project:

[xof@blog ~]$ sudo /usr/sbin/adduser thebuild
[xof@blog ~]$ sudo passwd thebuild
[xof@blog ~]$ sudo /usr/sbin/usermod -a -G apache thebuild

Note: Depending on the permissions your installation sets on /home directories by default, you may need to adjust them so that mod_wsgi can get at the files it needs.

Set up the Django project

Logging in as the project user, first, we’ll make sure that we get the right Python installation by adding an alias to our .bash_profile:

alias python='/usr/bin/python2.6'

Just checking!

[thebuild@blog ~]$ which python
alias python='/usr/bin/python2.6'
    /usr/bin/python2.6

And we can now create the project:

[thebuild@blog ~]$ cd thebuild.com
[thebuild@blog thebuild.com]$ python /opt/python2.6/lib/python2.6/site-packages/django/bin/django-admin.py startproject thebuild

We also need to create a directory for any Python eggs that our application might download (this is, by the way, a “feature” of Python that I really do not care for).

[thebuild@blog thebuild.com]$ mkdir .python-eggs
[thebuild@blog thebuild.com]$ chmod 770 .python-eggs

Now, we need to create a .wsgi file. The .wsgi file is the glue that connects the mod_wsgi module in Apache to our particular application. The Django project itself doesn’t (and shouldn’t) live under the webroot of the web server, but the .wsgi file does:

[thebuild@blog thebuild.com]$ cd ~/thebuild.com
[thebuild@blog ~]$ mkdir apache
[thebuild@blog ~]$ cd apache
[thebuild@blog apache]$ cat >>django.wsgi
#!/opt/python2.6/bin/python

import os, sys

sys.path.append('/home/thebuild/thebuild.com')

os.environ['DJANGO_SETTINGS_MODULE'] = 'thebuild.settings'
os.environ['PYTHON_EGG_CACHE'] = '/home/thebuild/thebuild.com/.python-eggs'

import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()
[thebuild@blog apache]$ 

Next, let’s create the media and admin-media directories, and move the admin media over from the source distribution:

[thebuild@blog apache]$ mkdir admin-media
[thebuild@blog apache]$ mkdir media
[thebuild@blog apache]$ cd admin-media
[thebuild@blog admin-media]$ cp -R /opt/python2.6/lib/python2.6/site-packages/django/contrib/admin/media/* .

Creating a User in PostgreSQL for the Blog

Right now, the only user in the PostgreSQL cluster is the default superuser; that’s not a great choice for the user that Django will use to connect to the database.

It’s also not idea to allow the Django user to create objects in the public schema, but it needs to be able to create objects in some schema if we’re going to use the cool Django model mechanism for table and field creation.

First, let’s create a database for the site:

[postgres@blog ~]$ createdb thebuild
[postgres@blog ~]$ psql thebuild
psql (8.4.1)
Type "help" for help.

thebuild=# CREATE USER django ENCRYPTED PASSWORD 'password';
CREATE ROLE

Then, let’s create a schema just to hold objects related to the site:

thebuild=# CREATE SCHEMA thebuild AUTHORIZATION django;
CREATE SCHEMA

And set an appropriate search path:

thebuild=# ALTER ROLE django SET search_path TO thebuild, public;
ALTER ROLE

… and revoke CREATE on the public schema from the user:

thebuild=# REVOKE CREATE ON SCHEMA public FROM  django;
REVOKE

This means that, by default, anything that the django user creates will go into the schema thebuild, while it still has access to the schema public. However, it won’t be able to create new objects in public.

(Note that we still have rather a barn door open here: We haven’t done anything with pg_hba.conf to require passwords or other security to log in.)

Django Settings

We’ll need to update the settings.py file to match all of the various changes we made. So far, we’ve changed:

DATABASE_ENGINE = 'postgresql_psycopg2'
DATABASE_NAME = 'thebuild'
DATABASE_USER = 'django'
DATABASE_PASSWORD = 'password'

TIME_ZONE = 'America/Los_Angeles'

MEDIA_ROOT = '/home/thebuild/thebuild.com/apache/media/'
MEDIA_URL = 'http://new.thebuild.com/media/'

ADMIN_MEDIA_PREFIX = '/admin-media/'

INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
)

Note that we’ve turned on the admin site by adding django.contrib.admin to installed apps. Now, we can do a syncdb to create the databases the installed apps require:

thebuild@blog thebuild]$ python manage.py syncdb
Creating table django_admin_log
Creating table auth_permission
Creating table auth_group
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (Leave blank to use 'thebuild'): admin
E-mail address: xof@thebuild.com
Password: 
Password (again): 
Superuser created successfully.
Installing index for admin.LogEntry model
Installing index for auth.Permission model
Installing index for auth.Message model
[thebuild@blog thebuild]$ 

Configure Apache

Apache can be tweaked endlessly, of course, but we’re going to do the minimum required to get the server up and ticking. Assuming we’ve done the basics (ServerName, etc.), we add the module info for mod_wsgi:

LoadModule wsgi_module modules/mod_wsgi.so
AddHandler wsgi-script .wsgi

Now, we need to point to the .wsgi file we created above. Moving over to /etc/httpd/conf.d, we can create a file thebuild.conf to contain the virtual host:

<IfModule mod_alias.c>
    Alias /media /home/thebuild/thebuild.com/apache/media
    Alias /admin-media /home/thebuild/thebuild.com/apache/admin-media
</IfModule>

<IfModule mod_wsgi.c>
    WSGIScriptAlias / /home/thebuild/thebuild.com/apache/django.wsgi
    WSGIDaemonProcess thebuild processes=5 threads=6 display-name=%{GROUP}
    WSGIProcessGroup thebuild
    WSGIApplicationGroup %{GLOBAL}

    WSGISocketPrefix run/wsgi
</IfModule>

… restart Apache …

[xof@blog conf]$ sudo /etc/init.d/httpd graceful

… and did it work?

Exciting! Now, time to start actually designing our blog application.

Installing Apache couldn’t be easier:

[xof@blog ~]$ sudo yum install httpd httpd-devel

(We need httpd-devel to build mod_wsgi, the module we’ll use to get at Python programs from Apache.)

 Python and Centos: A Brief Diversion

The good news about Python on Centos is that it comes pre-installed. (It has to be; yum, among other tools, is written in Python.) The bad news is that the version that it comes with is …

[xof@blog ~]$ python -V
Python 2.4.3

… not exactly current. The most recent version of Python on the 2.x line as of this writing is 2.6.3. The very bad news is that you cannot just upgrade the base system Python install to 2.6; it breaks some of the system tools.

Fortunately, a solution exists: Doing a parallel installation of Python. I’m deeply indebted to this posting from Perplexed Labs for guidance on the process.

On with it: Installing Python 2.6

Since the RPMs for Centos are all Python 2.4, we need to install Python 2.6 from source. So, we grab the tarball, download it, and install it.

[xof@blog ~]$ cd builds
[xof@blog builds]$ wget http://www.python.org/ftp/python/2.6.3/Python-2.6.3.tgz
[xof@blog builds]$ tar xvfz Python-2.6.3.tgz
[xof@blog builds]$ cd Python-2.6.3

Note the --prefix option to put the 2.6 install in a different location.

[xof@blog Python-2.6.3]$ ./configure --prefix=/opt/python2.6 --with-threads --enable-shared
[xof@blog Python-2.6.3]$ make
[xof@blog Python-2.6.3]$ sudo make install

Add a symbolic link to put Python 2.6 in our path:

[xof@blog Python-2.6.3]$ sudo ln -s /opt/python2.6/bin/python /usr/bin/python2.6

Add the shared libraries to ldconfig and to /usr/lib64:

[xof@blog Python-2.6.3]$ sudo sh
sh-3.2# cat >> /etc/ld.so.conf.d/opt-python2.6.conf
/opt/python2.6/lib
sh-3.2# /sbin/ldconfig
sh-3.2# exit
[xof@blog Python-2.6.3]$ cd /usr/lib64
[xof@blog Python-2.6.3]$ sudo ln -s /opt/python2.6/lib/libpython2.6.so.1.0 libpython2.6.so.1.0 
[xof@blog Python-2.6.3]$ sudo ln -s libpython2.6.so.1.0  libpython2.6.so

How does it look?

[xof@blog Python-2.6.3]$ python2.6
Python 2.6.3 (r263:75183, Oct 17 2009, 18:29:55) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Great. Onwards.

Installing setuptools

It’s very handy to have the Python package setuptools, so let’s download that and install it from the appropriate Python egg:

[xof@blog Python-2.6.3]$ cd ..
[xof@blog builds]$ wget http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c9-py2.6.egg
[xof@blog builds]$ sudo sh setuptools-0.6c9-py2.6.egg --prefix=/opt/python2.6

Installing psycopg2

And now we can install psycopg2. First, I have to say that the documentation on pretty much any aspect of psycopg2 is not exactly everything one can ask for; reading the INSTALL file in the tarball is your friend…

[xof@blog builds]$ wget http://initd.org/pub/software/psycopg/psycopg2-2.0.13.tar.gz
[xof@blog builds]$ tar xvfz psycopg2-2.0.13.tar.gz
[xof@blog builds]$ cd psycopg2-2.0.13

One of the files in the tarball is setup.cfg; this contains a setting we need to fix. We point pg_config to the right location for the binary.

# "pg_config" is the preferred method to locate PostgreSQL headers and
# libraries needed to build psycopg2. If pg_config is not in the path or
# is installed under a different name uncomment the following option and
# set it to the pg_config full path.
pg_config=/usr/local/pgsql/bin/pg_config

And away we go!

[xof@blog psycopg2-2.0.13]$ python2.6 setup.py build
[xof@blog psycopg2-2.0.13]$ sudo python2.6 setup.py install

So, did that work?

[xof@blog psycopg2-2.0.13]$ python2.6
Python 2.6.3 (r263:75183, Oct 17 2009, 18:29:55) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg2
>>> conn = psycopg2.connect("dbname='template1' user='postgres' host='localhost'")
>>> cur = conn.cursor()
>>> cur.execute("""SELECT datname from pg_database""")
>>> rows = cur.fetchall()
>>> print rows
[('template1',), ('template0',), ('postgres',)]

Installing mod_wsgi

mod_wsgi is an Apache module serving up Python programs. It’s generally considered a superior candidate to mod_python.

[xof@blog builds]$ wget http://modwsgi.googlecode.com/files/mod_wsgi-2.6.tar.gz
[xof@blog builds]$ tar xvfz mod_wsgi-2.6.tar.gz
[xof@blog builds]$ cd mod_wsgi-2.6
[xof@blog mod_wsgi-2.6]$ ./configure --with-python=/opt/python2.6/bin/python
[xof@blog mod_wsgi-2.6]$ make
[xof@blog mod_wsgi-2.6]$ sudo make install

To be paranoid, we should check two things. One: Did it link against the correct Python shared library?

xof@blog mod_wsgi-2.6]$ ldd /usr/lib64/httpd/modules/mod_wsgi.so
    linux-vdso.so.1 =>  (0x00007fffcbbfe000)
    libpython2.6.so.1.0 => /opt/python2.6/lib/libpython2.6.so.1.0 (0x00007f9ac3553000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9ac332c000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f9ac3128000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007f9ac2f25000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f9ac2ca1000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f9ac294b000)
    /lib64/ld-linux-x86-64.so.2 (0x000000323e400000)

Looks good. Second, how big is the shared library it created?

xof@blog mod_wsgi-2.6]$ ls -l /usr/lib64/httpd/modules/mod_wsgi.so
-rwxr-xr-x 1 root root 339625 Oct 17 23:02 /usr/lib64/httpd/modules/mod_wsgi.so

We’re looking for a size in the 332 kilobyte or smaller range, to confirm that it linked against the Python library as a shared, not static, library; for a discussion of this issue, see the “Lack of Shared Library” heading in the installation instructions.

In the next part, we’ll configure Apache, install Django, and bring up a test application.

Since we built PostgreSQL from the tarball, rather than using an RPM, we didn’t get some of the conveniences that the RPM provides, like creating the user that the server will run as. So let’s do that!

[xof@blog ~]$ sudo /usr/sbin/adduser postgres
[xof@blog ~]$ sudo passwd postgres
Changing password for user postgres.
New UNIX password: 
Retype new UNIX password: 
passwd: all authentication tokens updated successfully.

Logging in as our shiny new user, we add the PostgreSQL binaries to our $PATH in the .bash_profile:

PATH=$PATH:/usr/local/pgsql/bin:$HOME/bin

Well, no time like the present… let’s create a cluster:

[postgres@blog ~]$ initdb db

We want the locale and encoding to be UTF8, and that happens to be the default:

The database cluster will be initialized with locale en_US.UTF-8.
The default database encoding has accordingly been set to UTF8.
The default text search configuration will be set to "english".

Nota bene that the default authentication is “trust”:

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the -A option the
next time you run initdb.

That’s fine for now (we’ll change it later). As a smoke test, let’s start up the server and make sure all is well:

[postgres@blog ~]$ pg_ctl -D ~postgres/db -l logfile start

We’re just writing the log to a file ‘logfile’ in the postgres user’s home director, which is not particularly sophisticated log file management. (That’s also something we’ll fix later.)

So, did it work?

[postgres@blog ~]$ psql postgres
psql (8.4.1)
Type "help" for help.

postgres=#

Looks good!

Now, before we start creating new databases, let’s set up a couple of useful defaults.

[postgres@blog ~]$ psql template1
psql (8.4.1)
Type "help" for help.

template1=# 

We’ll be using PL/pgSQL functions in the future, so let’s add that language to template1. The template1 database is the template from which new databases are created, so changes here will be propagated to all new databses we create in the future.

template1=# CREATE LANGUAGE plpgsql;
CREATE LANGUAGE

We can also add the UUID-OSSP functions:

template1=# \i /usr/local/pgsql/share/contrib/uuid-ossp.sql
SET
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION

Great. PostgreSQL has a ton more configuration options, of course, but this will get us started.

One of the other things that you don’t get when you build PostgreSQL from a tarball is an automatic init.d script so that PostgreSQL starts on system boot. Fortunately, the source tarball comes packaged with a suitable startup script:

[xof@blog ~]$ cd builds/postgresql-8.4.1/contrib/start-scripts
[xof@blog start-scripts]$ sudo cp linux /etc/rc.d/init.d/postgresql
[xof@blog start-scripts]$ sudo chmod +x /etc/rc.d/init.d/postgresql

The standard script doesn’t have the right location for the database directory, so we have to edit the appropriate line:

# Data directory
PGDATA="/home/postgres/db"

and use chkconfig to add the appropriate symlinks:

[xof@blog start-scripts]$ sudo /sbin/chkconfig --add postgresql

(We went back to the xof account because the postgres account isn’t a member of wheel, and can’t use sudo, which is just the way it should be.)

So, did that work? Let’s find out!

[xof@blog ~]$ sudo reboot

(wait wait wait.) Back up!

[postgres@blog ~]$ psql postgres
psql (8.4.1)
Type "help" for help.

postgres=# 

Great. Note that the server logs are now being written to the default for the init.d file, which is $PGDATA/serverlogs rather than ~/logfile as before.

Next, in part 3, we’ll install Python 2.6, Apache, mod_wsgi and psycopg2.

For downloading, the direct link is here, and it is also available on Vimeo if that’s more your thing.

This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.

In this part, I’m installing PostgreSQL. First, an idiosyncrasy warning: I very much prefer building PostgreSQL from source, even on package-based systems like Centos. I like the extra bit of control, and knowing just what is going into my binary.

I’ve already created a user account (’xof’), as I much prefer to do everything through sudo (assuming root access is required) rather than logging in as root.

So, let’s get the development environment set up.

Step one, of course, is the base development tools group:

[xof@blog ~]$ sudo yum groupinstall 'Development Tools'

Some of the pieces that PostgreSQL needs to build aren’t in that group, so let’s install those.

[xof@blog ~]$ sudo yum install readline-devel
[xof@blog ~]$ sudo yum install zlib-devel

I’m planning to build PostgreSQL with OpenSSL support, so let’s install that as well:

[xof@blog ~]$ sudo yum install openssl-devel

In addition, I’m going to be building PostgreSQL with OSSP-UUID support, so we’ll need the packages for those. They’re not in the standard repository set, so we’ll need to find them elsewhere. (Many thanks to Devrim Gündüz for pointing me in the right place.)

[xof@blog ~]$ sudo rpm --install http://download.fedora.redhat.com/pub/epel/5/x86_64/uuid-1.5.1-3.el5.x86_64.rpm
[xof@blog ~]$ sudo rpm --install http://download.fedora.redhat.com/pub/epel/5/x86_64/uuid-devel-1.5.1-3.el5.x86_64.rpm

And now we’re ready to build! I created a directory “builds” to hold source tarballs, and downloaded the PostgreSQL source from the appropriate place:

[xof@blog builds]$ tar xvfz postgresql-8.4.1.tar.gz
[xof@blog builds]$ cd postgresql-8.4.1
[xof@blog postgresql-8.4.1]$ ./configure --with-ossp-uuid --with-openssl
[xof@blog postgresql-8.4.1]$ make

And away it goes! The configure and make worked cleanly on the first try. So, time to install it:

[xof@blog postgresql-8.4.1]$ sudo make install

Since I’ll be using the uuid-ossp contrib module, I’ll build and install that now:

[xof@blog postgresql-8.4.1]$ cd contrib/uuid-ossp
[xof@blog uuid-ossp]$ make
[xof@blog uuid-ossp]$ sudo make install

For the convenience of building other things that might depend on the PostgreSQL library, I like to include the PostgreSQL libraries in ldconfig:

[xof@blog ~]$ sudo sh
sh-3.2# cat >> /etc/ld.so.conf.d/usr-local-pgsql-lib.conf
/usr/local/pgsql/lib
sh-3.2# /sbin/ldconfig
sh-3.2# exit
[xof@blog ~]$ 

And PostgreSQL is installed. In part 2, I’ll configure the database.

On my blog running WordPress?

Well, that won’t do at all.

So, I’m going to be migrating this blog to use Django and PostgreSQL, and will blog about the adventure here. I’ll be starting with a bare VPS “slice” at Slicehost, running Centos 5.3, and document each step.

Onwards to step 1: Installing PostgreSQL on the slice.

As it happens, such a setting exists:

listen_addresses=''

Among other things, this appears to prevent local instances of pgAdmin from connecting (although psql works fine), so its utility is not clear, but there it is if you wish it.

CREATE TABLE orders (
   order_id bigserial PRIMARY KEY,
   customer_id bigint REFERENCES customers(customer_id),
   ...
)

and order lines:

CREATE TABLE order_lines (
   ordinal integer NOT NULL,
   order_id bigint REFERENCES orders(order_id),
   item_id bigint REFERENCES items(item_id),
   quantity decimal(10,2) NOT NULL CHECK(VALUE >= 0),
   ...
   PRIMARY KEY(order_id, ordinal)
)

… to get a list of customer_id, item_id pairs:

SELECT orders.customer_id, order_lines.item_id
   FROM orders, order_lines
   WHERE orders.order_id = order_lines.order_id;

And that works great. However, there is another syntax, which is to use a JOIN clause in the FROM list:

SELECT orders.customer_id, order_lines.item_id
   FROM orders NATURAL JOIN order_lines;

NATURAL JOIN, in the words of the wonderful PostgreSQL documentation, “is shorthand for a USING list that mentions all columns in the two tables that have the same names,” so it’s equivalent to:

SELECT orders.customer_id, order_lines.item_id
   FROM orders JOIN order_lines USING(order_id);

which in turn is equivalent to:

SELECT orders.customer_id, order_lines.item_id
   FROM orders JOIN order_lines ON orders.order_id=order_lines.order_id;

NATURAL JOIN is nifty, but it does have a signficant danger. Suppose later we decided to add notes to orders:

ALTER TABLE orders ADD COLUMN note text;

And to order lines, too:

ALTER TABLE order_lines ADD COLUMN note text;

Suddenly, our natural join is much less, well, natural; it’s equivalent to:

SELECT orders.customer_id, order_lines.item_id
   FROM orders JOIN order_lines ON (orders.order_id=order_lines.order_id)
      AND (orders.note=order_lines.note);

… which is almost certainly not what you want. In absence of amazingly good naming discipline, NATURAL is a bit too surprising.

That being said, I’ve gotten very fond of USING and ON. I find it very intuitive to separate join conditions from other filtering conditions. For example, if we only wanted order lines with a quantity greater than zero:

SELECT orders.customer_id, order_lines.item_id
   FROM orders JOIN order_lines USING(order_id)
   WHERE order_lines.quantity > 0;

I like the separation of the join condition that assembles the product of the input tables from the filtration conditions, as I tend to think of those independently when designing a query.

This starts coming apart when you have extremely complicated join conditions, or are joining over many tables (more than three or four), it’s often clearer to move the join conditions into the WHERE clause. Otherwise, see if separating out the join conditions from the other conditions makes your queries easier to read; it did for me.

For downloading, the direct link is here.

This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.

“The locking on this application is not so much pessimistic as clinically depressed.” — me

“Hovercraft are just as happy hovering over the surface of land as of water. The authorities forbid their use on roads. And when you appreciate that they have no brakes, tend to slither off the camber down into the gutter and are deflected sideways by the slightest breeze, you can see that the authorities have made a wise move for once.” — Chris Roper of the Steam Boat Willy project, dedicated to developing a human-powered hovercraft

For downloading, the direct link is here.

This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.

“The question of whether a machine can think is no more interesting than the question of whether or not a submarine can swim.” — Edsger Dijkstra

And this gives me an opportunity to vent one of my pet peeves about the original Tron.

In Tron, you have (in essence) a battle between three programmers: Flynn (hero), Bradley (hero), and Dillinger (villain). Now, what actual programs have these three delivered, per the movie?

1. Flynn: Some games (and those he wrote after hours on the company’s computer).
2. Bradley: A pretty bad-ass security program.
3. Dillinger: An operating system which becomes sentient.

Unsporting though it may be of me, I know which one I’d hire. With appropriate code reviews, of course.

The executive summary is, he doesn’t like it.

He’s making two arguments against dot notation:

1. While it is now legal Objective-C, it’s an uneasy fit to an enormous history of Objective-C code, and existing coding patterns in the language.
2. It blurs the lines between accessing fields in a structure and invoking code, and that’s bad in and of itself.

I have no argument at all with point 1. I personally dislike the brackets notation, but that’s just years of writing C++. Objective-C is a different language, different syntax, get over it, write some code: no problem.

Point 2… well. Maybe this is also my years of C++, but I really don’t see the problem. For example, he writes:

In Objective-C, the square brackets ([]) were added for message sending. While the square brackets had previously only been used for indexing an array, the intent of the brackets can be easily determined by the context.

That’s true, but the original reason that Objective-C used brackets was not to differentiate message sending from structure access; it was because Objective-C was (and largely still is) a preprocessor on C, and using brackets in that way made it easy to parse and do the substitutions. I’m entirely in favor of making a virtue of that necessity, but let’s not forget that the original reason was arbitrary, and unique in the history of object-oriented extensions to C.

He goes on to say, in the example:

int x = foo.value;

What does that mean? Are we getting the value field out of the structure object foo? Are we executing a simple method that returns a value? Or, in this case, are we creating a network connection, pulling a value from a web server, turning that data in to an integer and then returning it?

Well, yes, but we really don’t any more in:

int x = [foo value];

except that we’re running some code. He writes:

Our first glance tells us we are definitely sending a message. That clues us in that more work is being done, not just a simple memory offset and an assignment.

OK, good, we do know that we’re sending a message, but… well, we still don’t know anything at all about the nature of the message than we did in the first example. Calling these things “foo” and “value” is stacking the deck a bit, too; if the example was either:

int remoteMemoryTotal = remoteConnectionPartner.getRemoteMemoryUse;

or

int remoteMemoryTotal = [remoteConnectionPartner getRemoteMemoryUse];

it’s a bit harder to argue that we have zero information about what’s going on.

My preference for dot notation largely comes from the desire for encapsulation. I like the idea that if I need to swap out an implementation detail, I can do so without a syntax change. For me, the fact that accessing a field off of an object, and invoking a method on that object use the same syntax is a feature, not a bug.

Part of it is cultural, I suspect. Objective-C is an interesting hybrid language. The object model is very dynamic and fluid, very Smalltalk-y, but that’s added on top of C. Not a “C-like language,” but C. K&R C. Thus, there’s a huge difference (in philosophy as well as performance) between reading an int out of a structure and sending a message to an object that returns an int, and the syntax reflects that. In other languages, like C++ and Java, the philosophical difference is reduced (somewhat in C++, greatly in Java), and a founding principle of those languages was to make “get something from an object” be the fundamental operation.

He does make some good points about returning l-values from functions (something that C++ wrestled with unhappily), which is a requirement for writing really nice setters as well as gettings using dot notation. All in all, I agree with his fundamental point that:

You aren’t at home. You’re working with another language.

… even if I don’t accept the premise that using dot notation to invoke methods is fundamentally a bad idea.