No, really. They are great vehicles. You can use them for all sorts of really useful things: Bringing your tools out to a construction gig. Delivering refrigerators. Helping your friend move a sofa. Carting away a reasonable amount of construction debris.

But if you need to deliver 75,000 pounds of steel beams to a construction site, in a single run? A pickup truck will not do it. Not even a big pickup. Not even if you add a new engine. Not even if you are willing to get three pickups. You need equipment designed for that. (And, as a note, the equipment that could handle delivering the steel beams would be a terrible choice for helping a friend move their sofa.)

“But,” I hear you say, “I already know how to drive a pickup! And we have a parking space for it. Can’t we just use the pickup? You’re a truck expert; tell us how to get our pickup to pull that load!”

And I say, “Being a truck expert, I will tell you again, a pickup the wrong kind of truck. There are other trucks that will handle that load with no trouble, but a pickup isn’t one of them. The fact that you have a pickup doesn’t make it the right truck.”

We have many clients that run PostgreSQL, happily, on Amazon Web Services.

Some clients, however, are not happy. They are attempting to haul tractor-trailer loads (such as high volume data warehouses) using pickup trucks (Amazon EC2 instances). They wish us to fix their problem, but are not willing to move off of Amazon in order to get the problem fixed.

I like AWS for a lot of things; it has many virtues, which I will discuss in detail soon. However, AWS is not the right solution for every problem. In particular, if you require a high read or write data rate in order to get the performance you need from your database, you will ultimately not be happy on AWS. AWS has a single block-device storage mechanism, Elastic Block Storage, which simply does not scale up to very high data rates.

That doesn’t mean that AWS is useless, it just means it isn’t the right tool for every job. The problem arises when AWS is considered the fixed point, like the pickup was the fixed point above. At some point, you have to decide:

1. That being on AWS is so important (for whatever reason) that you are willing to sacrifice the performance you want; or,
2. The performance you want is so important that you will need to move off of AWS.

Sadly, even the best of consultants do not have the magic engine in our back room that will cause EBS to perform as well as high-speed direct attached storage.

More soon.

It’s time for Apple to admit that the traditional model of a singular WWDC either requires a venue that can handle the crowd, or split it into multiple, regional events. It would lose the “gathering of the tribe” aspect that has always been one of the best parts of WWDC, but that’s lost now, anyway; the “tribe” is not defined on who was by their keyboards for 90 minutes at 6am on a Wednesday.

If Apple views the WWDC as a way to pack people into seats to create excitement for their announcements early in the week, then I suppose the current system is as good as anything. From any other perspective, it’s time to find another way of doing this.

Then, view this photo set.

• PGCon, Ottawa, Ontario, Canada, May 17-18.
• DjangoCon Europe, Zurich, Switzerland, June 4-6.
• Southwest LinuxFest, Charlotte, North Carolina, June 8-10.
• EuroPython, Florence, Italy, July 2-8.
• OSCON, Portland, Oregon, USA, July 16-20.

Do not mix ‘n’ match override styles.

There are two valid styles for overriding the default values in postgresql.conf: Putting your changes as a cluster at the end, or uncommenting the defaults and overriding in place. Both have advantages and disadvantages. Having some settings one way and some another is pure disadvantage. Do not do this.

Use units.

Quick, what is log_min_duration_statement set to here?

log_min_statement_duration = 2000

Now, what is it set to here?

log_min_statement_duration = 2s

Always use units with numeric values if a unit is available.

Do not remove the default settings.

If you strip out all of the defaults, it becomes impossible to tell what a particular value is set to. Leave the defaults in place, and if you comment out a setting, reset the value to the default (or at least include comments that make it clear what is going on).

Do not leave junk postgresql.conf files scattered around.

If you need to move postgresql.conf (and the other configuration files) to a different location from where the package for your system puts it, don’t leave the old, dead postgresql.conf lying around. Delete any trace of the old installation hierarchy.

Thank you.

I have some acquaintance with the Instagram people, and they are among the smartest technologists I’ve met. Really nice, too. (Of course, they mention this blog in the post, so I’m biased.)

The goals are:

• Database operations that do not modify the database aren’t wrapped in a transactions at all.
• Database operations that modify the database are wrapped in a transactions.
• We have a lot of fine-grained control over sections that modify the databse vs those that don’t.

The bits of the recipe are:

• Use the autocommit option in your database configuration.
• Do not use the transaction middleware.
• Wrap the sections of code which modify the database in the xact() decorator / context manager below, using it like you would the commit_on_success() decorator.
• Profit!

The quick reasons behind each step:

• Turning on autocommit prevents psycopg2 from automatically starting a new transaction on the first database operation on each connection; this means that the transaction only starts when we want it to.
• Similarly, the transaction middleware will set the connection state to “managed,” which will defeat the autocommit option above, so we leave it out.
• The xact() decorator will set up the connection so that a transaction is started in the relevant block, which is what we want for database-modifying operations.

This recipe a few other nice features:

• xact() operates like commit_on_success(), in that it will issue a rollback if an exception escapes from the block or function it is wrapping.
• xact() ignores the dirty flag on the Django connection. Since we’re deliberately wrapping stuff that modifies the database with it, the chance of it being dirty is near 100%, and a commit on a transaction that did not modify the database is no more expensive in PostgreSQL than a rollback. It also means you can do raw SQL inside an xact() block without the foot-gun of forgetting to call set_dirty.
• Like the built-in Django transaction decorators, it can be used either as a decorator or as a context manager with the with statement.
• xact() can be nested, giving us nested transactions! If it sees that there is already a transaction open when it starts a new block, it will use a savepoint to set up a nested transaction block. (PostgreSQL does not have nested transactions as such, but you can use savepoints to get 99.9% of the way there.)
• By not wrapping operations that do not modify the database, we get better behavior when using pgPool II (more on that in a future post).
• xact() works around an outstanding bug in Django’s transaction handling on psycopg2.

xact() also supports the using parameter for multiple databases.

Of course, a few caveats:

• xact() requires the postgresql_psycopg2 backend, and PostgreSQL 8.2 or higher. It’s possible it can be hacked to work on other backends that support savepoints.
• xact() works just the way you want if it is nested inside a commit_on_success() block (it will properly create a savepoint insted of a new transaction). However, a commit_on_success() block nested inside of xact() will commit or rollback the entire transaction, somewhat defeating the outer xact(). To the extent possible, use only xact() in code you write.
• Be sure you catch exceptions outside of the xact() block; otherwise, the automatic rollback will be defeated. Allow the exception to escape the xact() block, and then catch it. (Of course, if the intention is to always commit and to defeat the rollback, by all means catch the exception inside the block.)

To use, just drop the source (one class definition, one function) into a file somewhere in your Django project (such as the omni-present utils application every Django project seems to have), and include it.

Examples:

from utils.transaction import xact

@xact
def my_view_function1(request):
   # Everything here will be in a transaction.
   # It'll roll back if an exception escapes, commits otherwise.

def my_view_function2(request):
   # This stuff won't be in a transaction, so don't modify the database here.
   with xact():
      # This stuff will be, and will commit on normal completion, roll back on a exception

def my_view_function3(request):
   with xact():
      # Modify the database here (let's call it "part 1").
      try:
         with xact():
            # Let's call this "part 2."
            # This stuff will be in its own savepoint, and can commit or
            # roll back without losing the whole transaction.
      except:
         # Part 2 will be rolled back, but part 1 will still be available to
         # be committed or rolled back.  Of course, if an exception
         # inside the "part 2" block is not caught, both part 2 and
         # part 1 will be rolled back.

The source is available on GitHub. It’s licensed under the PostgreSQL License.

To keep your Bucardo installation clean and tidy, a few regular cron jobs are required. One of them cleans up the archived replicated data (stored in a separate database by Bucardo) once you know you are done with it.

The Bucardo page above has a recommended script using all sorts of bashing, but I wanted something a bit more pure-PostgreSQL; it also doesn’t purge more than one old table at a time. So, I whipped up the following PL/pgSQL function.

(Note that this is for Bucardo 4.4. I haven’t played with the forthcoming Bucardo 5, so I’m not sure if this is still required.)

CREATE OR REPLACE FUNCTION bucardo.purge_frozen_child_qs(far_back interval)
    RETURNS SETOF TEXT AS
$purge_frozen_child_qs$
DECLARE
    t TEXT;
    qt TEXT;
BEGIN

    IF far_back IS NULL THEN
        RAISE EXCEPTION 'Interval cannot be null.'
            USING HINT = 'So, do not do that.';
    END IF;

    IF (now() + far_back) > now() THEN
        RAISE EXCEPTION 'Interval must be negative.'
            USING HINT = 'Consider using the "ago" form of intervals.';
    END IF;

    FOR t IN 
        SELECT tablename 
            FROM pg_tables
            WHERE schemaname='freezer' 
                  AND tablename like 'child_q_%' 
                  AND (replace(tablename, 'child_q_', '')::timestamp with time zone) < now() + far_back::interval
            ORDER BY tablename
    LOOP
        qt := 'freezer.' || t;
        EXECUTE 'DROP TABLE ' || qt;
        RETURN NEXT qt;
    END LOOP;

    DELETE FROM bucardo.q 
        WHERE (started < now() + far_back::interval 
                OR ended < now() + far_back::interval 
                OR aborted < now() + far_back::interval 
                OR cdate < now() + far_back::interval) 
              AND (ended IS NULL OR aborted IS NULL);

    RETURN;

END
$purge_frozen_child_qs$
LANGUAGE plpgsql
    VOLATILE;

To use it, just call it repeatedly from a cron job with the appropriate argument, along the lines of:

SELECT * FROM bucardo.purge_frozen_child_qs('7 days ago'::interval);

It returns the names of the tables it deleted.

This particular function doesn’t need to be run more often than once a day. And it keeps your Bucardo goats nice and clean.

(A “bucardo” is a now-extinct species of goat. For why Bucardo is goat-related, ask Greg.)

Update: Well, that was quick. WWDC sold out in 10 hours, while I was dithering.

• Justin Vincent wrote a post offering an opinion about the downsides of the chase of tech entrepreneurs over VC funding.
• Amy Hoy wrote a post expanding on Mr Vincent’s post.
• Alex Payne wrote a post criticizing this position, while finding it necessary to describe “long time acquaintance” Amy Hoy’s product as “duping credulous customers into overpaying for a time-tracking tool styled with this month’s CSS trends.”
• Unaccountably, he seems to have been surprised by the negative reaction this post generated, so he posted an explanation and partial retraction here.

Sadly, I find his last post as incoherent as his first one is vitriolic.

Rather than go through it point by point, the crux of his argument is:

Building a business around maximizing your individual happiness is not particularly useful or admirable. That is my position, and I’m well aware that it may be unpopular with some.

I am pleased to report, then, that Mr Payne has absolutely nothing to worry about, because no business that is built around the happiness of the owner as a primary goal has a hope of every getting anywhere, unless the business consists of the owner taking money out of one pocket and putting it in the other. Any business, unless it is operating in a grotesquely distorted marketplace, is primarily about pleasing its customers in exchange for their money.

I’m really not sure what these vaguely masturbatory companies Mr Payne is talking about do for a living, but every (successful) micro-business I know of is insanely, intensely focused on pleasing its customers. They have to be, because they don’t have an installed base, government-granted advantages, or (yes) piles of venture capital in the bank to fall back on if they fail to do so.

Mr Payne wants to run a big company. I wish him all the best. He seems to have his young heart in the right place. I have to say, though, that his emotional overreaction to the idea that someone might want to run a micro-business instead strikes me as the Puritan reacting to the idea that someone, somewhere, might be happy.

Anyone with the technical sense of a rock would pause for a moment, and then burst out in laughter at the utter absurdity of this proclamation. No one would ever attempt to make such a ludicrous and obviously self-serving claim, would they?

Verifone would. Verifone is very very concerned about Square’s iPhone card scanner, because someone could run a malicious app on the iPhone and collect card data using it. The fact that Square just announced new pricing undercutting Verifone’s is, of course, completely coincidental.

Where to begin?

• It is true that Square’s attachment does not encrypt the card track information between the iPhone and the card reader. This is true of pretty much every single card reader in the entire world. It is not the job of the card reader to encrypt data, any more than it is the job of the keyboard to encrypt your password. Verifone seems unconcerned at all of the other card readers you can buy from, say, Amazon (just for example).

• For Verifone’s apocalyptic scenario to occur, the iPhone into which the card reader is plugged must be running a malicious app. This pretty much requires the iPhone user to be in on the scam, which means that they could be using any hardware they wish to collect this card data. If the merchant is crooked, then they’ll find a way to collect the card data, since they have possession of the card (on which is printed essentially all of the relevant data that is on the mag tracks, plus the CVV printed on the back).

• Verifone’s competing solution, if the brochure is to be believed, encrypts the data at swipe-time. That’s nice, but the chance of card data being compromised between the reader and the iPhone, or during that extremely limited time that it is sitting unencrypted in the iPhone’s memory, is essentially zero. Again, Verifone seems unconcerned that Square’s app works exactly like every other PC-based credit card processing application in the entire world; indeed, Square’s is considerably more secure than most, since the merchant doesn’t have access to the card information. (For example, on my completely certified, authorized, and every-spec-compliant Nurit wireless card processing terminal, I can retrieve credit card numbers from a batch with no hassle whatsoever.)

In short, Verifone is bashing a competitor because the competitor’s pricing is more consumer-friendly than Verifone’s. Their technical arguments are nonsense, and they should be ashamed of launching a FUD campaign that plays on credit card security paranoia.

It’ll cover general development in Django, with a lot of PostgreSQL-specific details.

And, of course, the whole conference will be a fount of great PostgreSQL geekery.

In a flash of snarkiness, I posted this to my Twitter account, based on a colleague’s attempt to solve a YouTube password problem:

Google doesn’t want to hear from its users any more than Tyson wants to hear from its chickens.

The slightly-expanded message is: The users of Google’s free services are a product to be delivered to advertisers, not customers towards whom Google feels an obligation. I think this is more than just fair; I think it’s self-evident.

There was a certain amount of chatter form those who insist on viewing the world as a Manichean struggle between Apple and Google (which had nothing whatsoever to do with it), but it more productively resulting in a reply from no less that Matt Cutts (and I appreciate his taking the time to bounce this around with me):

@Xof that’s certainly not the case.

I replied:

@mattcutts Glad to hear it. What’s the tech support number for YouTube users?

Mr Cutts replied:

@Xof phone lines are 1:1, which means other users can’t benefit from answers/feedback. The forum for YouTube is http://goo.gl/pApu

And here we get to the lies we tell ourselves.

You can’t do everything all the time. No person, product or company can hit every single mark, can fulfill every single desire or request. No one should ever be ashamed of that; everyone has to prioritize.

Google’s prioritization is, “For our free services, we can’t provide individualized support because it would simply be too expensive. Instead, we mobsource our support. That’s what you are getting when you use our free services. Did we mention that they’re free?” That’s not insane, and you can agree with it (and use Google’s services), or disagree with it (and not, or do so anyway and whine about the terrible support, your choice). But that’s what you get.

But attempting to portray that decision as, “Forums are so much cooler than an actual human being on the phone who can actually address your particular item” is absurd. Can you imagine, say, a bank with that position? ”An unexplained debit to your account? No problem! Post a comment to our Forum, and maybe someone has had a similar problem and you can share your experience. Sometimes, an employee might see the post and comment. It’s rad!”

Tech support forums are great, and every company should sponsor one for their products, but they are not a substitute for a support system that does not rely on the kindness of strangers and does involve someone who can safely access details relevant to your individual situation and who can access internal expertise within the company. I would hope that’s self-evident, too.

(And, of course, a rep can share their redacted experience from a call on a forum.)

I’m reminded of the apologia for why qmail doesn’t send multiple messages on one SMTP connection, or the long diatribe in the Camel Book about why Perl 5 doesn’t have simple, reliable bytecode compilation. In both cases, a designer made a choice that feature was complex to implement and not a high priority; reasonable people can disagree with their choices, but they made them for what I assume are sound reasons. However, rather than just say, “Nope, not getting to that one right away,” what was a issue of priority was tarted up to look like a wise technical architecture decision, which neither one was. (Lots of MTAs can do just fine sending multiple messages on one connection, and if bytecode compilation for Perl was such a dumb idea, what’s up with Parrot?)

Every business (and person) needs to make choices. Nothing wrong with that. Just be honest with yourself about why you are doing it, so you can be honest with other people.

If a model is ordered, either by .order_by() on the QuerySet or a Meta ordering value, it will always include that field in the QuerySet. This is true even if the query uses .distinct(). To quote the documentation:

Any fields used in anorder_by() call are included in the SQL SELECT columns. This can sometimes lead to unexpected results when used in conjunction with distinct().

(The documentation as written implies that is only problem with related models, but as we’ll see, it’s a problem in general. A documentation patch is probably in order here.)

By way of illustration, let’s assume you have the following models:

from django.db import models

class Publisher(models.Model):
    name = models.TextField()

    class Meta:
        ordering = [ 'name', ]

class Book(models.Model):
    title = models.TextField()
    topic = models.TextField()
    publisher = models.ForeignKey(Publisher)

    class Meta:
        ordering = [ 'title', ]

And we create some rows:

pub = Publisher(name="Strange But True Publications")
pub.save()

And some books:

book1 = Book(title="New Topics in Industrial Meringue Production",
             topic="Cooking",
             publisher=pub)
book1.save()

book2 = Book(title="Your Chicken's First Song Book",
             topic="Animal Husbandry",
             publisher=pub)
book2.save()

Now, we want to get the list of IDs of the publishers, and we’re using the cool optimization that I described earlier, with the optimization a commenter suggested (thanks!):

>>> q = Book.objects.values_list('publisher_id', flat=True).distinct()
>>> print q
[1, 1]

Um, wait. That’s not right. Why would it return 1 twice when we said .distinct()? Let’s look at the SQL (you are doing a tail -f on the PostgreSQL logs while you develop, right?):

LOG:  statement: SELECT DISTINCT "x_book"."publisher_id", "x_book"."title" FROM "x_book" ORDER BY "x_book"."title" ASC LIMIT 21

And there we have it. It includes the title field in the query, even though it doesn’t return it. Since the DISTINCT thus applies to both, we have two distinct rows, rather than one.

The fix, fortunately, is easy; just clear the ordering with a .order_by() without any parameters:

>>> q = Book.objects.values_list('publisher_id', flat=True).distinct().order_by()
>>> print q
[1]

And the query:

LOG:  statement: SELECT DISTINCT "x_book"."publisher_id" FROM "x_book" LIMIT 21