Extra columns when doing .distinct() in a Django QuerySet

tl;dr: If you are doing a .distinct() query and limiting the results using .values() or .values_list(), you may be in for a surprise if your model has a default ordering using the Meta value ordering. You probably want to clear the ordering using .order_by() with no parameters.

If a model is ordered, either by .order_by() on the QuerySet or a Meta ordering value, it will always include that field in the QuerySet. This is true even if the query uses .distinct(). To quote the documentation:

Any fields used in anorder_by() call are included in the SQL SELECT columns. This can sometimes lead to unexpected results when used in conjunction with distinct().

(The documentation as written implies that is only problem with related models, but as we’ll see, it’s a problem in general. A documentation patch is probably in order here.)

By way of illustration, let’s assume you have the following models:

1
from django.db import models
2

3
class Publisher(models.Model):
4
    name = models.TextField()
5

6
    class Meta:
7
        ordering = [ 'name', ]
8

9
class Book(models.Model):
10
    title = models.TextField()
11
    topic = models.TextField()
12
    publisher = models.ForeignKey(Publisher)
13

14
    class Meta:
15
        ordering = [ 'title', ]

And we create some rows:

1
pub = Publisher(name="Strange But True Publications")
2
pub.save()

And some books:

1
book1 = Book(title="New Topics in Industrial Meringue Production",
2
             topic="Cooking",
3
             publisher=pub)
4
book1.save()
5

6
book2 = Book(title="Your Chicken's First Song Book",
7
             topic="Animal Husbandry",
8
             publisher=pub)
9
book2.save()

Now, we want to get the list of IDs of the publishers, and we’re using the cool optimization that I described earlier, with the optimization a commenter suggested (thanks!):

1
>>> q = Book.objects.values_list('publisher_id', flat=True).distinct()
2
>>> print q
3
[1, 1]

Um, wait. That’s not right. Why would it return 1 twice when we said .distinct()? Let’s look at the SQL (you are doing a tail -f on the PostgreSQL logs while you develop, right?):

1
LOG:  statement: SELECT DISTINCT "x_book"."publisher_id", "x_book"."title" FROM "x_book" ORDER BY "x_book"."title" ASC LIMIT 21

And there we have it. It includes the title field in the query, even though it doesn’t return it. Since the DISTINCT thus applies to both, we have two distinct rows, rather than one.

The fix, fortunately, is easy; just clear the ordering with a .order_by() without any parameters:

1
>>> q = Book.objects.values_list('publisher_id', flat=True).distinct().order_by()
2
>>> print q
3
[1]

And the query:

1
LOG:  statement: SELECT DISTINCT "x_book"."publisher_id" FROM "x_book" LIMIT 21