Don't Read Settings at Import Time

Python doesn't make a distinction between import time and run time. As such, it's possible to read settings at import time, but this can lead to subtle bugs. Reading a setting at import time will use or copy its initial value, and will not account for any later changes. Settings don't often change, but when they do, you definitely want to use the new value.

For example, take this views file:

from django.conf import settings

...

page_size = settings.PAGE_SIZE


def book_list(request):
    paginator = Paginator(Book.objects.all(), page_size)
    ...

page_size is set to the value of the setting once, at import time. It won't account for any changes made in tests, such as with override_settings:

from django.test import TestCase, override_settings


class BookListTests(TestCase):
    def test_pagination(self):
        with override_settings(PAGE_SIZE=2):
            response = self.client.get(...)

The previous view code would not use the changed PAGE_SIZE setting, since it won't affect the module-level page_size variable. You can fix this problem by changing the view to read the setting when it runs:

from django.conf import settings

...


def book_list(request):
    paginator = Paginator(Book.objects.all(), settings.PAGE_SIZE)
    ...

Straightforward when you know how.

This problem can also manifest itself with classes, as class-level attributes are also set once, at import time. For example, in this class-based view:

from django.conf import settings


class BookListView(ListView):
    ...
    paginate_by = settings.PAGE_SIZE

In this case, you can avoid the problem by reading the setting in the relevant view method:

from django.conf import settings


class BookListView(ListView):
    def get_paginate_by(self, queryset):
        return settings.PAGE_SIZE

Two thumbs up!

Avoid Direct Setting Changes

Thanks to Python's flexibility, it's possible to change a setting at runtime by directly setting it. But you should avoid doing so, since this does not trigger the django.test.signals.setting_changed signal, and any signal receivers will not be notified. Django's documentation warns against this pattern.

For example, tests should not directly change settings:

from django.conf import settings
from django.test import TestCase


class SomeTests(TestCase):
    def test_view(self):
        settings.PAGE_SIZE = 2
        ...

Instead, you should always use override_settings, and related tools:

from django.conf import settings
from django.test import TestCase
from django.test import override_settings


class SomeTests(TestCase):
    @override_settings(PAGE_SIZE=2)
    def test_view(self):
        ...

Coolio.

(Note, making direct changes to pytest-django's settings fixture is okay. It's a special object that uses override_settings under the hood.)

Don't Import Your Project Settings Module

It's possible to directly import your settings module and read from it directly, bypassing Django's settings machinery:

from example import settings


def index(request):
    if settings.DEBUG:
        ...

You should avoid doing so though. If settings are loaded from a different place, or overridden with override_settings etc., they won't be reflected in the original module.

You should always import and use Django's settings object:

from django.conf import settings


def index(request):
    if settings.DEBUG:
        ...

Alrighty.

It can occasionally be legitimate to import your settings modules directly, such as when testing them (covered later in the book). If you don't have such usage, you can guard against this pattern with the flake8-tidy-imports package's banned-modules option. This allows you to specify a list of module names for which it will report errors wherever they are imported.

Avoid Creating Custom Settings Where Module Constants Would Do

When you define a custom setting, ensure that it is something that can change. Otherwise, it's not really a setting but a constant, and it would be better to define it in the module that uses it. This can keep your settings file smaller and cleaner.

For example, take this "setting":

EXAMPLE_API_BASE_URL = "https://api.example.com"

(When using a single settings file.)

It only defines a code uses a constant base URL, so it's not configurable. Therefore there's no need for this variable to be a setting. It can instead live in the Example API module, keeping the settings file trim.

Avoid Creating Dynamic Module Constants Instead of Settings

This is kind of the opposite to the above. In order to avoid cluttering the settings file, you might be tempted to define some constants that read from environment variables in other modules. For example, at the top of an API client file:

EXAMPLE_API_BASE_URL = os.environ.get(
    "EXAMPLE_API_BASE_URL",
    "https://api.example.com",
)

As a separated module-level constant, it loses several of the advantages of being a setting:

1. Since it's not centralized in the settings file, it's not discoverable. This makes it hard to determine the full set of environment variables that your project reads.
2. When the environment variable is read is a bit less determined. The settings file is imported by Django at a predictable time in its startup process. Other modules can be imported earlier or later, and this can change as your project evolves. This non-determinism makes it harder to ensure that environment variable sources like a .env file have been read.
3. override_settings cannot replace the constant. You can use unittest.mock instead, but this gets tricky if the constant is imported elsewhere.

Such a constant deserves to be a setting instead. Make it so!

Name Your Custom Settings Well

As settings live in a single flat namespace, settings names are important, even more than regular variable names. When you create a custom setting, use a highly specific name to help guide future readers. Avoid abbreviations, and use a prefix where applicable. When reading from an environment variable, match the environment variable name to the setting name.

For example, here's a poorly named custom setting:

EA_TO = float(os.environ.get("EA_TIMEOUT", 5.0))

It begs some questions:

• What does "EA" stand for?
• What units is the timeout in?
• What does TO stand for?

Here's a better definition of that setting:

EXAMPLE_API_TIMEOUT_SECONDS = float(os.environ.get("EXAMPLE_API_TIMEOUT_SECONDS", 10))

A fine, informative name.

Override Complex Settings Correctly

Some settings take complex values, such as nested dictionary structures. When overriding them in tests, you need to be careful to only affect the part of the setting that you care about. If you don't, you may erase vital values from the setting.

For example, imagine a package configured with a dictionary setting:

EXAMPLE = {
    "VERSIONING": "example.SimpleVersioning",
    "PAGE_SIZE": 20,
    # ...
}

In a test, you might want to reduce the value of PAGE_SIZE to improve test speed. You might try this:

from django.test import TestCase, override_settings


class SomeTests(TestCase):
    def test_something(self):
        with override_settings(EXAMPLE={"PAGE_SIZE": 2}):
            # Do test
            ...

…unfortunately, this would erase the value of the VERSIONING key, and any others. This might affect the test behaviour now, or in the future when the setting is changed.

Instead, you should create a copy of the existing setting value with the appropriate modification. One way to do this, on Python 3.9+, is with the dictionary merge operator:

from django.conf import settings
from django.test import TestCase, override_settings


class SomeTests(TestCase):
    def test_something(self):
        with override_settings(EXAMPLE=settings.EXAMPLE | {"PAGE_SIZE": 2}):
            # Do test
            ...

On older Python versions, you can use dict(base, key=value, ...) to copy a dictionary and replace the given keys:

from django.conf import settings
from django.test import TestCase, override_settings


class SomeTests(TestCase):
    def test_something(self):
        with override_settings(EXAMPLE=dict(settings.EXAMPLE, PAGE_SIZE=2)):
            # Do test
            ...

w00t.

Fin

May your settings be tidy and clear,

-Adam

My books

My two books, Boost Your Django DX and Speed Up Your Django Tests, have 50% off their base prices.

The deal stacks with the regional discount that offers a 50% discount for those living outside the GDP top 50. If you live in such a country, email me to get a total of 75% off the base prices.

This discount is available until the end of Cyber Monday, "Anywhere on Earth" (AoE). That is, it's live as long as it's still Cyber Monday or before, in one time zone, anywhere on earth.

Buy now:

• Boost Your Django DX
• Speed Up Your Django Tests

(You can buy both in one transaction by clicking "Buy this" on one, then "Continue shopping" at the top right, which will let you add the other to your cart.)

William Vincent's Three Book Bundle

William Vincent is the author of three fantastic Django books:

• Django for Beginners
• Django for APIs
• Django for Professionals

He's offering a 50% discount on the three book bundle, from $97 to $49.

Buy it on Gumroad

testdriven.io Four Course Sale

The educational site testdriven.io are running a sale on a bundle of four Django courses:

• Developing RESTful APIs with Django REST Framework by Špela Giacomelli
• Test-Driven Development with Django, Django REST Framework, and Docker by Michael Herman
• The Definitive Guide to Celery and Django by Michael Yin
• Full-text Search in Django with Postgres and Elasticsearch by Jason Parent

Normally these four courses would total $120, but the bundle has a 30% discount to $84. This discount is available until the end of the 27th November.

Buy it on testdriven.io

Tango with Django

Tango with Django is an end-to-end beginner's guide to Django, by Leif Azzopardi and David Maxwell. It was recently updated for Django 4.

The book is "pay what you want" on Leanpub, with the minimum price discounted 30% from $12.99 to $9.99. This disocunt is available until the end of Cyber Monday.

Buy it on Leanpub

SaaS Pegasus

Corey Zue's SaaS Pegasus, is a Django project template with many preset niceties, including teams, Stripe subscriptions, a JavaScript pipeline, and multiple CSS themes. It can massively accelerate setting up a SaaS in Django.

The "unlimited lifetime license" is discounted 50%, from $750 to $375. This deal is available through 29 November.

Buy it on saaspegasus.com

Michael Yin's Books

Michael Yin, creator of one of the testdriven.io course above, has also written several books on Django related topics. He's offering a 30% discount on two of them, one about building a blog with wagtail, and one about using Hotwire (HTML-over-the-wire) and Turbo with Django.

Buy them on Leanpub with the discount links below:

• Build Blog With Wagtail CMS (4.0.0)
• The Definitive Guide to Hotwire and Django

Forge Pro

David Gaeddert's Forge is "an opinionated way to build with Django". Forge Pro is the paid product that contains six third party apps for your production projects. For example, "Query Stats" can show you SQL query statistics in production on any URL, whilst the "Stripe" app provides helpers for billing customers with Stripe. You also gain access to a community of Django developers where you can discuss Forge Pro and its future development.

Use the discount code BLACKFRIDAY22 for 50% off your first year of subscription, taking the personal license from $149/year to $74.50/year. This deal is available through 30 November. 10% of Forge's cost goes to support the Django Software Foundation (more on that below).

Buy it on forgepackages.com

Aidas Bendoraitis' GDPR Cookie Consent Package

Aidas Bendoraitis of djangotricks.com created this paid third-party app for Django last year. The package takes the pain out of setting up and customizing legally-mandated GDPR Cookie Consent screens. Compared to commercial "one size fits all" solutions, it's much simpler to use this third party app to host and tweak your own cookie consent screen.

Use the discount code BLACKFRIDAY22 for 20% off, from €150 to €120, until the end of November.

Buy it on Gumroad

Bonus: Django Itself

Okay, there's no discount here, but there is a good deal! You can fund the framework that you love to ensure it continues into the future.

If you are able to spend some money on Django-related products this Black Friday, please do consider sponsoring Django itself. You can support it by donating to the charity that runs the framework, the Django Software Foundation.

Your money will go towards:

• Paying the Django Fellows, Carlton and Mariusz, who merge respond to tickets, merge code, and make releases.
• Helping organize Djangocons in Africa, America, and Europe, and other events.
• Hosting costs of the documentation, source code, ticketing system, and CI system.
• A long tail of activities for keeping the framework alive and thriving.

You can sponsor Django on:

• GitHub Sponsors - adds to your GitHub bill.
• Threadless - grab some custom-printed merchandise.
• The Fundraising Page - bills from a credit card with Stripe. This page also links to other schemes: Amazon Smile and Benevity Workplace Giving.

If you're working with Django professionally, please consider sponsoring a few dollars a month. But better yet, get your organization to sponsor to pay back for all the advantages that Django offers you.

At time of writing, Django is 79% towards its 2022 funding goal:

Let's try get it to 100%!

Fin

Go forth and support Django creators, and Django itself!

-Adam

P.S. If you are also a creator, email me with details of your offer and I'll add it here.

A time-saving example

Take this test case, which uses my time-mocking library time-machine to fix the current microsecond to 0 (for whatever reason):

import datetime as dt
from unittest import TestCase

import time_machine


class ExampleTests(TestCase):
    def setUp(self):
        now = dt.datetime.now().replace(microsecond=0)

        self.traveller = time_machine.travel(now)
        self.traveller.start()
        self.addCleanup(self.traveller.stop)

        super().setUpClass()

    def test_microsecond(self):
        self.assertEqual(dt.datetime.now().microsecond, 0)

It takes three steps to set up the mocking:

1. Create a "traveller" by calling time_machine.travel().
2. Make it mock time by calling its start() start method.
3. Schedule un-mocking by passing its stop() method to addCleanup().

This is verbose. Plus, it's all too easy to too easy to forget the cleanup step, leading to test pollution.

You can merge those three steps into one with Python 3.11's enterContext() method:

import datetime as dt
from unittest import TestCase

import time_machine


class ExampleTests(TestCase):
    def setUp(self):
        now = dt.datetime.now().replace(microsecond=0)

        self.traveller = self.enterContext(time_machine.travel(now))

        super().setUpClass()

    def test_microsecond(self):
        self.assertEqual(dt.datetime.now().microsecond, 0)

Much tidier.

enterContext() executes three steps:

1. Call the context manager's __enter__ method.
2. Call addClassCleanup to schedule the context manager's __exit__ to run at class cleanup.
3. Return whatever __enter__ returned.

The example works because the time machine traveller class also behaves as a context manager. It has an __enter__() method equivalent to start(), and an __exit__() method equivalent to stop(). Many, maybe most, mocking tools work as context managers, such as unittest.mock patchers and requests_mock.Mocker.

A second class-based example

Here's another example test case that mocks an environment variable with unittest.mock at the class-level:

import os
from unittest import mock
from unittest import TestCase

from example import get_text_colour


class ExampleTests(TestCase):
    @classmethod
    def setUpClass(cls):
        cls.text_colour = get_text_colour()

        patcher = mock.patch.dict(os.environ, {"TEXT_COLOUR": cls.text_colour})
        patcher.start()
        cls.addClassCleanup(patcher.stop)

        super().setUpClass()

    def test_env_var(self):
        self.assertEqual(os.environ["TEXT_COLOUR"], self.text_colour)

The class-level mocking applies once for the whole test case. This can save time, and also ensures the mocking applies during any test data creation.

You can simplify this test case similarly by using Python 3.11's new enterClassContext() method:

import os
from unittest import mock
from unittest import TestCase

from example import get_text_colour


class ExampleTests(TestCase):
    @classmethod
    def setUpClass(cls):
        cls.text_colour = get_text_colour()

        cls.enterClassContext(
            mock.patch.dict(os.environ, {"TEXT_COLOUR": cls.text_colour})
        )

        super().setUpClass()

    def test_env_var(self):
        self.assertEqual(os.environ["TEXT_COLOUR"], self.text_colour)

Once more, a couple lines saved.

All the methods

Python 3.11 added four different enter*Context() methods, for different test levels, plus async.

1. The module-level unittest.enterModuleContext(), which you can use from within a setUpModule() function:

   from unittest import enterModuleContext
   from unittest import TestCase
   
   
   def setUpModule():
       enterModuleContext(...)
   
   
   class ExampleTests(TestCase):
       ...
   

2. The class-level TestCase.enterClassContext(), which you can use within setUpClass() (and setUpTestData() in Django test cases):

   from unittest import TestCase
   
   
   class ExampleTests(TestCase):
       @classmethod
       def setUpClass(cls):
           cls.enterClassContext(...)
           super().setUpClass()
   
       ...
   

3. The test-level TestCase.enterContext(), which you can use within setUp() or test methods:

   from unittest import TestCase
   
   
   class ExampleTests(TestCase):
       def test_something(self):
           self.enterContext(...)
   
           ...
   

4. The async test-level IsolatedAsyncioTestCase.enterAsyncContext(), which you can use within asyncSetUp() or async tests to enter an asynchronous context manager:

   from unittest import IsolatedAsyncioTestCase
   
   
   class ExampleTests(IsolatedAsyncioTestCase):
       async def test_something(self):
           await self.enterAsyncContext(...)
   
           ...
   

That's the lot.

A backport for plain unittest projects

If you're not on Python 3.11 yet, you can backport this feature to use it to simplify your tests. Since it builds on the long-existing add*Cleanup() methods, it doesn't take that much code.

Below is the code to copy to backport the functions on a plain unittest project. Add or merge it into a base test file, from which you import TestCase throughout your project. If you are working on a Django project, see the next section for a more specific backport.

The code is sourced from unittest/case.py (and async_case.py) on the 3.11 branch of CPython. The type hints come from the corresponding files in typeshed. (I couldn't properly type _enter_context(), due to this Mypy issue.)

The if sys.version_info ... lines mean that when you upgrade Python, the native functions will be used. You can use the pypugrade tool to automatically remove old versioned blocks like these.

from collections.abc import Callable
from contextlib import AbstractAsyncContextManager
from contextlib import AbstractContextManager
import sys
from typing import Any
from typing import TypeVar
import unittest

_T = TypeVar("_T")

if sys.version_info < (3, 11):

    def _enter_context(cm: Any, addcleanup: Callable[..., None]) -> Any:
        # We look up the special methods on the type to match the with
        # statement.
        cls = type(cm)
        try:
            enter = cls.__enter__
            exit = cls.__exit__
        except AttributeError:
            raise TypeError(
                f"'{cls.__module__}.{cls.__qualname__}' object does "
                f"not support the context manager protocol"
            ) from None
        result = enter(cm)
        addcleanup(exit, cm, None, None, None)
        return result


if sys.version_info < (3, 11):

    def enterModuleContext(cm: AbstractContextManager[_T]) -> _T:
        result: _T = _enter_context(cm, unittest.addModuleCleanup)
        return result

else:
    enterModuleContext = unittest.enterModuleContext


class TestCase(unittest.TestCase):
    if sys.version_info < (3, 11):

        def enterContext(self, cm: AbstractContextManager[_T]) -> _T:
            result: _T = _enter_context(cm, self.addCleanup)
            return result

        @classmethod
        def enterClassContext(cls, cm: AbstractContextManager[_T]) -> _T:
            result: _T = _enter_context(cm, cls.addClassCleanup)
            return result


class IsolatedAsyncioTestCase(unittest.IsolatedAsyncioTestCase):
    if sys.version_info < (3, 11):

        async def enterAsyncContext(
            self,
            cm: AbstractAsyncContextManager[_T],
        ) -> _T:
            """Enters the supplied asynchronous context manager.
            If successful, also adds its __aexit__ method as a cleanup
            function and returns the result of the __aenter__ method.
            """
            # We look up the special methods on the type to match the with
            # statement.
            cls = type(cm)
            try:
                enter = cls.__aenter__
                exit = cls.__aexit__
            except AttributeError:
                raise TypeError(
                    f"'{cls.__module__}.{cls.__qualname__}' object does "
                    f"not support the asynchronous context manager protocol"
                ) from None
            result = await enter(cm)
            self.addAsyncCleanup(exit, cm, None, None, None)
            return result

Enjoy.

A backport for Django projects

When testing a Django project, you normally use Django's test case classes. I recommend adding your own subclasses of these classes in your projects, to allow customization, such as extra assertion methods or a custom test client.

Below is a version of the backport, excluding enterAsyncContext since Django doesn't have a dedicated async test case. This backport sets up a custom SimpleTestCase and its subclasses. If you already have a custom class, you can merge the changes.

from collections.abc import Callable
from contextlib import AbstractContextManager
import sys
import unittest
from typing import Any
from typing import TypeVar

from django import test


_T = TypeVar("_T")


def _enter_context(cm: Any, addcleanup: Callable[..., None]) -> Any:
    # We look up the special methods on the type to match the with
    # statement.
    cls = type(cm)
    try:
        enter = cls.__enter__
        exit = cls.__exit__
    except AttributeError:
        raise TypeError(
            f"'{cls.__module__}.{cls.__qualname__}' object does "
            f"not support the context manager protocol"
        ) from None
    result = enter(cm)
    addcleanup(exit, cm, None, None, None)
    return result


if sys.version_info < (3, 11):

    def enterModuleContext(cm: AbstractContextManager[_T]) -> _T:
        result: _T = _enter_context(cm, unittest.addModuleCleanup)
        return result

else:
    enterModuleContext = unittest.enterModuleContext


class SimpleTestCase(test.SimpleTestCase):
    if sys.version_info < (3, 11):

        def enterContext(self, cm: AbstractContextManager[_T]) -> _T:
            result: _T = _enter_context(cm, self.addCleanup)
            return result

        @classmethod
        def enterClassContext(cls, cm: AbstractContextManager[_T]) -> _T:
            result: _T = _enter_context(cm, cls.addClassCleanup)
            return result


class TestCase(test.TestCase, SimpleTestCase):
    pass


class TransactionTestCase(test.TransactionTestCase, SimpleTestCase):
    pass


class LiveServerTestCase(test.LiveServerTestCase, SimpleTestCase):
    pass

There you go!

Fin

Thanks to Serhiy Storchaka for contributing this feature, and Andrew Svetlov for reviewing. And thanks to Alex Waygood for contributing the type hints to typeshed, and Jelle Zijlstra for reviewing that.

May your tests be tidier,

-Adam