fidzu : python

The last two posts, on releases and canaries, were all about the Muru daemon running more than one release of an application side by side and moving traffic between them. When the application runs in a container, Muru deliberately does none of that. A container image is already a fixed release of an application, and starting new containers, moving traffic to them and putting the old ones back is the job of whatever runs the containers. Doing the same thing again inside the container would only get in its way.

So in a container Muru runs as muructl serve, which serves the single application held in the image. The edge, the controller, the host and its workers are all still there, as in development mode, but releases and traffic policies aren't. Instead, it concentrates on being a good main process for a container, which mostly means handling things people otherwise end up doing by hand when using gunicorn or uvicorn.

An image

The image for this post holds a small WSGI application with a few pages which misbehave in different ways, which I will get to later. The Dockerfile follows the pattern from the Muru documentation:

FROM python:3.13-slim AS build
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
RUN UV_TOOL_BIN_DIR=/usr/local/bin uv tool install muructl
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN muructl build --deployment --deps-only --no-runtime --target /app/.venv
COPY . .
RUN muructl build --deployment --target /app/.venv --runtime-dir /opt/muru

FROM python:3.13-slim
COPY --from=build /usr/local/bin/muructl /usr/local/bin/muructl
COPY --from=build /app /app
COPY --from=build /opt/muru /opt/muru
WORKDIR /app
RUN useradd --uid 10001 app && install -d -o 10001 -g 0 -m 0770 /run/muru
USER 10001
EXPOSE 8080 9090
CMD ["muructl", "serve"]

muructl is a single executable, so the final image copies just the binary. The two muructl build steps install the application's environment from uv.lock. The first installs only the dependencies, so that its layer can be reused until they change. Each says what it is running:

muru: building /app/.venv with uv sync --frozen --no-dev --no-build --compile-bytecode --no-install-project
...
muru: building /app/.venv with uv sync --frozen --no-dev --no-build --compile-bytecode
muru: built /app/.venv on Python 3.13.18 (cp313)
muru: fetched the runtime library muru-1.0.0-cp313-cp313-manylinux_2_28_aarch64.whl from PyPI into /opt/muru

The last line is the part which is specific to Muru. The runtime library which runs inside the workers has to match both the version of muructl and the interpreter, so it is fetched when the image is built. It is kept outside the application's environment, so the application doesn't need to list Muru as a dependency.

The important thing is that this is the only point at which anything is fetched or installed. When the container starts, muructl serve never builds an environment, downloads a Python or fetches the runtime library, and anything it runs gets UV_OFFLINE=1 and UV_PYTHON_DOWNLOADS=never in its environment. If something is missing, it stops with a message naming the build step which would have provided it. For example, pointing it at the wrong runtime directory gives:

muru: error: no runtime library for /app/.venv/bin/python (cp313): build the image with muructl build --runtime-dir /nowhere, or add muru 1.0.0 to the environment

That matters more than it might seem. It isn't unusual to find images where an entrypoint script pip installs something, or where a missing package is quietly downloaded at start. That works until the network is unavailable or the package index changes, at which point two containers started from the same image are no longer the same.

Telling it how many workers

Starting the image with limits of two CPUs and a gigabyte of memory, with no swap, on a Docker host with ten CPUs, I gave it the number of worker processes and the threads in each:

docker run --name shop --cpus 2 --memory 1g --memory-swap 1g \
  -e MURU_SERVE_PROCESSES=2 -e MURU_SERVE_THREADS=4 -p 8080:8080 -p 9090:9090 shop

The output says what it found and what it is running:

muru: container limits: 2 CPUs from cpu.max 200000 100000, memory.max 1.0GB with 128MB of headroom
...
muru.host[12]: preloaded app:application in 1 ms
muru.host[12]: app:application is served as WSGI
muru.host[12]: worker 0 started as pid 15
muru.host[12]: worker 1 started as pid 16
...
muru: serving app:application on http://0.0.0.0:8080 with 2 workers of 4 threads
...
muru: the host holds 34MB once the application is imported

These settings should always be supplied, chosen to suit the resources given to the container. If they are left out, Muru runs a single worker. Two processes for two CPUs suits an application whose requests mostly use the CPU, while one whose requests spend most of their time waiting on a database can use more threads in each. The memory has to fit as well. Two workers, along with the host which imported the application, leave plenty of room in a gigabyte, but an application whose workers each need 750MB would have to run a single process in a container this size, or be given a bigger container. The other examples in this post start the container with the same settings, which I leave out to keep the commands short.

Muru doesn't try to choose these numbers itself, because they can't be reliably worked out from inside the container. Asking Python how many CPUs it has, in the same container, gives:

docker run --rm --cpus 2 shop python3 -c "import os; print(os.cpu_count(), os.process_cpu_count())"

10 10

A CPU limit set with --cpus, or resources.limits.cpu in Kubernetes, is a CFS quota in the cgroup's cpu.max file, which neither of those functions reads. Gunicorn's default is a single worker unless WEB_CONCURRENCY says otherwise, and uvicorn's is the same. Gunicorn's documentation suggests (2 x CPU cores) + 1, and the image the Gunicorn project publishes uses that formula by default, working out the count with nproc, which also says 10 here, so it would start 21 workers in a container allowed two CPUs. On a large Kubernetes node, with a pod limited to a couple of CPUs, it is far worse. Even a count which got the CPUs right would say nothing about how much memory each worker needs, or whether the requests use much CPU at all.

Muru does read the limits from the cgroup, as the first line of its output showed, but only to check the settings against them. Asking for four workers in the same container still starts four, but with a warning:

muru: warning: 4 workers is more than the 2 CPUs cpu.max allows; see https://docs.muruhq.dev/en/latest/howto/containers.html#container-limits

It also warns if the workers wouldn't fit in memory once each holds its own copy of what the host imported. A forked Python process tends to end up with such a copy over time, as it gradually stops sharing memory with the process it was forked from.

Running short of memory

A Python process tends to hang on to memory it has used, whether through a cache, a leak or one unusually large request. In a container, the usual outcome of this is that the container runs out of memory and the kernel kills a process outright.

When the container has a memory limit, Muru gets in first. Every few seconds it checks how much memory is left under the limit. If that drops below a margin it keeps in reserve, which it calls the headroom, it takes the largest worker out of service and replaces it with a fresh one. The old worker finishes the requests it already has before it goes, so nobody sees an error, and its memory is given back. Nothing needs to be set for this beyond the container's memory limit.

To see it in action, I used a page of the application which holds on to another 10MB each time it is requested, and asked for it once a second:

muru.host[12]: worker 0 (pid 15) the container was short of memory, 102MB left under memory.max 1.0GB, less than the headroom of 128MB; it was the largest worker, uss 424MB; replacing it
muru.host[12]: worker 0 (pid 15) stops before its replacement starts: 101MB is left under the memory limit, less than its size
muru.host[12]: worker 0 started as pid 38
muru.host[12]: worker.recycled: worker 0 (pid 15) replaced by pid 38: the container was short of memory, 102MB left under memory.max 1.0GB, less than the headroom of 128MB; it was the largest worker, uss 424MB; most objects: cell 31, tuple 20, builtin_function_or_method 16, function 11, list 11
muru.host[12]: worker 0 (pid 38) is ready
...
muru.host[12]: worker 1 (pid 16) the container was short of memory, 100MB left under memory.max 1.0GB, less than the headroom of 128MB; it was the largest worker, uss 645MB; replacing it
muru.host[12]: worker 1 (pid 16) stops before its replacement starts: 100MB is left under the memory limit, less than its size
muru.host[12]: worker 1 started as pid 49
muru.host[12]: worker.recycled: worker 1 (pid 16) replaced by pid 49: the container was short of memory, 100MB left under memory.max 1.0GB, less than the headroom of 128MB; it was the largest worker, uss 645MB; most objects: cell 31, tuple 20, builtin_function_or_method 16, function 11, list 11
muru.host[12]: worker 1 (pid 49) is ready

Both workers had been growing at the same rate, so when the container first ran short, worker 0 was only just the larger of the two, at 424MB. Replacing it gave that memory back, which left room for worker 1 to keep growing. By the time the container ran short again, worker 1 was at 645MB, and it was replaced in turn.

Normally Muru starts the new worker before the old one leaves, so there is never a gap. Here less memory was left in the container than the old worker was using, so rather than risk two large workers at once, the old worker stopped first, finishing the requests it already had, and its replacement started once it had gone. The other worker carried on serving in the meantime, and all 150 requests succeeded.

Without this, the first anyone knows of a worker growing is when the kernel's out-of-memory killer picks a process in the container to kill, and that can just as easily be one in the middle of a perfectly good request. Neither gunicorn nor uvicorn keeps any watch on memory. The best either offers is restarting a worker after a set number of requests, however much memory it is using.

Muru calls this its memory guard. Running muructl status inside the container shows the limits, how much memory is used and left, and what the guard last did:

docker exec shop muructl status

...
cpu      2 CPUs from cpu.max
memory   memory.max 1.0GB, 402MB used, 622MB left, headroom 128MB
         workers 329MB, host 34MB
guard    watching; it last replaced shop@r1 worker 1 (pid 16) at 07:31:18 UTC

The guard looks at the container as a whole. If a single worker should never grow beyond a certain size, however much room is left, it can also be given a hard memory limit, such as MURU_SERVE_MEMORY=hard_limit=300MB. A worker which goes over that limit is replaced in the same way. The value is chosen to suit the application, and should be low enough that all the workers together still fit in the container.

Memory is only checked every five seconds, so if it grows by more than the headroom between checks, the out-of-memory killer can still strike first. A request which allocated a gigabyte all at once showed what happens then:

502 Bad Gateway
reason: worker_crashed
request id: 01M6TF11TNSGS7R9AW4G704V4X

The worker sent no response head.

muru.host[12]: worker.oom_killed: worker 0 (pid 15) was killed by the kernel for memory, restarting
muru.host[12]: worker 0 started as pid 47
muru.host[12]: worker 0 (pid 47) is ready

Each worker raises its own oom_score_adj as it starts, which makes the workers the processes the kernel picks first. So it was the worker which was killed, rather than the host or muructl, and the host started a new one. The kill is also reported for what it was, as a worker.oom_killed event and in the muru_worker_exits_total metric with the reason oom_killed, rather than leaving a worker which mysteriously went away.

Probes on their own port

The output when it started also included this line:

muru: admin port on http://0.0.0.0:9090/

Port 9090 is separate from the application's port and answers the questions a container platform asks. /livez says whether the process is alive, /readyz whether enough workers are ready to take requests, and /startupz whether the application has finished loading. /metrics has the metrics in Prometheus format.

Gunicorn and uvicorn have nothing like this. The usual answer is to add a /health view to the application, which means the health check is served by the same workers as everything else. When the application is slow, perhaps because its database is, every worker is busy. The health check then queues behind real requests and times out, and the platform restarts containers which were doing nothing wrong, just when they are needed most.

To check that Muru doesn't suffer from this, I sent twelve requests to a page which takes six seconds to answer, more than the two workers with four threads each could handle at once, and then queried the admin port:

livez 200 0.001336s
readyz 200 0.001337s

The metrics confirmed every slot was busy, with requests queuing:

muru_queue_depth{app="shop",release="r1",group="default"} 4
muru_slots_busy{app="shop",release="r1",group="default",side="threads"} 8
muru_slots_total{app="shop",release="r1",group="default",side="threads"} 8

The probes are answered by muructl itself, from what it knows about the host and its workers, without a request having to go through a worker. A container whose workers are all busy is still alive, so it isn't killed for being busy. Having a port of their own also keeps the probes and metrics out of reach of the application's Ingress or load balancer.

An image without curl, such as one based on the slim Python images, can use muructl itself for a health check:

HEALTHCHECK --interval=5s CMD ["muructl", "probe", "ready"]

Process 1

Looking at the processes inside the container (by reading /proc, since the slim image has no ps), each of Muru's processes has set a title saying what it is:

    1     0 S  muructl serve
    7     1 S  muru serve shop
   11     7 S  muru relay shop@r1
   12     7 S  muru host shop@r1
   15    12 S  muru worker shop@r1 default/0
   16    12 S  muru worker shop@r1 default/1

Process 1 is muructl acting as an init, in the way tini or docker run --init would. It forwards signals to the real muructl serve, which is its child, and collects any process in the container which exits, so nothing is left as a zombie. Process 1 in a container has to do this, because orphaned processes are handed to it, and if it never collects them, zombies fill up the process table. Below muructl serve are the host and its two workers, and a log relay which passes on what they write.

The application has a page which starts a report by running a shell command, which leaves a process running in the background. That happens in real applications more often than anyone would like. Just after I requested the page three times, muructl status --processes --all, where --all adds the processes Muru didn't start itself, showed them:

docker exec shop muructl status --processes --all

edge  3.0 req/s  0 err/s  p50 2.1 ms  p95 4.6 ms  in flight 0  ws 0  conns 0  tls 0/s  in 0B/s  out 53B/s

PID   ROLE           WHAT                         UP    CPU      RSS   REQ/S   BUSY   QUEUE      P95   GIL/LAG
7     serve          controller and edge, :8080   5s   2.3%   19.5MB
12    └─ host        app:application (r1)         5s   0.0%   34.7MB     3.0    0/8       0   4.6 ms
11       ├─ relay    output to muructl            5s   0.0%   11.4MB
15       ├─ worker   worker 0                     5s   0.0%   28.7MB     2.0    0/4           4.8 ms
16       ├─ worker   worker 1                     5s   0.0%   28.9MB     1.0    0/4           2.4 ms
54       ├─ other    sleep 1                      1s           1.5MB
56       ├─ other    sleep 1                      1s           1.5MB
58       └─ other    sleep 1                      1s           1.5MB

The orphaned sleep processes were taken over by the host, which collects any process its workers leave behind, and a couple of seconds later they were gone, with no zombies. Gunicorn does collect any child that exits, including ones it didn't start. Uvicorn doesn't, so when run as process 1 it leaves a zombie behind for each one. Neither project's documentation says anything about tini or --init. An image which uses its own init can turn off Muru's by setting MURU_NO_INIT=1.

Stopping in order

When docker stop or Kubernetes wants a container gone, it sends SIGTERM, waits, and then sends SIGKILL. Docker waits ten seconds and Kubernetes thirty by default. For a server, what happens in between decides whether anyone sees an error.

I started four requests to the page which takes three seconds, and half a second later ran docker stop, while checking /readyz every 300 milliseconds:

readyz 200
readyz 503
readyz 503
readyz 503
readyz 503

The four requests all completed:

 200 3.004430s
 200 3.004411s
 200 3.004890s
 200 3.004973s

and docker stop returned after 2.76 seconds, with the container's exit status 0. On SIGTERM, readiness starts failing at once, so nothing new is sent to the container. The listener then closes, the requests in flight finish, and the workers stop. All of that has to happen within a budget, eight seconds unless --shutdown-budget changes it. That is less than the ten seconds docker stop allows, so anything still running at the end is stopped by Muru rather than killed by Docker.

Compare that with the defaults elsewhere. Gunicorn's graceful_timeout is 30 seconds, so under docker stop a slow request is cut off by SIGKILL at ten seconds and gunicorn never gets to finish shutting down. Uvicorn has no graceful shutdown timeout by default, so it waits as long as it takes, which under Docker means until the SIGKILL arrives. With either, the fix is to remember to set a timeout which fits within the platform's.

Kubernetes adds a wrinkle, as a pod can keep receiving new connections for a few seconds after SIGTERM, while the endpoints catch up. That is what --shutdown-delay is for, and I will come back to it in the next post.

A read-only root and any user

A security conscious platform will run the container with a read-only root filesystem, and OpenShift will run it as an arbitrary user ID in group 0, with no entry for it in /etc/passwd. Everything muructl serve writes at run time goes under one directory, /run/muru. With a read-only root and nothing mounted there, it stops at once and says what it needs:

docker run --rm --read-only --user 123456:0 shop

muru: error: the run directory /run/muru cannot be written (Read-only file system (os error 30)); with a read-only root filesystem give it a writable volume, a tmpfs or an emptyDir mounted at /run/muru, or name another with --run-dir

Giving it what it asks for:

docker run --read-only --tmpfs /run/muru --tmpfs /tmp --user 123456:0 ... -p 8080:8080 -p 9090:9090 shop

it runs as normal under a user ID which doesn't exist in the image:

hello from pid 15

Gunicorn in the same situation fails at start, with No usable temporary directory found, as it writes a heartbeat file for each worker to the temporary directory. That can be fixed with a tmpfs on /tmp, or by pointing --worker-tmp-dir at /dev/shm, but you have to know to do it.

Settings in layers

Every setting of muructl serve can be given as a command line flag. An image can also set defaults in the project's pyproject.toml, under [tool.muru.serve], and a deployment can override any of them with an environment variable named after the flag. --print-config shows the settings in force and where each came from, and works inside a running container too:

docker run --name shop -e MURU_SERVE_PROCESSES=3 -e MURU_SERVE_LOG_FORMAT=json ... shop
docker exec shop muructl serve --print-config

processes = 3                     # MURU_SERVE_PROCESSES
...
log_format = "json"              # MURU_SERVE_LOG_FORMAT

--config, or MURU_SERVE_CONFIG, names a settings file to read instead of the one in the image, such as a file mounted from a ConfigMap.

Logs a collector can use

With the JSON log format, every line is a JSON object. When the environment has POD_NAME, POD_NAMESPACE and NODE_NAME, which the Kubernetes Downward API can provide, each line also says where it came from. I set those by hand for this run. More useful is what happens to a traceback. The application has a page which raises an exception, and the error shows up as a single record, with the whole traceback in its exception field, alongside the request, release and worker it happened in:

{"time":"2026-11-07T07:37:52.296Z","kind":"app","source":"muru.worker[0]","app":"shop","exception":"Traceback (most recent call last):\n  File \"/app/app.py\", line 25, in application\n    raise RuntimeError(\"something went wrong\")\nRuntimeError: something went wrong","group":"default","level":"ERROR","logger":"muru","message":"the application raised an exception","release":"r1","request_id":"01M6TF2DXFEHAEQZN1C1JEXGBE","worker":0,"pod":"shop-7d4b9c-x2k8p","namespace":"shop","node":"worker-2"}

Anyone who has had to piece a Python traceback back together from a log system which treated each line as a separate entry will appreciate that.

Inside the container

Finally, the commands from looking inside the workers work inside the container through docker exec, talking to muructl serve over its admin socket. With a slow request under way:

docker exec shop muructl debug stacks

muru-app-0 in worker 0:
  serving GET /slow?seconds=4 for 1.2 s (request 01M6TF2SRXPXTHSAPPDG6MVNJQ)
  File "/opt/muru/muru/_pool.py", line 30, in serve_jobs
    serve()
  File "/app/app.py", line 15, in application
    time.sleep(seconds)

What's next

None of this needs anything outside the image. There is no init to add, no health check view to write, no timeouts to line up with the platform's and no temporary directory to remember. What it does need to be told is how many workers of how many threads to run, chosen to fit the resources the container is given.

Most containers end up on Kubernetes though, and there are a few more things there where Muru can work with what the platform provides rather than around it, such as the shutdown delay in place of the usual preStop hook, autoscaling on how busy the workers really are rather than on CPU, and running migrations as a Job from the new image. That is the subject of the next post, working with Kubernetes, not around it.

14 Nov 2026 12:00am GMT

This post has been updated since it was originally written. The most recent update was in November 2026; a summary of what changed is at the end.

This is the first in hopefully a series of posts I intend to write about how to build/manage/deploy/etc. Python applications in as boring a way as possible.

So before I go any further, I want to be absolutely clear on what I mean by "boring": I don't mean "reliable" or "bug-free" or "no incidents". While there is some overlap, and some of the things I'll be recommending can help to reduce bugs, I also want to be clear: there will be bugs. There will be incidents where a feature or maybe an entire service is down. "Boring", to me, is about the sources of those incidents. It's difficult enough to manage your own code and the bugs and other problems that will inevitably pop up in it from time to time; you don't want to compound that by having bugs or other surprises coming from the tools and processes you use to build, manage, and deploy it. So when I call something "boring", I mean it's unlikely to add another source of bugs and nasty surprises that cause a pager to go off at 2AM; the pager will, of course, eventually go off at 2AM, but when it does you'll be able to feel reasonably confident (if that's the right word for such a situation) that the source of it was something in your own code that you can diagnose and fix.

And so I'm planning several posts exploring different aspects of making Python development boring (in this sense). But for this first installment, I'll be talking about one of the internet's favorite topics: managing dependencies.

If you're just interested in the end result, the process I recommend is pretty simple and can be found in the section titled "Putting it all together". The rest of this post exists to explain, in as much detail as I can manage, all the things going on behind those simple-looking recommendations, and why I make those specific recommendations.

Python packaging: a brief overview

I've written about this before, but to make sure it's clear I want to do a quick refresher.

As I see it, there are three basic aspects to "packaging", and one of the difficult things about "Python packaging" discussions is lack of clarity around which are being discussed. They are:

1. Given some working code, define and produce from it a distributable artifact.
2. Given that someone has produced a distributable artifact, use it to cause the corresponding code to appear, in working form, somewhere else.
3. Given the existence of potentially many separate projects or versions of projects each with their own independent and potentially-conflicting sets of dependencies, make it possible to work on and run more than one at a time, on the same system.

Most of the actual remaining complexity in "Python packaging" today is concentrated in item (1), and even then only really kicks in if you're packaging extensions written in non-Python languages.

A lot of the perceived complexity of "Python packaging" is due to its long history, evolving expectations over the course of that history, and the resulting pile of strata of different things people have come up with to try to suit the expectations of different eras.

Much of the public grumbling and complaining about "Python packaging" these days is due to the lack of a single, standard first-party, all-in-one high-level tool to handle all aspects of packaging, and the proliferation of third-party tools to try to fill that niche.

Finally, most of the meaningful work in Python packaging, for quite a while now, has been the unglamorous, underappreciated slog of people on mailing lists and discussion forums patiently working out what Python packaging really should look like and thoroughly specifying/standardizing it, but with the added thrill of all that having to occur in a way that doesn't break things for the vast number of people relying on Python and its huge package ecosystem for their day-to-day work.

To the extent that we now have a bunch of shiny fancy flashy third-party all-in-one packaging tools, it's because of that heroic unsung work that's been quietly going on for years. The fancy new tools get to be so fancy and flashy in large part because modern Python packaging has such strong foundations (and also because, unlike the pre-existing tools, the new ones don't have to maintain compatibility with literal decades of historical features and configuration options, all of which have been relied on, at some point, by someone).

If I went into detail on everything I'd wind up with several posts' worth of material, so let me just pick out a few things that I think are the most crucial underpinnings of modern Python packaging:

• The "wheel" (.whl) package format (PEP 427), which turned Python package installation into a simple matter of unpacking an archive and moving files to their destinations. Any extension modules in other languages are already pre-compiled inside the package, and there is no install-time build step or scripting hook.
• The development of virtual environments, and the addition of a basic implementation to the Python standard library (as the venv module, PEP 405). Virtual environments solve the "how do I have a bunch of projects at once" issue: they're essentially lightweight Python workspaces that you can install packages into, isolated from each other and from the rest of the host system.
• The standardization of the pyproject.toml file as the place to record packaging configuration and metadata (PEP 621). This allowed package authors to finally stop writing setup.py scripts (which were the main way to define Python packages for two decades) and start keeping packaging information in a single standard static file that can be read without being executed as code.
• The standardization of the pylock.toml lock file format (PEP 751). People had been trying to figure out a standard Python lock file format for a long time; every third-party tool had its own take on this, and people (like me) who tried to stick to first-party tools had to rely on a pile of hacks built on top of a feature (pip requirements files) that was never intended to support this functionality. Now there's one single clear answer, and a lot of packaging tools have support for it.

Boring Python packaging: a philosophy

When I wrote the first version of this post in 2022, I focused on defining a workflow that relied as much as possible on standard first-party (as in, included with Python itself) tooling. I recommended exactly one third-party tool, and only as an optional convenience; the functionality it provided could also be obtained from chaining together invocations of first-party tooling.

That focus on first-party tooling was deliberate. As I wrote at the time:

[The] core default tools I've mentioned have all been around and stable for a long time, in software terms, at least - they're all over a decade old, and their end-user interfaces evolve incredibly slowly, when they evolve at all. They're reliable. They're well-understood. That's exactly what I want and exactly what I recommend.

This is in line with what I said at the start of this post about wanting things to be boring. And my stance hasn't changed, but advances in the foundational standards of Python packaging do let me be a bit more flexible. So although I'm still going to be focused on first-party tools I'll now point out a couple of places where I think it's safe to use some of the hip and trendy new third-party tools if you want the added conveniences they offer.

Also, I should note here that my use case for Python is deploying networked (primarily web) services, on servers. If you primarily use Python for, say, machine learning or data science, you are likely to already be a happy user of a completely different world of tools optimized for those use cases (in particular, for the much greater amount of non-Python code wrapped by/accessed through Python in those fields and for their different patterns of code reuse and sharing). Please continue using them! They suit your use case really well, and replacing them with my preferred tools and workflow would probably be a regression for you.

Now, let's get started.

Lock it up

The standard Python package manager is pip. And my eventual goal is to invoke pip install to, well, do what it says: install a bunch of Python packages. But how to tell pip which packages to install? There are a couple of options:

• Passing a list of package names on the command line: pip install package1 package2 ...
• Passing a file containing a list of packages: pip install -r filename

Historically, the second option was referred to as a "requirements file", and by convention was generally named requirements.txt. Now that PEP 751 has defined a standard Python lock file format, and now that pip supports producing and consuming that standard lock file format, that's the format I recommend.

This prompts a question, though: since dependencies could already be specified as part of the metadata in pyproject.toml, why did we need another file format to record them? By way of an answer, consider a web application written using the Django web framework. You might think it's simple enough to just declare Django as a dependency in your pyproject.toml file, like so:

[project]
dependencies = ["Django"]

But if I create a new virtual environment and run pip install Django inside it, three packages get installed:

$ pip install Django
Collecting Django
  Using cached django-6.1-py3-none-any.whl.metadata (3.9 kB)
Collecting asgiref>=3.9.1 (from Django)
  Using cached asgiref-3.12.1-py3-none-any.whl.metadata (9.4 kB)
Collecting sqlparse>=0.5.0 (from Django)
  Using cached sqlparse-0.6.0-py3-none-any.whl.metadata (6.0 kB)
Using cached django-6.1-py3-none-any.whl (8.4 MB)
Using cached asgiref-3.12.1-py3-none-any.whl (25 kB)
Using cached sqlparse-0.6.0-py3-none-any.whl (50 kB)
Installing collected packages: sqlparse, asgiref, Django
Successfully installed Django-6.1 asgiref-3.12.1 sqlparse-0.6.0

And which versions of those packages I get will depend on when I run pip install. Right now I get Django 6.1, but Django does frequent bug fix releases, so if I re-run that pip install in the future I'll get Django 6.1.1, and further in the future I'll get Django 6.1.2, and so on. If I test my application locally with the set of packages I installed today, and then deploy it to a server later using pip install to fetch dependencies, the deployment will very likely get a different set of packages than I used during development, which is a potential source of problems.

So the key difference is that the pyproject.toml file typically specifies a list of direct dependencies, often with broad version constraints or none at all, while a lock file specifies an entire environment, with all direct and transitive dependencies resolved and pinned to exact versions along with the expected checksums of the packages, to be reproduced as exactly as possible.

There are two main approaches to generating a lock file:

• Create a virtual environment, use pip to install all the direct dependencies you care about, then run pip freeze to export the full resolved package list, and pip lock on that list to produce a pylock.toml lock file, or
• Use a third-party all-in-one packaging frontend which manages the local virtual environment and package syncing and lock file for you.

Managing your local environment and producing your lock file is one of the areas where I think it's generally OK to use a third-party tool. If something goes wrong with it, it might temporarily mess with your ability to get work done on your own computer, but that won't break your actual deployed application.

The hip and trendy third-party all-in-one tool these days is uv, which has its own tool-specific lock file format (uv.lock), but also has a uv export command which can export to the standard pylock.toml format.

My own personal preference, and the tool I use on my own projects, is PDM, which has a similar feature set to uv and can export to pylock.toml from its own lock format, but also can just manage a project directly in a pylock.toml file.

As long as the tool you choose is one you and your team are all happy with, I think you can pick anything that can produce the standard lock file format and be OK. Also, you should probably automate the process of exporting the lock file, to ensure developers don't have to remember how to manually do it. I personally like to include a Makefile in most projects, with a target specifically to refresh the lock file (make lock, for example), but the idea is portable to a lot of task-automation tools, so it's easy to make it work with whatever your team likes to use.

I also want to point out here that if you're deploying services written in Python, you can build them as distributable packages and then install those packages to automatically pull their dependencies, rather than using a lock file. I just don't recommend this, for a few reasons:

• The main convenience of distributable packages is that you can re-use them across multiple projects/services/etc. For example, you might have an auth setup you want to use in all your services, and build it as a package that can be installed by those services. When the thing you're deploying is the service, this is much less important.
• It was already extremely common in 2022 when I first wrote this post, and is even more common in 2026 as I rewrite it, that deployment of a Python service will consist of building some type of container and then pushing it to some type of container runtime/orchestration system, and that the manifest which defines that container will be version-controlled along with the service's code. In which case it doesn't make much sense to build a package from the code and then pull it into the container to be installed; just copy the source tree into the container, rather than adding extra steps for no benefit.
• Copying the source tree as-is to its destination without an intermediate trip through packaging avoids some potential issues, primarily around imports: a common error made by people building packages for the first time is relying-often without realizing it!-on the fact that in Python the current working directory is (usually) implicitly on the import path. As a result, they end up building packages that only "work" in the original local development environment with a specific working directory (the solution to this, if you're curious, is to force your local development workflow to depend on the installability of your package).
• Ultimately, what you want is to take a known-good, working environment from a developer's machine, and reproduce exactly that environment elsewhere. And as I already mentioned above, that's exactly what lock files are for.

Getting testy

In addition to direct dependencies to run an application, it's fairly common to have additional dependencies needed to run a test suite, linters, or other development and quality-control tasks. But there's no need for a production deployment to have all those extra dependencies installed, and in general the less stuff you put in your production environment the fewer things you have that can go wrong, so I generally like to install those only when the test suite (or other task requiring extra dependencies) is actually going to be run.

The standard solution for this is dependency groups, which were first defined for library-style package dependencies but are also part of the lock file specification. And as of pip version 26.3, selecting groups when installing from a lock file is now supported. Popular third-party package-manager frontends also seem to have broad support now, in late 2026, for specifying dependency groups and exporting them to a standard lock file, so you can manage multiple logical groups of dependencies with ease.

The general consensus among the third-party tools seems to be to automatically support a group named dev for dependencies used only in development/testing, so my recommendation is to go with that convention rather than manually creating your own group with a different name.

Here's an example with PDM of how you could set up a project to use Django as the web framework in production, and the pytest-django library for testing:

pdm add Django
pdm add --dev pytest-django

And then you can export to a standard lock file like so:

pdm export --format pylock -o pylock.toml

The lock file format is pretty readable, so you can look inside and see that it specifies multiple groups (one named default and one named dev) and tracks which dependencies belong to which group(s). The exact commands for adding packages to and exporting a "dev" group with other tools will vary, but should generally look pretty similar to what's above.

Using the right invocations

Before putting all this together, it's worth covering one more detail: how to invoke the tools. This may seem a bit silly, since they all provide executable entry points. Just run pip install, right?

But there's a potential issue here: in a moment I'm going to recommend creating a Python virtual environment, which opens up the possibility of multiple Python environments coexisting on the same machine. This is certainly a useful feature, but it comes with a new concern, which is how to ensure you're using and running things in the environment you expect to be using.

For example, one way to run into trouble with multiple Python environments is to have one environment's package directory be first on your $PYTHONPATH (which controls Python import locations) while a different one's bin/ directory is first on your general $PATH (which is where executable scripts will be found). If you just run pip install you'll get the second one's instance of pip, which may not be at all what you want.

This is why the official Python packaging guides, and official documentation for tools like pip, always use a different approach: they tell you to run python -m pip instead of pip, and python -m venv instead of a standalone script like virtualenv. The -m flag allows a Python module to be run directly like a script (as long as it's been written to provide an entry point for this, which pip and venv both have), and can prevent a lot of potential hard-to-debug issues that can accidentally result from things like manually hacking around with paths, by ensuring you're getting the version of pip or venv that actually goes with the Python environment you invoked.

And to guarantee that you get the Python environment you actually want, you can specify the full path. For example, many base Python container images will put the Python interpreter in /usr/local/bin, so invoking /usr/local/bin/python instead of just python ensures you get that interpreter.

There's also one more thing I like to do that isn't (currently) in some of the popular guides, and that's invoking Python with the -I flag. This runs Python in "isolated" mode, which removes some of the automatic implicit directories from the import path and also ignores environment variables like PYTHONPATH. Once again, this reduces the number of things that can go wrong (for example, with -I the current working directory won't be implicitly added to the import path, so you can't accidentally depend on it being importable). So whenever you can invoke Python in isolated mode, I generally recommend that you do.

Putting it all together

That was a lot of explanation for what ends up being, ultimately, a pretty simple process to actually use. So now let's finally take a look at it.

First things first: always work in, and always deploy in, a virtual environment. Even if you think you don't need one. In fact, especially if you don't think you need one. This may seem like strange advice if you're already using a container or other virtual machine, since you're probably thinking that provides all the isolation you'll need. But virtual environments don't cost you anything to create, and if you ever do end up with multiple Python interpreters-which is easy to accidentally do, if you use a base system with a purpose-built Python and then install a system package that turns out to depend on the distro's own Python, for example-using one from the start will help to save you from potentially having a pager go off one night when suddenly the wrong Python is being invoked.

Virtual environments also provide a useful "Python environment" artifact that can be copied between stages of a container build, and many tools automatically recognize and can work with them. And ever since PEP 668 started to be adopted by operating-system vendors, many "system" Python installations will require you to use a virtual environment in order to install packages with pip (and only allow you to interact with the "system" Python environment through the system's own package manager). So use a virtual environment, even if you're working in a container or other VM.

When you're working in a local directory on your own computer, you can invoke the correct Python version with -m venv to create a virtual environment and install things into it, but again this is an area where third-party package tools can be useful, because they'll manage this for you automatically. If you do create your local virtual environment manually, you should almost certainly put it inside your project in a subdirectory named .venv, because that's already the common unofficial convention supported by a lot of IDEs and other tools, and likely to become the official standard convention once PEP 832 finalizes.

In a container, I think the choice that's most in line with Linux filesystem hierarchy standards is to put a Python virtual environment somewhere under /opt. Generally I like to create an /opt/venvs in a base "build" stage, which can populate multiple virtual environments with different package sets to be copied into later stages for tasks like CI or deployment.

So, taking as an example current Debian stable (as I write this, Debian 13 "trixie") and the most recent Python (as I write this, Python 3.15), let's see what this actually looks like. This example assumes you're producing a standard Python lock file as described above:

# syntax=docker/dockerfile:1

ARG DISTRO="slim-trixie"
ARG PYTHON_VERSION="3.15"

FROM python:${PYTHON_VERSION}-${DISTRO} AS dependencies

# Turn off pip upgrade reminders, since we're about to upgrade it
# anyway, and also specify the root cache directory pip under which
# will create its package cache.
ENV PIP_DISABLE_VERSION_CHECK=1 \
    XDG_CACHE_HOME=/var/cache

RUN <<END
mkdir -p /opt/venvs/deploy
mkdir -p /opt/venvs/tests
# Ensure the pip cache directory exists.
mkdir -p ${XDG_CACHE_HOME}/pip
/usr/local/bin/python -Im venv /opt/venvs/deploy
/usr/local/bin/python -Im venv /opt/venvs/tests
END

COPY pylock.toml  /opt/venvs

The next step is to ensure pip is present at the latest version in each virtual environment, and invoke it to install packages.

At this point you might be wondering: if I was willing to recommend using a third-party packaging frontend earlier for local development use and producing lock files, why am I insisting on pip here? And the answer is this is one of the places where I think using boring standard default tools really matters. As I said above, if something goes wrong with your fancy third-party package tool in a local development setup, it just causes a problem for that local development setup. If it goes wrong in your production build/deploy process, it breaks all your builds and deployments until you resolve the issue. I want to avoid that, so I stick to pip here, because even if it's not as fancy as some of the newer third-party tools, in my experience it's much less likely to be a source of unexpected issues than the newer third-party alternatives.

Also, I'm not going to go into a full explanation of an ideal Python application Dockerfile here, but the example below does at least use a Docker cache mount to store the downloaded packages, so that subsequent rebuilds only have to download new or changed packages, and can pull everything else from cache.

The pip invocation adds a few flags, mostly to ensure maximum safety and reproducibility:

• --no-deps tells pip not to try to resolve the dependency tree for the given list of packages. Because this setup is installing from a lock file, that tree is already fully resolved, so all pip should need to do is fetch and install the packages listed in the lock file. This speeds up the installation process, and also prevents any unexpected packages from suddenly being requested due to re-running dependency resolution.
• --only-binary :all: is the only one you might need to skip, but try using it first and only remove it if you know it doesn't work. This tells pip to only use "wheel" (.whl) format packages; the wheel format has been around for long enough most of your dependencies are likely to provide it, but if for some reason something you depend on doesn't, you can change to --prefer-binary (which will use .whl packages whenever available) instead. Just be aware that any dependency which doesn't provide a .whl will likely have to fall back to executing a setup.py script in order to install, which is slower and comes with some risks.
• --require-hashes should be redundant, since this setup uses a lock file which will include the package hashes, but it's useful to have as a safety net: if you ever accidentally produce a file that doesn't contain package hashes, using this flag will cause installation to fail with an error message saying the hashes couldn't be found.

If you're using GitHub Actions as your CI/CD, Python core developer Brett Cannon has written a reusable action which automatically invokes pip with these flags.

But continuing with the example of a Dockerfile containing a "dependencies" stage, here's what it looks like. Note the difference between the two pip install invocations, though. Asking pip to install from a lock file with -r pylock.toml will only install the default dependency group, and for the deployment dependencies, that's the right behavior, but for the test dependencies it isn't, so the test dependencies use the --group flag to explicitly select the set of groups to install (both default and dev, in this case).

# Install test dependencies first since they're likely to be a
# superset of the deployment dependencies and will populate the
# package cache.
RUN --mount=type=cache,sharing=locked,target=${XDG_CACHE_HOME}/pip,id=pip <<END
/opt/venvs/tests/bin/python -Im pip install --upgrade pip
/opt/venvs/tests/bin/python -Im pip install \
    --no-deps \
    --only-binary :all: \
    --require-hashes \
    --group /opt/venvs/pylock.toml:default \
    --group /opt/venvs/pylock.toml:dev
END

RUN --mount=type=cache,sharing=locked,target=${XDG_CACHE_HOME}/pip,id=pip <<END
/opt/venvs/deploy/bin/python -Im pip install --upgrade pip
/opt/venvs/deploy/bin/python -Im pip install \
    --no-deps \
    --only-binary :all: \
    --require-hashes \
    -r /opt/venvs/pylock.toml
END

Now you can pull the installed packages into later stages of your build by copying the virtual environment. For example, in a deployment stage:

FROM python:${PYTHON_VERSION}-${DISTRO} AS deploy

COPY --from=dependencies /opt/venvs/deploy /opt/venvs/deploy

# Do the rest of your deployment stage setup here: copying in
# your application source code, setting the entry point, etc.

Once again, I'm not going to go into a ton of detail here on other things that make a good Python Dockerfile. If you want to learn more about that, I'd recommend reading Itamar Turner-Trauring's articles on Python and Docker or Hynek Schlawack's guide to Python and Docker. Both are regularly updated, and they'll both give you a solid education in how to containerize Python.

Be up-to-date, but stay cool

The only thing still missing here is how to handle updates as new versions of your dependencies are released. For security updates this is crucial, but it's also important as a general practice. If you make dependency updates a regular, routine part of your development process that's easy for developers to do, then it'll also be routine and easy to apply critical updates when they appear. And by applying updates as they come, a couple dependencies at a time, you avoid building up a huge backlog of deferred updates that will make a critical issue even more difficult to address when one inevitably occurs.

And luckily, all of the popular third-party package management tools have straightforward commands you can run to identify and update outdated packages. But there is one wrinkle: you probably don't want to always eagerly accept most routine dependency updates as soon as they're released, since that can expose you to potential security issues if a dependency has been compromised but nobody's noticed it yet. The solution to this is dependency cooldowns: when upgrading, only consider packages which have had at least a bit of time for security researchers and scanners to take a look. The typical recommended window is three days; anything newer than that should be excluded from your package updates unless it's a critical security patch.

At this point, most popular tools in the Python ecosystem support dependency cooldowns. For example:

• If you're using uv you can set the exclude-newer configuration option, like so: exclude-newer = "3 days"
• PDM also supports this, calls the configuration option strategy.exclude-newer, and you could set it to "3d" to get the same three-day cooldown rule for new releases.
• And pip, as of version 26.1, supports the --uploaded-prior-to flag with a relative duration. Its relative syntax would be --uploaded-prior-to "P3D" for a three-day cooldown.

You also can configure third-party dependency update bots to apply updates for you as they appear. GitHub's Dependabot automatically applies a three-day cooldown for everything except critical security updates, and so does the Renovate update bot.

Whether to have a dependency update be a periodic manual task for a developer or something to automate via a tool like Dependabot is up to you (though I personally prefer automation). The important thing is you do it and commit to it, on a cadence that lets you take in updates only one or a few at a time, rather than falling behind and needing to update dozens of packages all in one go.

And that's a wrap

As promised, that was a lot of words for what's really a pretty simple set of recommendations. Unfortunately there's a lot of complexity-necessary complexity, of a sort that pops up in any software packaging ecosystem-lurking in this topic, and explaining all that is what drives up the word count.

But hopefully you now understand how to do "boring" Python dependency management, relying primarily on standard first-party tooling. Even if you don't want to adopt my recommendations, I'd like to think that learning what they are and why I make them is helpful to you. For me, these recommendations are the result of over a decade of work, across multiple employers, to develop a dependency management workflow that keeps things up-to-date with minimal risk of causing pagers to go off.

Meanwhile I've got some ideas for further "boring Python" articles, but those will have to wait for another day.

Changelog

The first version of this post was written and published in May 2022. A major revision was written during late August 2026 and published in early September 2026. The key changes between those versions were:

• The 2022 version recommended a workflow using a pip requirements file with a fully-resolved, version-pinned and hashed package set. This was the closest it was possible to get, using standard tools in 2022, to a real lock file. Now, in late 2026, a true Python lock file format (PEP 751) has been standardized and has support (at least for basic creating and consuming of lock files) across the packaging ecosystem, so I recommend using it.
• The 2022 version suggested using the pip-compile script from the pip-tools project to make it easier to produce the requirements file. Since I'm now recommending the use of the standardized Python lock file format, and multiple tools can produce that format, it's no longer needed.
• As of 2026, my recommendation on third-party packaging tools is that you can use an all-in-one tool like uv or pdm as a frontend for local development, to get the convenience of a single tool with a high-level interface managing the local virtual environment, adding/removing/updating packages, and producing the lock files, but just as in 2022 I still recommend using pip to install packages for production deployments.
• The 2022 version recommended the python -m invocation pattern, and suggested the --only-binary :all: option for pip. The 2026 version expands on this by recommending explicit invocation of the desired Python interpreter, the use of Python in "isolated mode" whenever possible, and a more comprehensive set of pip command-line flags to maximize safety and reproducibility of package installs.
• The 2022 version had a section on applying package updates which mentioned Dependabot and touched on the importance of applying updates regularly. As of 2026, that section is expanded with discussion of the importance of dependency cooldowns, and gives examples of how to configure popular third-party package-management tools.

A second update was written and published in November 2026. This was a smaller change, in response to pip 26.3 adding support for selecting and installing specific dependency groups from a lock file. Previously, the lack of this support led me to recommend the use of multiple lock files to track separate deployment/test/etc. dependencies; now, with dependency groups supported in pip, I can recommend the simpler approach of a single lock file containing multiple dependency groups, and instruct pip to install the correct group(s) for a given use case.

13 Nov 2026 8:23pm GMT

At the FOSS4G UK 2026 conference in October, I gave a talk announcing that I'd re-released the Environment Agency DEM data for England in cloud-native formats. This was covering some work I did for my client Rebalance Earth and that we've decided to release openly.

If you already know what I'm talking about then you can find the data on Source Cooperative as COGs with STAC indexes. The README on that page will explain how to access the data, and you can access an interactive webmap of the data too. If that didn't make full sense to you, then read on…

I'm not going to go into full details here, as sometime soon the video recording of my talk at the conference should be released, and I can link to that here. In the meantime, my slides are available and may help with understanding some of what I explain below.

So, basically, the Environment Agency (EA) collect LiDAR data across England and process it to produce Digital Elevation Model (DEMs). DEMs are rasters where every pixel contains a height value, and the EA produce two types: a Digital Terrain Model which is just the height of the terrain, and a Digital Surface Model which includes the height of everything on top of the ground - like buildings, trees and so on.

The EA provide access to this data under the Open Government License, but their web interface for getting hold of the data is a bit of a pain - you can't download much data at once. You can ask for bulk access to the data, but even then you have to work out exactly what OS 5x5km grid squares you want, and the whole thing is a pain.

As the data is released under the OGL, people are allowed to re-release processed versions of it under the same license. So, I converted the files to modern cloud-native formats, specifically Cloud Optimized GeoTIFF with a Spatio-Temporal Asset Catalog. These formats store the data efficiently, allowing small parts of the image to be easily downloaded, as well as 'zoomed-out' lower-resolution versions of the data.

The upshot of all of this is that you can use a few lines of Python or a GDAL command-line to download and mosaic the DEM over any area you choose, and you can also display the data easily on a web map. The latter is quite impressive - you can see an example webmap here, which does live rescaling in the browser, along with hillshading and colour blending, producing a far nicer output than the EA's viewer - the example below is the DSM over Bristol (the location of next year's FOSS4G international conference!)

I'll leave the detailed documentation on how to use this to the README file in the data repository, but a few quick examples will show how easy it is to use.

You can use a command like this to download and mosaic all of the DSM data in a particular bounding box:

gdal raster clip \
  --bbox 438000,108000,446000,116000 \
  GTI:/vsicurl/https://data.source.coop/rebalance-earth/ea-dem/ea-lidar-fz-dsm-1m-items.parquet output.tif

You can also do similarly in Python using the lazycogs library:

import lazycogs

da = lazycogs.open(
            "items.parquet",
            bbox=lonlat_to_bng_bbox(bbox_lonlat),
            crs="EPSG:27700", 
            resolution=1.0, 
            chunks=chunks)

da = da.squeeze(["band", "time"], drop=True)

which gives you a nice xarray DataArray to play with.

The talk went down very well, and was described by some people as the "highlight of the conference" (including Alastair Graham on the Geomob Podcast). The data has also been downloaded quite a lot since the conference - there have been over 1 million individual downloads, with over 4TB of data transferred. Given that each tile of DEM data is only around 80MB in size then that's the equivalent of half a million full tiles (the whole of England is only around 5,500 tiles).

I'm quite chuffed with this - it's something I've wanted to do for ages but never quite got round to, until we found that my client Rebalance Earth needed something like this, and decided we could release it openly. I want to thank Rebalance Earth for paying me to work on this (alongside lots of other stuff) and covering my costs of attending the conference to present it.

13 Nov 2026 7:13pm GMT

News

In Memoriam: Tim Peters, 1952-2026

Guido van Rossum shared the news that Tim Peters died on July 15. He wrote the Zen of Python (PEP 20) and gave Python Timsort, and as "Timbot" he answered countless questions and set the tone for early core development. Pythonistas are sharing their memories in the thread.

Wagtail CMS News

DRF vs Django Ninja: the Wagtail 8.1 API schema

Wagtail's new API runs on Django Ninja, not DRF, and Sage Abdullah and Thibaud Colas explain why: schema generation is first-party rather than deprecated in favor of drf-spectacular, and Pydantic schemas follow the type hints Wagtail already uses.

Djangonaut Space News

Picking Up Django #26355, Part 1: Understanding the Problem

Emanuel Jaimes explores Django ticket #26355, investigating previous implementation attempts and possible approaches to supporting PostgreSQL's array_append functionality in ArrayField.

Landing My First Commit in Django Core.

Goodluck Igbokwe shares his journey to his first Django core commit, exploring ImageField validation and the importance of understanding the code behind the documentation.

Python Software Foundation

Python 3.16.0 alpha 2 is now available

uuid4() and uuid7() now have C implementations and run 15-30x faster, re picked up nested character sets, and startup and garbage collection are quicker. Alpha 3 follows on December 15.

PyPI's redesigned project pages with clearer security signals are now live

Each PyPI project now has a Security tab that shows whether a release has full provenance, meaning every file has an attestation, or none. The redesign also adds search and filters to the release files list, and shows file counts and yanked or quarantined status in the release history.

Joshua Mcshannon Joins the PSF as Infrastructure Engineer

The PSF's new Infrastructure Engineer will spend his time cutting operational toil across python.org, PyPI, and the systems behind PyCon US. He has already started on Kubernetes cost savings.

Updates to Django

Today, "Updates to Django" is presented by Raffaella from Djangonaut Space! 🚀

Last week we had 17 pull requests merged into Django by 15 different contributors - including 8 first-time contributors! Congratulations to ines besrour, himanshu0870, ShabirAhmadLinux, Patel Namraa, Dmitry, Sahil Advani, Kunal Kumar and Hossein Shams for having their first commits merged into Django - welcome on board!

New in Django 6.2:

• An input_type of "textarea" was added to the Textarea widget. Like other widgets, when set to "hidden" it now renders the hidden HTML attribute. (PR #21857)
• django is now the preferred name for Django's command-line interface. django-admin remains available as an alias. Both names provide the same functionality. (PR #21260)

New in Django 6.1:

• Fixed a bug in Django 6.1 where accessing a deferred field with the FETCH_PEERS fetch mode overwrote values that peer instances had already modified locally or loaded via refresh_from_db with an explicit fields argument. The fix also addresses a KeyError when a peer's row had been deleted (#37436).

Django Fellow Reports

Django Fellow Report - Jacob

Between release blocker and security work, Jacob ran a small Trac sprint over Sarah's triage workflow changes and this year's Google Summer of Code dark mode project, and followed up with the GDAL maintainers on findings from the security releases. The main management command is now named django.

Django Fellow Report - Sarah

A four-day week spent mostly on triage and the "Ready for Checkin" backlog, accepting tickets on StepValueValidator crashing for large numbers, order_by("pk") on a union of composite-pk models, and NegatedExpression mishandling FullResultSet. She also enforced ticket assignment limits and triage restrictions in Trac.

Sponsored

Prod-like data for your coding agents

Coding agents write Django code against fixtures and guess what your data looks like. Baseshift gives every agent, PR and developer its own masked copy of your production database, ready in seconds, with no infrastructure changes. Start free and point DATABASE_URL at your first clone.

Articles

Django's Generic Views: FormView

Ben Cardy's fourth class-based view post walks through FormView, which renders a form, validates it, and redirects on success.

Making Django admin easier to use every day

Some examples of a team customizing the admin as needed for their specific use cases. A worthwhile effort if you use it heavily.

Django: introducing django-launch-editor for jump-to-file debugging

Adam Johnson's new package makes the file paths on Django's debug page clickable, opening your editor right at that line. When the editor is out of reach, in Docker or on a server, clicking copies the location instead.

Authenticating Django Channels WebSockets without putting the JWT in the URL

A browser cannot send an Authorization header on a WebSocket handshake. The common fix puts the token in the query string, where every access log keeps a copy. This is the other way, and the four places it can quietly go wrong.

Using OpenTofu to manage your Infrastructure as Code

Ryan Cheley finally learned the OpenTofu that Django Commons runs on by importing his own AWS, Coolify, DigitalOcean, GitHub, Hetzner, and Tailscale setup.

Lock your GitHub Actions

Action tags can be moved, so a workflow pinned to a safe-looking tag can start running an attacker's code. Hugo van Kemenade walks through GitHub's new workflow lockfiles.

Catch the import. Name the culprit. Keep threads free: Introducing gilguard

gilguard is a tiny, zero-dependency package for free-threaded Python (3.13t, 3.14t, 3.15t). Import it first. If any later import quietly turns the GIL back on, you get an error naming the extension module, the package that shipped it, and the line of your code that pulled it in.

What makes uv slow?

uv maintainer Konstin walks through where uv still spends its time, from source distribution builds and network requests down to TOML parsing (about 30ms just to read Airflow's lockfile), hashing, NTFS, and memory allocation.

Events

Wagtail Space 2026

Wagtail Space is next week, November 18-20. The event is 100% online and you can watch live or watch the recordings at your convenience. Expect talks from Wagtail and Django experts plus guest speakers on AI, sustainable tech, and other trends shaping the web. Tickets are free, so grab one before Wednesday.

Videos

People First, Code Always - Kenneth Love

Longtime Django developer Kenneth is the guest in this episode from Jon Gould at Foxley Talent. Lots of insights from coding in the pre/post AI world.

Databases, Django APIs, and AI - Haki Benita

The video cut of Django Chat #209, just over an hour with Haki Benita on getting more out of the ORM and PostgreSQL indexes, where Django Ninja fits, and what Django still needs for typing and APIs. That last part is the ground DEP 0017 covers, adding content type parsing and a request.data attribute that handles JSON, and the forum thread has the wider debate.

Django Job Board

The PSF's CPython Developer in Residence opening is still accepting applications, alongside senior Python roles at Manychat in Barcelona and Proxify, plus a founding engineering role at Alloy.

CPython Developer in Residence at Python Software Foundation

Senior Python Engineer (Barcelona-based / Relocation support to Barcelona) at Manychat

Senior Backend Developer (Python) at Proxify AB

Founding Forward Deployed Engineer at Alloy

Projects

prafulgulani/django-experimental

A playground for trying experimental Django features behind feature flags, installed as django.experimental.

swarfte/django-ninja-expose

Add @expose() to a Django model. Get a REST API.

Sponsor Django News

Reach 4,300+ Django developers every Friday. See sponsorship details and rates.

Planning something for the holidays? Reply and ask about our Black Friday deal.

13 Nov 2026 5:00pm GMT

I've made another package! django-launch-editor makes the file locations on Django's debug error page into links that open your editor.

Here's how it looks, on a KeyError from the package's example project:

(Click to enlarge.)

The change against stock Django is subtle, but can you spot the underlines on the file locations? They're now links that open your editor at the right line, and even the right column when Python knows it.

I think this is gonna be a neat time saver. Before, I would normally remember the filename and line number, then go open that in my editor. Or I would fiddle around copying the file path to then open it. Now it's just a click!

Setup is as easy as installing the package, setting Django's DEFAULT_EXCEPTION_REPORTER setting to the package's reporter class, and adding its URLs. See the README for the details.

When DEBUG is False, the reporter adds nothing to error pages, and the URL isn't even registered, so there's zero overhead in production.

When no editor is found

django-launch-editor launches your editor from the Django server process, so it can only launch an editor directly when you run Django on the same machine as your editor. If you run your development Django server elsewhere, such as in a Docker container or on a remote server, it can't find your editor. In that case, clicking a location shows this dialog and copies the location to your clipboard instead:

That way, you can still paste the location into your editor's "go to file" prompt, without fiddling around selecting text.

from launch_editor import LaunchEditorError, launch_editor

try:
    launch_editor("src/example.py", line=12, column=4)
except LaunchEditorError as exc:
    print(f"Could not open editor: {exc}")

$ python -m launch_editor --guess
code

$ python -m launch_editor example/views.py:20:15

$ uvx --from launch-editor-py python -m launch_editor --guess

08 Nov 2026 6:00am GMT

News

Django security releases issued: 6.1.3, 6.0.10, and 5.2.19

Six CVEs, three of them in GeoDjango: two server-side request forgery holes where GDAL resolved URLs or files from GeoJSON CRS members and SpatialReference strings (full protection needs GDAL 3.9+, flagged by the new gis.W001 check), plus log injection through GeometryField. The rest cover base64 uploads that ignored FILE_UPLOAD_MAX_MEMORY_SIZE, cache keys that confused missing and empty Vary headers, and password hashes shown to view-only admin users. Upgrade soon.

Django Software Foundation

Django Steering Council Meeting, November 2

The Steering Council met with the Fellows ahead of the mid-January feature freeze, with Carlton Gibson aiming to land the async transactions PR before then. They also weighed what to do now that sqlparse is no longer maintained (it backs --debug-sql and inspectdb), how to surface DEP discussions on the Forum, and GSoC 2027 ideas.

Python Software Foundation

Announcing new Security Developer at the PSF

Brett Smith, formerly of the W3C and Software Freedom Conservancy, joins Seth Larson and Mike Fiedler on the PSF security team in a new role funded by Anthropic. The Python Security Response Team is handling 3-4× last year's volume of reports and CVEs, and PyPI malware is up 60%, so Brett will help with remediation and scale PyPI's autonomous quarantine program.

How the Python ecosystem responded to the eMudhra intermediate compromise

Within two days of the October 20 CA compromise, fixes shipped for certifi, Python, pip, uv, sigstore, virtualenv, and more, and Python added a last-resort denylist for the bad intermediates (CVE-2026-115557) that was backported to older releases. Seth Larson explains why certifi, pip, and Python sat in the critical path, and why browsers were safe from day one thanks to certificate revocation lists, which Python may adopt.

Wagtail CMS News

Wagtail 8.1: automate more with an improved API and agent-ready tools

The preview REST API can now write more content types, expose StreamField block schemas, and drive moderation workflows (submit, approve, reject, cancel), alongside official agent skills and a prototype CLI. Editors also get SEO checks for meta titles and descriptions, pages can be served as JSON or Markdown from the same URL, and multilingual sitemaps now include translated pages.

An official headless demo for Wagtail

bakerydemo-headless rebuilds the bakery demo as a fully decoupled Next.js front-end in TypeScript, with live preview still working in the CMS. Thibaud Colas notes it runs about 95% on Wagtail's built-in v3 API, whose OpenAPI schema generates typed clients down to individual StreamField blocks, with no GraphQL required.

Updates to Django

Today, "Updates to Django" is presented by Hwayoung from Djangonaut Space! 🚀

Last week we had 17 pull requests merged into Django by 12 different contributors - including 2 first-time contributors! Congratulations to Alex Henman and Spoko Dev for having their first commits merged into Django - welcome on board!

News in Django:

• Django now supports the HTTP QUERY method, allowing safe, idempotent requests with query content in the request body (#37232).
• Django templates now support template tags spanning multiple lines, making tags with several arguments easier to read (#35899).
• Fixed a regression where BoundField.legend_tag() could add a custom required CSS class more than once when passed attrs (#37463).

Django Fellow Reports

Django Fellow Report - Jacob

Jacob Tyler Walls finalized the patches and sent prenotifications for Django's November security releases, then started drafting patches for the next round. He also fixed attrs mutation in legend_tag(), opened a follow-up optimizing lookup preparation for GeoJSON geometries, and sat in on Executive Director interviews.

Django Fellow Report - Sarah

Sarah Boyce accepted five tickets, including consistent database selection for subqueries, dropping GIS support on 32-bit Windows, and a parallel test runner crash, and reviewed fixes for XOR join promotion, select_for_update() after combined queries, and warning assertions. Her own PR allowing multiline template tags was merged, and she worked on three security reports and interviewed final candidates for Executive Director.

Django Fellow Report - Natalia

Natalia Bidart prepared the branch that bootstraps Django 2028, the first calendar-versioned release: it drops Python 3.12 and 3.13, removes the EMAIL_* settings in favor of MAILERS, and finalizes several ORM deprecations, and it lands in mid-January once the 6.2 branch is cut. She also pushed docs PRs that give local builds a new theme and better accessibility, and joined three Executive Director interviews.

Sponsored

Django for Beginners, 6th Edition

The sixth edition is out now, available in digital and paperback formats. It features two new chapters, completely updated for Django 6.1+, and more. Read the first four chapters for free to try it out.

Articles

Soft delete in Django without making a mess

Stein Ove Helset builds soft delete on a deleted_at timestamp and works through where it breaks: bulk QuerySet.delete() still removes rows, unique=True blocks re-creating a deleted record (a conditional UniqueConstraint fixes that, though MySQL and MariaDB ignore the condition), restores can collide with newer rows, and CASCADE never fires for related objects. A filtered default manager paired with an unfiltered base manager keeps related lookups working.

N+1 queries in Django Admin

If a related model's __str__ reads another foreign key, every option in an admin dropdown runs its own query, and select_related() on the edited object doesn't reach those options. Override formfield_for_foreignkey() and formfield_for_manytomany() to add select_related() to the querysets behind the widgets.

Django's Generic Views: TemplateView

Ben Cardy's series on class-based views reaches TemplateView, which needs only a template_name. He uses extra_context for static values, get_context_data() for computed ones, and template_engine to render a single view with Jinja2.

Taking Website Screenshots with Django, Celery, and Playwright

Nik Tomazic builds a Dockerized Django app that takes full-page or viewport screenshots with Playwright. Each capture runs in a Celery task so the form returns right away, and the page polls a small JSON status view until the image is ready. An optional last section moves the browser to Browser Use's paid hosted Chromium for when local browsers eat worker memory or hit bot detection.

Looking for a job? Start a blog.

Tim Schilling argues that a blog shows a hiring manager how you communicate before they ever talk to you, and suggests starter topics like a recent technical decision or how you review code. He says the host doesn't matter (GitHub Pages, dev.to, and Medium all work) and asks you to write it yourself instead of with AI, since the point is to show how you think.

How Caktus Runs and Uses Local Models

Tobias McNulty and Colin Copeland describe the two-GPU workstation Caktus uses to serve open-weight models over Tailscale, so client code stays on its own network. A dense Qwen3.8-27B handles planning and long sessions, and a Qwen3.6 MoE with 3B active parameters runs everyday work at 200-250 tokens per second, both on vLLM in Docker Compose. The post includes their benchmarks, why Gemma 4 31B didn't make the cut, and the pi.dev and VS Code configs they use daily.

Per-Object Permissions in Django That Roll Back with Atomic()

Joshua Fields wrote pemba, a Zanzibar-style authorization library that keeps per-object permission relationships in your own Django database. A grant is just rows in the current transaction, so it commits or rolls back with the data it protects, which a separate service like SpiceDB can't guarantee.

Artisanal Podcasting in the Age of AI

Will Vincent has hand-edited Django Chat in GarageBand for nearly seven years, even with AI tools that promise to handle the whole thing. Editing gives him a second listen to each conversation and a kind of deep focus he no longer gets when coding with a handful of AI agents.

Caching at Scale in Django: Lessons from Production

Johnny Metz's team handled about 20x traffic growth on a multi-tenant Django API by caching aggressively: per-user lists of document IDs paired with one shared entry per document, seven-day TTLs with invalidation on Django signals, a cached identity lookup, and failing open when Redis goes down. During the rollout, database queries per second halved, and Redis now serves most reads at a 97% hit rate.

Events

DjangoCon Europe 2027 Call for Participation

DjangoCon Europe runs February 17-21, 2027, in Innsbruck, Austria, and wants talk and workshop proposals on technical and non-technical topics from new and experienced speakers. Proposals close November 15, 2026.

Podcasts

Django Chat #209: Databases, Django APIs, and AI - Haki Benita

Haki Benita talks about getting more out of Django's ORM and PostgreSQL indexes, why Django Ninja appeals for APIs, and what Django needs for typing and a built-in API story. They also get into how AI is changing the way developers learn and share what they know.

Videos

From librarian to Djangonaut: My Open Source journey with Annabelle Wiegart

In a guest talk for Djangonaut Space Session 7, Annabelle Wiegart traces her path from librarian to Django contributor, PyLadies Zurich organizer, and Djangonaut Space captain. She talks about working on ORM tickets and nullable expressions with mentors, and how contributing went from intimidating to something she could do.

Django Forum

A Path to Modernizing APIs in Django (DEP 0017 Content Type Parsing)

Picking up from the Haki Benita episode of Django Chat, Will Vincent lists what stands between Django and a built-in API: a modernized HttpRequest through DEP 0017, a pluggable serialization layer (the hardest part), and then the view layer. Carlton Gibson wants those primitives updated first, while Haki would settle on an official API interface and tune performance later.

Django Job Board

The PSF's CPython Developer in Residence opening leads the board this week, alongside senior Python roles at webbyfox consultancy and at Manychat in Barcelona, with relocation support.

CPython Developer in Residence at Python Software Foundation

Senior Python Engineer at webbyfox consultancy

Senior Python Engineer (Barcelona-based / Relocation support to Barcelona) at Manychat

Projects

amureki/django-cmdbar

Adds a Cmd+K / Ctrl+K palette to the Django admin for jumping to models, searching records (user alice), or opening one by primary key (user#42). It respects admin permissions and needs no frontend dependencies, middleware, URLs, or migrations: just add django_cmdbar above django.contrib.admin in INSTALLED_APPS.

Metron-Project/izaya

An htmx-powered autocomplete widget for Django forms that ships no required styles, so it fits any CSS framework. Register a ModelLookup with your search_fields, drop LookupWidget into a form (with multiple=True for multi-selects), and use rank_fields to put the closest matches first; it needs Python 3.14+ and Django 6.0+.

Sponsor Django News

Reach 4,300+ Django developers every Friday. See sponsorship details and rates.

06 Nov 2026 5:00pm GMT

Twisted was originally the network component of a text-based role-playing game engine. In the late 90s when it was first developed, the idea of a video game being able to send you actual emails and instant messages¹ seemed pretty neat, rather than the valence that it would have today (nightmare). We never ended up making an actual game, so we eventually removed the "game engine" piece, and just kept the protocol implementations of the email and the AOL Instant Messenger, and the DNS and eventually SSH and so on. The residual pieces of that project have not seen maintenance for a decade.

So it's not really a way to make a text-based game any more.

Then, Twisted was the premier longest-lived and best-maintained event-driven I/O library for Python. If you wanted to make high-performance servers, it was the best and most obvious way to do it. Even if it still acquits itself relatively well in this regard, its claim to fame, and to indispensability, has been somewhat obviated by the inclusion of asyncio in the standard library. Even if it has some stuff to recommend it - better-than-average testing facilities, for example - you can get better-documented and better-integrated low-level asynchronous networking tools without even installing any packages.

So it's not really the premier choice for an event-loop these days.

Twisted does still contain an SSH server and client, an email server and client, a DNS server and client, a web server and client (now including websockets). But you can cobble these together by installing a variety of packages from other sources.

So... it's a bunch of protocols? And I guess it doesn't use AI? But... why would anyone care about that? Why contribute to such a thing?

What is Twisted doing, and who is it doing it for? What is the political project of Twisted, nigh on 30 years into its existence?

Most of all, why should you care about it, or want to contribute?

If you're asking what you could use Twisted for today, it's an excellent choice if you're implementing a new network service and you need more than one protocol. If you need to do HTTP, DNS, packet capture, and a custom SSH server, it would be very hard to arrange a collage of other Python packages to manage this. It's still much better at deterministic testing than most of the standard library. It's a bit easier to hook into a GUI.

But that's not what Twisted is doing.

What I want Twisted to do for the world is still an unfinished project. It's honestly kind of bad at doing what I want it to, but it could be great. The overarching goal of having and maintaining all this stuff is to, as much as possible, demystify networked software operation and let you build Internet facing systems at a human scale.

It's all in a memory-safe language with good error handling, so you can avoid the scariest security pitfalls. It's all integrated into a single paradigm with built-in tools for testing. It's designed to scale down to be runnable on the average laptop, and not require large elaborate server farms or distributed clusters to run. Via the twist command line utility, it provides a Swiss army knife of simple tools that you can use out of the box.

In a world where all of our compute is in massive distant opaque datacenters, where a tiny cartel of gigantic tech companies control nearly the whole supply chain, one of the few things that you still can have agency over is your own server on the Internet.

It's still possible to get a domain name for a few bucks a year, a free-tier or very low-cost VPS, then stick a glue record in and run your own DNS. From that point forward, you can run your own email, your own web server, your own terminal server. But doing this requires deep expertise, and that expertise requirement is massively inflated by the operational concerns that go with running twenty different pieces of software from twenty different projects, probably in twenty different languages, with completely inconsistent aesthetics.

The reason that Twisted is event-driven is because as much as it can be a pain to wrap your head around async/await (or worse yet, Deferred directly), it is much easier for a single person to understand the concurrency requirements of an async I/O multiplexed system than to understand what is going on with an application that requires multiple thread pools. It is possible to understand concurrent code that manages parallel tasks in that idiom. The reason that Twisted integrates so many different protocol implementations and attempts² to unify them with a common authentication layer is to reduce the overhead required to manage a comprehensive suite of network services.

Even the not-using-AI thing is not an aside due to some accidental personal preference about tools - it's a core part of building an engine that you can personally understand if you dive into it, that's built with a consistent style and layered in a way that you can take apart if you need to. It's not built with AI because we're not building for AI, we're building for people and the entire point is that people are supposed to be able to understand it.

We used to have a tag line, which is somewhat less emphasized these days because its meaning is not immediately clear without this background, but was once quite prominently featured on the home page: "Twisted: Building the Engine of Your Internet". That's what I want to bring to you: an engine for you to build your Internet, that is small enough that you can make it personal, that doesn't require massive investment in management tooling just to get started.

Now, for various reasons, I no longer use twisted.names to host my own DNS (although I did for many years!), and we never wrote enough of a twisted.mail suite for it to be usable for email hosting on its own. But, if you're reading this article, it was probably delivered to you directly via Twisted's own twisted.web HTTP server, using twisted.protocols.tls to configure our own OpenSSL TLS termination. I have run with extremely high reliability for years and next to zero management overhead, in a single simple docker container. Which I then administer and upload files to via Twisted's own twisted.conch SSH server.

If that seems like an appealing project and you'd like to volunteer to help out, drop me a line and maybe I can point you at some tasks to get started. Twisted falls far short of this aspiration despite 30 years of work, and we could use all the help we can get.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!

05 Nov 2026 10:51pm GMT

Creative Work

Writers went on strike to get protections against "AI". Thousands of artists have signed open letters in protest of "AI". There are so many copyright lawsuits from creative industry groups against AI that there's a whole dedicated website for it. Popular YouTubers absolutely hate it. If they're also musicians, they REALLY hate it. Across all creative industries, there is a concerted push to reject this technology.

Yet, almost unique among creative fields, many experienced programmers remain convinced that it's fine to use "AI" for programming. We do seem to hate it, and what it's doing to our industry. It's making us all miserable, but we are using it anyway.

A lot of the justification of this resignation seems to be to be because programming is not Art. If the tool can get the job done, and the job is just functional, then why does it matter?

It does matter, though. It matters because we shouldn't be using AI to produce Art, and programming is Art.

Art can be Mundane

Some people will say that programs cannot be art because programs are functional, rather than being expressive. Programs are mundane whereas art is transcendent.

This is based on a distorted understanding of what Art actually is.

In John Berger's "Ways of Seeing", he names this type of distortion "mystification". His example of this process is both amusing and illustrative. I encourage you to read it in its entirety.

In summary, though: Berger critiques the florid prose of an art historian describing a commissioned group portrait, including phrases like "subtle modulations of the deep, glowing blacks" and "harmonious fusion". The portrait is described as sublime, in nearly ecstatic terms.

Berger reveals that the reality of this portrait is that a poor old painter needed some work, and some officials probably thought it might be nice to have an official portrait. So they paid some money to the poor old man, and he painted it, and then they had a painting. It's a well-executed portrait of a group of people. Beautiful, even. But it was work commissioned for a fairly mundane purpose and it suited that purpose just fine. It was not, and is not, a divine relic.

Culturally, we are prone to mystifying painting, and sculpture, and film, and music. We imbue them with "subtle modulations". We ignore their functional aspects - we desire decoration, amusement, and distraction - and focus on their emotional impact.

Don't get me wrong: I love me some good aesthetic philosophy. I think it's great to really examine our reactions to artwork and to try and gain a deeper understanding of our culture and our selves through media analysis. If anything we really need to do more of it.

This does not mean that the creation of such works is mystical or that it should be venerated beyond any other sort of labor.

Not least of which other types of labor that are adjacent to, but not as culturally venerated, as fine art. We tend to mystify the work of a novelist, but to denigrate the work of a journalist. In reality, the functional prose of the journalist is no less important and deserves no less respect.

Although they might be far below the ethereal realm that novelists inhabit in our collective imagination, even journalists receive more respect and thus more mystification than lowly copywriters. Yet, there is no transcendental distinction between "novelist" and "copywriter"; the many of the skills are the same, and the distinction is merely an accident of commerce and opportunity.

In fact, many famous writers have famously inhabited both roles. This is not an accident! Working with words professionally, even (perhaps especially) mundane words, is excellent practice for working with words in a more purely artistic context, because even mundane creativity is still artistic.

Code can be Beautiful

One thousand Internet years ago, when I was in my late teens, I would describe myself as a "code poet". I was relentlessly mocked for this as what the Youth would today call "being cringe", and at the time was referred to as "pretentious".

I succumbed to the peer pressure, removed it from my email signature and my bio. While I still believed strongly in the parallels, I accepted that - socially, at least - comparing code to poetry, or indeed to Art, was a silly thing to do.

However, I never abandoned the idea, in my heart.

The thing that I am most well-known for, the invention of Deferred, was specifically an aesthetic reaction to the tedium of passing callback and errback parameters to every single remote procedure call in an RPC client/server application. Those two callbacks got the job done just fine. But they were ugly, and annoying to work with.

Deferred is an intentional poem about asynchronous task execution, with a deliberate eye to the aesthetics of the problem and the experience of using it. It was influential because of its focus on aesthetics.

I do not want to overstate the beauty or profundity of this minor contribution, or indeed its durability. That a poem exists does not mean it is a great poem, merely that it is a poem.

Our aesthetic culture around programs is more like folk epic poetry than fine art, so the influence of this contribution is less about its specific enduring power than it is about its influence on what came next; from MochiKit.Async to JQuery Deferred to JavaScript Promises and eventually to async/await; a long chain of different artisans each adding something of their own until the original has all but dissolved. (And I wasn't the "original" here, either, as I drew heavily from the E language's Promises, among other things.)

In order to make code into a deliberate artistic expression, one must have spent quite a bit of time contemplating the problem domain. Without having experienced the tedium of manually passing a thousand callback parameters, I would have had neither the skill, nor indeed the motivation, to bother creating such a thing.

Now, most code does not have to be like this. Most code does not get to be like this. Most code is functional, workday code. Most code could not make a lady weep. It's just copy-writing, if you will.

As I explained previously, most writing couldn't do that either. Most writing is just copy-writing, too. Most visual art is advertising. Most live music performance is background music in bars that will go largely ignored.

However, code that is intentionally aesthetically designed tends to be important, both socially and technologically.

We do have some tradition of self-mystification in software. As Abelson memorably put it, "Programs must be written for people to read, and only incidentally for machines to execute.", so we have long had some conception of programs as highly expressive, even if we can't always agree on what they're expressing or to whom. We will occasionally wax poetical about the philosophical implications of a particular piece of software. This is not unique to a single piece of software, either; more than one community has indulged in similar philosophizing.

The expressive and aesthetic qualities of software are not limited to reading source code or interacting with other programmers via APIs, either. For example, every year, Federico Viticci does a review of Apple's new operating system, which is (among other things) an aesthetic critique. Such a project would not be possible if the software did not have an aesthetic impact on its users.

Not to mention that every video game review is also a software review.

Defend The Mundane

If we use AI to erase all the copy-writing, all the graphic design, all the boring mundane art, and yes, all the boring custom WordPress theme development, then we will be removing all the practical opportunities for the vast amounts of practice and contemplation required for people to elevate their craft to eventually achieve great things. Education is great, but the majority of true skill development happens on the job and always has.

This doesn't mean that we can't use abstractions, or automation, to make our work easier. Programming is the art of abstraction, of understanding how to compose smaller ideas into bigger ones, of how to understand the automation of a larger system by understanding the rules that automate smaller ones and understanding how to combine them.

When we use "AI" to eliminate that understanding rather than raise it up to a higher level, to entirely destroy that creative decision-making process, we do a disservice both to ourselves as programmers and to our users. We would be doing a disservice to our users and our downstream fellow developers in the same way that a visual artist would be doing a disservice to their viewers or a musician would be doing a disservice to their listeners if they served them auto-generated filler instead of their own creative output.

Slop is slop, no matter the medium.

Each mundane project has some tiny chance - let's say, something like 0.1% - of achieving greatness. If we do a single project with AI, then sure, whatever, there's almost no chance that that project was going to be the one hit to create that career-defining moment for an engineer working on it. If we make a habit of doing all projects that way, though, we take the total likelihood of those moments of greatness from "definitely sometimes" to "never".

The precisely appropriate ways in which to resist AI encroachment on all software development lie well beyond the margins of this one short post. How much you can resist and which specific uses you should resist are up to you. But it is worth resisting in software just as much as it would be worth resisting in any creative medium.

Programming isn't special. It's just Art, and Art is the most human - and thus, the most universal - thing that there is.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!

09 Oct 2026 7:00am GMT

One of the issues that I have with the current generation of "AI" products is that they do not appear to take their own premises seriously. I look at a plethora of obsequious chatbots claiming to be serious tools for problem solving, and I think, this is not what a problem-solving tool would look like.

Even before we get to the tremendous ethical problems with the frontier labs, it is this impression of their composition as a product that makes me feel, constantly, whenever I am interacting with them, that they are less a software product than that they are a grift, a scam designed to make me feel like I am interacting with a product that has capabilities that it simply does not, to try to lull me into a false sense of security that I can trust it.

The frontier labs are of course the worst offenders, but every criticism here applies just as much to Ollama, which (if anything, due to the obviously poorer quality of the available models themselves) needs these features even more than the frontier labs do.

Here, I will set down a few features that might convince me that an LLM-based product, particularly one focused on research or software development, was actually serious about helping me do useful things with it.

Make "Checking For Mistakes" A First-Class Feature

This is the biggest issue, and the major reason that I was inspired to write this post.

It is a truth universally acknowledged, that AIs cannot reliably provide information.

I could cite a ton of news articles and studies about this fact, but there is no need. Every single chatbot admits this, up front, in a fine-print disclaimer as a core part of their user interface. Gemini says "AI can make mistakes, so double-check responses", Claude says "Claude is AI and can make mistakes. Please double-check responses.¹" ChatGPT says "ChatGPT can make mistakes. Check important info.".

Every time I see that last one, I wonder how I'm supposed to know what "info" is supposed to be "important".

All of these warnings are all small, gray text, painfully obviously included as legalese to push responsibility back onto the user rather than to help with anything. This is a core limitation of all these products. Checking their output is a part of the workflow for using them that:

1. you absolutely cannot skip or skimp on without creating risks to yourself and whoever you are conveying its output to, and,
2. it is very easy to skip or skimp on and you are encouraged at every turn to do so, because "just trust the output" is one of the quickest ways to save time.

A chatbot product that took this weakness seriously, as an actual consideration for using it, would put a checkbox next to every claim in its output. It would be a 2-column worksheet, where you've got the LLM output in the first column, and next to it, human notes in the second column, explaining what work went into checking this claim, and a big checkbox that you would only check off after you believe you'd checked its claims thoroughly enough.

Coding assistants would need to have some version of this as well. Right now, this is pushed off into code review, which means it is a dark pattern which subtly encourages the "author"² to offload this work to their code reviewer without ever looking. Once again, "it's probably fine, I don't need to check" is the quickest way to save time and churn out those PRs faster.

It might even be useful for coding harnesses to have some affordance for checking code before it even runs tests. As the vendors themselves have admitted, it's not just expensive to burn tokens on your "AI", you also end up burning far more compute on the AI. Being able to check your diffs before sending them over to uselessly exhaust your testing compute cluster would be useful.

If your product tells me that it makes mistakes and I must be the one to check for the mistakes, but then gives me zero tools to check for mistakes, I cannot take it seriously.

More Citations to Check, And More Details

Most chatbots prefer to give an answer, rather than a citation. In my own personal use, I find that when asked to provide a list of citations with clearly marked sources for each one, they will appear to "get bored" halfway through the list and simply stop including citations at some point.

When the bots include citations at all, present them as inline annotations that say nothing but the domain name of the search result, in a font so small that it's barely legible, and an equally indecipherable icon that is fewer than 16 pixels on a side.

This is backwards.

Now, I am aware that these citations do come from somewhere, and in an attempt to reduce hallucinations, all of the major providers support some form of "grounding"³, and that those little barely-readable citation links are referencing actual structures in the RAG pipeline and not just potentially-hallucinated tokens, but I'm not talking about the underlying machinery in the model, I'm talking about the presentation to the user.

Plus, regardless of whether a snippet of text came from a RAG query, we know that LLMs can never provide an authoritative result; it's a fundamental limitation of the technology. They can still garble the results of RAG as much as they can misrepresent any other training data. This means that it must never present its results as authoritative.

If you ask an AI to do research queries, every result should be presented as a list of citations. Moreover, the presentation should display each citation as a large object of in its own right, with clearly identified metadata, including not just the site where it was found but its publication date and, if possible, the name of the author. The literal, unmodified quotation (not from RAG, not a summary: a quotation extracted with a regular program and not an LLM) should be front-and-center, larger than any AI-generated text.

If the AI product wants to editorialize or summarize (which should not always be necessary!), the AI-generated text should be presented as small text underneath the citation that has been found, de-emphasized as much as the disclaimer is right now, at the very least until the user has verified that the summary is accurate. Perhaps, for a research project, a "did you read the citation" checkbox might even be helpful.

If your product openly tells me that it will scramble, misrepresent, or omit its citations in its summaries, and I must read the original human-authored citations to be sure, but then gives me no tools to track my reading of those citations or even any way to find them, I cannot take it seriously.

No First-Person Output, No Apologies

There is no reason for a software development or research tool to use first-person language to describe itself. They should not do so. In fact they should not be allowed to do so.

There is also no reason that they should ever apologize. It is a waste of everyone's time; it's a waste for the chatbot to generate the apology, it's a waste for the user to read the apology, and it's a waste for the user to respond to the apology. Yet they unfailingly do this upon every correction.

The vendors of these tools know that they are routinely causing mental-health crises. In response, they have added non-functional "guard rails" that can still, in 2026, easily be bypassed.⁴

A product seriously interested in helping with productivity would correct this glaringly obvious flaw, focus on the task at hand, and stop emitting useless verbiage.

In the previous two sections, I tried to focus on ways in which the harness would be constructed differently even if the LLM technology is fundamentally impossible to improve; in this case, I have to assume that the labs have some control over the model itself. But unless they are truly incapable of influencing their output (and all their "benchmarks" and "capabilities" seem to indicate that they can control it very tightly) they ought to be building models that are much less verbose.

More Non-Natural-Language User Interfaces

Although natural language could hypothetically be a powerful interface for interacting with a computer system, the practical upshot of LLM natural language interfaces is that these interfaces are imprecise and repetitive, full of superstitions masquerading as "best practices". The inputs are a mess and the resulting outputs are a mess.

The general way of addressing this unstructured mess is to allow the chatbot to directly take action in response to the user's input; in other words to supply it with "tools" via an MCP server. But again, this is backwards. If we cannot even express our intent clearly in the first place, why are we trusting this system to take potentially destructive and harmful actions on our behalf?

Instead, I would expect a product that was seriously invested in helping me accomplish specific tasks, to have user interfaces specific to those tasks. Is it supposed to be able to be a security scanner that can discover OWASP top 10 bugs in a codebase? Have a button for that. Build that functionality into your harness, train it directly into the model, use smaller models that can satisfy that functionality more effectively than throwing it at the planet-sized brain of Fable or whatever.

I'm aware that there are small software startups that do something like this, but they are bolted on to the side of the main model providers' APIs, not integrated into the core of the product and not using their own models and AI systems to achieve consistent and repeatable results.

Strong Data Provenance Indicators

Chatbots produce data tables pulled from websites, from APIs, from MCP tools or from summarizing and scrambling the user's input. In order to provide the illusion of a seamless interface, this data is presented in-line regardless of where it comes from. But some of these outputs are produced mechanically via regular old API calls, for example, from the result of calling a tool or querying a website, but presented uniformly.

But there is a huge difference between an authoritative data source being inlined as part of a chatbot conversation, being treated as input by the chatbot, and some ad-hoc hallucinated data being treated as output of the chatbot.

If a product is trying to help me make accurate, empirically-grounded, data-driven decisions, the source of the data is critical.

Integrated into the "check for mistakes" and "verify citations" workflow I described above, there's a necessary "verify data programmatically" pass as well; to have tools that will treat portions of the output as a regular spreadsheet, allowing regular-old computer arithmetic to verify things and showing where such arithmetic was used, and how.

Better User Control of Reproducibility

Anyone familiar with the technical specifics of LLMs will know that they have a variable called "temperature" which controls the degree of randomness that the LLM uses to produce its outputs. But most users don't know this, because it isn't exposed as part of the user interface by default.

This leads to a subjective impression that you asked ChatGPT, and you got ChatGPT's authoritative answer.

You can't just set the temperature to zero and still get useful results - I am aware that it does more than just scramble the output at random, and there are perhaps good reasons that simply exposing just a temperature setting would not be that useful to users. But if we followed some more of my earlier recommendations for making more structured UI elements to solve specific problems rather than having long back-and-forth chats where each refinement depends on the previous response, perhaps those elements could also re-play the process so that users can see how reliable the bot is at a particular task and develop a sense of how the stochastic nature of the process actually affects it.

Similarly, if a user is trying to solve the same problem repeatedly with a chatbot, and the chatbot product has numerous computational tools that don't really have anything to do with the LLM, such as deterministic data-processing tools, then having a way to freeze the non-deterministic parts of the transcript but re-populate a particular data frame with updated information and fork / continue the conversation from there would be a way to avoid introducing pointless additional randomness when you already know what tool you're trying to use.

The fact that every conversation is presented as this flat chat prompt that doesn't let me interact with any of the widgets that were previously produced except through more chatting, really makes me feel like the whole product is just doing predatory social-media style "increase time on site" optimization, just trying to lure me into further repetitive and unreliable chats, rather than letting me get in, solve my problem, and get out.

Context Visibility

Managing the LLM context is the ongoing challenge facing organizations that are trying to use "agentic" workflows. Filling up the context with too much information causes well-known problems. In response, advanced LLM users attempting to solve larger problems must break up very long prompts into "skills", give access to lengthy information via "tools", and delegating sub-problems to "sub-agents" rather than simply extending a single prompt indefinitely.

All of these strategies have flaws, because even on the largest models, compared to the breadth and depth of knowledge-work problems, LLM contexts are quite small.

And yet, none of these products will show the context to the user by default. There are third-party addons that can show you a simple progress bar but for addressing the premier engineering difficulty with this technology, that is below the bare minimum.

This lack of visibility means that almost all of the tools for extending the context are flying blind. Rather than responding meaningfully to a full context, everyone just kind of guesses how much state they need by guessing and trying over and over again with progressively more elaborate skill and sub-agent layouts. Even managing context compaction ends up being an advanced API-driven workflow⁵.

A serious product that was trying to help the user understand would not only show "available context" but explain the impact of context compactions, make it easier to see harness-generated prompts, and so on. This would be a first-class feature, combined with the aforementioned reproducibility / replay tools, would allow users to do real experiments to develop an understanding about how to make good use of the context window.

A Sandbox That Actually Works

I've been focused on the chatbot interface here because it is the most immediately egregious upon looking at the UI. But the "agentic loop" tools used for coding are equally dangerous, if not more so. Coding tools keep destroying everyone's data, over the course of years.

These catastrophic incidents that become front-page news are relatively rare compared to the amount of coding-agent use out there. But they also aren't the only kind of sandbox violation. Coding models will so routinely edit test code instead of the system under test that there are "pro tips" articles all over the web giving you the flawed advice to simply ask the agent not to cheat. News write-ups of the catastrophic incidents themselves will also offer glib and wrong advice, like "use a docker container". That might prevent it from literally deleting your operating system, but it won't prevent it from destroying all the local work you have in your codebase (it needs access to a checkout, after all!)

There is a flurry of activity in the infosec space where people are rushing to plug the gaps left by these coding harnesses. Everyone's got their own version of an MCP approval gateway where you can optionally place a proxy between your agent and your production infrastructure.

In the best case, though, all these mitigations and proxies and prompts simply turn the user into an auto-approval automaton, hitting Y, Y, Y, Y over and over again, until you finally are driven mad and hit "yes to all", turn on full-auto mode and submit yourself to the void. With nothing between your personal vigilance and disaster, there are no workflows left beyond decrementing your own vigilance until there's nothing left and then hoping the disaster never arrives.

The fact that some mitigations exist that can be deployed by extra-cautious users does not change the fact that "agentic coding" is an unsafe-by-default technology deployed without concern or guidance. Every frontier lab has tied a spring-loaded shotgun to a dog; the fact that dog owners can publish thoughtful blog posts explaining how you can teach your dog the basics of gun safety or how you can have your dogs play in a bullet-proof room does not mitigate the fact that the product should not have been allowed in the first place, nor should it continue to exist without VERY strong security controls.

I might believe that a frontier lab were seriously interested in providing developers with a useful tool if they shipped something that had safety built-in.

That means tools in the harness, detached from any LLM, independent of the prompt, that could:

• sandbox all filesystem operations and strictly limit ANY deletions outside of specified scopes, regardless of operating system,
• enforce snapshotting of the entire repo on every operation for easy rollbacks and minimal lost work,
• remove the disaster of "auto mode" (not to mention nonsense like --dangerously-skip-permissions) entirely, and
• carefully consider a structure for presenting plans to the user where, rather than provoking immediate alert fatigue by asking for checks on every action, make structured plans which can be submitted to the user as a group of actions and reviewed and approved as a batch.

In the same way that I suggested above that research-based tasks should have a way of re-issuing prompts to determine how reproducible a result is, or whether other sources might be found, agent-based tasks should have a way of being executed against mock services for popular APIs, so that the verification can match both on the front-end (review the plan for making the API calls before they're executed) and the back end (review the API calls that were issued to the mock service and verify that they matched).

Instead, the frontier labs provide us products that are disasters out of the box, give us "best practices" to build massive and elaborate, as well as incomplete and error-prone, security perimeters of our own design. Then they blame "operator error" when it inevitably goes wrong. I cannot believe that these design choices are intended to help us be productive.

Bonus: Human Processes

Organizations deploying AI also frequently come across as unserious, for similar reasons. In 2023, naive exuberance could perhaps be forgiven. But today, as we near the close of 2026, there are several well-known problems, that have been extremely well-covered in the press. None of these things should be surprising, but most orgs deploying these tools are still just letting them rip and hoping it all works out.

Organizations deploying these tools would need at least three kinds of major modifications to their internal processes, if they wanted to be serious about using them safely:

1. Shift Rotations to Prevent Vigilance Decrement

There have been several high-profile incidents where software developers' gradual acquiescence to accepting LLM output have lead to serious economic consequences for the companies deploying them, perhaps best typified by Amazon's "millions of lost orders" due to a gradual decay of their engineering processes from LLM use.

These outages, and other AI-related failures, are due to the difficulty of maintaining focus on the same problems. In other words, as I described above, vigilance decrement is a constant problem, because AI outputs are most often correct, but continue to be incorrect in surprising and non-intuitive ways. As I have previously written, you cannot trust yourself to catch every bug with code review, and LLM output.

Aviation, for example, has very strict rules around rest requirements. There is also a specific rule that "No certificate holder may operate an aircraft without a second in command if that aircraft has a passenger seating configuration, excluding any pilot seat, of ten seats or more.". Other safety-critical professions have similar rules.

And yet, even in the age of the supposed "AI revolution", most software teams are still assigning every engineer a full feature load, not planning for any rest, and telling people to review code whenever they happen to have some "free time".

Maintenance of vigilance has to be your top priority. Regular, scheduled, inviolable rest periods where people do work without AI assistance, and are not exposed to any AI output for review or otherwise, would be crucial in order to stay mentally sharp enough.

The tools themselves should have this sort of thing built in. The mistake-review process described above should have a periodic spot-check mode where a second reviewer periodically reviews a chatbot log, doing their own independent verification of claims, to see if they spot the same errors. This could provide a feedback loop to determine how much rest is necessary to maintain continuous attention and actually spot hallucinations.

2. Skill Practice To Prevent Skill Loss

It is also well-known that AI use leads to AI reliance, and AI reliance leads to skill loss.

I like to use the analogy to dockworkers at a seaport⁶ adopting automation.

If you employ dockworkers to load and unload ships all day long, they are going to be getting tons of exercise. They will be able to lift heavy objects on demand, whenever. They might have plenty of health problems and injuries from this type of work, but "lack of exercise" will not be a problem.

With the development of standardized container ships and mechanized cranes, you are going to be changing their job description substantially: now they mostly spend all day sitting in a small cubicle moving a control lever back and forth, not lifting heavy stuff. They will get worse at lifting heavy objects.

In this analogy however, the cranes are not all that reliable. We know they break, and they drop their payloads sometimes, and the stuff needs to be manually moved. But this only happens a few times a week, at most. If you need whoever is driving the crane to be able to jump out at any moment and still move stuff around manually, then you need to make an affordance for that. You need to give them time to go to the gym and do some lifting for practice, or every crane failure is going to be a major emergency.

An organization doing an AI transformation would also need a massive increase to learning & development budget, both in terms of resources and in terms of schedule. If your people are going to lose skills because they've lost regular practice in the incidental course of doing their duties, then they are going to need deliberate, intentional, non-incidental practice of those skills to keep them sharp.

But rather than trying to accommodate new workflows and give time for people to adjust, most AI mandates are simply dropped on workers like a ton of bricks, with no time to adapt and no affordance for maintaining their skills. Operate the crane and stay fit and healthy and ready to switch back to manual lifting at any time and then get back in the crane cockpit right afterwards. Don't mess up.

Then an accident happens and everyone is surprised, as if this process weren't practically designed to produce a terrible result.

3. Mental Health Resources to Deal with Mental Health Risks

AI psychosis often begins with practical problem-solving, and beyond that, it can start specifically at work. Not to mention the more pedestrian condition of "AI brain fry".

If you are mandating your employees to use a hazardous tool that may seriously and directly damage their mental health, you need trainings and resources. You need in-house therapists and you need to be making sure to check in with people actively to make sure that this is not happening.

Again, the tool itself ought to have some way of dealing with this. An occasional "take a break" popup is easily dismissed; they need a user-visible AI personal dosimeter so you can see your cumulative usage over time.

I don't even know if "usage over time" is a sufficient metric to gauge risk. Maybe if your work chatbot start to talk about resonance too much, unless you literally work as an acoustic engineer, that should be flagged for someone.

We are, again, years into dealing with these tools, and we know these risks exist. Yet no serious mitigations are provided. Not even any way of measuring the risk exposure.

And More

There are also many other risks associated with the technology. There are intellectual property risks with the foundation models, due to recklessness with their training data. There are existential financial risks associated with the infrastructure build-out. The extent to which most "open" models are simply derivatives of frontier models is an open question.

What I Think

If any one of these things were regularly overlooked by AI vendors or users, that would be a totally normal product oversight. Room for improvement for the next version, but nothing catastrophic.

Shipping without any of them doesn't seem like lean product management, it seems like a careless attitude towards risk and a product design philosophy oriented entirely towards short-term demos, with no regard for how to realize actual productivity gains.

Furthermore, being available for years without anything like these features, despite hundreds of incidents demonstrating the risks, with hundreds of billions of dollars of funding, makes it seem to me like if they were to add all the features that would make their product actually safe and hypothetically useful, these features would reveal that it is actually not an improvement to productivity.

In the year since I first wrote about measuring the cost/benefit ratio of AI, I have heard from numerous people who have shown this to management to try to illustrate why their AI initiatives - like almost all AI initiatives - were either failing or burning out their engineers.

I've also heard from lots of people that have told me that it's obviously useful and they don't need to measure so carefully, because they are getting lots of work done that they couldn't have otherwise.⁷

I have yet to hear from a single person who has said "yeah, we measured according to your methodology⁸, and it turns out that our AI work is going great and that our ratio is 0.75".

Obviously, I cannot say for sure why this is; absence of evidence is not evidence of absence. But at this point I think the null hypothesis is that AI tools provide, in aggregate, zero value. They make mistakes too often, and the externalities they produce are so bad and so difficult to control that even before we get to the places where they are just physically poisoning people, even the negative effects on their direct users end up cancelling out whatever benefit to they provide to their organizations.

If I were wrong, then including tools to measure an AI's effectiveness at the tasks their users are actually trying to accomplish, rather than meaningless benchmarks, would show big productivity gains. The frontier labs would be champing at the bit to add such features, and crowing about their fantastic results.

I think the labs know that if they did that, it would present a grim picture to their users. Such tools would let their users see that it's making mistakes much more often than they realized, that they're spending much more time with it than they want to be, and that it's just generally not fit for purpose.

If they prove me wrong by adding in all of these safety mechanisms, and in the process, they make all of their AI technology less harmful, I'll be thrilled to be debunked.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!

28 Sep 2026 4:27am GMT