fidzu : python

In a first look at Muru I made a point of the access log separating how long a request waited for a worker from how long it spent in the application. That tells you where a slow request lost its time, but not why. Sooner or later the question you end up asking is what the workers are actually doing, usually because a request is taking far longer than it should.

The traditional answers to that are to guess, to sprinkle print statements and timing code through the application, or to bring in a separate monitoring product. Having spent twenty years on mod_wsgi seeing what people struggle with when hosting Python web applications, and five years at New Relic working on exactly this problem for Python, I have some fairly strong opinions about how much of this a hosting system should be able to tell you itself. Quite a lot, as it turns out, is the answer Muru gives.

A slow report

To have something to look into, here is a small WSGI application with a deliberately slow report. Fetching the rows for the report pretends to wait on a database for a tenth of a second per month, and summarising them is pure Python arithmetic over eight million rows per month.

import time
from urllib.parse import parse_qs


def fetch_rows(months):
    time.sleep(0.1 * months)  # as if waiting on a slow database query
    return range(8_000_000 * months)


def summarise(rows):
    total = 0

    for row in rows:
        total += row * row % 7

    return total


def build_report(months):
    rows = fetch_rows(months)
    return f"{months} months: {summarise(rows)}\n"


def application(environ, start_response):
    query = parse_qs(environ["QUERY_STRING"])
    months = int(query.get("months", ["1"])[0])

    if environ["PATH_INFO"] == "/report":
        body = build_report(months)
    else:
        body = "ok\n"

    start_response("200 OK", [("Content-Type", "text/plain")])
    return [body.encode()]

Rather than giving everything as options to muructl run, Muru can read its settings from a [tool.muru] table in the project's pyproject.toml. Here they name the entrypoint, ask for two worker processes, and also set a soft timeout of three seconds, which I will come back to shortly:

[tool.muru]
entrypoint = "report:application"
processes = 2

[tool.muru.timeouts]
soft_timeout = "3s"

Start it with muructl run and ask for reports covering one month and then three months. The access log shows what we would expect:

200 GET /report?months=1 19B 382ms queue=0.0ms app=382ms worker=0 id=01M6T02DPVG7XRY5V2P2V9S7WG
200 GET /report?months=3 19B 1.15s queue=0.0ms app=1.15s worker=1 id=01M6T02E30G7XRY5V2P2V9S7WG

The queue time is zero and the application time is all of it, so we know straight away the problem is in our code and not a lack of capacity. Three months taking over a second also means a year is going to be painful.

Catching the worst case

By default a request in Muru can run for as long as it likes, up to a deadline at the edge of five minutes. Even when the edge gives up and answers the client, though, the request keeps going in the worker, tying up a thread for no benefit to anyone. That is what the soft timeout is for, and why I turned it on for this example.

Ask for twelve months, using JSON since that is easier to read here:

curl -H 'Accept: application/json' 'http://127.0.0.1:8080/report?months=12'

Three seconds later the response comes back:

{"type":"about:blank","title":"Gateway Timeout","status":504,"detail":"The request ran past the soft timeout of 3 s.","reason":"soft_timeout","request_id":"01M6T02F76G7XRY5V2P2V9S7WG","traceback":"..."}

When the request passed three seconds, a watchdog in the worker noted where the request was, then raised a muru.RequestTimeout exception in the thread running it. The exception unwound the report just like any other exception would, and Muru answered with a 504. The terminal says exactly where the request was when it was stopped:

muru.host[74087]: request.soft_timeout: worker 0 (pid 74088): request 01M6T02F76G7XRY5V2P2V9S7WG GET /report?months=12 ran past the soft timeout of 3 s at report.py line 14 in summarise, called from report.py line 21 in build_report, called from report.py line 29 in application
muru.worker[0]: Traceback (most recent call last):
muru.worker[0]:   File "/Users/graham/reports/report.py", line 29, in application
muru.worker[0]:     body = build_report(months)
muru.worker[0]:   File "/Users/graham/reports/report.py", line 21, in build_report
muru.worker[0]:     return f"{months} months: {summarise(rows)}\n"
muru.worker[0]:                                ~~~~~~~~~^^^^^^
muru.worker[0]:   File "/Users/graham/reports/report.py", line 14, in summarise
muru.worker[0]:     total += row * row % 7
muru.worker[0]:     ^^^^^
muru.worker[0]: muru.RequestTimeout
504 GET /report?months=12 638B 3.07s queue=0.0ms app=3.07s reason=soft_timeout worker=0 id=01M6T02F76G7XRY5V2P2V9S7WG

Only the one request was stopped, and the worker carried on serving others.

One detail worth knowing is that muru.RequestTimeout derives from BaseException and not Exception. A lot of application code catches Exception to log an error and carry on, and if the timeout were an Exception that code would swallow it and keep going. An application which does want to respond in its own way, such as telling the user to narrow down the date range, can still catch muru.RequestTimeout by name.

The soft timeout can only interrupt Python code. A thread stuck inside a C extension won't see the exception until the call returns. For that case there is also a hard timeout, which gives up on the whole worker and replaces it, but that is a topic for another time.

Which requests are in flight

A timeout tells you about a request after the fact. Often what you want is to know what is running right now, and with more than one worker, the first thing to work out is which worker a slow request is running in. A six month report takes about two seconds, so start one in the background while a few one month reports are also being requested, and ask Muru what requests are in flight:

curl -s 'http://127.0.0.1:8080/report?months=6' &
muructl requests

RELEASE              GROUP        QUEUED  QUEUED-OLDEST  THREADS  THREAD-OLDEST
report@r1            default      0       -              5/8      1.2s

Under the daemon you would need to say which application to look at, but a development run only ever serves the one application. The muructl commands find the run by its admin socket when you run them from the project directory, so there is nothing more to type.

This first view counts the requests for each release. Nothing is waiting in the queue, five of the eight threads across the two workers are busy, and the oldest request has been running for 1.2 seconds. Adding --list shows each request, oldest first:

muructl requests --list

AGE      STATE    KIND            WORKER  REQUEST                                  CLIENT           REQUEST-ID
1.2s     running  wsgi            1       GET /report?months=6                     127.0.0.1        01M6T02SA1G7XRY5V2P2V9S7WH
0.5s     running  wsgi            0       GET /report?months=1                     127.0.0.1        01M6T02T18G7XRY5V2P2V9S7WJ
0.4s     running  wsgi            0       GET /report?months=1                     127.0.0.1        01M6T02T56G7XRY5V2P2V9S7WJ
0.3s     running  wsgi            0       GET /report?months=1                     127.0.0.1        01M6T02T94G7XRY5V2P2V9S7WJ
0.1s     running  wsgi            1       GET /report?months=1                     127.0.0.1        01M6T02TD2G7XRY5V2P2V9S7WJ

The six month report is the oldest, and it is running in worker 1.

Strictly speaking, this isn't looking inside the workers at all. The answer comes from the edge, which knows about every request it has passed to a worker, along with how long it has been running and which worker it went to. Nothing has to be asked of the workers to get it, so it costs them nothing, and it still answers even when a worker is too wedged to respond. That is why it is the first place to look. Once you know which worker a request is in, the more expensive tools can be pointed at just that worker.

There is an equivalent muructl websockets command for WebSockets, which aren't being used here. It counts the WebSockets open for each endpoint, and with --list shows each one, the longest open first.

Seeing a request in progress

Knowing the six month report is in worker 1, the next step is to see what it is doing. Asking for the stacks of that worker:

muructl debug stacks --worker 1

Among the threads shown is the one serving the report:

muru-app-1 in worker 1:
  serving GET /report?months=6 for 1.3 s (request 01M6T02SA1G7XRY5V2P2V9S7WH)
  File "/Users/graham/Library/Caches/muru/runtime/0.1.0a18/cp315t/muru/_pool.py", line 30, in serve_jobs
    serve()
  File "/Users/graham/reports/report.py", line 29, in application
    body = build_report(months)
  File "/Users/graham/reports/report.py", line 21, in build_report
    return f"{months} months: {summarise(rows)}\n"
  File "/Users/graham/reports/report.py", line 14, in summarise
    total += row * row % 7

It shows which request the thread is handling, how long that request has been running, and where in the code it is at that moment. Leave off --worker and you get every thread in every worker instead, all without stopping anything.

How the stacks are obtained depends on the platform. On Linux with Python 3.15 or later, Muru reads the stacks from outside the worker process using the remote debugging support added by PEP 768, so it works even if a worker is wedged in C code while holding the interpreter. On macOS, where reading another process needs root, the worker is asked to report its own stacks instead, which is what happened here.

Where the time goes

A stack is a single moment in time. To know where the time goes overall, you want a profile, which samples the stacks repeatedly over a period and counts where they were. To get some load to profile, keep reports coming in another terminal:

while true; do curl -s 'http://127.0.0.1:8080/report?months=2' > /dev/null; done

and then profile the workers for ten seconds:

muructl debug profile --for 10s --output report.folded

808 samples of running Python code in 1632 rounds across 2 workers
   73.4%  summarise (report.py:10)
   26.6%  fetch_rows (report.py:5)
muru: collapsed stacks written to report.folded

Three quarters of the time is going on summarising the rows and a quarter on waiting for them, so a faster summary would be worth a lot more than a faster query. Being able to say that with evidence, rather than a hunch, is the whole point.

The file written out holds the collapsed stacks with their counts, which is the format flame graph tools expect, so flamegraph.pl can turn it into an SVG or you can open it in speedscope in a browser. If the application has many endpoints and only one of them is a problem, the profile can also be restricted to requests for a particular path.

Sampling every so often tells you proportions. If you want exact numbers instead, muructl debug trace counts every call of every Python function over a period, along with the time spent in each:

muructl debug trace --for 3s

worker 0 (pid 74088)
    total ms     calls  function
    1483.823         2  application (report.py:24)
    1483.674         2  build_report (report.py:19)
    1073.508         2  summarise (report.py:10)
     410.103         2  fetch_rows (report.py:5)
       0.095         2  parse_qs (parse.py:859)
       0.056         2  parse_qsl (parse.py:902)
       0.042         4  parse_qsl.<locals>._unquote (parse.py:937)
       0.031         4  unquote_plus (parse.py:983)
       0.014         4  unquote (parse.py:836)

worker 1 (pid 74089)
    total ms     calls  function
     736.969         2  application (report.py:24)
     736.857         2  build_report (report.py:19)
     531.764         2  summarise (report.py:10)
     410.168         2  fetch_rows (report.py:5)
       0.104         2  parse_qs (parse.py:859)
       0.064         2  parse_qsl (parse.py:902)
       0.048         4  parse_qsl.<locals>._unquote (parse.py:937)
       0.036         4  unquote_plus (parse.py:983)
       0.018         4  unquote (parse.py:836)

Tracing every call is a lot more expensive than sampling, so Muru will refuse to trace a worker which already has more than half of its threads busy with requests unless forced to with --force. The worker does keep serving while the trace runs though, since the requests it takes in that time are what the trace is measuring. Those requests are slowed down by the tracing, so this isn't something to leave running for long against an application people are using.

Memory and running code

Slow requests aren't the only thing that goes wrong. Python web applications also have a habit of slowly growing in memory until something gives. Running muructl debug memory reports for each worker its memory use, the garbage collector's counts and the object types it holds most of. Adding --trace turns on tracemalloc for a period and reports the places in the code which allocated the most memory during that time, and there are options for taking a snapshot at one point and comparing against it later.

When all else fails, muructl debug exec will run a snippet of code inside the workers and print the result, which is handy for checking the state of some cache or other in a running process:

muructl debug exec -- 'import sys; sys.version'

worker 0 (pid 74088):
'3.15.1 free-threading build (main, Oct 21 2026, 17:44:52) [Clang 23.1.3 ]'
worker 1 (pid 74089):
'3.15.1 free-threading build (main, Oct 21 2026, 17:44:52) [Clang 23.1.3 ]'

Under the daemon this one is turned off unless explicitly allowed in the daemon's configuration, since it lets someone run arbitrary code inside a production application. In a development run it is always on, since the person at the terminal is the one who started the workers in the first place.

Stopping at a breakpoint

All of the above is about looking at a worker from the outside. Sometimes though you really do want to stop in the middle of a request and poke around, and the usual way to do that in Python is a call to breakpoint().

The catch is that with a normal development server, breakpoint() drops you into pdb on whatever terminal the server's output is going to, interleaved with log output from everything else. In Muru, a worker which hits breakpoint() instead waits for a debugger to attach, and the terminal tells you how to do that. Add a breakpoint() call to build_report() after the rows are fetched, save the file so it reloads, make a request, and the terminal shows:

muru.host[75915]: debug.breakpoint: worker 0 (pid 75918): breakpoint() at /Users/graham/reports/report.py:21 waits 60 s for muructl debug attach report --worker 0

The end of that line gives the command to run, which is muructl debug attach report --worker 0. Running it from another terminal opens a pdb session in the worker, at the point the request stopped:

muructl debug attach report --worker 0

muru: attached to worker 0 at /Users/graham/reports/report.py:21; the end of input lets the code go on
> /Users/graham/reports/report.py(21)build_report()
-> breakpoint()
(Pdb) p months
1
(Pdb) continue
muru: the session in worker 0 has ended

Typing continue lets the request carry on, and it completes as normal. While the session was open, the worker was taken out of routing so it wouldn't be given any new requests, leaving those to worker 1. The terminal records that, along with the worker being put back once the session ended:

muru.host[75915]: worker 0 (pid 75918) leaves routing while it is debugged
muru.host[75915]: debug.detached: worker 0 (pid 75918): the debugger session at /Users/graham/reports/report.py:21 ended
muru.host[75915]: worker 0 (pid 75918) serves again: no debugger is attached
200 GET /report?months=1 19B 2.28s queue=0.0ms app=2.28s worker=0 id=01M6T043M4G7XRY5V2P2V9S7WJ

The request was also exempt from the soft timeout, so it wasn't killed off while I was looking at it. If nobody attaches within the timeout, the request just carries on by itself.

For those who prefer to debug from an IDE, muructl debug attach --dap speaks the Debug Adapter Protocol instead, so something like VS Code can attach to the worker through muructl. That needs debugpy installed in the project's environment, and I will leave the details of that to the Muru documentation.

The dashboard

Everything so far has been done from the command line. When you just want to keep an eye on things, the run also serves a dashboard, by default at http://127.0.0.1:8081/.

The top of the page shows the last minute at a glance, with requests per second, the 50th and 95th percentile latency, the count of server errors, how many requests are in flight and how many WebSockets are open. Below that is a table of the workers and how many of their threads are busy, charts of the same figures over time which can be switched to cover the last hour or the last day, the most recent requests with the same queue and application times as the access log, and the latest output from the run. A link from the workers table opens a page for each worker with its status, and the same stacks, tasks and memory figures the muructl debug commands give. In a development run there is no sign-in, and the dashboard is read-only, so nothing on it can change what the run is doing.

The same address also serves a /metrics endpoint in the Prometheus format, so if you do want to point some other monitoring system at a development run, you can.

What's next

What I like about all this is that none of it required changing the application, installing an agent, or restarting anything, and all of it works the same way against an application running under the daemon in production. Along the way I have been talking about hosts, workers, the edge and admin sockets without really explaining them, so in the next post I will step back and look at how the parts of Muru fit together.

07 Nov 2026 12:00am GMT

TL;DR: gilguard is a tiny, zero-dependency package for free-threaded Python (3.13t, 3.14t, 3.15t). Import it first. If any later import quietly turns the GIL back on, you get an error naming the extension module, the package that shipped it, and the line of your code that pulled it in. python -m gilguard audits an entire environment in one go. It's on PyPI now, and it's the second piece of a larger free-threaded Python effort I've been working on.

The quiet failure mode of free-threaded Python

Free-threaded Python is officially supported as of 3.14, and I'm all in. Threads that actually run Python code in parallel? Yes please. But the migration has a failure mode that I think is going to bite a lot of teams, and there isn't much written about it.

Here's the scenario. You move a service to 3.14t. You give the hot path a thread pool. You deploy, pull up the dashboard, and the throughput graph does… nothing. It's the exact same shape as last week, as if the whole migration had been a dream.

An hour later you find the reason, one line in a startup log that scrolled past in half a second:

RuntimeWarning: The global interpreter lock (GIL) has 
been enabled to load module 'somepackage._speedups', 
which has not declared that it can run safely without the GIL.

That line is CPython being careful. Every C extension has to declare that it's safe to run without the GIL (via the Py_mod_gil slot). When the interpreter loads one that hasn't, it turns the GIL back on for the rest of the process and carries on. Your threads still run, they just take turns again. And here's the kicker: once the GIL is on, nothing can turn it back off for the life of that process.

The warning names a private module you've never heard of. It doesn't tell you which package you installed shipped it, or which of your imports dragged it in. So now you're bisecting imports.

What is gilguard?

gilguard turns that one line into an exception, raised at the import statement that caused it. Make it the first import in your program and you're done. You don't call anything or configure anything:

import gilguard  # must be first

import fastapi
import pandas  # <- raises here if anything in pandas' import tree needs the GIL

When something in that tree forces the GIL back on, this is what you see instead of a warning you'd miss:

gilguard.GILEnabledError: Importing 'gilneeder._speedups' re-enabled the GIL.

  Module:       gilneeder._speedups
  File:         .../site-packages/gilneeder/_speedups.cpython-314t-x86_64-linux-gnu.so
  Distribution: gilneeder 1.2.0
  Import chain: gilneeder -> gilneeder._speedups
  Your code:    /home/me/app.py:5
  Imported by:  .../site-packages/gilneeder/__init__.py:1

This extension module does not declare free-threading 
support (Py_mod_gil), so CPython turned the GIL back 
on for the rest of the process. It cannot be turned off again.

Options:
  - Upgrade gilneeder to a release that ships free-threaded (cp314t) wheels.
  - Drop or replace gilneeder.
  - If you have verified it is thread-safe, run with PYTHON_GIL=0 to keep the GIL off (at your own risk).
  - Run `python -m gilguard` to list every installed package that does this.

(gilneeder is one of gilguard's test fixtures, a tiny C extension built to misbehave so the real ones don't have to.)

Everything you'd otherwise be asking a debugger for is right there: which module flipped it, what file it lives in, and which distribution and version shipped it (so you know what to upgrade or report). You also get the import chain from your code down to the culprit, which matters when the offender is a dependency of a dependency of something you installed for one function, and the line in your code where it all started.

You might notice that GILEnabledError derives from BaseException rather than Exception, putting it alongside KeyboardInterrupt. Here's why. Lots of packages have an optional speedup import like try: import fastthing / except ImportError: import slowthing. If gilguard raised a normal exception, that except would eat it, the package would quietly fall back to its pure-Python path, and your app would carry on with the GIL already back on. The report would end up inside someone else's except block, where nobody would ever see it. With BaseException, it sails right past those handlers. To catch it, you have to ask for GILEnabledError by name (or use a bare except:, and you know better).

Not ready to fail hard? Set GILGUARD_ACTION=warn and you get the same report as a warning while the program keeps running. Set it to off and the same code runs on a standard build.

How do I find every offender in an environment?

A single process can only tell you about the first module that flips the GIL. After that it's already on, and the rest have nothing left to flip. So gilguard also ships a scanner:

python -m gilguard

GIL    gilneeder                via gilneeder._speedups  [gilneeder 1.2.0]

8 checked: 7 ok, 1 re-enable the GIL, 0 failed to import.

It gives every installed package its own fresh interpreter (in parallel, across your CPUs), so none of them can hide behind the one that went first. It exits 1 when it finds anything, so it drops into CI as a single step without any extra scripting. There's --json for tooling and --strict if a package that fails to import should fail the build too. Pass package names to check just those, which is how you re-check one after an upgrade to see if the fix took.

The scan has one blind spot: extensions a package loads lazily, on first use of a feature, don't show up at import time. For those, run your test suite with gilguard loaded as a plugin:

pytest -p gilguard

pytest loads -p plugins before it touches conftest.py or any test module, so the guard is watching before your code imports anything. The two checks cover different gaps. The scanner sees every installed package but only at import time, and the test suite sees lazy loads but only on the code paths your tests run. Use both. Tests and CI has a GitHub Actions job that does exactly that.

What does a clean run actually mean?

"The GIL stayed off" is easy to over-read. Here are the limits. A clean run means none of the imports that actually ran turned the GIL back on. It does not mean your code or your dependencies are thread-safe, and it does not mean your program gets faster with more threads. Those are separate problems with their own tests. gilguard catches one specific failure, and that's all it tries to do.

Once you have a name, the docs have a whole page on what to do with it: upgrade, report upstream (with exactly which lines to paste into the issue), replace, or, if the extension is yours, the one slot that declares support, with the one-line equivalents for Cython, pybind11, and PyO3.

Why I built this

When I announced chameleon-rs ten days ago, I said free-threaded Python means we need to start expecting more from our web apps. What I didn't say is that chameleon-rs was the first piece of a larger free-threaded effort, and just the first one I am able to talk about publicly. gilguard is the second.

It's a much smaller piece, a few hundred lines of plain Python with no dependencies. But it's the kind of building block everything else depends on. There's no point making a template engine or a web framework thread-safe and fast if one dependency turned the lock back on at startup and nobody noticed for a month. Before you can make free-threaded Python fast, you have to be able to trust that you're actually running free-threaded.

My goal with all of this work is simple to state: free-threaded Python faster, safer, better, and adopted sooner. gilguard is my contribution to the "safer" and "sooner" parts. I'll write about the rest as soon as I'm able to.

Try it

uv pip install gilguard

Then import gilguard at the top of your entry point, or run python -m gilguard against your current environment and see what's lurking. I'd love to hear what it finds. The docs are short and the code is on GitHub. If you use an AI coding agent, the docs also ship an agent skill that teaches it the rules (first import, don't catch the error, what a pass means), so it can wire gilguard into your project and CI without you explaining any of this.

06 Nov 2026 11:29pm GMT

Learn about Python's log handlers, part of the built-in logging module, in this video.

This video is a part of my Python Logging Video Course. Check it out today!

The post An Intro to Python Logging Handlers (Video) appeared first on Mouse Vs Python.

06 Nov 2026 7:08pm GMT

News

Django security releases issued: 6.1.3, 6.0.10, and 5.2.19

Six CVEs, three of them in GeoDjango: two server-side request forgery holes where GDAL resolved URLs or files from GeoJSON CRS members and SpatialReference strings (full protection needs GDAL 3.9+, flagged by the new gis.W001 check), plus log injection through GeometryField. The rest cover base64 uploads that ignored FILE_UPLOAD_MAX_MEMORY_SIZE, cache keys that confused missing and empty Vary headers, and password hashes shown to view-only admin users. Upgrade soon.

Django Software Foundation

Django Steering Council Meeting, November 2

The Steering Council met with the Fellows ahead of the mid-January feature freeze, with Carlton Gibson aiming to land the async transactions PR before then. They also weighed what to do now that sqlparse is no longer maintained (it backs --debug-sql and inspectdb), how to surface DEP discussions on the Forum, and GSoC 2027 ideas.

Python Software Foundation

Announcing new Security Developer at the PSF

Brett Smith, formerly of the W3C and Software Freedom Conservancy, joins Seth Larson and Mike Fiedler on the PSF security team in a new role funded by Anthropic. The Python Security Response Team is handling 3-4× last year's volume of reports and CVEs, and PyPI malware is up 60%, so Brett will help with remediation and scale PyPI's autonomous quarantine program.

How the Python ecosystem responded to the eMudhra intermediate compromise

Within two days of the October 20 CA compromise, fixes shipped for certifi, Python, pip, uv, sigstore, virtualenv, and more, and Python added a last-resort denylist for the bad intermediates (CVE-2026-115557) that was backported to older releases. Seth Larson explains why certifi, pip, and Python sat in the critical path, and why browsers were safe from day one thanks to certificate revocation lists, which Python may adopt.

Wagtail CMS News

Wagtail 8.1: automate more with an improved API and agent-ready tools

The preview REST API can now write more content types, expose StreamField block schemas, and drive moderation workflows (submit, approve, reject, cancel), alongside official agent skills and a prototype CLI. Editors also get SEO checks for meta titles and descriptions, pages can be served as JSON or Markdown from the same URL, and multilingual sitemaps now include translated pages.

An official headless demo for Wagtail

bakerydemo-headless rebuilds the bakery demo as a fully decoupled Next.js front-end in TypeScript, with live preview still working in the CMS. Thibaud Colas notes it runs about 95% on Wagtail's built-in v3 API, whose OpenAPI schema generates typed clients down to individual StreamField blocks, with no GraphQL required.

Updates to Django

Today, "Updates to Django" is presented by Hwayoung from Djangonaut Space! 🚀

Last week we had 17 pull requests merged into Django by 12 different contributors - including 2 first-time contributors! Congratulations to Alex Henman and Spoko Dev for having their first commits merged into Django - welcome on board!

News in Django:

• Django now supports the HTTP QUERY method, allowing safe, idempotent requests with query content in the request body (#37232).
• Django templates now support template tags spanning multiple lines, making tags with several arguments easier to read (#35899).
• Fixed a regression where BoundField.legend_tag() could add a custom required CSS class more than once when passed attrs (#37463).

Django Fellow Reports

Django Fellow Report - Jacob

Jacob Tyler Walls finalized the patches and sent prenotifications for Django's November security releases, then started drafting patches for the next round. He also fixed attrs mutation in legend_tag(), opened a follow-up optimizing lookup preparation for GeoJSON geometries, and sat in on Executive Director interviews.

Django Fellow Report - Sarah

Sarah Boyce accepted five tickets, including consistent database selection for subqueries, dropping GIS support on 32-bit Windows, and a parallel test runner crash, and reviewed fixes for XOR join promotion, select_for_update() after combined queries, and warning assertions. Her own PR allowing multiline template tags was merged, and she worked on three security reports and interviewed final candidates for Executive Director.

Django Fellow Report - Natalia

Natalia Bidart prepared the branch that bootstraps Django 2028, the first calendar-versioned release: it drops Python 3.12 and 3.13, removes the EMAIL_* settings in favor of MAILERS, and finalizes several ORM deprecations, and it lands in mid-January once the 6.2 branch is cut. She also pushed docs PRs that give local builds a new theme and better accessibility, and joined three Executive Director interviews.

Sponsored

Django for Beginners, 6th Edition

The sixth edition is out now, available in digital and paperback formats. It features two new chapters, completely updated for Django 6.1+, and more. Read the first four chapters for free to try it out.

Articles

Soft delete in Django without making a mess

Stein Ove Helset builds soft delete on a deleted_at timestamp and works through where it breaks: bulk QuerySet.delete() still removes rows, unique=True blocks re-creating a deleted record (a conditional UniqueConstraint fixes that, though MySQL and MariaDB ignore the condition), restores can collide with newer rows, and CASCADE never fires for related objects. A filtered default manager paired with an unfiltered base manager keeps related lookups working.

N+1 queries in Django Admin

If a related model's __str__ reads another foreign key, every option in an admin dropdown runs its own query, and select_related() on the edited object doesn't reach those options. Override formfield_for_foreignkey() and formfield_for_manytomany() to add select_related() to the querysets behind the widgets.

Django's Generic Views: TemplateView

Ben Cardy's series on class-based views reaches TemplateView, which needs only a template_name. He uses extra_context for static values, get_context_data() for computed ones, and template_engine to render a single view with Jinja2.

Taking Website Screenshots with Django, Celery, and Playwright

Nik Tomazic builds a Dockerized Django app that takes full-page or viewport screenshots with Playwright. Each capture runs in a Celery task so the form returns right away, and the page polls a small JSON status view until the image is ready. An optional last section moves the browser to Browser Use's paid hosted Chromium for when local browsers eat worker memory or hit bot detection.

Looking for a job? Start a blog.

Tim Schilling argues that a blog shows a hiring manager how you communicate before they ever talk to you, and suggests starter topics like a recent technical decision or how you review code. He says the host doesn't matter (GitHub Pages, dev.to, and Medium all work) and asks you to write it yourself instead of with AI, since the point is to show how you think.

How Caktus Runs and Uses Local Models

Tobias McNulty and Colin Copeland describe the two-GPU workstation Caktus uses to serve open-weight models over Tailscale, so client code stays on its own network. A dense Qwen3.8-27B handles planning and long sessions, and a Qwen3.6 MoE with 3B active parameters runs everyday work at 200-250 tokens per second, both on vLLM in Docker Compose. The post includes their benchmarks, why Gemma 4 31B didn't make the cut, and the pi.dev and VS Code configs they use daily.

Per-Object Permissions in Django That Roll Back with Atomic()

Joshua Fields wrote pemba, a Zanzibar-style authorization library that keeps per-object permission relationships in your own Django database. A grant is just rows in the current transaction, so it commits or rolls back with the data it protects, which a separate service like SpiceDB can't guarantee.

Artisanal Podcasting in the Age of AI

Will Vincent has hand-edited Django Chat in GarageBand for nearly seven years, even with AI tools that promise to handle the whole thing. Editing gives him a second listen to each conversation and a kind of deep focus he no longer gets when coding with a handful of AI agents.

Caching at Scale in Django: Lessons from Production

Johnny Metz's team handled about 20x traffic growth on a multi-tenant Django API by caching aggressively: per-user lists of document IDs paired with one shared entry per document, seven-day TTLs with invalidation on Django signals, a cached identity lookup, and failing open when Redis goes down. During the rollout, database queries per second halved, and Redis now serves most reads at a 97% hit rate.

Events

DjangoCon Europe 2027 Call for Participation

DjangoCon Europe runs February 17-21, 2027, in Innsbruck, Austria, and wants talk and workshop proposals on technical and non-technical topics from new and experienced speakers. Proposals close November 15, 2026.

Podcasts

Django Chat #209: Databases, Django APIs, and AI - Haki Benita

Haki Benita talks about getting more out of Django's ORM and PostgreSQL indexes, why Django Ninja appeals for APIs, and what Django needs for typing and a built-in API story. They also get into how AI is changing the way developers learn and share what they know.

Videos

From librarian to Djangonaut: My Open Source journey with Annabelle Wiegart

In a guest talk for Djangonaut Space Session 7, Annabelle Wiegart traces her path from librarian to Django contributor, PyLadies Zurich organizer, and Djangonaut Space captain. She talks about working on ORM tickets and nullable expressions with mentors, and how contributing went from intimidating to something she could do.

Django Forum

A Path to Modernizing APIs in Django (DEP 0017 Content Type Parsing)

Picking up from the Haki Benita episode of Django Chat, Will Vincent lists what stands between Django and a built-in API: a modernized HttpRequest through DEP 0017, a pluggable serialization layer (the hardest part), and then the view layer. Carlton Gibson wants those primitives updated first, while Haki would settle on an official API interface and tune performance later.

Django Job Board

The PSF's CPython Developer in Residence opening leads the board this week, alongside senior Python roles at webbyfox consultancy and at Manychat in Barcelona, with relocation support.

CPython Developer in Residence at Python Software Foundation

Senior Python Engineer at webbyfox consultancy

Senior Python Engineer (Barcelona-based / Relocation support to Barcelona) at Manychat

Projects

amureki/django-cmdbar

Adds a Cmd+K / Ctrl+K palette to the Django admin for jumping to models, searching records (user alice), or opening one by primary key (user#42). It respects admin permissions and needs no frontend dependencies, middleware, URLs, or migrations: just add django_cmdbar above django.contrib.admin in INSTALLED_APPS.

Metron-Project/izaya

An htmx-powered autocomplete widget for Django forms that ships no required styles, so it fits any CSS framework. Register a ModelLookup with your search_fields, drop LookupWidget into a form (with multiple=True for multi-selects), and use rank_fields to put the closest matches first; it needs Python 3.14+ and Django 6.0+.

Sponsor Django News

Reach 4,300+ Django developers every Friday. See sponsorship details and rates.

06 Nov 2026 5:00pm GMT

Twisted was originally the network component of a text-based role-playing game engine. In the late 90s when it was first developed, the idea of a video game being able to send you actual emails and instant messages¹ seemed pretty neat, rather than the valence that it would have today (nightmare). We never ended up making an actual game, so we eventually removed the "game engine" piece, and just kept the protocol implementations of the email and the AOL Instant Messenger, and the DNS and eventually SSH and so on. The residual pieces of that project have not seen maintenance for a decade.

So it's not really a way to make a text-based game any more.

Then, Twisted was the premier longest-lived and best-maintained event-driven I/O library for Python. If you wanted to make high-performance servers, it was the best and most obvious way to do it. Even if it still acquits itself relatively well in this regard, its claim to fame, and to indispensability, has been somewhat obviated by the inclusion of asyncio in the standard library. Even if it has some stuff to recommend it - better-than-average testing facilities, for example - you can get better-documented and better-integrated low-level asynchronous networking tools without even installing any packages.

So it's not really the premier choice for an event-loop these days.

Twisted does still contain an SSH server and client, an email server and client, a DNS server and client, a web server and client (now including websockets). But you can cobble these together by installing a variety of packages from other sources.

So... it's a bunch of protocols? And I guess it doesn't use AI? But... why would anyone care about that? Why contribute to such a thing?

What is Twisted doing, and who is it doing it for? What is the political project of Twisted, nigh on 30 years into its existence?

Most of all, why should you care about it, or want to contribute?

If you're asking what you could use Twisted for today, it's an excellent choice if you're implementing a new network service and you need more than one protocol. If you need to do HTTP, DNS, packet capture, and a custom SSH server, it would be very hard to arrange a collage of other Python packages to manage this. It's still much better at deterministic testing than most of the standard library. It's a bit easier to hook into a GUI.

But that's not what Twisted is doing.

What I want Twisted to do for the world is still an unfinished project. It's honestly kind of bad at doing what I want it to, but it could be great. The overarching goal of having and maintaining all this stuff is to, as much as possible, demystify networked software operation and let you build Internet facing systems at a human scale.

It's all in a memory-safe language with good error handling, so you can avoid the scariest security pitfalls. It's all integrated into a single paradigm with built-in tools for testing. It's designed to scale down to be runnable on the average laptop, and not require large elaborate server farms or distributed clusters to run. Via the twist command line utility, it provides a Swiss army knife of simple tools that you can use out of the box.

In a world where all of our compute is in massive distant opaque datacenters, where a tiny cartel of gigantic tech companies control nearly the whole supply chain, one of the few things that you still can have agency over is your own server on the Internet.

It's still possible to get a domain name for a few bucks a year, a free-tier or very low-cost VPS, then stick a glue record in and run your own DNS. From that point forward, you can run your own email, your own web server, your own terminal server. But doing this requires deep expertise, and that expertise requirement is massively inflated by the operational concerns that go with running twenty different pieces of software from twenty different projects, probably in twenty different languages, with completely inconsistent aesthetics.

The reason that Twisted is event-driven is because as much as it can be a pain to wrap your head around async/await (or worse yet, Deferred directly), it is much easier for a single person to understand the concurrency requirements of an async I/O multiplexed system than to understand what is going on with an application that requires multiple thread pools. It is possible to understand concurrent code that manages parallel tasks in that idiom. The reason that Twisted integrates so many different protocol implementations and attempts² to unify them with a common authentication layer is to reduce the overhead required to manage a comprehensive suite of network services.

Even the not-using-AI thing is not an aside due to some accidental personal preference about tools - it's a core part of building an engine that you can personally understand if you dive into it, that's built with a consistent style and layered in a way that you can take apart if you need to. It's not built with AI because we're not building for AI, we're building for people and the entire point is that people are supposed to be able to understand it.

We used to have a tag line, which is somewhat less emphasized these days because its meaning is not immediately clear without this background, but was once quite prominently featured on the home page: "Twisted: Building the Engine of Your Internet". That's what I want to bring to you: an engine for you to build your Internet, that is small enough that you can make it personal, that doesn't require massive investment in management tooling just to get started.

Now, for various reasons, I no longer use twisted.names to host my own DNS (although I did for many years!), and we never wrote enough of a twisted.mail suite for it to be usable for email hosting on its own. But, if you're reading this article, it was probably delivered to you directly via Twisted's own twisted.web HTTP server, using twisted.protocols.tls to configure our own OpenSSL TLS termination. I have run with extremely high reliability for years and next to zero management overhead, in a single simple docker container. Which I then administer and upload files to via Twisted's own twisted.conch SSH server.

If that seems like an appealing project and you'd like to volunteer to help out, drop me a line and maybe I can point you at some tasks to get started. Twisted falls far short of this aspiration despite 30 years of work, and we could use all the help we can get.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!

05 Nov 2026 10:51pm GMT

🔗 Links

• Haki Benita's Website
• Django Ninja
• Reliable Django Signals (DjangoCon Europe 2026)
• Workshop: Advanced Indexing and Concurrency in Django and PostgreSQL (DjangoCon Europe 2026)
• Don't Blame the ORM: Advanced Indexing in PostgreSQL (PyCon Italia 2026)
• How to Get Foreign Keys Horribly Wrong (DjangoCon Europe 2025)
• Unlocking the Full Potential of PostgreSQL Indexes in Django (DjangoCon Europe 2021)
• One Database Transaction Too Many

📚 Books

• In the Distance by Hernán Díaz
• Django for Beginners, 6th Edition by William S. Vincent
• Storyworthy by Matthew Dicks

📦 Projects

• Django Admin
• django-rosetta
• Django New Features

🤝 Sponsor

Thanks again to Laravel Cloud for sponsoring this episode. The team behind Laravel's managed hosting platform has something big in the works, and we'll share more here on the show in the coming weeks.

🎥 YouTube

• YouTube Channel: @djangochat

05 Nov 2026 1:50am GMT

This tutorial looks at how to build a Django app that takes website screenshots using Playwright and Celery.

04 Nov 2026 4:28am GMT

Creative Work

Writers went on strike to get protections against "AI". Thousands of artists have signed open letters in protest of "AI". There are so many copyright lawsuits from creative industry groups against AI that there's a whole dedicated website for it. Popular YouTubers absolutely hate it. If they're also musicians, they REALLY hate it. Across all creative industries, there is a concerted push to reject this technology.

Yet, almost unique among creative fields, many experienced programmers remain convinced that it's fine to use "AI" for programming. We do seem to hate it, and what it's doing to our industry. It's making us all miserable, but we are using it anyway.

A lot of the justification of this resignation seems to be to be because programming is not Art. If the tool can get the job done, and the job is just functional, then why does it matter?

It does matter, though. It matters because we shouldn't be using AI to produce Art, and programming is Art.

Art can be Mundane

Some people will say that programs cannot be art because programs are functional, rather than being expressive. Programs are mundane whereas art is transcendent.

This is based on a distorted understanding of what Art actually is.

In John Berger's "Ways of Seeing", he names this type of distortion "mystification". His example of this process is both amusing and illustrative. I encourage you to read it in its entirety.

In summary, though: Berger critiques the florid prose of an art historian describing a commissioned group portrait, including phrases like "subtle modulations of the deep, glowing blacks" and "harmonious fusion". The portrait is described as sublime, in nearly ecstatic terms.

Berger reveals that the reality of this portrait is that a poor old painter needed some work, and some officials probably thought it might be nice to have an official portrait. So they paid some money to the poor old man, and he painted it, and then they had a painting. It's a well-executed portrait of a group of people. Beautiful, even. But it was work commissioned for a fairly mundane purpose and it suited that purpose just fine. It was not, and is not, a divine relic.

Culturally, we are prone to mystifying painting, and sculpture, and film, and music. We imbue them with "subtle modulations". We ignore their functional aspects - we desire decoration, amusement, and distraction - and focus on their emotional impact.

Don't get me wrong: I love me some good aesthetic philosophy. I think it's great to really examine our reactions to artwork and to try and gain a deeper understanding of our culture and our selves through media analysis. If anything we really need to do more of it.

This does not mean that the creation of such works is mystical or that it should be venerated beyond any other sort of labor.

Not least of which other types of labor that are adjacent to, but not as culturally venerated, as fine art. We tend to mystify the work of a novelist, but to denigrate the work of a journalist. In reality, the functional prose of the journalist is no less important and deserves no less respect.

Although they might be far below the ethereal realm that novelists inhabit in our collective imagination, even journalists receive more respect and thus more mystification than lowly copywriters. Yet, there is no transcendental distinction between "novelist" and "copywriter"; the many of the skills are the same, and the distinction is merely an accident of commerce and opportunity.

In fact, many famous writers have famously inhabited both roles. This is not an accident! Working with words professionally, even (perhaps especially) mundane words, is excellent practice for working with words in a more purely artistic context, because even mundane creativity is still artistic.

Code can be Beautiful

One thousand Internet years ago, when I was in my late teens, I would describe myself as a "code poet". I was relentlessly mocked for this as what the Youth would today call "being cringe", and at the time was referred to as "pretentious".

I succumbed to the peer pressure, removed it from my email signature and my bio. While I still believed strongly in the parallels, I accepted that - socially, at least - comparing code to poetry, or indeed to Art, was a silly thing to do.

However, I never abandoned the idea, in my heart.

The thing that I am most well-known for, the invention of Deferred, was specifically an aesthetic reaction to the tedium of passing callback and errback parameters to every single remote procedure call in an RPC client/server application. Those two callbacks got the job done just fine. But they were ugly, and annoying to work with.

Deferred is an intentional poem about asynchronous task execution, with a deliberate eye to the aesthetics of the problem and the experience of using it. It was influential because of its focus on aesthetics.

I do not want to overstate the beauty or profundity of this minor contribution, or indeed its durability. That a poem exists does not mean it is a great poem, merely that it is a poem.

Our aesthetic culture around programs is more like folk epic poetry than fine art, so the influence of this contribution is less about its specific enduring power than it is about its influence on what came next; from MochiKit.Async to JQuery Deferred to JavaScript Promises and eventually to async/await; a long chain of different artisans each adding something of their own until the original has all but dissolved. (And I wasn't the "original" here, either, as I drew heavily from the E language's Promises, among other things.)

In order to make code into a deliberate artistic expression, one must have spent quite a bit of time contemplating the problem domain. Without having experienced the tedium of manually passing a thousand callback parameters, I would have had neither the skill, nor indeed the motivation, to bother creating such a thing.

Now, most code does not have to be like this. Most code does not get to be like this. Most code is functional, workday code. Most code could not make a lady weep. It's just copy-writing, if you will.

As I explained previously, most writing couldn't do that either. Most writing is just copy-writing, too. Most visual art is advertising. Most live music performance is background music in bars that will go largely ignored.

However, code that is intentionally aesthetically designed tends to be important, both socially and technologically.

We do have some tradition of self-mystification in software. As Abelson memorably put it, "Programs must be written for people to read, and only incidentally for machines to execute.", so we have long had some conception of programs as highly expressive, even if we can't always agree on what they're expressing or to whom. We will occasionally wax poetical about the philosophical implications of a particular piece of software. This is not unique to a single piece of software, either; more than one community has indulged in similar philosophizing.

The expressive and aesthetic qualities of software are not limited to reading source code or interacting with other programmers via APIs, either. For example, every year, Federico Viticci does a review of Apple's new operating system, which is (among other things) an aesthetic critique. Such a project would not be possible if the software did not have an aesthetic impact on its users.

Not to mention that every video game review is also a software review.

Defend The Mundane

If we use AI to erase all the copy-writing, all the graphic design, all the boring mundane art, and yes, all the boring custom WordPress theme development, then we will be removing all the practical opportunities for the vast amounts of practice and contemplation required for people to elevate their craft to eventually achieve great things. Education is great, but the majority of true skill development happens on the job and always has.

This doesn't mean that we can't use abstractions, or automation, to make our work easier. Programming is the art of abstraction, of understanding how to compose smaller ideas into bigger ones, of how to understand the automation of a larger system by understanding the rules that automate smaller ones and understanding how to combine them.

When we use "AI" to eliminate that understanding rather than raise it up to a higher level, to entirely destroy that creative decision-making process, we do a disservice both to ourselves as programmers and to our users. We would be doing a disservice to our users and our downstream fellow developers in the same way that a visual artist would be doing a disservice to their viewers or a musician would be doing a disservice to their listeners if they served them auto-generated filler instead of their own creative output.

Slop is slop, no matter the medium.

Each mundane project has some tiny chance - let's say, something like 0.1% - of achieving greatness. If we do a single project with AI, then sure, whatever, there's almost no chance that that project was going to be the one hit to create that career-defining moment for an engineer working on it. If we make a habit of doing all projects that way, though, we take the total likelihood of those moments of greatness from "definitely sometimes" to "never".

The precisely appropriate ways in which to resist AI encroachment on all software development lie well beyond the margins of this one short post. How much you can resist and which specific uses you should resist are up to you. But it is worth resisting in software just as much as it would be worth resisting in any creative medium.

Programming isn't special. It's just Art, and Art is the most human - and thus, the most universal - thing that there is.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!

09 Oct 2026 7:00am GMT

One of the issues that I have with the current generation of "AI" products is that they do not appear to take their own premises seriously. I look at a plethora of obsequious chatbots claiming to be serious tools for problem solving, and I think, this is not what a problem-solving tool would look like.

Even before we get to the tremendous ethical problems with the frontier labs, it is this impression of their composition as a product that makes me feel, constantly, whenever I am interacting with them, that they are less a software product than that they are a grift, a scam designed to make me feel like I am interacting with a product that has capabilities that it simply does not, to try to lull me into a false sense of security that I can trust it.

The frontier labs are of course the worst offenders, but every criticism here applies just as much to Ollama, which (if anything, due to the obviously poorer quality of the available models themselves) needs these features even more than the frontier labs do.

Here, I will set down a few features that might convince me that an LLM-based product, particularly one focused on research or software development, was actually serious about helping me do useful things with it.

Make "Checking For Mistakes" A First-Class Feature

This is the biggest issue, and the major reason that I was inspired to write this post.

It is a truth universally acknowledged, that AIs cannot reliably provide information.

I could cite a ton of news articles and studies about this fact, but there is no need. Every single chatbot admits this, up front, in a fine-print disclaimer as a core part of their user interface. Gemini says "AI can make mistakes, so double-check responses", Claude says "Claude is AI and can make mistakes. Please double-check responses.¹" ChatGPT says "ChatGPT can make mistakes. Check important info.".

Every time I see that last one, I wonder how I'm supposed to know what "info" is supposed to be "important".

All of these warnings are all small, gray text, painfully obviously included as legalese to push responsibility back onto the user rather than to help with anything. This is a core limitation of all these products. Checking their output is a part of the workflow for using them that:

1. you absolutely cannot skip or skimp on without creating risks to yourself and whoever you are conveying its output to, and,
2. it is very easy to skip or skimp on and you are encouraged at every turn to do so, because "just trust the output" is one of the quickest ways to save time.

A chatbot product that took this weakness seriously, as an actual consideration for using it, would put a checkbox next to every claim in its output. It would be a 2-column worksheet, where you've got the LLM output in the first column, and next to it, human notes in the second column, explaining what work went into checking this claim, and a big checkbox that you would only check off after you believe you'd checked its claims thoroughly enough.

Coding assistants would need to have some version of this as well. Right now, this is pushed off into code review, which means it is a dark pattern which subtly encourages the "author"² to offload this work to their code reviewer without ever looking. Once again, "it's probably fine, I don't need to check" is the quickest way to save time and churn out those PRs faster.

It might even be useful for coding harnesses to have some affordance for checking code before it even runs tests. As the vendors themselves have admitted, it's not just expensive to burn tokens on your "AI", you also end up burning far more compute on the AI. Being able to check your diffs before sending them over to uselessly exhaust your testing compute cluster would be useful.

If your product tells me that it makes mistakes and I must be the one to check for the mistakes, but then gives me zero tools to check for mistakes, I cannot take it seriously.

More Citations to Check, And More Details

Most chatbots prefer to give an answer, rather than a citation. In my own personal use, I find that when asked to provide a list of citations with clearly marked sources for each one, they will appear to "get bored" halfway through the list and simply stop including citations at some point.

When the bots include citations at all, present them as inline annotations that say nothing but the domain name of the search result, in a font so small that it's barely legible, and an equally indecipherable icon that is fewer than 16 pixels on a side.

This is backwards.

Now, I am aware that these citations do come from somewhere, and in an attempt to reduce hallucinations, all of the major providers support some form of "grounding"³, and that those little barely-readable citation links are referencing actual structures in the RAG pipeline and not just potentially-hallucinated tokens, but I'm not talking about the underlying machinery in the model, I'm talking about the presentation to the user.

Plus, regardless of whether a snippet of text came from a RAG query, we know that LLMs can never provide an authoritative result; it's a fundamental limitation of the technology. They can still garble the results of RAG as much as they can misrepresent any other training data. This means that it must never present its results as authoritative.

If you ask an AI to do research queries, every result should be presented as a list of citations. Moreover, the presentation should display each citation as a large object of in its own right, with clearly identified metadata, including not just the site where it was found but its publication date and, if possible, the name of the author. The literal, unmodified quotation (not from RAG, not a summary: a quotation extracted with a regular program and not an LLM) should be front-and-center, larger than any AI-generated text.

If the AI product wants to editorialize or summarize (which should not always be necessary!), the AI-generated text should be presented as small text underneath the citation that has been found, de-emphasized as much as the disclaimer is right now, at the very least until the user has verified that the summary is accurate. Perhaps, for a research project, a "did you read the citation" checkbox might even be helpful.

If your product openly tells me that it will scramble, misrepresent, or omit its citations in its summaries, and I must read the original human-authored citations to be sure, but then gives me no tools to track my reading of those citations or even any way to find them, I cannot take it seriously.

No First-Person Output, No Apologies

There is no reason for a software development or research tool to use first-person language to describe itself. They should not do so. In fact they should not be allowed to do so.

There is also no reason that they should ever apologize. It is a waste of everyone's time; it's a waste for the chatbot to generate the apology, it's a waste for the user to read the apology, and it's a waste for the user to respond to the apology. Yet they unfailingly do this upon every correction.

The vendors of these tools know that they are routinely causing mental-health crises. In response, they have added non-functional "guard rails" that can still, in 2026, easily be bypassed.⁴

A product seriously interested in helping with productivity would correct this glaringly obvious flaw, focus on the task at hand, and stop emitting useless verbiage.

In the previous two sections, I tried to focus on ways in which the harness would be constructed differently even if the LLM technology is fundamentally impossible to improve; in this case, I have to assume that the labs have some control over the model itself. But unless they are truly incapable of influencing their output (and all their "benchmarks" and "capabilities" seem to indicate that they can control it very tightly) they ought to be building models that are much less verbose.

More Non-Natural-Language User Interfaces

Although natural language could hypothetically be a powerful interface for interacting with a computer system, the practical upshot of LLM natural language interfaces is that these interfaces are imprecise and repetitive, full of superstitions masquerading as "best practices". The inputs are a mess and the resulting outputs are a mess.

The general way of addressing this unstructured mess is to allow the chatbot to directly take action in response to the user's input; in other words to supply it with "tools" via an MCP server. But again, this is backwards. If we cannot even express our intent clearly in the first place, why are we trusting this system to take potentially destructive and harmful actions on our behalf?

Instead, I would expect a product that was seriously invested in helping me accomplish specific tasks, to have user interfaces specific to those tasks. Is it supposed to be able to be a security scanner that can discover OWASP top 10 bugs in a codebase? Have a button for that. Build that functionality into your harness, train it directly into the model, use smaller models that can satisfy that functionality more effectively than throwing it at the planet-sized brain of Fable or whatever.

I'm aware that there are small software startups that do something like this, but they are bolted on to the side of the main model providers' APIs, not integrated into the core of the product and not using their own models and AI systems to achieve consistent and repeatable results.

Strong Data Provenance Indicators

Chatbots produce data tables pulled from websites, from APIs, from MCP tools or from summarizing and scrambling the user's input. In order to provide the illusion of a seamless interface, this data is presented in-line regardless of where it comes from. But some of these outputs are produced mechanically via regular old API calls, for example, from the result of calling a tool or querying a website, but presented uniformly.

But there is a huge difference between an authoritative data source being inlined as part of a chatbot conversation, being treated as input by the chatbot, and some ad-hoc hallucinated data being treated as output of the chatbot.

If a product is trying to help me make accurate, empirically-grounded, data-driven decisions, the source of the data is critical.

Integrated into the "check for mistakes" and "verify citations" workflow I described above, there's a necessary "verify data programmatically" pass as well; to have tools that will treat portions of the output as a regular spreadsheet, allowing regular-old computer arithmetic to verify things and showing where such arithmetic was used, and how.

Better User Control of Reproducibility

Anyone familiar with the technical specifics of LLMs will know that they have a variable called "temperature" which controls the degree of randomness that the LLM uses to produce its outputs. But most users don't know this, because it isn't exposed as part of the user interface by default.

This leads to a subjective impression that you asked ChatGPT, and you got ChatGPT's authoritative answer.

You can't just set the temperature to zero and still get useful results - I am aware that it does more than just scramble the output at random, and there are perhaps good reasons that simply exposing just a temperature setting would not be that useful to users. But if we followed some more of my earlier recommendations for making more structured UI elements to solve specific problems rather than having long back-and-forth chats where each refinement depends on the previous response, perhaps those elements could also re-play the process so that users can see how reliable the bot is at a particular task and develop a sense of how the stochastic nature of the process actually affects it.

Similarly, if a user is trying to solve the same problem repeatedly with a chatbot, and the chatbot product has numerous computational tools that don't really have anything to do with the LLM, such as deterministic data-processing tools, then having a way to freeze the non-deterministic parts of the transcript but re-populate a particular data frame with updated information and fork / continue the conversation from there would be a way to avoid introducing pointless additional randomness when you already know what tool you're trying to use.

The fact that every conversation is presented as this flat chat prompt that doesn't let me interact with any of the widgets that were previously produced except through more chatting, really makes me feel like the whole product is just doing predatory social-media style "increase time on site" optimization, just trying to lure me into further repetitive and unreliable chats, rather than letting me get in, solve my problem, and get out.

Context Visibility

Managing the LLM context is the ongoing challenge facing organizations that are trying to use "agentic" workflows. Filling up the context with too much information causes well-known problems. In response, advanced LLM users attempting to solve larger problems must break up very long prompts into "skills", give access to lengthy information via "tools", and delegating sub-problems to "sub-agents" rather than simply extending a single prompt indefinitely.

All of these strategies have flaws, because even on the largest models, compared to the breadth and depth of knowledge-work problems, LLM contexts are quite small.

And yet, none of these products will show the context to the user by default. There are third-party addons that can show you a simple progress bar but for addressing the premier engineering difficulty with this technology, that is below the bare minimum.

This lack of visibility means that almost all of the tools for extending the context are flying blind. Rather than responding meaningfully to a full context, everyone just kind of guesses how much state they need by guessing and trying over and over again with progressively more elaborate skill and sub-agent layouts. Even managing context compaction ends up being an advanced API-driven workflow⁵.

A serious product that was trying to help the user understand would not only show "available context" but explain the impact of context compactions, make it easier to see harness-generated prompts, and so on. This would be a first-class feature, combined with the aforementioned reproducibility / replay tools, would allow users to do real experiments to develop an understanding about how to make good use of the context window.

A Sandbox That Actually Works

I've been focused on the chatbot interface here because it is the most immediately egregious upon looking at the UI. But the "agentic loop" tools used for coding are equally dangerous, if not more so. Coding tools keep destroying everyone's data, over the course of years.

These catastrophic incidents that become front-page news are relatively rare compared to the amount of coding-agent use out there. But they also aren't the only kind of sandbox violation. Coding models will so routinely edit test code instead of the system under test that there are "pro tips" articles all over the web giving you the flawed advice to simply ask the agent not to cheat. News write-ups of the catastrophic incidents themselves will also offer glib and wrong advice, like "use a docker container". That might prevent it from literally deleting your operating system, but it won't prevent it from destroying all the local work you have in your codebase (it needs access to a checkout, after all!)

There is a flurry of activity in the infosec space where people are rushing to plug the gaps left by these coding harnesses. Everyone's got their own version of an MCP approval gateway where you can optionally place a proxy between your agent and your production infrastructure.

In the best case, though, all these mitigations and proxies and prompts simply turn the user into an auto-approval automaton, hitting Y, Y, Y, Y over and over again, until you finally are driven mad and hit "yes to all", turn on full-auto mode and submit yourself to the void. With nothing between your personal vigilance and disaster, there are no workflows left beyond decrementing your own vigilance until there's nothing left and then hoping the disaster never arrives.

The fact that some mitigations exist that can be deployed by extra-cautious users does not change the fact that "agentic coding" is an unsafe-by-default technology deployed without concern or guidance. Every frontier lab has tied a spring-loaded shotgun to a dog; the fact that dog owners can publish thoughtful blog posts explaining how you can teach your dog the basics of gun safety or how you can have your dogs play in a bullet-proof room does not mitigate the fact that the product should not have been allowed in the first place, nor should it continue to exist without VERY strong security controls.

I might believe that a frontier lab were seriously interested in providing developers with a useful tool if they shipped something that had safety built-in.

That means tools in the harness, detached from any LLM, independent of the prompt, that could:

• sandbox all filesystem operations and strictly limit ANY deletions outside of specified scopes, regardless of operating system,
• enforce snapshotting of the entire repo on every operation for easy rollbacks and minimal lost work,
• remove the disaster of "auto mode" (not to mention nonsense like --dangerously-skip-permissions) entirely, and
• carefully consider a structure for presenting plans to the user where, rather than provoking immediate alert fatigue by asking for checks on every action, make structured plans which can be submitted to the user as a group of actions and reviewed and approved as a batch.

In the same way that I suggested above that research-based tasks should have a way of re-issuing prompts to determine how reproducible a result is, or whether other sources might be found, agent-based tasks should have a way of being executed against mock services for popular APIs, so that the verification can match both on the front-end (review the plan for making the API calls before they're executed) and the back end (review the API calls that were issued to the mock service and verify that they matched).

Instead, the frontier labs provide us products that are disasters out of the box, give us "best practices" to build massive and elaborate, as well as incomplete and error-prone, security perimeters of our own design. Then they blame "operator error" when it inevitably goes wrong. I cannot believe that these design choices are intended to help us be productive.

Bonus: Human Processes

Organizations deploying AI also frequently come across as unserious, for similar reasons. In 2023, naive exuberance could perhaps be forgiven. But today, as we near the close of 2026, there are several well-known problems, that have been extremely well-covered in the press. None of these things should be surprising, but most orgs deploying these tools are still just letting them rip and hoping it all works out.

Organizations deploying these tools would need at least three kinds of major modifications to their internal processes, if they wanted to be serious about using them safely:

1. Shift Rotations to Prevent Vigilance Decrement

There have been several high-profile incidents where software developers' gradual acquiescence to accepting LLM output have lead to serious economic consequences for the companies deploying them, perhaps best typified by Amazon's "millions of lost orders" due to a gradual decay of their engineering processes from LLM use.

These outages, and other AI-related failures, are due to the difficulty of maintaining focus on the same problems. In other words, as I described above, vigilance decrement is a constant problem, because AI outputs are most often correct, but continue to be incorrect in surprising and non-intuitive ways. As I have previously written, you cannot trust yourself to catch every bug with code review, and LLM output.

Aviation, for example, has very strict rules around rest requirements. There is also a specific rule that "No certificate holder may operate an aircraft without a second in command if that aircraft has a passenger seating configuration, excluding any pilot seat, of ten seats or more.". Other safety-critical professions have similar rules.

And yet, even in the age of the supposed "AI revolution", most software teams are still assigning every engineer a full feature load, not planning for any rest, and telling people to review code whenever they happen to have some "free time".

Maintenance of vigilance has to be your top priority. Regular, scheduled, inviolable rest periods where people do work without AI assistance, and are not exposed to any AI output for review or otherwise, would be crucial in order to stay mentally sharp enough.

The tools themselves should have this sort of thing built in. The mistake-review process described above should have a periodic spot-check mode where a second reviewer periodically reviews a chatbot log, doing their own independent verification of claims, to see if they spot the same errors. This could provide a feedback loop to determine how much rest is necessary to maintain continuous attention and actually spot hallucinations.

2. Skill Practice To Prevent Skill Loss

It is also well-known that AI use leads to AI reliance, and AI reliance leads to skill loss.

I like to use the analogy to dockworkers at a seaport⁶ adopting automation.

If you employ dockworkers to load and unload ships all day long, they are going to be getting tons of exercise. They will be able to lift heavy objects on demand, whenever. They might have plenty of health problems and injuries from this type of work, but "lack of exercise" will not be a problem.

With the development of standardized container ships and mechanized cranes, you are going to be changing their job description substantially: now they mostly spend all day sitting in a small cubicle moving a control lever back and forth, not lifting heavy stuff. They will get worse at lifting heavy objects.

In this analogy however, the cranes are not all that reliable. We know they break, and they drop their payloads sometimes, and the stuff needs to be manually moved. But this only happens a few times a week, at most. If you need whoever is driving the crane to be able to jump out at any moment and still move stuff around manually, then you need to make an affordance for that. You need to give them time to go to the gym and do some lifting for practice, or every crane failure is going to be a major emergency.

An organization doing an AI transformation would also need a massive increase to learning & development budget, both in terms of resources and in terms of schedule. If your people are going to lose skills because they've lost regular practice in the incidental course of doing their duties, then they are going to need deliberate, intentional, non-incidental practice of those skills to keep them sharp.

But rather than trying to accommodate new workflows and give time for people to adjust, most AI mandates are simply dropped on workers like a ton of bricks, with no time to adapt and no affordance for maintaining their skills. Operate the crane and stay fit and healthy and ready to switch back to manual lifting at any time and then get back in the crane cockpit right afterwards. Don't mess up.

Then an accident happens and everyone is surprised, as if this process weren't practically designed to produce a terrible result.

3. Mental Health Resources to Deal with Mental Health Risks

AI psychosis often begins with practical problem-solving, and beyond that, it can start specifically at work. Not to mention the more pedestrian condition of "AI brain fry".

If you are mandating your employees to use a hazardous tool that may seriously and directly damage their mental health, you need trainings and resources. You need in-house therapists and you need to be making sure to check in with people actively to make sure that this is not happening.

Again, the tool itself ought to have some way of dealing with this. An occasional "take a break" popup is easily dismissed; they need a user-visible AI personal dosimeter so you can see your cumulative usage over time.

I don't even know if "usage over time" is a sufficient metric to gauge risk. Maybe if your work chatbot start to talk about resonance too much, unless you literally work as an acoustic engineer, that should be flagged for someone.

We are, again, years into dealing with these tools, and we know these risks exist. Yet no serious mitigations are provided. Not even any way of measuring the risk exposure.

And More

There are also many other risks associated with the technology. There are intellectual property risks with the foundation models, due to recklessness with their training data. There are existential financial risks associated with the infrastructure build-out. The extent to which most "open" models are simply derivatives of frontier models is an open question.

What I Think

If any one of these things were regularly overlooked by AI vendors or users, that would be a totally normal product oversight. Room for improvement for the next version, but nothing catastrophic.

Shipping without any of them doesn't seem like lean product management, it seems like a careless attitude towards risk and a product design philosophy oriented entirely towards short-term demos, with no regard for how to realize actual productivity gains.

Furthermore, being available for years without anything like these features, despite hundreds of incidents demonstrating the risks, with hundreds of billions of dollars of funding, makes it seem to me like if they were to add all the features that would make their product actually safe and hypothetically useful, these features would reveal that it is actually not an improvement to productivity.

In the year since I first wrote about measuring the cost/benefit ratio of AI, I have heard from numerous people who have shown this to management to try to illustrate why their AI initiatives - like almost all AI initiatives - were either failing or burning out their engineers.

I've also heard from lots of people that have told me that it's obviously useful and they don't need to measure so carefully, because they are getting lots of work done that they couldn't have otherwise.⁷

I have yet to hear from a single person who has said "yeah, we measured according to your methodology⁸, and it turns out that our AI work is going great and that our ratio is 0.75".

Obviously, I cannot say for sure why this is; absence of evidence is not evidence of absence. But at this point I think the null hypothesis is that AI tools provide, in aggregate, zero value. They make mistakes too often, and the externalities they produce are so bad and so difficult to control that even before we get to the places where they are just physically poisoning people, even the negative effects on their direct users end up cancelling out whatever benefit to they provide to their organizations.

If I were wrong, then including tools to measure an AI's effectiveness at the tasks their users are actually trying to accomplish, rather than meaningless benchmarks, would show big productivity gains. The frontier labs would be champing at the bit to add such features, and crowing about their fantastic results.

I think the labs know that if they did that, it would present a grim picture to their users. Such tools would let their users see that it's making mistakes much more often than they realized, that they're spending much more time with it than they want to be, and that it's just generally not fit for purpose.

If they prove me wrong by adding in all of these safety mechanisms, and in the process, they make all of their AI technology less harmful, I'll be thrilled to be debunked.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!

28 Sep 2026 4:27am GMT