16 Nov 2026

feedPlanet Python

Ed Crewe: From Routing Checks to Trajectory Testing: Evaluating an Agentic Chatbot

Which Agentic Chatbot?

I have been working on a Python based AI test framework for a chatbot interface for my company's product, Postgres AI Hybrid Manager. The manager allows the setup of Postgres clusters across cloud or on-prem and attaching various AI tools such as Langflow. So a combination of more traditional Postgres backup, migration, telemetry and analytics features along with LLM workflows leveraging the data it holds.

The product already has a control plane UI for managing Postgres estates. It also has full help for the product, all Postgres versions, analytics, AI and add ons. The chatbot brings all these things together: ask a question, get the relevant help, or ask it to do something such as migrate a cluster, or evaluate telemetry that would otherwise require clicking through the UI.

That makes it a pretty handy interface, especially for the less technical. However it is not simple to test and ensure good quality responses.

A normal deterministic API test is simple. Send a request, check the status code, check the JSON body, perhaps check the database state. An LLM-backed agent does not pass or fail so clearly. It can route to the wrong capability and still return fluent text. It can pick a plausible but wrong tool. It can miss half the task and still sound confident. It can complete the first turn of a conversation and lose the plot on the second. It could get malformed or missing data from tooling that leads it to deliver a misleading conclusion. It might only provide help to something that should be from tool data or was a request for an action such as create a cluster.

So the testing problem was not "does the chatbot return a reasonable response?" It was "how do we test the whole chat path is doing the right thing?"

This is the story of how our agent-eval test framework evolved as we worked to see that our chatbot was not only getting the right answer, 42 , but whether it was asking all the right questions of the right tools to get that answer. Known as trajectory testing ...


You're Golden

Before we can tell our story we need to define some terms.

A Golden is an example of a perfect desired output from a test input. They often refer to more complex outputs that may need saving as separate files, but a simple assertable output such as 42, is a golden too!
Whilst complex goldens may be used and marked for semantic similarity against the test output. It is more common for complex outputs to be described by a rubric. A rubric is a checklist of qualitative properties a good answer must exhibit, written in plain English as opposed to a golden example of an answer.

For AI testing the tests are termed evals, ie they evaluate the tool, but not by strict assertions, because one thing you can be sure of with an LLM is that given the same input, you usually get subtly different output, ie they are non-deterministic. Which means for LLM outputs the only way to test them is to use an LLM-as-judge, ie give that LLM the test output and a rubric or golden and let it mark it against that. Then you set a pass threshold for that mark, to translate your complex output into a pass or fail.

You can also total up all the passes to give you a Task Completion Rate, TCR. So with complex AI agentic LLM interactions a 100% pass of all evals is often not realistic. Hence you set a TCR below 100% for the whole test suite of evals to pass. Start with the smallest useful test. The core principle of evals is not complicated, you want the input to give you the expected output.

But for an Agentic application this may require a sequence of LLM calls and tools: Making the final output dependent on the route that should be chosen, the tool(s) that should be called, the actions to be taken, further LLM calls that may be necessary and finally the core data that the response to the user should contain.

Our first version did not try to solve every part of that. It started with routing, simple and deterministic.

Routing is the starting point

The chatbot originally had an agent per tool. The tool being the code and API calls that performed actions or returned data or help.

Different specialist agents owned different parts of the product surface: Control-plane actions, Postgres database operations, schema design, roles and permissions, cluster reporting, migration, and so on.

Before any specialist can help, something has to choose the right specialist.

So the first eval suite asked a narrow question:

Given this user prompt, did the chatbot route to the expected tool?

That gave us a fast health check. We could keep a corpus of prompts, map each one to an expected destination, run them through either a direct model path or against the real deployment and its tools, and score whether the selected destination tool matched the golden.

A golden here is just the name of the tool:
- id: "core-iam-001"
    prompt: "List all my projects"
    expected_tool: "control-plane"
    tags: ["core", "control-plane", "project"]

And the check on the other end is deliberately dumb - an equality test, not a semantic one:

self.success = tool_match(predicted_tool, expected_tool)

Agents became skills, but routing remained

The design moved away from "one agent per tool family" toward a more consolidated orchestrating agent with skills that could better compose different tool use for common tasks.

That is a better fit for how modern agent systems are evolving. A skill = instructions, constraints, and a subset of tools that are relevant for a task. It is a form of progressive disclosure. Give the model the minium it needs at each step to save tokens.

But this did not make routing irrelevant.

Instead of asking "did we transfer to the right sub-agent?", the eval asks "was the right skill made visible and selected for this task?" The labels changed but a skill could still use the wrong tool.

Routing evals stayed valuable because they were fast, explainable, and easy to run in CI. But they are limited, routing should always be correct but it doesn't mean that the final agent response is too.

TCR jumps to the endpoint, the response

Task Completion Rate, or TCR, was the next step.

The user asked for a cluster comparison, or a schema recommendation, or help diagnosing a database issue. We need to know whether the full response actually completed these tasks.

Responses are complex goldens so they need the LLM-as-a-judge pattern: run the chatbot, take the actual response, and ask a judge model to score it against expected sections.

The eval has a rubric here for judging the output:

- id: "tcr-core-014"
  prompt: "Compare CPU usage between these two clusters"
  expected_sections:
    - "identifies which cluster has higher CPU usage"
    - "cites at least one supporting metric"
    - "suggests a plausible next step"

The judge gets one simple instruction: score each expected_sections between 0.0-1.0 A metric class then just thresholds it for pass / fail:

self.success = score >= 0.7

The judge must be calibrated and a consistent model used for comparing runs over time. Consistent judging enables skill and/or prompt tuning from metric trends. The rubric must be specific enough to avoid marking waffle as success. But it turns a non-deterministic complex output into a simple pass and fail. It also separated two different levels of QA:

That led to two execution modes.

Direct mode calls the model with simulated context. It is faster and useful for prompt and rubric development.

Proxy mode calls the real chatbot. It is slower, but it exercises the production path: routing, skill selection, tool calls, guardrails, streaming responses, conversation state, and the actual service wiring.

Both matter. Direct mode tells you whether the model is capable of the answer. Proxy mode tells you whether your product is capable of really delivering it via agents running your deployment's tools.

This is the major difference from standard AI LLM testing, the model is only a small pluggable engine for the full agentic skill set that requires the actual deployment domain of data, actions and tools. Direct mode testing of only the model, is occasionally useful but E2E testing of the Chatbot deployment is required for agentic AI Chatbot QA, tuning and validation.

Multi-step conversations changed the unit of testing

Single-turn TCR is still too small for many real chatbot tasks.

Users do not always provide all required information in one message. They ask to create a cluster, then pick a project, then choose a size, then confirm. They ask for a schema review, then refine the problem, then ask for a migration path. They troubleshoot by adding information over time.

So the framework has to exercise test cases that are conversations, not just single prompts.

That sounds like a minor data-model change. It was not. Once a test has steps, the eval runner has to preserve conversation state. In proxy mode, that means carrying the real conversation_id returned by the chatbot and sending each follow-up as part of the same server-side conversation. In direct mode, it means building a synthetic conversation history so the model sees the prior turns.

In code that split is about as literal as it sounds. Proxy mode threads a real id through each call:

response = client.send_message(prompt=msg, conversation_id=conversation_id)
conversation_id = response.conversation_id  # captured on turn 1, reused after

Direct mode has no server-side conversation to lean on, so it fakes one by re-rendering the transcript into the prompt itself, every turn:

full_prompt = f"## Conversation History\n{render(history)}\n\n{next_prompt}"

Same test case, same expected outcome, but a different code path depending on which half of the system is actually holding the conversation state. That's impacts multi-turn evals because conversation memory is part of the harness code for the actual deployment not just a model issue.

The scoring also becomes more interesting. You want per-step checks, because the assistant should ask the right clarifying question at the right time. You also want an overall score, because a conversation can have reasonable individual turns and still fail to complete the user's goal.

Coding it yourself: deepeval underneath

Everything above sits on top of deepeval, the open-source LLM eval library. We add a Synthesize β†’ Execute β†’ Evaluate pipeline, a plugin system, YAML goldens, CI wiring, and Langfuse push on top of it But the core library underneath is plain deepeval, and you do not need any of the surrounding machinery we used. Here are routing, TCR and multi-step just built directly on deepeval (simplified deepeval 3.6.9)
NB: Once tests have stabilised, the synthesize output should be retained and the step skipped and just rerun as part of test modification PR merges.

A test case is just an input/output pair. LLMTestCase is the base unit everything else scores:

from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="List all my projects",
    actual_output=chatbot_response_text,       # what the system under test said
    expected_output="control-plane",                  # the golden - a skill label here, not prose
    additional_metadata={"predicted_skill": predicted_skill},
)

Routing is a custom metric, not a built-in one. deepeval ships plenty of semantic metrics, but "did it route to the right skill" is an exact-match business rule, so you write your own BaseMetric. This is a simplified version of the same shape our real AgentMatch metric takes:

from deepeval.metrics import BaseMetric
from deepeval.test_case import LLMTestCase

class AgentMatch(BaseMetric):
    def __init__(self, threshold: float = 1.0):
        self.threshold = threshold
        self.async_mode = False  # routing checks are cheap; no need for async here

    def measure(self, test_case: LLMTestCase) -> float:
        predicted = test_case.additional_metadata["predicted_skill"]
        expected = test_case.expected_output
        self.score = 1.0 if tool_match(predicted, expected) else 0.0
        self.success = self.score >= self.threshold
        return self.score

    async def a_measure(self, test_case: LLMTestCase) -> float:
        return self.measure(test_case)

    def is_successful(self) -> bool:
        return bool(self.success)

    @property
    def __name__(self):
        return "Agent Match"

tool_match is the check from earlier. Run it with deepeval's own runner rather than hand-rolled assertions, and you get retries, pretty output, and a result object for free:

from deepeval import evaluate

evaluate(test_cases=[test_case], metrics=[AgentMatch()])

TCR is where deepeval's built-in GEval earns its keep. GEval is deepeval's off-the-shelf LLM-as-judge metric, you give it criteria (or explicit evaluation steps) and it handles the judge prompt, the JSON parsing, and the scoring for you. Our rubric-per-line expected_sections maps onto evaluation_steps almost directly:

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

task_completion = GEval(
    name="TaskCompletion",
    evaluation_steps=[
        "Check whether the response identifies which cluster has higher CPU usage",
        "Check whether the response cites at least one supporting metric",
        "Check whether the response suggests a plausible next step",
    ],
    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
    threshold=0.7,
)

test_case = LLMTestCase(
    input="Compare CPU usage between these two clusters",
    actual_output=chatbot_response_text,
)

evaluate(test_cases=[test_case], metrics=[task_completion])

Multi-step conversations get their own test case type. ConversationalTestCase takes a list of Turns instead of a single input/output pair, and pairs with a BaseConversationalMetric instead of BaseMetric:

from deepeval.test_case import ConversationalTestCase, Turn

convo = ConversationalTestCase(
    turns=[
        Turn(role="user", content="Create a new cluster"),
        Turn(role="assistant", content="Sure - which project should it go in?"),
        Turn(role="user", content="acme-prod"),
        Turn(role="assistant", content=final_response_text),
    ],
    expected_outcome="A cluster is created in acme-prod after resolving the missing project name",
)

deepeval has a conversational counterpart to GEval too (ConversationalGEval), scored against the whole turn sequence rather than a single response which is the natural fit for "did the assistant ask the right clarifying question at the right time", the per-step-plus-overall shape TCR needed once prompts became conversations.

Put together, that is the whole starting kit: LLMTestCase plus a hand-written BaseMetric for hard business rules like routing, GEval for rubric-style task completion, ConversationalTestCase plus ConversationalGEval once a prompt becomes a conversation, and evaluate() to run the lot and get a result object back.

Everything else we built, the YAML goldens, the plugin architecture, the CI wiring, the Langfuse push exists to run more of these at scale and make the failures easy to find. But none of it is required to get started. If you are testing your own agentic chatbot, this is how to begin.

This is where instrumentation started to matter much more.

For a single-turn answer, a markdown report with pass/fail rows is often enough to start debugging. For multi-step conversations, that is thin. You need to know which turn failed, whether the route changed, whether the wrong tool was called, whether the tool call used correct arguments, whether the model forgot earlier context, or whether the final answer simply missed a required section.

That is why we added span-level telemetry and pushed eval traces into Langfuse.

Langfuse made the failures inspectable

The useful thing about Langfuse is not just having another pretty dashboard. Although that is important for spotting quality regressions over time via regular CI/CD automated runs.

The vital thing was being able to treat an eval run as a set of traces. A run becomes a session. Each test case becomes a trace. The trace carries the prompt, response, scores, tags, model, mode, scenario, and the spans emitted by the proxy.

For a chatbot path, those spans are where the debugging starts. You can see routing, tool execution, LLM calls, latency, and token usage where it is available. You can filter by scenario and model. You can compare runs. You can look at a failing conversation and see whether the problem began at route selection, tool selection, tool arguments, or final synthesis.

That changes the tuning loop.



Without traces, an eval failure says "this case failed". With traces, it can say why it failed.

That distinction matters because the fix lands in different places...
Is it a routing rule?
Is it a skill description?
Is it a tool schema?
Is it the judge rubric?
Is it that the eval has has an expectation that the product has never actually promised?

Trajectory testing -> knitted the pieces together

Routing and TCR started as separate signals.

Routing asked whether the right capability was selected. TCR asked whether the final task was completed. Multi-step testing asked whether that held across a conversation. Instrumentation showed what happened between those points.

Trajectory testing is the next natural step: score the path itself.

For an agentic product, the fully correct path is essential to response quality.
So trajectory tests add expectations about intermediate actions:

The label-based routing tests are still useful as fast canaries. They tell us whether the classifier shape has drifted and distinguish tiers - see the next section.
But full trajectory tests judge the route by consequence: did the system actually follow the tool path that would satisfy the user?

So retain the fast determisitc routing tests, but move more user-visible behavioural coverage into trajectory and TCR.

Sovereign AI makes the eval problem tiered S/M/L/XL

There is one more constraint that makes this more than a generic chatbot-testing story.

Our chatbot has to work for sovereign and air-gapped deployments. In those environments, prompts, tool results, schema details, and operational data cannot be sent to a hosted frontier model outside the customer's trust boundary. The inference model may run inside the customer's environment.

That usually means a smaller model.

Smaller models are not just cheaper versions of larger ones. They have different context limits, weaker tool-selection behaviour, and less tolerance for an over-wide capability surface. If you show a smaller model every possible tool and skill, you have increased the chance that it chooses a bad one.

So the architecture becomes tiered. Models are effectively T-shirt sized. A small self-hosted model sees a curated subset of reliable skills. A larger model can be allowed to see more. Some experimental or complex skills only make sense for the highest tiers.

That changes the meaning of a routing eval again.

The correct visible skill set is no longer universal. It depends on the model tier. A prompt that should route to an advanced skill for an XL model may need to be dropped, refused, or handled differently for a smaller model that should not see that skill at all.

This is why trajectory testing and routing need to be tier-aware. We are not only asking whether the chatbot can complete a task. We are asking whether it can complete the task through the capability surface that a deployment's LLM size allows.

What I would keep from the journey

The final shape was not obvious at the start.

We began with routing because it was the first integration failure point and the cheapest one to isolate. We added TCR because correct routing did not prove task completion. We added multi-step cases because real users have conversations, not isolated prompts. We added telemetry because multi-step failures are otherwise too hard to debug. We moved toward trajectory testing because the route, tools, arguments, and answer need to be judged as one path.

If I were starting another agentic product eval framework, I would keep that order.

Do not start by trying to build a grand universal benchmark. Start with the smallest failure point that would embarrass the product if it regressed. Then move the signal closer to the user's actual goal.

For a chatbot wired into a real control plane, that means testing more than the output text. It means testing the route, the skill, the tool call, the arguments, the conversation state, the final answer, and the model tier that made those options visible in the first place.

That is the difference between checking that an AI system said something vaguely relevant and checking that it actually did all the things the user asked of it.

16 Nov 2026 9:13am GMT

Graham Dumpleton: Muru: scaling from what the workers report

In the Kubernetes post the scaling was done from outside. Muru published how busy its worker slots were, and a HorizontalPodAutoscaler added pods when the number went over a target. That works, but it can only ever add pods, and it has no idea why the slots were busy. Back on a host running the Muru daemon, Muru decides for itself how many worker processes each release has, and because it sits between the requests and the workers, it can say rather more about why requests are waiting than a single number. The same scaling policy also applies under muructl serve in a container, where another worker is only added if it fits under the container's memory limit, but here I am back with the daemon.

Most Python web servers don't scale the number of processes at all. Gunicorn starts the number of workers it is told to, and the only way to change that while it runs is to send it SIGTTIN or SIGTTOU, which add or remove one worker. Uvicorn and granian both take a fixed --workers count, and mod_wsgi's daemon mode has a fixed number of processes set in the Apache configuration. The exception is uWSGI, whose cheaper subsystem can spawn and stop workers based on how many are busy, or on the backlog of connections waiting to be accepted.

So adding workers when they are all busy isn't new, and if that were all Muru did it wouldn't be worth a post. What is different is what Muru does with everything else it knows about the workers, and whether more workers is even the right answer.

Growing and shrinking

The application for this post has a few pages which each misbehave in a different way. The first, /orders, sleeps for a fifth of a second as if waiting on a database. The release starts with one worker of two threads, and has a scaling policy in spare mode, which tries to keep between one and three idle workers in hand, much as Apache httpd does with spare processes or threads:

[tool.muru]
entrypoint = "app:application"
processes = 1
threads = 2

[tool.muru.scaling]
mode = "spare"
min_workers = 1
max_workers = 12
min_spare = 1
max_spare = 3
scale_in_window = "30s"
hold_after_growth = "30s"
reap_interval = "5s"
min_lifetime = "10s"
idle_reap_after = "45s"

The last five settings shorten windows which default to minutes, so it all happens quickly enough to watch. I deployed it to a daemon and set sixteen clients asking for /orders over and over for a minute. Three seconds in, muructl scale status showed the group to be well short:

shop@r1
GROUP        WORKERS      SLOTS      IDLE    QUEUE-P95     REAPS      POLICY
default      1            2/2        -7.00   under 2.5 s   0/6        spare 1-12 workers, 1-3 idle
  workers: cpu -, gc -, gil wait p95 -, loop lag p95 -
  diagnosis: waiting for the workers' first report

Both slots were busy, the requests waiting at the edge were counted as seven workers' worth of slots wanted, and the 95th percentile of the time requests waited for a slot was in the seconds. The workers hadn't yet sent their first report, so the diagnosis had to wait for it. Once the shortage had held for the five second scale-out window, the daemon forked more workers, at most four at a time:

default: scale out from 1 to 5 workers: short by 7.00 workers for 5999ms; diagnosis: waiting for the workers' first report
default: scale out from 5 to 9 workers: short by 3.00 workers for 5999ms; diagnosis: I/O bound with exhausted slots; ...

Each new worker is a fork of the host which already imported the application, as covered in the import once, fork many post, so the second step came only seven seconds after the first. From then on the group sat at nine workers, sixteen of eighteen slots busy and one worker in hand, and the queue wait was under a millisecond.

After the load stopped, the workers were taken away again, but slowly, one at a time:

default: scale in from 9 to 8 workers: at least 9.00 idle workers over the last 30s, over max_spare 3; diagnosis: idle
default: scale in from 8 to 7 workers: at least 8.00 idle workers over the last 30s, over max_spare 3; diagnosis: idle
...
default: scale in from 4 to 3 workers: at least 4.00 idle workers over the last 30s, over max_spare 3; diagnosis: idle

and then stopped at three, with more than it needed:

default      3            0/6        3.00    -             6/6 halt   spare 1-12 workers, 1-3 idle
  ...
  last: suspended: 6 workers reaped in the last hour, the churn budget; scale-in is suspended and the scale-in window doubled until it recovers (1s ago)

Each release has a churn budget of six reaps an hour. I had shortened every window but left that alone, so it was spent and scale-in was suspended. It is there so that a badly tuned policy can't keep a group swinging up and down all day.

When the answer is more threads

Once the workers had reported, the status had more to say. This is from while the group was still growing:

default      5            10/10      -1.00   under 2.5 s   0/6        spare 1-12 workers, 1-3 idle
  workers: cpu 0%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: I/O bound with exhausted slots; for the next release, more threads per worker, or more concurrency for an ASGI application

The group's slots were all busy, but its workers were using no CPU at all. That is what waiting on a database looks like, and nine processes is a costly way to hold sixteen requests that are each just waiting. The suggestion was more threads per worker, and that can't be done by the autoscaler. A release's processes and threads are fixed when it starts, so the change goes into the next release.

I deployed the same application with threads = 8 and ran the same load. Instead of nine workers it settled at three:

default      3            16/24      1.00    under 1 ms    0/6        spare 1-12 workers, 1-3 idle
  workers: cpu 1%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: within capacity

Over the minute it served 4440 requests compared with 4007, the 99th percentile response time was 0.409 seconds where before it was 1.623, as it needed only one step to scale out rather than two, and each of the three workers had 12MB of memory of its own:

WORKER  PID      GROUP      UP       REQUESTS   THREADS  THREAD-OLDEST  RSS     USS     ROUTED  STATE
0       67546    default    38s      1090       6/8      0.2s           18MB    12MB    yes     ready
1       67729    default    31s      813        5/8      0.0s           17MB    12MB    yes     ready
2       67730    default    31s      790        5/8      0.2s           17MB    12MB    yes     ready

When the answer is fewer threads

More threads isn't always the answer. In the import once, fork many post I showed four threads of CPU bound code running four times slower than one, because they had to take turns holding the global interpreter lock, and said Muru kept track of time spent waiting for the GIL. This is where that comes in.

The /prices page does about 13 milliseconds of pure Python arithmetic. With the eight thread release still serving, and running on CPython 3.15 with the GIL, eight clients asking for it gave:

default      3            8/24       2.00    under 1 ms    0/6        spare 1-12 workers, 1-3 idle
  workers: cpu 100%, gc 0%, gil wait p95 under 50 ms, loop lag p95 -
  diagnosis: GIL bound; for the next release, more processes with fewer threads each

By the measure the autoscaler uses, there was nothing to do. Only eight of the twenty-four slots were busy and there were two idle workers in hand. Yet the three workers were each using all of the one core the GIL lets them have, and the 95th percentile of the time threads with work to run waited for their turn was put at under 50 milliseconds. A request which takes 13 milliseconds on its own had a median time of 20 milliseconds.

An autoscaler working from busy slots or a connection backlog, Muru's own or uWSGI's, can't see this. Adding threads would make it worse, and adding workers wouldn't happen because the slots aren't full. The diagnosis says what would help, which is more processes with fewer threads each. I deployed the release again with one thread per worker:

default      9            8/9        1.00    under 1 ms    0/6        spare 1-12 workers, 1-3 idle
  workers: cpu 86%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: CPU bound; for the next release, more processes, up to the node's cores; more threads will not help

Now each request occupied a whole worker, the slots filled, and the autoscaler took the group to nine workers on this ten core machine. It served 24863 requests in the 45 seconds compared with 18626, and the median time was 13 milliseconds.

The other way around the GIL is to not have one. I deployed the eight thread release again, with nothing changed but a .python-version of 3.15t, so uv built its environment with the free-threaded build of Python 3.15:

default      2            7/16       1.12    under 1 ms    0/6        spare 1-12 workers, 1-3 idle
  workers: cpu 48%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: within capacity

Two workers of eight threads served 26040 requests in the same 45 seconds, a little more than the nine single threaded processes, with nothing waiting for a GIL. The CPU figure is now a share of a core per thread, as threads on a free-threaded build really can run in parallel.

When the answer is less garbage

The /search page makes short lived objects, 40,000 dictionaries per request, in pairs which refer to each other. With a release of four threads per worker and four clients:

default      2            4/8        1.00    under 1 ms    0/6        spare 1-12 workers, 1-3 idle
  workers: cpu 100%, gc 73%, gil wait p95 under 5 ms, loop lag p95 -
  diagnosis: paused by garbage collection of short-lived reference cycles; for the next release, break the cycles, or use weakref for the references back

Close to three quarters of the workers' time was going into garbage collection. A collection stops every thread in the process, so that time is lost to every request in flight.

The diagnosis says why. A dictionary only needs the garbage collector, rather than being freed as soon as nothing refers to it, when it is part of a reference cycle, and muructl debug memory showed the youngest generation's collections freeing hundreds of millions of objects. Long lived objects being scanned over and over don't get freed, so this was garbage being made and collected. Removing the reference back was a one line change:

default      2            4/8        1.00    under 1 ms    0/6        spare 1-12 workers, 1-3 idle
  workers: cpu 100%, gc 0%, gil wait p95 under 1 ms, loop lag p95 -
  diagnosis: CPU bound; for the next release, more processes, up to the node's cores; more threads will not help

The garbage collection share went to zero, and in 40 seconds the release served 73637 requests where before it had served 17977, with the median time down from 8 milliseconds to 2. The diagnosis then moved on to the next thing holding the group back, which was now plain CPU.

When the event loop is blocked

An ASGI application has one event loop per worker rather than a pool of threads, and one blocking call in a coroutine holds up everything else on that loop. This release had an /invoice page calling a function which blocks for a fifth of a second, as a synchronous client library would, directly from the coroutine. With eight clients on a page which awaits for 50 milliseconds, and one client on /invoice:

default      2            9/2000     1.99    under 1 ms    0/6        spare 1-12 workers, 1-3 idle
  workers: cpu 2%, gc 0%, gil wait p95 under 0.5 ms, loop lag p95 under 500 ms
  diagnosis: event loop held by blocking code; for the next release, await blocking calls, or run them in a thread, so the loop keeps serving

Each worker's event loop has a heartbeat twice a second, and the lag is how late it runs. Here it was hundreds of milliseconds late. The slots barely registered, nine busy of two thousand, which is the concurrency limit of two event loops, so nothing based on slots would ever have noticed. The page which should have taken 50 milliseconds had a 95th percentile of 0.410 seconds, as it kept getting stuck behind an invoice.

Changing the one line to body = await asyncio.to_thread(render_invoice), so the blocking call ran in a thread, brought the loop lag down to under 5 milliseconds and the 95th percentile back to 0.054 seconds. The eight clients got through 6064 requests in 40 seconds instead of 2960.

When the answer is a worker group

The last case is one slow endpoint making everything else wait. Here /reports takes two seconds and is used in bursts, ten requests at once every six seconds, alongside two clients steadily asking for /orders. The release had two workers of four threads and no scaling policy, but even a static release gets a diagnosis. Every few seconds it read:

default      2            2/8        1.50    under 2.5 s   -          static
  workers: cpu 0%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: a few heavy requests; for the next release, a worker group of their own for the slow paths

Most of the time the slots were mostly free, but the queue wait was in the seconds, because each burst of reports took every slot and /orders requests arriving then waited behind them. For /orders the median response time was fine at 0.208 seconds, but one in a hundred requests took over two seconds.

Scaling out would help for a moment, until the next burst took all the extra slots as well. What the diagnosis suggests is separating the slow paths, so I gave them a worker group of their own, forked from the same host:

[tool.muru]
entrypoint = "app:application"
processes = 2
threads = 4

[tool.muru.groups.reports]
paths = ["/reports"]
processes = 1
threads = 12

Each group then gets its own line, and its own diagnosis:

default      2            2/8        1.50    under 1 ms    -          static
  workers: cpu 0%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: within capacity
reports      1            10/12      0.17    -             -          static
  workers: cpu 0%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: I/O bound with exhausted slots; for the next release, more threads per worker, or more concurrency for an ASGI application

The reports now only compete with each other. For /orders, the slowest of the 389 requests took 0.212 seconds, where before the 99th percentile was 2.093. Each group can have its own scaling policy too, so a group like reports could be allowed to grow on its own when a burst arrives.

Scaling to a target

Spare mode is about keeping idle workers in hand. The other mode, target mode, keeps the share of busy slots near a target, and can also scale on how long requests wait:

[tool.muru.scaling]
mode = "target"
min_workers = 1
max_workers = 8
slot_utilization = 0.6
queue_wait_p95 = "100ms"

With eight threads per worker and the same sixteen clients on /orders, it went straight to the count which would carry the load at the target:

default      4            16/32      2.00    under 1 ms    0/6        target 1-8 workers, 60% of slots busy, queue wait p95 100 ms
  workers: cpu 0%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: within capacity
  last: scaled out to 4: 100% of slots busy, over the target of 60% for 6s (19s ago)

Sixteen busy slots at 60 percent needs about 27 slots, which is four workers of eight threads. This is much the same rule the HorizontalPodAutoscaler applied in the Kubernetes post, except that adding a worker here costs a fork rather than a new pod, and the diagnosis comes along with it.

What's next

What the daemon does about the worker count is fairly conventional. Grow quickly when requests wait, shrink slowly, and don't thrash. The part I find more useful is the diagnosis, because in every case above except the first, adding workers was either not going to happen or not going to help. Whether the answer was more threads, fewer threads, a free-threaded interpreter, less garbage, a thread for a blocking call or a group of its own for a slow path, it was a change to the next release, and you only know which by measuring what the workers are actually doing.

Scaling also goes the other way. A server with many small sites, previews or staging copies doesn't want each of them holding workers while nobody is using it. The next post, scaling to zero, is about taking a release all the way down to no workers, and what it then costs for the next request to wake it up again.

16 Nov 2026 12:00am GMT

15 Nov 2026

feedPlanet Python

Speed Matters: zar-rs

ZAR: A New Archive Format with a Python API

Have you ever been bored because of slow compression/decompression tools? Because speed matters, I've developed a new archive format with very fast tools.

ZAR is a new archive format, not just another Python wrapper around ZIP or TAR. It is designed to combine fast parallel Zstandard compression with features that the familiar formats don't offer together: content deduplication, versioned commits, historical restores, and recovery checks.

The format is implemented in Rust and comes with a native Python API. Create archives quickly, keep earlier file versions in the same archive, and restore part of the archive without unpacking everything.

And Python is just one way to use it: the same Rust library also powers a command-line tool, a desktop archive manager, and bindings for many other languages.

What makes the format different?

Most archive formats are great at packaging files. ZAR is built to manage changes to those files over time, too. Each update can add a commit while preserving earlier states, so you can inspect or restore history without keeping a separate full archive for every version.

Capability ZAR ZIP tar.gz 7z
Parallel, block-based Zstandard compression Yes No (Deflate) No (single-threaded gzip) No (typical LZMA workflows)
Commit-based updates and historical states Yes Appendable, not versioned No No
Within-commit content deduplication Yes No No Solid blocks only
Random access to a file Yes Yes No Limited by solid blocks
Per-block checksums and recovery scan past damaged regions Yes CRC; no recovery scan No Limited
Authenticated encryption with per-commit passwords Yes Varies No (requires a wrapper) AES-CBC

These are not just checkboxes: commits let you update an archive without discarding the old state, deduplication avoids storing duplicate content within a create or add operation, and block framing lets ZAR compress in parallel and verify individual blocks. The result is a format for people who want both quick archive operations and useful history.

Why Python developers should take a look

πŸš€ Fast compression, without leaving Python

The Python package, pyzarlib, is a PyO3 extension for the Rust archive library. ZAR splits data into blocks and compresses them in parallel with Zstandard. Long-running Rust operations release Python's GIL, so other Python threads can keep working while an archive operation is running.

In one unencrypted, hot-cache benchmark on a Linux Core i7-10700, ZAR compressed a 525 MiB Linux source tree in 0.86 seconds. In the same run, ZIP took 11.01 seconds and 7z took 24.91 seconds. ZAR's archive was 115.5 MiB-smaller than ZIP's 151.1 MiB, though not as small as 7z's 81.1 MiB. If you need a much smaller archive, 7z may be the better fit; if you want a compelling balance of compression time and size, ZAR is worth a try.

ZAR archive benchmark results on a Linux Core i7-10700 One hot-cache run on a Linux Core i7-10700; results vary by machine, storage, and data.

The full benchmark report includes extraction times, cold-cache runs, and details about the test setup.

🐍 Python API - and a CLI for Python workflows

Create, update, inspect, verify, and extract archives directly from Python. Inputs accept pathlib.Path as well as strings:

from pathlib import Path

import pyzarlib

archive = Path("backup.zar")

created = pyzarlib.create_archive(
    archive,
    [Path("src"), Path("README.md")],
)
print(f"Created with {len(created['warnings'])} warning(s)")

# Add a new commit. Earlier file versions remain in the archive history.
pyzarlib.add_archive(archive, [Path("README.md")])

# Store and read custom archive-level metadata.
pyzarlib.update_custom_metadata(
    archive,
    set={"project": "example", "revision": 1},
)
metadata = pyzarlib.read_custom_metadata(archive)
print(metadata["project"], metadata["revision"])

for entry in pyzarlib.list_archive(archive):
    print(entry["path"], entry["type"])

check = pyzarlib.check_archive(archive)
print(f"Verified {check['commits']} commit(s)")

# Preview a restore before writing files.
plan = pyzarlib.extract_archive(
    archive,
    "restored",
    dry_run=True,
)
print(f"Would extract {len(plan['files'])} files")

Custom metadata values can be strings, signed integers, or finite floats. Updating metadata appends a metadata-only commit; keys beginning with zar. are reserved. Metadata updates require an unencrypted archive.

To append an encrypted commit to the archive above and read its file back:

from getpass import getpass

password = getpass("Commit password: ").encode("utf-8")
pyzarlib.add_archive(
    archive,
    [Path("secrets.txt")],
    password=password,
    encryption_scope="header-and-data",
)

contents = pyzarlib.extract_archive_to_memory(archive, password=password)
print(f"Read {len(contents['secrets.txt'])} bytes")

Only the new commit is encrypted; earlier commits remain unchanged. Without the password, commits with encrypted headers are skipped, so a listing or extraction can show only part of the archive. For commits with different passwords, pass a passwords mapping from commit IDs to password bytes.

Need a terminal workflow? The package also provides zarc-py, a Python CLI for creating, adding, removing, joining, undoing, checking, listing, inspecting, and extracting archives.

The package supports Python 3.11 and newer, with CI coverage for CPython 3.11-3.15 and free-threaded CPython 3.14t and 3.15t. Native work releases the GIL, and the extension is tested with concurrent Python calls.

πŸ—œοΈ Compression controls and familiar archive inputs

Choose a Zstandard compression level from -7 to 22 and a block size from 64 KiB to 32 MiB, or let ZAR select a block size automatically. Default compression level is 3. Files with identical content can be deduplicated within a create or add operation. Some already-compressed formats and media are stored without an extra compression pass to improve compression speed.

You don't have to start with a ZAR archive. Python create_archive and add_archive can import entries from ZIP, 7z, RAR, ISO, TAR, TAR.GZ, TGZ, and GZ files, preserving their internal paths. Encrypted ZIP/7z/RAR inputs and some special archive entries are not supported; see the project documentation for the precise limits.

More than a compressed file

ZAR archives are built around commits:

That makes ZAR useful for more than a one-off download bundle: it can serve as a compact, inspectable history of project snapshots or application data.

One Rust core, many ways to use it

The shared archive library is available through:

Language or runtime Integration
Rust Native zarlib crate
Python PyO3 pyzarlib extension
C and C++ czarlib C ABI and cxxzarlib
Go gozarlib through cgo
Java FFM and JNI bindings
C# P/Invoke binding
WASI WebAssembly Component; Node.js integration is available

You can use the zarc command-line tool, manage archives in the zarg desktop application, or connect an AI client to the local zarmcp server. The companion zcopy tool transfers file trees over a network using compressed, optionally encrypted ZAR batches. Try it out. You'll be impressed how fast you can transfer big file trees over the network without creating (temporary) archives.

Installation

Build the Python extension from the project source with Maturin:

python -m pip install maturin
cd pyzarlib
maturin develop --release

Then import pyzarlib in your Python code. The repository includes Python installation and API documentation, examples, and tests.

Get started

If you're a Python developer who wants quicker archives without giving up a comfortable Python API, try ZAR on a representative dataset. The benchmark includes machine and method details so you can compare it against your own workload-not just trust a headline number.

Explore the source, complete documentation, and benchmark reports:

https://github.com/brmmm3/zar-rs

Licensed under the MIT or Apache-2.0 License.

15 Nov 2026 7:00am GMT

13 Nov 2026

feedDjango community aggregator: Community blog posts

Issue 363: Wagtail's new API

News

In Memoriam: Tim Peters, 1952-2026

Guido van Rossum shared the news that Tim Peters died on July 15. He wrote the Zen of Python (PEP 20) and gave Python Timsort, and as "Timbot" he answered countless questions and set the tone for early core development. Pythonistas are sharing their memories in the thread.


Wagtail CMS News

DRF vs Django Ninja: the Wagtail 8.1 API schema

Wagtail's new API runs on Django Ninja, not DRF, and Sage Abdullah and Thibaud Colas explain why: schema generation is first-party rather than deprecated in favor of drf-spectacular, and Pydantic schemas follow the type hints Wagtail already uses.


Djangonaut Space News

Picking Up Django #26355, Part 1: Understanding the Problem

Emanuel Jaimes explores Django ticket #26355, investigating previous implementation attempts and possible approaches to supporting PostgreSQL's array_append functionality in ArrayField.

Landing My First Commit in Django Core.

Goodluck Igbokwe shares his journey to his first Django core commit, exploring ImageField validation and the importance of understanding the code behind the documentation.


Python Software Foundation

Python 3.16.0 alpha 2 is now available

uuid4() and uuid7() now have C implementations and run 15-30x faster, re picked up nested character sets, and startup and garbage collection are quicker. Alpha 3 follows on December 15.

PyPI's redesigned project pages with clearer security signals are now live

Each PyPI project now has a Security tab that shows whether a release has full provenance, meaning every file has an attestation, or none. The redesign also adds search and filters to the release files list, and shows file counts and yanked or quarantined status in the release history.

Joshua Mcshannon Joins the PSF as Infrastructure Engineer

The PSF's new Infrastructure Engineer will spend his time cutting operational toil across python.org, PyPI, and the systems behind PyCon US. He has already started on Kubernetes cost savings.


Updates to Django

Today, "Updates to Django" is presented by Raffaella from Djangonaut Space! πŸš€

Last week we had 17 pull requests merged into Django by 15 different contributors - including 8 first-time contributors! Congratulations to ines besrour, himanshu0870, ShabirAhmadLinux, Patel Namraa, Dmitry, Sahil Advani, Kunal Kumar and Hossein Shams for having their first commits merged into Django - welcome on board!

New in Django 6.2:

New in Django 6.1:


Django Fellow Reports

Django Fellow Report - Jacob

Between release blocker and security work, Jacob ran a small Trac sprint over Sarah's triage workflow changes and this year's Google Summer of Code dark mode project, and followed up with the GDAL maintainers on findings from the security releases. The main management command is now named django.

Django Fellow Report - Sarah

A four-day week spent mostly on triage and the "Ready for Checkin" backlog, accepting tickets on StepValueValidator crashing for large numbers, order_by("pk") on a union of composite-pk models, and NegatedExpression mishandling FullResultSet. She also enforced ticket assignment limits and triage restrictions in Trac.


Sponsored

Prod-like data for your coding agents

Coding agents write Django code against fixtures and guess what your data looks like. Baseshift gives every agent, PR and developer its own masked copy of your production database, ready in seconds, with no infrastructure changes. Start free and point DATABASE_URL at your first clone.


Articles

Django's Generic Views: FormView

Ben Cardy's fourth class-based view post walks through FormView, which renders a form, validates it, and redirects on success.

Making Django admin easier to use every day

Some examples of a team customizing the admin as needed for their specific use cases. A worthwhile effort if you use it heavily.

Django: introducing django-launch-editor for jump-to-file debugging

Adam Johnson's new package makes the file paths on Django's debug page clickable, opening your editor right at that line. When the editor is out of reach, in Docker or on a server, clicking copies the location instead.

Authenticating Django Channels WebSockets without putting the JWT in the URL

A browser cannot send an Authorization header on a WebSocket handshake. The common fix puts the token in the query string, where every access log keeps a copy. This is the other way, and the four places it can quietly go wrong.

Using OpenTofu to manage your Infrastructure as Code

Ryan Cheley finally learned the OpenTofu that Django Commons runs on by importing his own AWS, Coolify, DigitalOcean, GitHub, Hetzner, and Tailscale setup.

Lock your GitHub Actions

Action tags can be moved, so a workflow pinned to a safe-looking tag can start running an attacker's code. Hugo van Kemenade walks through GitHub's new workflow lockfiles.

Catch the import. Name the culprit. Keep threads free: Introducing gilguard

gilguard is a tiny, zero-dependency package for free-threaded Python (3.13t, 3.14t, 3.15t). Import it first. If any later import quietly turns the GIL back on, you get an error naming the extension module, the package that shipped it, and the line of your code that pulled it in.

What makes uv slow?

uv maintainer Konstin walks through where uv still spends its time, from source distribution builds and network requests down to TOML parsing (about 30ms just to read Airflow's lockfile), hashing, NTFS, and memory allocation.


Events

Wagtail Space 2026

Wagtail Space is next week, November 18-20. The event is 100% online and you can watch live or watch the recordings at your convenience. Expect talks from Wagtail and Django experts plus guest speakers on AI, sustainable tech, and other trends shaping the web. Tickets are free, so grab one before Wednesday.


Videos

People First, Code Always - Kenneth Love

Longtime Django developer Kenneth is the guest in this episode from Jon Gould at Foxley Talent. Lots of insights from coding in the pre/post AI world.

Databases, Django APIs, and AI - Haki Benita

The video cut of Django Chat #209, just over an hour with Haki Benita on getting more out of the ORM and PostgreSQL indexes, where Django Ninja fits, and what Django still needs for typing and APIs. That last part is the ground DEP 0017 covers, adding content type parsing and a request.data attribute that handles JSON, and the forum thread has the wider debate.


Django Job Board

The PSF's CPython Developer in Residence opening is still accepting applications, alongside senior Python roles at Manychat in Barcelona and Proxify, plus a founding engineering role at Alloy.

CPython Developer in Residence at Python Software Foundation

Senior Python Engineer (Barcelona-based / Relocation support to Barcelona) at Manychat

Senior Backend Developer (Python) at Proxify AB

Founding Forward Deployed Engineer at Alloy


Projects

prafulgulani/django-experimental

A playground for trying experimental Django features behind feature flags, installed as django.experimental.

swarfte/django-ninja-expose

Add @expose() to a Django model. Get a REST API.


Sponsor Django News

Reach 4,300+ Django developers every Friday. See sponsorship details and rates.

Planning something for the holidays? Reply and ask about our Black Friday deal.

13 Nov 2026 5:00pm GMT

08 Nov 2026

feedDjango community aggregator: Community blog posts

Django: introducing django-launch-editor for jump-to-file debugging

I've made another package! django-launch-editor makes the file locations on Django's debug error page into links that open your editor.

Here's how it looks, on a KeyError from the package's example project:

Screenshot of Django’s debug error page, with the file locations in the traceback underlined as links.

(Click to enlarge.)

The change against stock Django is subtle, but can you spot the underlines on the file locations? They're now links that open your editor at the right line, and even the right column when Python knows it.

I think this is gonna be a neat time saver. Before, I would normally remember the filename and line number, then go open that in my editor. Or I would fiddle around copying the file path to then open it. Now it's just a click!

Setup is as easy as installing the package, setting Django's DEFAULT_EXCEPTION_REPORTER setting to the package's reporter class, and adding its URLs. See the README for the details.

When DEBUG is False, the reporter adds nothing to error pages, and the URL isn't even registered, so there's zero overhead in production.

When no editor is found

django-launch-editor launches your editor from the Django server process, so it can only launch an editor directly when you run Django on the same machine as your editor. If you run your development Django server elsewhere, such as in a Docker container or on a remote server, it can't find your editor. In that case, clicking a location shows this dialog and copies the location to your clipboard instead:

Screenshot of a dialog headed β€œRunning editor not detected”, saying the file location was copied.

That way, you can still paste the location into your editor's "go to file" prompt, without fiddling around selecting text.

Origins: profiling-explorer and launch-editor-py

django-launch-editor grew out of work on profiling-explorer, my browser-based interface for exploring Python profiling data. Its table shows a filename and line number for each function, with the same problem that it would be nice to open the file in my editor at that line. Initially, I added a copy button next to each location, for copying <filename>:<line> to the clipboard, but that still left the switch, paste, and wait steps. But after digging through hundreds of functions with the tool, I really wanted to be able to jump straight to the code with one click.

After a little exploration, I found that the JavaScript world had solved this problem years ago with the launch-editor npm package, which powers "click to open in editor" features in Vite and other frameworks. It detects which editor is running, by scanning the process list, and knows the command line syntax for passing a line and column to many editors. For example, VS Code wants code --goto file:line:column, PyCharm wants pycharm --line line --column column file, and Vim wants vim +line file.

Rather than reinvent this knowledge, I ported the package to Python, as launch-editor-py. Translating code between languages is a task where LLMs shine, so I used my gift Claude subscription to do the bulk of the port. That made the project small enough to be worth doing, leaving me to focus on reviewing and testing the result, adding tests, and making the API Pythonic. The README has a full list of differences from the npm package.

profiling-explorer 1.3.0, released last week, uses launch-editor-py to open file locations on click. Once I had that working, Django's error page was the obvious next target, hence django-launch-editor.

launch-editor-py's API

The main API for launching an editor at a given file, line, and column is a single function:

from launch_editor import LaunchEditorError, launch_editor

try:
    launch_editor("src/example.py", line=12, column=4)
except LaunchEditorError as exc:
    print(f"Could not open editor: {exc}")

launch-editor-py on the command line

launch-editor-py also comes with a command line interface, which is handy for checking which editor it detects. Use --guess to see which editor it would pick:

$ python -m launch_editor --guess
code

That's the code command for VS Code.

Then pass a filename, optionally with a line and column, to open it:

$ python -m launch_editor example/views.py:20:15

…and VS Code jumps to line 20, column 15 of that file.

You can also try it without installing, using uv:

$ uvx --from launch-editor-py python -m launch_editor --guess

Fin

I hope django-launch-editor saves you some copying, switching, and pasting. And hopefully we can integrate launch-editor-py elsewhere in the Django ecosystem, such as in Django Debug Toolbar. Successful integration among packages could even provide good motivation to add the dependency to Django itself…

May your debugging be infrequent and fast,

-Adam

08 Nov 2026 6:00am GMT

06 Nov 2026

feedDjango community aggregator: Community blog posts

Issue 362: Django 6.1.3, 6.0.10, and 5.2.19 Security Releases

News

Django security releases issued: 6.1.3, 6.0.10, and 5.2.19

Six CVEs, three of them in GeoDjango: two server-side request forgery holes where GDAL resolved URLs or files from GeoJSON CRS members and SpatialReference strings (full protection needs GDAL 3.9+, flagged by the new gis.W001 check), plus log injection through GeometryField. The rest cover base64 uploads that ignored FILE_UPLOAD_MAX_MEMORY_SIZE, cache keys that confused missing and empty Vary headers, and password hashes shown to view-only admin users. Upgrade soon.


Django Software Foundation

Django Steering Council Meeting, November 2

The Steering Council met with the Fellows ahead of the mid-January feature freeze, with Carlton Gibson aiming to land the async transactions PR before then. They also weighed what to do now that sqlparse is no longer maintained (it backs --debug-sql and inspectdb), how to surface DEP discussions on the Forum, and GSoC 2027 ideas.


Python Software Foundation

Announcing new Security Developer at the PSF

Brett Smith, formerly of the W3C and Software Freedom Conservancy, joins Seth Larson and Mike Fiedler on the PSF security team in a new role funded by Anthropic. The Python Security Response Team is handling 3-4Γ— last year's volume of reports and CVEs, and PyPI malware is up 60%, so Brett will help with remediation and scale PyPI's autonomous quarantine program.

How the Python ecosystem responded to the eMudhra intermediate compromise

Within two days of the October 20 CA compromise, fixes shipped for certifi, Python, pip, uv, sigstore, virtualenv, and more, and Python added a last-resort denylist for the bad intermediates (CVE-2026-115557) that was backported to older releases. Seth Larson explains why certifi, pip, and Python sat in the critical path, and why browsers were safe from day one thanks to certificate revocation lists, which Python may adopt.


Wagtail CMS News

Wagtail 8.1: automate more with an improved API and agent-ready tools

The preview REST API can now write more content types, expose StreamField block schemas, and drive moderation workflows (submit, approve, reject, cancel), alongside official agent skills and a prototype CLI. Editors also get SEO checks for meta titles and descriptions, pages can be served as JSON or Markdown from the same URL, and multilingual sitemaps now include translated pages.

An official headless demo for Wagtail

bakerydemo-headless rebuilds the bakery demo as a fully decoupled Next.js front-end in TypeScript, with live preview still working in the CMS. Thibaud Colas notes it runs about 95% on Wagtail's built-in v3 API, whose OpenAPI schema generates typed clients down to individual StreamField blocks, with no GraphQL required.


Updates to Django

Today, "Updates to Django" is presented by Hwayoung from Djangonaut Space! πŸš€

Last week we had 17 pull requests merged into Django by 12 different contributors - including 2 first-time contributors! Congratulations to Alex Henman and Spoko Dev for having their first commits merged into Django - welcome on board!

News in Django:


Django Fellow Reports

Django Fellow Report - Jacob

Jacob Tyler Walls finalized the patches and sent prenotifications for Django's November security releases, then started drafting patches for the next round. He also fixed attrs mutation in legend_tag(), opened a follow-up optimizing lookup preparation for GeoJSON geometries, and sat in on Executive Director interviews.

Django Fellow Report - Sarah

Sarah Boyce accepted five tickets, including consistent database selection for subqueries, dropping GIS support on 32-bit Windows, and a parallel test runner crash, and reviewed fixes for XOR join promotion, select_for_update() after combined queries, and warning assertions. Her own PR allowing multiline template tags was merged, and she worked on three security reports and interviewed final candidates for Executive Director.

Django Fellow Report - Natalia

Natalia Bidart prepared the branch that bootstraps Django 2028, the first calendar-versioned release: it drops Python 3.12 and 3.13, removes the EMAIL_* settings in favor of MAILERS, and finalizes several ORM deprecations, and it lands in mid-January once the 6.2 branch is cut. She also pushed docs PRs that give local builds a new theme and better accessibility, and joined three Executive Director interviews.


Sponsored

Django for Beginners, 6th Edition

The sixth edition is out now, available in digital and paperback formats. It features two new chapters, completely updated for Django 6.1+, and more. Read the first four chapters for free to try it out.


Articles

Soft delete in Django without making a mess

Stein Ove Helset builds soft delete on a deleted_at timestamp and works through where it breaks: bulk QuerySet.delete() still removes rows, unique=True blocks re-creating a deleted record (a conditional UniqueConstraint fixes that, though MySQL and MariaDB ignore the condition), restores can collide with newer rows, and CASCADE never fires for related objects. A filtered default manager paired with an unfiltered base manager keeps related lookups working.

N+1 queries in Django Admin

If a related model's __str__ reads another foreign key, every option in an admin dropdown runs its own query, and select_related() on the edited object doesn't reach those options. Override formfield_for_foreignkey() and formfield_for_manytomany() to add select_related() to the querysets behind the widgets.

Django's Generic Views: TemplateView

Ben Cardy's series on class-based views reaches TemplateView, which needs only a template_name. He uses extra_context for static values, get_context_data() for computed ones, and template_engine to render a single view with Jinja2.

Taking Website Screenshots with Django, Celery, and Playwright

Nik Tomazic builds a Dockerized Django app that takes full-page or viewport screenshots with Playwright. Each capture runs in a Celery task so the form returns right away, and the page polls a small JSON status view until the image is ready. An optional last section moves the browser to Browser Use's paid hosted Chromium for when local browsers eat worker memory or hit bot detection.

Looking for a job? Start a blog.

Tim Schilling argues that a blog shows a hiring manager how you communicate before they ever talk to you, and suggests starter topics like a recent technical decision or how you review code. He says the host doesn't matter (GitHub Pages, dev.to, and Medium all work) and asks you to write it yourself instead of with AI, since the point is to show how you think.

How Caktus Runs and Uses Local Models

Tobias McNulty and Colin Copeland describe the two-GPU workstation Caktus uses to serve open-weight models over Tailscale, so client code stays on its own network. A dense Qwen3.8-27B handles planning and long sessions, and a Qwen3.6 MoE with 3B active parameters runs everyday work at 200-250 tokens per second, both on vLLM in Docker Compose. The post includes their benchmarks, why Gemma 4 31B didn't make the cut, and the pi.dev and VS Code configs they use daily.

Per-Object Permissions in Django That Roll Back with Atomic()

Joshua Fields wrote pemba, a Zanzibar-style authorization library that keeps per-object permission relationships in your own Django database. A grant is just rows in the current transaction, so it commits or rolls back with the data it protects, which a separate service like SpiceDB can't guarantee.

Artisanal Podcasting in the Age of AI

Will Vincent has hand-edited Django Chat in GarageBand for nearly seven years, even with AI tools that promise to handle the whole thing. Editing gives him a second listen to each conversation and a kind of deep focus he no longer gets when coding with a handful of AI agents.

Caching at Scale in Django: Lessons from Production

Johnny Metz's team handled about 20x traffic growth on a multi-tenant Django API by caching aggressively: per-user lists of document IDs paired with one shared entry per document, seven-day TTLs with invalidation on Django signals, a cached identity lookup, and failing open when Redis goes down. During the rollout, database queries per second halved, and Redis now serves most reads at a 97% hit rate.


Events

DjangoCon Europe 2027 Call for Participation

DjangoCon Europe runs February 17-21, 2027, in Innsbruck, Austria, and wants talk and workshop proposals on technical and non-technical topics from new and experienced speakers. Proposals close November 15, 2026.


Podcasts

Django Chat #209: Databases, Django APIs, and AI - Haki Benita

Haki Benita talks about getting more out of Django's ORM and PostgreSQL indexes, why Django Ninja appeals for APIs, and what Django needs for typing and a built-in API story. They also get into how AI is changing the way developers learn and share what they know.


Videos

From librarian to Djangonaut: My Open Source journey with Annabelle Wiegart

In a guest talk for Djangonaut Space Session 7, Annabelle Wiegart traces her path from librarian to Django contributor, PyLadies Zurich organizer, and Djangonaut Space captain. She talks about working on ORM tickets and nullable expressions with mentors, and how contributing went from intimidating to something she could do.


Django Forum

A Path to Modernizing APIs in Django (DEP 0017 Content Type Parsing)

Picking up from the Haki Benita episode of Django Chat, Will Vincent lists what stands between Django and a built-in API: a modernized HttpRequest through DEP 0017, a pluggable serialization layer (the hardest part), and then the view layer. Carlton Gibson wants those primitives updated first, while Haki would settle on an official API interface and tune performance later.


Django Job Board

The PSF's CPython Developer in Residence opening leads the board this week, alongside senior Python roles at webbyfox consultancy and at Manychat in Barcelona, with relocation support.

CPython Developer in Residence at Python Software Foundation

Senior Python Engineer at webbyfox consultancy

Senior Python Engineer (Barcelona-based / Relocation support to Barcelona) at Manychat


Projects

amureki/django-cmdbar

Adds a Cmd+K / Ctrl+K palette to the Django admin for jumping to models, searching records (user alice), or opening one by primary key (user#42). It respects admin permissions and needs no frontend dependencies, middleware, URLs, or migrations: just add django_cmdbar above django.contrib.admin in INSTALLED_APPS.

Metron-Project/izaya

An htmx-powered autocomplete widget for Django forms that ships no required styles, so it fits any CSS framework. Register a ModelLookup with your search_fields, drop LookupWidget into a form (with multiple=True for multi-selects), and use rank_fields to put the closest matches first; it needs Python 3.14+ and Django 6.0+.


Sponsor Django News

Reach 4,300+ Django developers every Friday. See sponsorship details and rates.

06 Nov 2026 5:00pm GMT

05 Nov 2026

feedPlanet Twisted

Glyph Lefkowitz: Why You Should Care About The Political Project of Twisted

Twisted was originally the network component of a text-based role-playing game engine. In the late 90s when it was first developed, the idea of a video game being able to send you actual emails and instant messages1 seemed pretty neat, rather than the valence that it would have today (nightmare). We never ended up making an actual game, so we eventually removed the "game engine" piece, and just kept the protocol implementations of the email and the AOL Instant Messenger, and the DNS and eventually SSH and so on. The residual pieces of that project have not seen maintenance for a decade.

So it's not really a way to make a text-based game any more.

Then, Twisted was the premier longest-lived and best-maintained event-driven I/O library for Python. If you wanted to make high-performance servers, it was the best and most obvious way to do it. Even if it still acquits itself relatively well in this regard, its claim to fame, and to indispensability, has been somewhat obviated by the inclusion of asyncio in the standard library. Even if it has some stuff to recommend it - better-than-average testing facilities, for example - you can get better-documented and better-integrated low-level asynchronous networking tools without even installing any packages.

So it's not really the premier choice for an event-loop these days.

Twisted does still contain an SSH server and client, an email server and client, a DNS server and client, a web server and client (now including websockets). But you can cobble these together by installing a variety of packages from other sources.

So... it's a bunch of protocols? And I guess it doesn't use AI? But... why would anyone care about that? Why contribute to such a thing?

What is Twisted doing, and who is it doing it for? What is the political project of Twisted, nigh on 30 years into its existence?

Most of all, why should you care about it, or want to contribute?

If you're asking what you could use Twisted for today, it's an excellent choice if you're implementing a new network service and you need more than one protocol. If you need to do HTTP, DNS, packet capture, and a custom SSH server, it would be very hard to arrange a collage of other Python packages to manage this. It's still much better at deterministic testing than most of the standard library. It's a bit easier to hook into a GUI.

But that's not what Twisted is doing.

What I want Twisted to do for the world is still an unfinished project. It's honestly kind of bad at doing what I want it to, but it could be great. The overarching goal of having and maintaining all this stuff is to, as much as possible, demystify networked software operation and let you build Internet facing systems at a human scale.

It's all in a memory-safe language with good error handling, so you can avoid the scariest security pitfalls. It's all integrated into a single paradigm with built-in tools for testing. It's designed to scale down to be runnable on the average laptop, and not require large elaborate server farms or distributed clusters to run. Via the twist command line utility, it provides a Swiss army knife of simple tools that you can use out of the box.

In a world where all of our compute is in massive distant opaque datacenters, where a tiny cartel of gigantic tech companies control nearly the whole supply chain, one of the few things that you still can have agency over is your own server on the Internet.

It's still possible to get a domain name for a few bucks a year, a free-tier or very low-cost VPS, then stick a glue record in and run your own DNS. From that point forward, you can run your own email, your own web server, your own terminal server. But doing this requires deep expertise, and that expertise requirement is massively inflated by the operational concerns that go with running twenty different pieces of software from twenty different projects, probably in twenty different languages, with completely inconsistent aesthetics.

The reason that Twisted is event-driven is because as much as it can be a pain to wrap your head around async/await (or worse yet, Deferred directly), it is much easier for a single person to understand the concurrency requirements of an async I/O multiplexed system than to understand what is going on with an application that requires multiple thread pools. It is possible to understand concurrent code that manages parallel tasks in that idiom. The reason that Twisted integrates so many different protocol implementations and attempts2 to unify them with a common authentication layer is to reduce the overhead required to manage a comprehensive suite of network services.

Even the not-using-AI thing is not an aside due to some accidental personal preference about tools - it's a core part of building an engine that you can personally understand if you dive into it, that's built with a consistent style and layered in a way that you can take apart if you need to. It's not built with AI because we're not building for AI, we're building for people and the entire point is that people are supposed to be able to understand it.

We used to have a tag line, which is somewhat less emphasized these days because its meaning is not immediately clear without this background, but was once quite prominently featured on the home page: "Twisted: Building the Engine of Your Internet". That's what I want to bring to you: an engine for you to build your Internet, that is small enough that you can make it personal, that doesn't require massive investment in management tooling just to get started.

Now, for various reasons, I no longer use twisted.names to host my own DNS (although I did for many years!), and we never wrote enough of a twisted.mail suite for it to be usable for email hosting on its own. But, if you're reading this article, it was probably delivered to you directly via Twisted's own twisted.web HTTP server, using twisted.protocols.tls to configure our own OpenSSL TLS termination. I have run with extremely high reliability for years and next to zero management overhead, in a single simple docker container. Which I then administer and upload files to via Twisted's own twisted.conch SSH server.

If that seems like an appealing project and you'd like to volunteer to help out, drop me a line and maybe I can point you at some tasks to get started. Twisted falls far short of this aspiration despite 30 years of work, and we could use all the help we can get.


Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!


  1. Yes, it was the 90s, we didn't have "texts" yet, it was all "IMs". ↩

  2. - with varying degrees of success - ↩

05 Nov 2026 10:51pm GMT

09 Oct 2026

feedPlanet Twisted

Glyph Lefkowitz: Programming Isn’t Special

Creative Work

Writers went on strike to get protections against "AI". Thousands of artists have signed open letters in protest of "AI". There are so many copyright lawsuits from creative industry groups against AI that there's a whole dedicated website for it. Popular YouTubers absolutely hate it. If they're also musicians, they REALLY hate it. Across all creative industries, there is a concerted push to reject this technology.

Yet, almost unique among creative fields, many experienced programmers remain convinced that it's fine to use "AI" for programming. We do seem to hate it, and what it's doing to our industry. It's making us all miserable, but we are using it anyway.

A lot of the justification of this resignation seems to be to be because programming is not Art. If the tool can get the job done, and the job is just functional, then why does it matter?

It does matter, though. It matters because we shouldn't be using AI to produce Art, and programming is Art.

Art can be Mundane

Some people will say that programs cannot be art because programs are functional, rather than being expressive. Programs are mundane whereas art is transcendent.

This is based on a distorted understanding of what Art actually is.

In John Berger's "Ways of Seeing", he names this type of distortion "mystification". His example of this process is both amusing and illustrative. I encourage you to read it in its entirety.

In summary, though: Berger critiques the florid prose of an art historian describing a commissioned group portrait, including phrases like "subtle modulations of the deep, glowing blacks" and "harmonious fusion". The portrait is described as sublime, in nearly ecstatic terms.

Berger reveals that the reality of this portrait is that a poor old painter needed some work, and some officials probably thought it might be nice to have an official portrait. So they paid some money to the poor old man, and he painted it, and then they had a painting. It's a well-executed portrait of a group of people. Beautiful, even. But it was work commissioned for a fairly mundane purpose and it suited that purpose just fine. It was not, and is not, a divine relic.

Culturally, we are prone to mystifying painting, and sculpture, and film, and music. We imbue them with "subtle modulations". We ignore their functional aspects - we desire decoration, amusement, and distraction - and focus on their emotional impact.

Don't get me wrong: I love me some good aesthetic philosophy. I think it's great to really examine our reactions to artwork and to try and gain a deeper understanding of our culture and our selves through media analysis. If anything we really need to do more of it.

This does not mean that the creation of such works is mystical or that it should be venerated beyond any other sort of labor.

Not least of which other types of labor that are adjacent to, but not as culturally venerated, as fine art. We tend to mystify the work of a novelist, but to denigrate the work of a journalist. In reality, the functional prose of the journalist is no less important and deserves no less respect.

Although they might be far below the ethereal realm that novelists inhabit in our collective imagination, even journalists receive more respect and thus more mystification than lowly copywriters. Yet, there is no transcendental distinction between "novelist" and "copywriter"; the many of the skills are the same, and the distinction is merely an accident of commerce and opportunity.

In fact, many famous writers have famously inhabited both roles. This is not an accident! Working with words professionally, even (perhaps especially) mundane words, is excellent practice for working with words in a more purely artistic context, because even mundane creativity is still artistic.

Code can be Beautiful

One thousand Internet years ago, when I was in my late teens, I would describe myself as a "code poet". I was relentlessly mocked for this as what the Youth would today call "being cringe", and at the time was referred to as "pretentious".

I succumbed to the peer pressure, removed it from my email signature and my bio. While I still believed strongly in the parallels, I accepted that - socially, at least - comparing code to poetry, or indeed to Art, was a silly thing to do.

However, I never abandoned the idea, in my heart.

The thing that I am most well-known for, the invention of Deferred, was specifically an aesthetic reaction to the tedium of passing callback and errback parameters to every single remote procedure call in an RPC client/server application. Those two callbacks got the job done just fine. But they were ugly, and annoying to work with.

Deferred is an intentional poem about asynchronous task execution, with a deliberate eye to the aesthetics of the problem and the experience of using it. It was influential because of its focus on aesthetics.

I do not want to overstate the beauty or profundity of this minor contribution, or indeed its durability. That a poem exists does not mean it is a great poem, merely that it is a poem.

Our aesthetic culture around programs is more like folk epic poetry than fine art, so the influence of this contribution is less about its specific enduring power than it is about its influence on what came next; from MochiKit.Async to JQuery Deferred to JavaScript Promises and eventually to async/await; a long chain of different artisans each adding something of their own until the original has all but dissolved. (And I wasn't the "original" here, either, as I drew heavily from the E language's Promises, among other things.)

In order to make code into a deliberate artistic expression, one must have spent quite a bit of time contemplating the problem domain. Without having experienced the tedium of manually passing a thousand callback parameters, I would have had neither the skill, nor indeed the motivation, to bother creating such a thing.

Now, most code does not have to be like this. Most code does not get to be like this. Most code is functional, workday code. Most code could not make a lady weep. It's just copy-writing, if you will.

As I explained previously, most writing couldn't do that either. Most writing is just copy-writing, too. Most visual art is advertising. Most live music performance is background music in bars that will go largely ignored.

However, code that is intentionally aesthetically designed tends to be important, both socially and technologically.

We do have some tradition of self-mystification in software. As Abelson memorably put it, "Programs must be written for people to read, and only incidentally for machines to execute.", so we have long had some conception of programs as highly expressive, even if we can't always agree on what they're expressing or to whom. We will occasionally wax poetical about the philosophical implications of a particular piece of software. This is not unique to a single piece of software, either; more than one community has indulged in similar philosophizing.

The expressive and aesthetic qualities of software are not limited to reading source code or interacting with other programmers via APIs, either. For example, every year, Federico Viticci does a review of Apple's new operating system, which is (among other things) an aesthetic critique. Such a project would not be possible if the software did not have an aesthetic impact on its users.

Not to mention that every video game review is also a software review.

A Brief Aside about Software Literacy

It does make me a bit sad that we don't have much of a critical reading tradition in the software community. Literate Programming is often praised, but rarely practiced.

Moreover, it makes me sad that users have a pretty jumbled idea of what goes into making software, that programming literacy is pretty low, and that modern programming practices often deliberately produce a bad mental model of what the software is doing so it's even harder for the user to understand. The aesthetic experience of software is often wildly detached from its internal state.

While all of these problems predate AI by years or indeed decades, that's no reason to enthusiastically make them worse.

Defend The Mundane

If we use AI to erase all the copy-writing, all the graphic design, all the boring mundane art, and yes, all the boring custom WordPress theme development, then we will be removing all the practical opportunities for the vast amounts of practice and contemplation required for people to elevate their craft to eventually achieve great things. Education is great, but the majority of true skill development happens on the job and always has.

This doesn't mean that we can't use abstractions, or automation, to make our work easier. Programming is the art of abstraction, of understanding how to compose smaller ideas into bigger ones, of how to understand the automation of a larger system by understanding the rules that automate smaller ones and understanding how to combine them.

When we use "AI" to eliminate that understanding rather than raise it up to a higher level, to entirely destroy that creative decision-making process, we do a disservice both to ourselves as programmers and to our users. We would be doing a disservice to our users and our downstream fellow developers in the same way that a visual artist would be doing a disservice to their viewers or a musician would be doing a disservice to their listeners if they served them auto-generated filler instead of their own creative output.

Slop is slop, no matter the medium.

Each mundane project has some tiny chance - let's say, something like 0.1% - of achieving greatness. If we do a single project with AI, then sure, whatever, there's almost no chance that that project was going to be the one hit to create that career-defining moment for an engineer working on it. If we make a habit of doing all projects that way, though, we take the total likelihood of those moments of greatness from "definitely sometimes" to "never".

The precisely appropriate ways in which to resist AI encroachment on all software development lie well beyond the margins of this one short post. How much you can resist and which specific uses you should resist are up to you. But it is worth resisting in software just as much as it would be worth resisting in any creative medium.

Programming isn't special. It's just Art, and Art is the most human - and thus, the most universal - thing that there is.


Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!

09 Oct 2026 7:00am GMT

28 Sep 2026

feedPlanet Twisted

Glyph Lefkowitz: What Would A Serious AI Product Look Like?

One of the issues that I have with the current generation of "AI" products is that they do not appear to take their own premises seriously. I look at a plethora of obsequious chatbots claiming to be serious tools for problem solving, and I think, this is not what a problem-solving tool would look like.

Even before we get to the tremendous ethical problems with the frontier labs, it is this impression of their composition as a product that makes me feel, constantly, whenever I am interacting with them, that they are less a software product than that they are a grift, a scam designed to make me feel like I am interacting with a product that has capabilities that it simply does not, to try to lull me into a false sense of security that I can trust it.

The frontier labs are of course the worst offenders, but every criticism here applies just as much to Ollama, which (if anything, due to the obviously poorer quality of the available models themselves) needs these features even more than the frontier labs do.

Here, I will set down a few features that might convince me that an LLM-based product, particularly one focused on research or software development, was actually serious about helping me do useful things with it.

Make "Checking For Mistakes" A First-Class Feature

This is the biggest issue, and the major reason that I was inspired to write this post.

It is a truth universally acknowledged, that AIs cannot reliably provide information.

I could cite a ton of news articles and studies about this fact, but there is no need. Every single chatbot admits this, up front, in a fine-print disclaimer as a core part of their user interface. Gemini says "AI can make mistakes, so double-check responses", Claude says "Claude is AI and can make mistakes. Please double-check responses.1" ChatGPT says "ChatGPT can make mistakes. Check important info.".

Every time I see that last one, I wonder how I'm supposed to know what "info" is supposed to be "important".

All of these warnings are all small, gray text, painfully obviously included as legalese to push responsibility back onto the user rather than to help with anything. This is a core limitation of all these products. Checking their output is a part of the workflow for using them that:

  1. you absolutely cannot skip or skimp on without creating risks to yourself and whoever you are conveying its output to, and,
  2. it is very easy to skip or skimp on and you are encouraged at every turn to do so, because "just trust the output" is one of the quickest ways to save time.

A chatbot product that took this weakness seriously, as an actual consideration for using it, would put a checkbox next to every claim in its output. It would be a 2-column worksheet, where you've got the LLM output in the first column, and next to it, human notes in the second column, explaining what work went into checking this claim, and a big checkbox that you would only check off after you believe you'd checked its claims thoroughly enough.

Coding assistants would need to have some version of this as well. Right now, this is pushed off into code review, which means it is a dark pattern which subtly encourages the "author"2 to offload this work to their code reviewer without ever looking. Once again, "it's probably fine, I don't need to check" is the quickest way to save time and churn out those PRs faster.

It might even be useful for coding harnesses to have some affordance for checking code before it even runs tests. As the vendors themselves have admitted, it's not just expensive to burn tokens on your "AI", you also end up burning far more compute on the AI. Being able to check your diffs before sending them over to uselessly exhaust your testing compute cluster would be useful.

If your product tells me that it makes mistakes and I must be the one to check for the mistakes, but then gives me zero tools to check for mistakes, I cannot take it seriously.

More Citations to Check, And More Details

Most chatbots prefer to give an answer, rather than a citation. In my own personal use, I find that when asked to provide a list of citations with clearly marked sources for each one, they will appear to "get bored" halfway through the list and simply stop including citations at some point.

When the bots include citations at all, present them as inline annotations that say nothing but the domain name of the search result, in a font so small that it's barely legible, and an equally indecipherable icon that is fewer than 16 pixels on a side.

This is backwards.

Now, I am aware that these citations do come from somewhere, and in an attempt to reduce hallucinations, all of the major providers support some form of "grounding"3, and that those little barely-readable citation links are referencing actual structures in the RAG pipeline and not just potentially-hallucinated tokens, but I'm not talking about the underlying machinery in the model, I'm talking about the presentation to the user.

Plus, regardless of whether a snippet of text came from a RAG query, we know that LLMs can never provide an authoritative result; it's a fundamental limitation of the technology. They can still garble the results of RAG as much as they can misrepresent any other training data. This means that it must never present its results as authoritative.

If you ask an AI to do research queries, every result should be presented as a list of citations. Moreover, the presentation should display each citation as a large object of in its own right, with clearly identified metadata, including not just the site where it was found but its publication date and, if possible, the name of the author. The literal, unmodified quotation (not from RAG, not a summary: a quotation extracted with a regular program and not an LLM) should be front-and-center, larger than any AI-generated text.

If the AI product wants to editorialize or summarize (which should not always be necessary!), the AI-generated text should be presented as small text underneath the citation that has been found, de-emphasized as much as the disclaimer is right now, at the very least until the user has verified that the summary is accurate. Perhaps, for a research project, a "did you read the citation" checkbox might even be helpful.

If your product openly tells me that it will scramble, misrepresent, or omit its citations in its summaries, and I must read the original human-authored citations to be sure, but then gives me no tools to track my reading of those citations or even any way to find them, I cannot take it seriously.

No First-Person Output, No Apologies

There is no reason for a software development or research tool to use first-person language to describe itself. They should not do so. In fact they should not be allowed to do so.

There is also no reason that they should ever apologize. It is a waste of everyone's time; it's a waste for the chatbot to generate the apology, it's a waste for the user to read the apology, and it's a waste for the user to respond to the apology. Yet they unfailingly do this upon every correction.

The vendors of these tools know that they are routinely causing mental-health crises. In response, they have added non-functional "guard rails" that can still, in 2026, easily be bypassed.4

A product seriously interested in helping with productivity would correct this glaringly obvious flaw, focus on the task at hand, and stop emitting useless verbiage.

In the previous two sections, I tried to focus on ways in which the harness would be constructed differently even if the LLM technology is fundamentally impossible to improve; in this case, I have to assume that the labs have some control over the model itself. But unless they are truly incapable of influencing their output (and all their "benchmarks" and "capabilities" seem to indicate that they can control it very tightly) they ought to be building models that are much less verbose.

More Non-Natural-Language User Interfaces

Although natural language could hypothetically be a powerful interface for interacting with a computer system, the practical upshot of LLM natural language interfaces is that these interfaces are imprecise and repetitive, full of superstitions masquerading as "best practices". The inputs are a mess and the resulting outputs are a mess.

The general way of addressing this unstructured mess is to allow the chatbot to directly take action in response to the user's input; in other words to supply it with "tools" via an MCP server. But again, this is backwards. If we cannot even express our intent clearly in the first place, why are we trusting this system to take potentially destructive and harmful actions on our behalf?

Instead, I would expect a product that was seriously invested in helping me accomplish specific tasks, to have user interfaces specific to those tasks. Is it supposed to be able to be a security scanner that can discover OWASP top 10 bugs in a codebase? Have a button for that. Build that functionality into your harness, train it directly into the model, use smaller models that can satisfy that functionality more effectively than throwing it at the planet-sized brain of Fable or whatever.

I'm aware that there are small software startups that do something like this, but they are bolted on to the side of the main model providers' APIs, not integrated into the core of the product and not using their own models and AI systems to achieve consistent and repeatable results.

Strong Data Provenance Indicators

Chatbots produce data tables pulled from websites, from APIs, from MCP tools or from summarizing and scrambling the user's input. In order to provide the illusion of a seamless interface, this data is presented in-line regardless of where it comes from. But some of these outputs are produced mechanically via regular old API calls, for example, from the result of calling a tool or querying a website, but presented uniformly.

But there is a huge difference between an authoritative data source being inlined as part of a chatbot conversation, being treated as input by the chatbot, and some ad-hoc hallucinated data being treated as output of the chatbot.

If a product is trying to help me make accurate, empirically-grounded, data-driven decisions, the source of the data is critical.

Integrated into the "check for mistakes" and "verify citations" workflow I described above, there's a necessary "verify data programmatically" pass as well; to have tools that will treat portions of the output as a regular spreadsheet, allowing regular-old computer arithmetic to verify things and showing where such arithmetic was used, and how.

Better User Control of Reproducibility

Anyone familiar with the technical specifics of LLMs will know that they have a variable called "temperature" which controls the degree of randomness that the LLM uses to produce its outputs. But most users don't know this, because it isn't exposed as part of the user interface by default.

This leads to a subjective impression that you asked ChatGPT, and you got ChatGPT's authoritative answer.

You can't just set the temperature to zero and still get useful results - I am aware that it does more than just scramble the output at random, and there are perhaps good reasons that simply exposing just a temperature setting would not be that useful to users. But if we followed some more of my earlier recommendations for making more structured UI elements to solve specific problems rather than having long back-and-forth chats where each refinement depends on the previous response, perhaps those elements could also re-play the process so that users can see how reliable the bot is at a particular task and develop a sense of how the stochastic nature of the process actually affects it.

Similarly, if a user is trying to solve the same problem repeatedly with a chatbot, and the chatbot product has numerous computational tools that don't really have anything to do with the LLM, such as deterministic data-processing tools, then having a way to freeze the non-deterministic parts of the transcript but re-populate a particular data frame with updated information and fork / continue the conversation from there would be a way to avoid introducing pointless additional randomness when you already know what tool you're trying to use.

The fact that every conversation is presented as this flat chat prompt that doesn't let me interact with any of the widgets that were previously produced except through more chatting, really makes me feel like the whole product is just doing predatory social-media style "increase time on site" optimization, just trying to lure me into further repetitive and unreliable chats, rather than letting me get in, solve my problem, and get out.

Context Visibility

Managing the LLM context is the ongoing challenge facing organizations that are trying to use "agentic" workflows. Filling up the context with too much information causes well-known problems. In response, advanced LLM users attempting to solve larger problems must break up very long prompts into "skills", give access to lengthy information via "tools", and delegating sub-problems to "sub-agents" rather than simply extending a single prompt indefinitely.

All of these strategies have flaws, because even on the largest models, compared to the breadth and depth of knowledge-work problems, LLM contexts are quite small.

And yet, none of these products will show the context to the user by default. There are third-party addons that can show you a simple progress bar but for addressing the premier engineering difficulty with this technology, that is below the bare minimum.

This lack of visibility means that almost all of the tools for extending the context are flying blind. Rather than responding meaningfully to a full context, everyone just kind of guesses how much state they need by guessing and trying over and over again with progressively more elaborate skill and sub-agent layouts. Even managing context compaction ends up being an advanced API-driven workflow5.

A serious product that was trying to help the user understand would not only show "available context" but explain the impact of context compactions, make it easier to see harness-generated prompts, and so on. This would be a first-class feature, combined with the aforementioned reproducibility / replay tools, would allow users to do real experiments to develop an understanding about how to make good use of the context window.

A Sandbox That Actually Works

I've been focused on the chatbot interface here because it is the most immediately egregious upon looking at the UI. But the "agentic loop" tools used for coding are equally dangerous, if not more so. Coding tools keep destroying everyone's data, over the course of years.

These catastrophic incidents that become front-page news are relatively rare compared to the amount of coding-agent use out there. But they also aren't the only kind of sandbox violation. Coding models will so routinely edit test code instead of the system under test that there are "pro tips" articles all over the web giving you the flawed advice to simply ask the agent not to cheat. News write-ups of the catastrophic incidents themselves will also offer glib and wrong advice, like "use a docker container". That might prevent it from literally deleting your operating system, but it won't prevent it from destroying all the local work you have in your codebase (it needs access to a checkout, after all!)

There is a flurry of activity in the infosec space where people are rushing to plug the gaps left by these coding harnesses. Everyone's got their own version of an MCP approval gateway where you can optionally place a proxy between your agent and your production infrastructure.

In the best case, though, all these mitigations and proxies and prompts simply turn the user into an auto-approval automaton, hitting Y, Y, Y, Y over and over again, until you finally are driven mad and hit "yes to all", turn on full-auto mode and submit yourself to the void. With nothing between your personal vigilance and disaster, there are no workflows left beyond decrementing your own vigilance until there's nothing left and then hoping the disaster never arrives.

The fact that some mitigations exist that can be deployed by extra-cautious users does not change the fact that "agentic coding" is an unsafe-by-default technology deployed without concern or guidance. Every frontier lab has tied a spring-loaded shotgun to a dog; the fact that dog owners can publish thoughtful blog posts explaining how you can teach your dog the basics of gun safety or how you can have your dogs play in a bullet-proof room does not mitigate the fact that the product should not have been allowed in the first place, nor should it continue to exist without VERY strong security controls.

I might believe that a frontier lab were seriously interested in providing developers with a useful tool if they shipped something that had safety built-in.

That means tools in the harness, detached from any LLM, independent of the prompt, that could:

In the same way that I suggested above that research-based tasks should have a way of re-issuing prompts to determine how reproducible a result is, or whether other sources might be found, agent-based tasks should have a way of being executed against mock services for popular APIs, so that the verification can match both on the front-end (review the plan for making the API calls before they're executed) and the back end (review the API calls that were issued to the mock service and verify that they matched).

Instead, the frontier labs provide us products that are disasters out of the box, give us "best practices" to build massive and elaborate, as well as incomplete and error-prone, security perimeters of our own design. Then they blame "operator error" when it inevitably goes wrong. I cannot believe that these design choices are intended to help us be productive.

Bonus: Human Processes

Organizations deploying AI also frequently come across as unserious, for similar reasons. In 2023, naive exuberance could perhaps be forgiven. But today, as we near the close of 2026, there are several well-known problems, that have been extremely well-covered in the press. None of these things should be surprising, but most orgs deploying these tools are still just letting them rip and hoping it all works out.

Organizations deploying these tools would need at least three kinds of major modifications to their internal processes, if they wanted to be serious about using them safely:

1. Shift Rotations to Prevent Vigilance Decrement

There have been several high-profile incidents where software developers' gradual acquiescence to accepting LLM output have lead to serious economic consequences for the companies deploying them, perhaps best typified by Amazon's "millions of lost orders" due to a gradual decay of their engineering processes from LLM use.

These outages, and other AI-related failures, are due to the difficulty of maintaining focus on the same problems. In other words, as I described above, vigilance decrement is a constant problem, because AI outputs are most often correct, but continue to be incorrect in surprising and non-intuitive ways. As I have previously written, you cannot trust yourself to catch every bug with code review, and LLM output.

Aviation, for example, has very strict rules around rest requirements. There is also a specific rule that "No certificate holder may operate an aircraft without a second in command if that aircraft has a passenger seating configuration, excluding any pilot seat, of ten seats or more.". Other safety-critical professions have similar rules.

And yet, even in the age of the supposed "AI revolution", most software teams are still assigning every engineer a full feature load, not planning for any rest, and telling people to review code whenever they happen to have some "free time".

Maintenance of vigilance has to be your top priority. Regular, scheduled, inviolable rest periods where people do work without AI assistance, and are not exposed to any AI output for review or otherwise, would be crucial in order to stay mentally sharp enough.

The tools themselves should have this sort of thing built in. The mistake-review process described above should have a periodic spot-check mode where a second reviewer periodically reviews a chatbot log, doing their own independent verification of claims, to see if they spot the same errors. This could provide a feedback loop to determine how much rest is necessary to maintain continuous attention and actually spot hallucinations.

2. Skill Practice To Prevent Skill Loss

It is also well-known that AI use leads to AI reliance, and AI reliance leads to skill loss.

I like to use the analogy to dockworkers at a seaport6 adopting automation.

If you employ dockworkers to load and unload ships all day long, they are going to be getting tons of exercise. They will be able to lift heavy objects on demand, whenever. They might have plenty of health problems and injuries from this type of work, but "lack of exercise" will not be a problem.

With the development of standardized container ships and mechanized cranes, you are going to be changing their job description substantially: now they mostly spend all day sitting in a small cubicle moving a control lever back and forth, not lifting heavy stuff. They will get worse at lifting heavy objects.

In this analogy however, the cranes are not all that reliable. We know they break, and they drop their payloads sometimes, and the stuff needs to be manually moved. But this only happens a few times a week, at most. If you need whoever is driving the crane to be able to jump out at any moment and still move stuff around manually, then you need to make an affordance for that. You need to give them time to go to the gym and do some lifting for practice, or every crane failure is going to be a major emergency.

An organization doing an AI transformation would also need a massive increase to learning & development budget, both in terms of resources and in terms of schedule. If your people are going to lose skills because they've lost regular practice in the incidental course of doing their duties, then they are going to need deliberate, intentional, non-incidental practice of those skills to keep them sharp.

But rather than trying to accommodate new workflows and give time for people to adjust, most AI mandates are simply dropped on workers like a ton of bricks, with no time to adapt and no affordance for maintaining their skills. Operate the crane and stay fit and healthy and ready to switch back to manual lifting at any time and then get back in the crane cockpit right afterwards. Don't mess up.

Then an accident happens and everyone is surprised, as if this process weren't practically designed to produce a terrible result.

3. Mental Health Resources to Deal with Mental Health Risks

AI psychosis often begins with practical problem-solving, and beyond that, it can start specifically at work. Not to mention the more pedestrian condition of "AI brain fry".

If you are mandating your employees to use a hazardous tool that may seriously and directly damage their mental health, you need trainings and resources. You need in-house therapists and you need to be making sure to check in with people actively to make sure that this is not happening.

Again, the tool itself ought to have some way of dealing with this. An occasional "take a break" popup is easily dismissed; they need a user-visible AI personal dosimeter so you can see your cumulative usage over time.

I don't even know if "usage over time" is a sufficient metric to gauge risk. Maybe if your work chatbot start to talk about resonance too much, unless you literally work as an acoustic engineer, that should be flagged for someone.

We are, again, years into dealing with these tools, and we know these risks exist. Yet no serious mitigations are provided. Not even any way of measuring the risk exposure.

And More

There are also many other risks associated with the technology. There are intellectual property risks with the foundation models, due to recklessness with their training data. There are existential financial risks associated with the infrastructure build-out. The extent to which most "open" models are simply derivatives of frontier models is an open question.

What I Think

If any one of these things were regularly overlooked by AI vendors or users, that would be a totally normal product oversight. Room for improvement for the next version, but nothing catastrophic.

Shipping without any of them doesn't seem like lean product management, it seems like a careless attitude towards risk and a product design philosophy oriented entirely towards short-term demos, with no regard for how to realize actual productivity gains.

Furthermore, being available for years without anything like these features, despite hundreds of incidents demonstrating the risks, with hundreds of billions of dollars of funding, makes it seem to me like if they were to add all the features that would make their product actually safe and hypothetically useful, these features would reveal that it is actually not an improvement to productivity.

In the year since I first wrote about measuring the cost/benefit ratio of AI, I have heard from numerous people who have shown this to management to try to illustrate why their AI initiatives - like almost all AI initiatives - were either failing or burning out their engineers.

I've also heard from lots of people that have told me that it's obviously useful and they don't need to measure so carefully, because they are getting lots of work done that they couldn't have otherwise.7

I have yet to hear from a single person who has said "yeah, we measured according to your methodology8, and it turns out that our AI work is going great and that our ratio is 0.75".

Obviously, I cannot say for sure why this is; absence of evidence is not evidence of absence. But at this point I think the null hypothesis is that AI tools provide, in aggregate, zero value. They make mistakes too often, and the externalities they produce are so bad and so difficult to control that even before we get to the places where they are just physically poisoning people, even the negative effects on their direct users end up cancelling out whatever benefit to they provide to their organizations.

If I were wrong, then including tools to measure an AI's effectiveness at the tasks their users are actually trying to accomplish, rather than meaningless benchmarks, would show big productivity gains. The frontier labs would be champing at the bit to add such features, and crowing about their fantastic results.

I think the labs know that if they did that, it would present a grim picture to their users. Such tools would let their users see that it's making mistakes much more often than they realized, that they're spending much more time with it than they want to be, and that it's just generally not fit for purpose.

If they prove me wrong by adding in all of these safety mechanisms, and in the process, they make all of their AI technology less harmful, I'll be thrilled to be debunked.


Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!


  1. It is also interesting that for the next section, sometimes it seems that Claude's disclaimer is "Please double-check cited sources." instead. ↩

  2. ... by which I mean the "prompter", since authorship is not what's happening here. ↩

  3. Claude has the "citations API", Google has various different kinds of "grounding" against its own APIs, and I guess Microsoft can check OpenAI's homework if you want. ↩

  4. Given the relatively slow speed of the justice system and the mainstream press around the world, we probably will not hear about whether people are managing to incidentally break through these guard rails to self harm right now, but there are no shortage of stories still being reported right now where people were still doing just that, such as in this story where the effect of the much vaunted "guard rails" in 2025 was that if you wanted it to write you a suicide note, it would refuse twice but acquiesce on the third try. I don't see any reason to believe this fundamental issue has been addressed in the meanwhile, since it had been happening for years at that point. ↩

  5. In this tutorial we can also see an incredibly rosy scenario presented, where a long-running workflow effortlessly compresses all of the necessary information into the new context, even if it uses a lower-fidelity model to do so, rather than the tangled and gnarly problem of problems which really are too big to fit in the context, which is to say, "most real-world problems". This presents the context limit instead as a minor speedbump to be worked around rather than the fundamental flaw in LLM tooling. ↩

  6. A heavily fictionalized seaport. This is not how actual dockworkers work. This is a simplistic metaphor about incidental benefits of instrumental tasks, it is not supposed to delve deeply into the mechanics of maritime shipping. In particular I know that cranes are more reliable than this and this is not actually how you would respond to a crane malfunction anyway. Feel free to share fun facts about maritime shipping if that is your special interest but please do not @ me to correct this metaphor. ↩

  7. To my knowledge, none of their publicly-traded employers have posted a measurable improvement to efficiency outside the margin of error. ↩

  8. Or any similar methodology. I don't need people to adopt the exact practice that I proposed there. ↩

28 Sep 2026 4:27am GMT