19 Nov 2026

feedPlanet Python

Graham Dumpleton: Muru: noticing and responding

The last post was about overlays, named pieces of configuration the Muru daemon lays over a site's routes for a while, activated by hand or on a schedule. Everything in it was known ahead of time. The backup runs at 2am, and the sale starts at 9am on Monday.

The things that cause the most trouble don't come with a schedule. AI crawlers turn up when they turn up, and some of them will happily take more of a site's capacity than all its real visitors put together. A payment provider goes down without telling anyone first. The usual answer is a monitoring system that notices, an alert that wakes someone up, and that person logging in to change something, which they then have to remember to change back once it is over. Automating that means a script hanging off an alerting webhook, which is one more thing to write, secure and keep working.

Muru has two pieces for this. A condition says when something holds over Muru's own figures, and an overlay can be triggered by a condition rather than by the time. Put together, Muru notices something, responds to it, and stops responding once it is over.

Telling crawlers apart

Before anything can respond to crawlers, the edge has to know which requests are theirs. Muru gives every request a client class, worked out from rules declared beside the routes:

[host.shop]
names = ["shop.localhost"]
default = true

[[route]]
name = "shop"
host = "shop"
app = "shop"

[[client_class]]
name = "ai-crawler"
user_agent = ["GPTBot", "ClaudeBot", "CCBot"]

[rate_limit.crawlers]
match = { client_class = ["ai-crawler"] }
key = "client_class"
rate = "2/s"
burst = 2

Matching on the User-Agent is the simplest rule. Rules can also match addresses, headers, cookies or the JA4 fingerprint of a TLS connection, or check a claimed search engine crawler by looking up its address in DNS, and community lists of AI crawler user agents can be dropped in as files. A request matching no rule is class default. The class goes to the application as muru.client_class in the WSGI environ, into the access log, where muructl logs --access shows it as class=ai-crawler for anything but default, and onto the client_class label of the request metrics.

The rate limit's match applies it to requests in the ai-crawler class, and key puts the whole class in one bucket, so the crawlers get two requests a second between them while everyone else is left alone. Declaring it doesn't put it in force, though. A rate limit takes effect only on a route that names it, and nothing names this one yet.

To have something to watch, I used a load script with two people's browsers asking for the shop's front page five times a second each for a whole run, and four crawlers, with a GPTBot user agent, asking as fast for the first part of it. Every five seconds it prints the answers so far, by status.

A condition watches

A condition is declared in the same files:

[condition.crawler-surge]
when = "request_share{app='shop',client_class='ai-crawler'}[10s] > 0.5"
for = "10s"
clear_when = "request_share{app='shop',client_class='ai-crawler'}[10s] < 0.2"
clear_for = "10s"
min_requests = 20

The expression is written to look like a Prometheus alerting rule, and that is deliberate, since it is the form people who run sites already know. request_share is the share of the shop's requests from the ai-crawler class, over a ten second window. The condition fires once the crawlers have been more than half the shop's requests for ten seconds, and resolves once they have been under a fifth for ten seconds. Having two thresholds with a gap between them stops a condition flapping around one value. min_requests is how many requests the window needs before a share counts as a figure at all, so a handful of requests can't fire it.

The windows here are short, where real ones would be minutes long. There are signals for request rates, error rates, latencies, queue depth, how busy the workers are, whether the autoscaler is at its limit, the node's memory and the health of upstream services, and terms can be joined with and and or. There is no arithmetic and no functions, which keeps a condition something you can read at a glance.

With no traffic, the share has no figure, and the condition is unknown:

$ muructl condition list
CONDITION     STATE     SINCE                 WHEN
crawler-surge unknown   -                     request_share{app='shop',client_class='ai-crawler'}[10s] > 0.5

A few seconds into a run of the load script, the crawlers were two thirds of the requests, and the condition was waiting out its for:

$ muructl condition show crawler-surge
condition crawler-surge (applied)
  state       pending: when has held for 5s of 10s
  when        request_share{app='shop',client_class='ai-crawler'}[10s] > 0.5
  for         10s
  clear_when  request_share{app='shop',client_class='ai-crawler'}[10s] < 0.2
  clear_for   10s
  min         20 requests
  term        request_share{app='shop',client_class='ai-crawler'}[10s] > 0.5 is 0.6667
  term        request_share{app='shop',client_class='ai-crawler'}[10s] < 0.2 is 0.6667

Ten seconds after that, it fired:

  state       firing since 2026-11-17T06:29:30Z

When the crawlers stopped, their share dropped away, and show counted down the clear_for wait:

  state       firing since 2026-11-17T06:29:30Z; clear_when has held for 6s of 10s

before the condition resolved. Each change is an event, carrying the figures that caused it:

$ muructl events
...
    10 17:29:30 warning  condition.firing     crawler-surge            firing, when held for 10s; request_share{app='shop',client_class='ai-crawler'}[10s] > 0.5: 0.6667; request_share{app='shop',client_class='ai-crawler'}[10s] < 0.2: 0.6667
    11 17:30:08 info     condition.resolved   crawler-surge            resolved, clear for 10s; request_share{app='shop',client_class='ai-crawler'}[10s] > 0.5: 0; request_share{app='shop',client_class='ai-crawler'}[10s] < 0.2: 0

And that was all that happened. The crawlers were served throughout:

 30s  people 296 x 200  crawlers 592 x 200

A condition does nothing by itself. That is the point of declaring one before anything acts on it, as it can be left watching real traffic, with its events going to wherever the other events go, until its thresholds and waits fire when you would actually want them to.

Acting on it

An overlay triggered by the condition gives the crawlers a route of their own while it fires, the same as the shop's route but drawing on the rate limit:

[overlay.crawler-response]
routes.shop.client_routes.ai-crawler = { rate_limits = ["crawlers"] }
trigger = { condition = "crawler-surge" }
min_active = "20s"
cooldown = "1m"

client_routes lets a route treat requests from a particular client class differently. Each entry is named after a class and gives only the settings that change for that class's requests, with everything else staying as the route has it. Here, requests from the ai-crawler class get the crawlers rate limit added, and requests from everyone else carry on as before. I ran the load again, and once the condition fired, the overlay was active:

$ muructl overlay list
OVERLAY                  STATE              TRIGGER                            CHANGES
crawler-response         active (condition) condition crawler-surge, firing    route shop

and the crawlers' answers turned to 429s while the people's stayed 200s:

  5s  people 50 x 200  crawlers 100 x 200
 10s  people 100 x 200  crawlers 200 x 200
 15s  people 148 x 200  crawlers 296 x 200
 20s  people 198 x 200  crawlers 307 x 200, 89 x 429
 25s  people 248 x 200  crawlers 317 x 200, 179 x 429
 30s  people 296 x 200  crawlers 327 x 200, 265 x 429

A refused request is still counted as one of the shop's, so the crawlers stayed most of its requests and the condition kept firing for as long as they kept coming. Once they stopped, the condition resolved and the overlay ended with it. The events tell the whole story:

    13 17:30:34 warning  condition.firing     crawler-surge            firing, when held for 10s; ...
    14 17:30:34 notice   overlay.activated    overlay crawler-response activated: its condition crawler-surge is firing; ...
    15 17:30:35 warning  ratelimit.enforcing  rate_limit crawlers      began refusing or holding requests: 14 so far
    16 17:31:08 info     condition.resolved   crawler-surge            resolved, clear for 10s; ...
    17 17:31:08 notice   overlay.ended        overlay crawler-response ended: its condition crawler-surge resolved; ...

Nobody ran a command, and nothing was left for anyone to undo.

Something acting automatically needs to be kept from acting too often. min_active keeps the overlay active for at least a while however soon its condition resolves, five minutes by default. cooldown stops the condition activating it again for a while after it ends, ten minutes by default, and max_per_hour caps how many times it can activate in an hour. I sent another burst of crawlers straight after, inside the cooldown. The condition fired again, but the overlay stayed as it was, and the crawlers were served:

    18 17:31:35 warning  condition.firing     crawler-surge            firing, when held for 10s; ...
    19 17:31:35 warning  overlay.skipped      overlay crawler-response not activated though its condition crawler-surge is firing: its cooldown after it last ended lasts until 2026-11-17T06:32:08.559Z

A response that hides its own cause

Rather than refusing the crawlers, an overlay could answer them from a static snapshot of the shop, which costs the application nothing at all:

[overlay.crawler-response]
routes.shop.client_routes.ai-crawler = { target = { static = "snapshot" } }
trigger = { condition = "crawler-surge" }
min_active = "20s"
cooldown = "1m"

This is the part of conditions I found most interesting, because it seems like an improvement and isn't. muructl apply accepted it, but warned:

muru: warning: overlay crawler-response takes shop's ai-crawler requests away from what condition crawler-surge reads, so the condition sees none of them and may resolve while they keep coming; rate-limit the class instead, or read the class's share of every request, without app

A request answered from a static directory isn't the application's. While the overlay is active, the crawlers vanish from the shop's figures, so their share of its requests falls to nothing, even though they are still coming. I ran it anyway, with the crawlers going for the whole run, to see what happens:

    24 17:33:34 notice   overlay.activated    overlay crawler-response activated: its condition crawler-surge is firing; ...
    25 17:33:53 info     condition.resolved   crawler-surge            resolved, clear for 10s; ...
    26 17:33:55 notice   overlay.ended        overlay crawler-response ended: its condition crawler-surge resolved; ...
    27 17:34:11 warning  condition.firing     crawler-surge            firing, when held for 10s; ...
    28 17:34:11 warning  overlay.skipped      overlay crawler-response not activated though its condition crawler-surge is firing: its cooldown after it last ended lasts until 2026-11-17T06:34:55.558Z

Twenty seconds after the overlay sent the crawlers to the snapshot, the condition saw no crawlers and resolved. The overlay ended and the crawlers were back on the shop, where the condition saw them and fired again, and only the cooldown stopped the overlay going straight back on. Without a cooldown, it would have flipped back and forth like this for as long as the crawlers kept coming.

An overlay must not take away what its own condition reads. There are two ways out. One is a condition on a figure the overlay doesn't change. Leaving app out of request_share counts every request the node takes, the snapshot's included. The other is latch = true, which keeps the overlay active after its condition resolves, until a person ends it:

$ muructl overlay list
OVERLAY                  STATE              TRIGGER                                    CHANGES
crawler-response         active (condition) condition crawler-surge, latched, resolved route shop

The condition resolved while the crawlers carried on, but the overlay stayed. Deciding when the crawlers have really gone is then left to a person, with muructl overlay preview showing the condition's figures beside what the overlay changes, and muructl overlay end ending it. A latch suits an overlay which responds to something going wrong, where Muru turning the response off on its own isn't wanted anyway.

Holding capacity while a dependency is down

Overlays can set capacity as well as routes, and one of the cases where that matters is less obvious than crawlers. The shop's checkout calls out to a payment service, and when that is down, checkout fails straight away:

def checkout(start_response):
    try:
        with urllib.request.urlopen(f"{PAYMENTS}/charge", timeout=2) as response:
            response.read()
    except (urllib.error.URLError, OSError):
        start_response("503 Service Unavailable", [("Content-Type", "text/plain")])
        return [b"Payments are unavailable, please try again shortly.\n"]

    start_response("200 OK", [("Content-Type", "text/plain")])
    return [b"Thank you for your order.\n"]

Failing fast is the right thing for the application to do, but it changes what the autoscaler sees. A checkout normally holds a worker thread while the payment goes through. When every checkout fails in a millisecond, the slots look idle, and the autoscaler does what it should with idle slots, as covered in the scaling from what the workers report post, and takes workers away. When the payment service comes back, the customers who were waiting all come back with it, and the shop is short of workers at exactly the wrong moment.

The shop has the same kind of spare mode policy as in that post, with its windows shortened so this happens in seconds rather than minutes, between one and four workers. I ran twelve clients doing checkouts, and the shop scaled out to four workers. Then I stopped the payment service:

 30s  1143 x 200
 40s  1603 x 200, 12 x 503
 50s  1603 x 200, 2189 x 503

and within half a minute the shop had given up half its workers:

    19 17:45:25 info     scaling.decision     shop@r1                  default: scale in from 4 to 3 workers: at least 3.50 idle workers over the last 20s, over max_spare 2; diagnosis: within capacity
    21 17:45:30 info     scaling.decision     shop@r1                  default: scale in from 3 to 2 workers: at least 2.75 idle workers over the last 20s, over max_spare 2; diagnosis: within capacity

With the payment service back, and the shop down to a single worker after a quiet spell, the next rush of checkouts had to wait on a scale-out:

    26 17:47:00 info     scaling.decision     shop@r1                  default: scale out from 1 to 3 workers: short by 2.00 workers for 5001ms; diagnosis: I/O bound with exhausted slots; ...

With a policy whose min_workers is zero, as in the scaling to zero post, the shop could have scaled all the way down while payments were out.

The autoscaler can't know that the idle slots are the symptom of something else. A condition can, if Muru is told about the payment service. Declaring it as an upstream with a health check gives the edge a figure for it:

[upstream.payments]
addresses = ["127.0.0.1:18540"]
health = { path = "/healthz", interval = "2s", unhealthy_after = 2, healthy_after = 2 }

[condition.payments-down]
when = "upstream_healthy{upstream='payments'} < 0.5"
for = "5s"
clear_when = "upstream_healthy{upstream='payments'} >= 1"
clear_for = "20s"

upstream_healthy is the share of the upstream's addresses in service, so the condition fires when more than half of them have failed their health checks, and resolves once all of them have been back for twenty seconds. The overlay holds the shop's capacity while it fires:

[overlay.shop-hold]
trigger = { condition = "payments-down" }
apps.shop = { min_workers = 4, scale_in = "hold" }

Running the same test again, the condition fired five seconds after the payment service stopped answering its health checks, and the overlay raised the floor:

    41 17:48:44 warning  condition.firing     payments-down            firing, when held for 5s; upstream_healthy{upstream='payments'} < 0.5: 0; upstream_healthy{upstream='payments'} >= 1: 0
    42 17:48:44 notice   overlay.activated    overlay shop-hold        activated: its condition payments-down is firing; ...
    43 17:48:45 info     scaling.decision     shop@r1                  default: scale out from 3 to 4 workers: raising toward the floor of 4 that shop-hold sets; diagnosis: within capacity

For the rest of the outage the shop held four workers, though every one of them was idle:

$ muructl scale status shop
shop@r1
GROUP        WORKERS      SLOTS      IDLE    QUEUE-P95     REAPS      POLICY
default      4            2/16       3.50    under 1 ms    5/6        spare 1-4 workers, 0-2 idle
  scheduled: 4-4 workers, no scale-in from shop-hold
  workers: cpu 1%, gc 0%, gil wait p95 under 0.1 ms, loop lag p95 -
  diagnosis: within capacity
  last: scaled out to 4: raising toward the floor of 4 that shop-hold sets (70s ago)

The health check doesn't depend on the shop's own requests, so this overlay can't hide its own cause the way the snapshot did. When the payment service came back, the condition resolved, and the overlay stayed on for the rest of its min_active, five minutes by default, which is time for the rush that follows an outage to arrive while the workers are still there. After that, the autoscaler goes back to deciding for itself.

The same pattern works the other way around. A cap, cap_workers, sets the most workers a group may have whatever its policy says, so while a struggling database is the bottleneck, a condition on its latency can stop the autoscaler adding workers that would only send it more queries at once.

Signals from outside

Muru's figures only cover what Muru can see. A database replica falling behind, or a payment provider's status page announcing an incident, is something a monitoring system knows about first. An overlay with trigger = { external = true } can be activated and ended through a signed webhook, served on an address of its own given in the daemon's configuration file:

[overlay_webhook]
listen = "127.0.0.1:9466"
secret_file = "/etc/muru/overlay-webhook.key"

This overlay takes the shop read-only while payments are out, with a page that says so:

[overlay.payments-outage]
routes.shop.error_pages = { "503:read_only" = { template = "<h1>Checkout is paused</h1><p>Payments are down. Your cart is safe.</p>" } }
apps.shop = { mode = "read-only", deny = ["POST", "PUT", "PATCH", "DELETE"] }
trigger = { external = true }

A request to the webhook carries a JSON body saying what to do, and the HMAC-SHA256 of the body with the shared secret in a header, the same signing Muru uses for the events it sends out. The body includes the time it was made, which must be within five minutes of the daemon's clock, so a captured request can't be replayed later:

$ body='{"action": "activate", "time": "2026-11-17T06:50:36Z", "for": "2h"}'
$ signature=$(printf '%s' "$body" | openssl dgst -sha256 -hmac "$(cat /etc/muru/overlay-webhook.key)" | sed 's/^.* //')
$ curl -H "X-Muru-Signature: sha256=$signature" -d "$body" http://127.0.0.1:9466/v1/overlays/payments-outage:trigger
{"message":"activated by the webhook","overlay":"payments-outage"}

Sending the same request again was refused, since the overlay was already active, and one with a bad signature was turned away:

{"error":"refused","message":"overlay payments-outage is already active"}
{"error":"bad_signature","message":"X-Muru-Signature is not sha256= and the body's HMAC with the webhook's secret"}

While it was active, the overlay list showed who had activated it and for how long:

payments-outage          active (external) until 2026-11-17T08:50:36Z by hand or webhook                 route shop, app shop

and an attempt to add to the cart got the overlay's page:

$ curl -si -X POST http://127.0.0.1:8080/cart
HTTP/1.1 503 Service Unavailable
content-type: text/html; charset=utf-8
content-length: 71
cache-control: no-store
x-request-id: 01M7M4B4NT1VVWMGEPKW48DD5P
date: Tue, 17 Nov 2026 06:50:36 GMT

<h1>Checkout is paused</h1><p>Payments are down. Your cart is safe.</p>

A request with "action": "end" ended it again. An overlay activated by the webhook, or by hand, is only ended by the webhook or by hand, so a monitoring system that resolves its alert gets to say when the response is over.

What's next

What I like about conditions is that the first step is only watching. A condition can sit against real traffic for a week, raising events, before anything is allowed to act on it, and when something does act, the events record what it saw, what it did, and when it stopped. The responses are the same overlays as in the last post, so everything about them, from the preview to the effective configuration, works the same whether a person, a schedule or a condition turned them on.

The overlay posts have been about the Muru daemon running a site. The next post, a toolbar on every page, goes back to development mode, and the toolbar Muru adds to each page under muructl run, with the timings, logs and errors of the request which produced it, and nothing changed in the application.

19 Nov 2026 12:00am GMT

18 Nov 2026

feedDjango community aggregator: Community blog posts

Weeknotes (2026 week 47)

Weeknotes (2026 week 47)

I haven't done any blogging in the last 4 weeks. Instead, I've been writing a lot of code and have let the agent write even more code. I have finally managed to implement some tools I would have liked to implement a long time ago.

Working with LLM agents

The relationship with LLM agents doesn't get any easier, I have to admit. That said, the quality argument against LLMs seems less and less plausible to me. Some people (especially on Mastodon) will say that's because I'm losing view of what's actually important. As I see it, human-written code was never perfect either (maybe apart from the Apollo Guidance Computer). I can actually do good and solve more problems for real users with more than acceptable code quality now and that seems like a net benefit to me.

Governance - a Django app for Holacracy

As some people may know, we are using Holacracy as our organizational model at Feinheit. Holacracy One, the company behind Holacracy, also offers a SaaS solution for managing the structure consisting of circles, roles, policies etc. The tool is unfortunately quite buggy and slow, and hasn't improved in recent years - to the contrary.

I have reimplemented the parts we actually use using Django, Bootstrap and htmx while profiting from years and years of product engineering going into the Workbench agency software.

It won't surprise anyone that after the first rush of getting a half-functioning, half-assed prototype out of the agent it was a lot of work to actually make the tool somewhat consistent, usable and correct. The bar to replace the tool we've been using before wasn't that high, so I was able to cross that in a few days of concentrated work.

I think that this project wouldn't have been doable without the deep knowledge of the organizational model (the domain knowledge) and the product engineering and UX background.

An experimental API and CLI for feincms3

The feincms3-api and feincms3-api-cli allow interacting with a feincms3-managed site. The API allows reading and writing pages and other objects with their regions and plugins, django-json-schema-editor data and django-cabinet files. It's meant for scripts, importers and agents, next to the Django admin, not instead of it. The CLI (f3api) makes the API available from the command line. Writes can be tried as a dry run first, and changes show up in the admin's history like any other edit.

Permissions, editable fields etc. are all handled by the Django admin. Since feincms3 and django-content-editor lean heavily on Django's administration interface this was an easy decision to make. Plugins can be arbitrary models and we don't have an obvious way to discover them, or even know what their structure is apart from introspection or manual work. The constraints of working in an agency are such that I cannot spend a lot of time on each individual project; I really need reusable tools which can be applied to many websites and projects at once. It can be acceptable for the result to not be perfect or a little awkward as long as it is simple and understandable.

Releases

feincms3 and django-content-editor

The rendering engine from feincms3 has been ported to django-content-editor and lives there now. Nested sections stay in feincms3 as a backwards compatible implementation, the new implementation in django-content-editor is much nicer because it doesn't require section closing plugins. Instead, all plugins have a depth field and form a tree structure that way; a new level is opened by a plugin which is marked as a container. Many such container plugins can coexist.

Also, the new releases fully build on the importmap DEP API, and include many bugfixes which will not be mentioned in detail here.

feincms3-sites has also received a small update and a few bugfixes.

django-cabinet

The media library now supports moving files with drag'n'drop and has a collapsible folder tree on the left for quicker navigation.

django-json-schema-editor

The proxy models created by JSONPluginBase.proxy() are automatically injected into the Python module now; this allows Django's model autoimporter in ./manage.py shell to discover those models, which avoids many, many warnings.

django-prose-editor

I am always tracking Tiptap and ProseMirror closely. The editor now has a deeper integration with django-cabinet - inserted files keep their ID around. Apart from that, mostly minor changes.

django-js-asset

I have released django-js-asset 5.0 implementing the ImportMap specification which I'm still hoping to bring to Django core.

form-designer

Bugfixes, French, Catalan and Spanish translations, and a small security fix (don't inadvertently add formulas to generated XLSX files, the fix is in xlsxdocument below).

django-authlib

Hardened the Microsoft Entra ID adapter. Fixed a few bugs and improved docs.

django-imagefield

I did finally release a new stable release of django-imagefield containing the still experimental vips backend.

django-translated-fields

TranslatedField now handles model fields with broken^Wcustomized deconstruct methods properly. For example, feincms3's ChoicesCharField hides the choices from Django's migrations, so changing choices doesn't create pointless migrations (they wouldn't produce any SQL anyway). However, the previous implementation required that fields are reconstructible from the return value of deconstruct, which means that such fields could not be used with TranslatedField. Now they can.

django-tree-queries

Fixed an edge case crash, allowed customizing the node moving machinery to reject targets and slightly modernized the code.

feincms3-cookiecontrol

The cookie banner now supports Google's advanced consent mode out of the box, if you opt in to it as the one implementing it (not as a visitor to the website!). I think supporting that mode is good, because while the mode itself is very questionable, having a simple and straightforward implementation for it is better than having to choose one of the really, really complex and user-hostile consent management platforms.

feincms3-forms

The proxy models created by the forms builder are now injected into their module. Same problem as above, in django-json-schema-editor.

Other releases

18 Nov 2026 6:00pm GMT

feedPlanet Python

The Python Coding Stack: The Human Touch


Subscribe now

stephengruppetta.com

18 Nov 2026 4:31pm GMT

Python Software Foundation: De-fanging dangerous triggers in GitHub Actions across an organization

Authors: Mike Fiedler, Seth Larson

On December 2, 2026, GitHub will start blocking the pull_request_target trigger by default in public repositories. If that sentence means nothing to you, this post is for you. If it fills you with dread, it is also for you.

The short version: a handful of GitHub Actions triggers let a stranger on the internet start a workflow that may have access to your secrets. On a GitHub Team or Enterprise plan, you can turn them off for every repository in your organization with one policy. On a Free plan, you can do it one repository at a time. Set the policy up, check what it would block, then enforce it.

Mike Fiedler, PyPI Safety and Security Engineer, and Seth Larson, Security Developer in Residence, have removed dangerous triggers and applied these policies to the Python Software Foundation (PSF) stewarded GitHub organizations (psf, python, pypi). They also wrote up technical documentation on how to roll these policies out to your own projects and organizations. The full details and technical documentation are available on the Alpha-Omega blog.

Have dangerous triggers caused real incidents?

Yes, repeatedly. A few that have affected projects hosted on PyPI directly:

  • Ultralytics, December 2024. The project's GitHub Actions workflows were compromised and the build cache was poisoned, which put malicious releases on PyPI. Seth wrote PyPI's analysis. William Woodruff's write-up traces the entry point to a pull_request_target workflow.

  • LiteLLM and Telnyx, March 2026. Malicious releases of both packages were published after an API token was exposed through an exploited Trivy dependency. StepSecurity traced the Trivy compromise back to a pull_request_target workflow exploited three weeks earlier. We worked with both maintainer teams on token rotation, release removals, and moving to Trusted Publishers, which both projects have since adopted.

  • elementary-data, April 2026. A workflow triggered by comments puts the comment text straight into a shell script. One crafted comment on a pull request ran with a token that could be written to the repository. The attacker used it to commit a malicious file and push a release tag, and the project's own release pipeline published the result to PyPI.

The maintainers behind these projects weren't careless. These triggers look harmless in a workflow file, and the unsafe version is one line away from the safe one.

Why are these triggers dangerous?

Insecure triggers can run workflows with access to your secrets, and a stranger can start them.

When someone opens a pull request from a fork, the pull_request trigger runs their code with a read-only token and no secrets. That's the safe one.

pull_request_target looks almost the same in a workflow file and behaves very differently. It runs with a read/write token and access to your repository and organization secrets. GitHub built it for chores like labeling pull requests from forks, where the workflow never touches the contributor's code. The trouble starts when a workflow checks out the pull request and runs it anyway:

on: pull_request_target
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
        with:
          # a stranger's code
          ref: ${{ github.event.pull_request.head.sha }}  
      # running with your secrets
      - run: make test                                    

Anyone with a GitHub account can open that pull request. Their code now runs with your credentials. This is called a "pwn request", and GitHub's docs describe it as the root cause of multiple supply-chain compromises.

The issue_comment trigger carries the same risk. This trigger fires when anyone comments on an issue or pull request, and the workflow runs with the repository's token and secrets. Sometimes the victim is a "comment /test to run CI" bot that checks out the pull request and runs their code. Sometimes the workflow pastes the comment text into a templated shell command, and the comment turns out to inject different instructions into the workflow shell.

How did Alpha-Omega support this work?

Alpha-Omega is a fund and catalyst for sustainable open source security improvements. Alpha-Omega supports Mike's role as PyPI Safety & Security Engineer and Seth's role as Security Developer in Residence at the PSF.

That support is what gives us the time for the work behind this post:

  • Seth wrote PyPI's analysis of the Ultralytics attack

  • Mike and Seth worked with the LiteLLM and Telnyx maintainers during their incident, and wrote up the guidance for everyone else

  • Mike set up the event policy for the pypi GitHub organization

  • Seth is working to migrate the few remaining repositories using insecure triggers in the python GitHub organization before enabling the event policy

  • Mike and Seth wrote this guide so other organizations can do the same!

About the Python Software Foundation

The PSF is a non-profit whose mission is to promote, protect, and advance the Python programming language, and to support and facilitate the growth of a diverse and international community of Python programmers. The PSF supports the Python community using corporate sponsorships, grants, and donations. Are you interested in sponsoring or donating to the PSF so it can continue supporting Python and its community? Check out our sponsorship program, donate directly here, or contact our team!


18 Nov 2026 12:46pm GMT

13 Nov 2026

feedDjango community aggregator: Community blog posts

Issue 363: Wagtail's new API

News

In Memoriam: Tim Peters, 1952-2026

Guido van Rossum shared the news that Tim Peters died on July 15. He wrote the Zen of Python (PEP 20) and gave Python Timsort, and as "Timbot" he answered countless questions and set the tone for early core development. Pythonistas are sharing their memories in the thread.


Wagtail CMS News

DRF vs Django Ninja: the Wagtail 8.1 API schema

Wagtail's new API runs on Django Ninja, not DRF, and Sage Abdullah and Thibaud Colas explain why: schema generation is first-party rather than deprecated in favor of drf-spectacular, and Pydantic schemas follow the type hints Wagtail already uses.


Djangonaut Space News

Picking Up Django #26355, Part 1: Understanding the Problem

Emanuel Jaimes explores Django ticket #26355, investigating previous implementation attempts and possible approaches to supporting PostgreSQL's array_append functionality in ArrayField.

Landing My First Commit in Django Core.

Goodluck Igbokwe shares his journey to his first Django core commit, exploring ImageField validation and the importance of understanding the code behind the documentation.


Python Software Foundation

Python 3.16.0 alpha 2 is now available

uuid4() and uuid7() now have C implementations and run 15-30x faster, re picked up nested character sets, and startup and garbage collection are quicker. Alpha 3 follows on December 15.

PyPI's redesigned project pages with clearer security signals are now live

Each PyPI project now has a Security tab that shows whether a release has full provenance, meaning every file has an attestation, or none. The redesign also adds search and filters to the release files list, and shows file counts and yanked or quarantined status in the release history.

Joshua Mcshannon Joins the PSF as Infrastructure Engineer

The PSF's new Infrastructure Engineer will spend his time cutting operational toil across python.org, PyPI, and the systems behind PyCon US. He has already started on Kubernetes cost savings.


Updates to Django

Today, "Updates to Django" is presented by Raffaella from Djangonaut Space! 🚀

Last week we had 17 pull requests merged into Django by 15 different contributors - including 8 first-time contributors! Congratulations to ines besrour, himanshu0870, ShabirAhmadLinux, Patel Namraa, Dmitry, Sahil Advani, Kunal Kumar and Hossein Shams for having their first commits merged into Django - welcome on board!

New in Django 6.2:

New in Django 6.1:


Django Fellow Reports

Django Fellow Report - Jacob

Between release blocker and security work, Jacob ran a small Trac sprint over Sarah's triage workflow changes and this year's Google Summer of Code dark mode project, and followed up with the GDAL maintainers on findings from the security releases. The main management command is now named django.

Django Fellow Report - Sarah

A four-day week spent mostly on triage and the "Ready for Checkin" backlog, accepting tickets on StepValueValidator crashing for large numbers, order_by("pk") on a union of composite-pk models, and NegatedExpression mishandling FullResultSet. She also enforced ticket assignment limits and triage restrictions in Trac.


Sponsored

Prod-like data for your coding agents

Coding agents write Django code against fixtures and guess what your data looks like. Baseshift gives every agent, PR and developer its own masked copy of your production database, ready in seconds, with no infrastructure changes. Start free and point DATABASE_URL at your first clone.


Articles

Django's Generic Views: FormView

Ben Cardy's fourth class-based view post walks through FormView, which renders a form, validates it, and redirects on success.

Making Django admin easier to use every day

Some examples of a team customizing the admin as needed for their specific use cases. A worthwhile effort if you use it heavily.

Django: introducing django-launch-editor for jump-to-file debugging

Adam Johnson's new package makes the file paths on Django's debug page clickable, opening your editor right at that line. When the editor is out of reach, in Docker or on a server, clicking copies the location instead.

Authenticating Django Channels WebSockets without putting the JWT in the URL

A browser cannot send an Authorization header on a WebSocket handshake. The common fix puts the token in the query string, where every access log keeps a copy. This is the other way, and the four places it can quietly go wrong.

Using OpenTofu to manage your Infrastructure as Code

Ryan Cheley finally learned the OpenTofu that Django Commons runs on by importing his own AWS, Coolify, DigitalOcean, GitHub, Hetzner, and Tailscale setup.

Lock your GitHub Actions

Action tags can be moved, so a workflow pinned to a safe-looking tag can start running an attacker's code. Hugo van Kemenade walks through GitHub's new workflow lockfiles.

Catch the import. Name the culprit. Keep threads free: Introducing gilguard

gilguard is a tiny, zero-dependency package for free-threaded Python (3.13t, 3.14t, 3.15t). Import it first. If any later import quietly turns the GIL back on, you get an error naming the extension module, the package that shipped it, and the line of your code that pulled it in.

What makes uv slow?

uv maintainer Konstin walks through where uv still spends its time, from source distribution builds and network requests down to TOML parsing (about 30ms just to read Airflow's lockfile), hashing, NTFS, and memory allocation.


Events

Wagtail Space 2026

Wagtail Space is next week, November 18-20. The event is 100% online and you can watch live or watch the recordings at your convenience. Expect talks from Wagtail and Django experts plus guest speakers on AI, sustainable tech, and other trends shaping the web. Tickets are free, so grab one before Wednesday.


Videos

People First, Code Always - Kenneth Love

Longtime Django developer Kenneth is the guest in this episode from Jon Gould at Foxley Talent. Lots of insights from coding in the pre/post AI world.

Databases, Django APIs, and AI - Haki Benita

The video cut of Django Chat #209, just over an hour with Haki Benita on getting more out of the ORM and PostgreSQL indexes, where Django Ninja fits, and what Django still needs for typing and APIs. That last part is the ground DEP 0017 covers, adding content type parsing and a request.data attribute that handles JSON, and the forum thread has the wider debate.


Django Job Board

The PSF's CPython Developer in Residence opening is still accepting applications, alongside senior Python roles at Manychat in Barcelona and Proxify, plus a founding engineering role at Alloy.

CPython Developer in Residence at Python Software Foundation

Senior Python Engineer (Barcelona-based / Relocation support to Barcelona) at Manychat

Senior Backend Developer (Python) at Proxify AB

Founding Forward Deployed Engineer at Alloy


Projects

prafulgulani/django-experimental

A playground for trying experimental Django features behind feature flags, installed as django.experimental.

swarfte/django-ninja-expose

Add @expose() to a Django model. Get a REST API.


Sponsor Django News

Reach 4,300+ Django developers every Friday. See sponsorship details and rates.

Planning something for the holidays? Reply and ask about our Black Friday deal.

13 Nov 2026 5:00pm GMT

08 Nov 2026

feedDjango community aggregator: Community blog posts

Django: introducing django-launch-editor for jump-to-file debugging

I've made another package! django-launch-editor makes the file locations on Django's debug error page into links that open your editor.

Here's how it looks, on a KeyError from the package's example project:

Screenshot of Django’s debug error page, with the file locations in the traceback underlined as links.

(Click to enlarge.)

The change against stock Django is subtle, but can you spot the underlines on the file locations? They're now links that open your editor at the right line, and even the right column when Python knows it.

I think this is gonna be a neat time saver. Before, I would normally remember the filename and line number, then go open that in my editor. Or I would fiddle around copying the file path to then open it. Now it's just a click!

Setup is as easy as installing the package, setting Django's DEFAULT_EXCEPTION_REPORTER setting to the package's reporter class, and adding its URLs. See the README for the details.

When DEBUG is False, the reporter adds nothing to error pages, and the URL isn't even registered, so there's zero overhead in production.

When no editor is found

django-launch-editor launches your editor from the Django server process, so it can only launch an editor directly when you run Django on the same machine as your editor. If you run your development Django server elsewhere, such as in a Docker container or on a remote server, it can't find your editor. In that case, clicking a location shows this dialog and copies the location to your clipboard instead:

Screenshot of a dialog headed “Running editor not detected”, saying the file location was copied.

That way, you can still paste the location into your editor's "go to file" prompt, without fiddling around selecting text.

Origins: profiling-explorer and launch-editor-py

django-launch-editor grew out of work on profiling-explorer, my browser-based interface for exploring Python profiling data. Its table shows a filename and line number for each function, with the same problem that it would be nice to open the file in my editor at that line. Initially, I added a copy button next to each location, for copying <filename>:<line> to the clipboard, but that still left the switch, paste, and wait steps. But after digging through hundreds of functions with the tool, I really wanted to be able to jump straight to the code with one click.

After a little exploration, I found that the JavaScript world had solved this problem years ago with the launch-editor npm package, which powers "click to open in editor" features in Vite and other frameworks. It detects which editor is running, by scanning the process list, and knows the command line syntax for passing a line and column to many editors. For example, VS Code wants code --goto file:line:column, PyCharm wants pycharm --line line --column column file, and Vim wants vim +line file.

Rather than reinvent this knowledge, I ported the package to Python, as launch-editor-py. Translating code between languages is a task where LLMs shine, so I used my gift Claude subscription to do the bulk of the port. That made the project small enough to be worth doing, leaving me to focus on reviewing and testing the result, adding tests, and making the API Pythonic. The README has a full list of differences from the npm package.

profiling-explorer 1.3.0, released last week, uses launch-editor-py to open file locations on click. Once I had that working, Django's error page was the obvious next target, hence django-launch-editor.

launch-editor-py's API

The main API for launching an editor at a given file, line, and column is a single function:

from launch_editor import LaunchEditorError, launch_editor

try:
    launch_editor("src/example.py", line=12, column=4)
except LaunchEditorError as exc:
    print(f"Could not open editor: {exc}")

launch-editor-py on the command line

launch-editor-py also comes with a command line interface, which is handy for checking which editor it detects. Use --guess to see which editor it would pick:

$ python -m launch_editor --guess
code

That's the code command for VS Code.

Then pass a filename, optionally with a line and column, to open it:

$ python -m launch_editor example/views.py:20:15

…and VS Code jumps to line 20, column 15 of that file.

You can also try it without installing, using uv:

$ uvx --from launch-editor-py python -m launch_editor --guess

Fin

I hope django-launch-editor saves you some copying, switching, and pasting. And hopefully we can integrate launch-editor-py elsewhere in the Django ecosystem, such as in Django Debug Toolbar. Successful integration among packages could even provide good motivation to add the dependency to Django itself…

May your debugging be infrequent and fast,

-Adam

08 Nov 2026 6:00am GMT

05 Nov 2026

feedPlanet Twisted

Glyph Lefkowitz: Why You Should Care About The Political Project of Twisted

Twisted was originally the network component of a text-based role-playing game engine. In the late 90s when it was first developed, the idea of a video game being able to send you actual emails and instant messages1 seemed pretty neat, rather than the valence that it would have today (nightmare). We never ended up making an actual game, so we eventually removed the "game engine" piece, and just kept the protocol implementations of the email and the AOL Instant Messenger, and the DNS and eventually SSH and so on. The residual pieces of that project have not seen maintenance for a decade.

So it's not really a way to make a text-based game any more.

Then, Twisted was the premier longest-lived and best-maintained event-driven I/O library for Python. If you wanted to make high-performance servers, it was the best and most obvious way to do it. Even if it still acquits itself relatively well in this regard, its claim to fame, and to indispensability, has been somewhat obviated by the inclusion of asyncio in the standard library. Even if it has some stuff to recommend it - better-than-average testing facilities, for example - you can get better-documented and better-integrated low-level asynchronous networking tools without even installing any packages.

So it's not really the premier choice for an event-loop these days.

Twisted does still contain an SSH server and client, an email server and client, a DNS server and client, a web server and client (now including websockets). But you can cobble these together by installing a variety of packages from other sources.

So... it's a bunch of protocols? And I guess it doesn't use AI? But... why would anyone care about that? Why contribute to such a thing?

What is Twisted doing, and who is it doing it for? What is the political project of Twisted, nigh on 30 years into its existence?

Most of all, why should you care about it, or want to contribute?

If you're asking what you could use Twisted for today, it's an excellent choice if you're implementing a new network service and you need more than one protocol. If you need to do HTTP, DNS, packet capture, and a custom SSH server, it would be very hard to arrange a collage of other Python packages to manage this. It's still much better at deterministic testing than most of the standard library. It's a bit easier to hook into a GUI.

But that's not what Twisted is doing.

What I want Twisted to do for the world is still an unfinished project. It's honestly kind of bad at doing what I want it to, but it could be great. The overarching goal of having and maintaining all this stuff is to, as much as possible, demystify networked software operation and let you build Internet facing systems at a human scale.

It's all in a memory-safe language with good error handling, so you can avoid the scariest security pitfalls. It's all integrated into a single paradigm with built-in tools for testing. It's designed to scale down to be runnable on the average laptop, and not require large elaborate server farms or distributed clusters to run. Via the twist command line utility, it provides a Swiss army knife of simple tools that you can use out of the box.

In a world where all of our compute is in massive distant opaque datacenters, where a tiny cartel of gigantic tech companies control nearly the whole supply chain, one of the few things that you still can have agency over is your own server on the Internet.

It's still possible to get a domain name for a few bucks a year, a free-tier or very low-cost VPS, then stick a glue record in and run your own DNS. From that point forward, you can run your own email, your own web server, your own terminal server. But doing this requires deep expertise, and that expertise requirement is massively inflated by the operational concerns that go with running twenty different pieces of software from twenty different projects, probably in twenty different languages, with completely inconsistent aesthetics.

The reason that Twisted is event-driven is because as much as it can be a pain to wrap your head around async/await (or worse yet, Deferred directly), it is much easier for a single person to understand the concurrency requirements of an async I/O multiplexed system than to understand what is going on with an application that requires multiple thread pools. It is possible to understand concurrent code that manages parallel tasks in that idiom. The reason that Twisted integrates so many different protocol implementations and attempts2 to unify them with a common authentication layer is to reduce the overhead required to manage a comprehensive suite of network services.

Even the not-using-AI thing is not an aside due to some accidental personal preference about tools - it's a core part of building an engine that you can personally understand if you dive into it, that's built with a consistent style and layered in a way that you can take apart if you need to. It's not built with AI because we're not building for AI, we're building for people and the entire point is that people are supposed to be able to understand it.

We used to have a tag line, which is somewhat less emphasized these days because its meaning is not immediately clear without this background, but was once quite prominently featured on the home page: "Twisted: Building the Engine of Your Internet". That's what I want to bring to you: an engine for you to build your Internet, that is small enough that you can make it personal, that doesn't require massive investment in management tooling just to get started.

Now, for various reasons, I no longer use twisted.names to host my own DNS (although I did for many years!), and we never wrote enough of a twisted.mail suite for it to be usable for email hosting on its own. But, if you're reading this article, it was probably delivered to you directly via Twisted's own twisted.web HTTP server, using twisted.protocols.tls to configure our own OpenSSL TLS termination. I have run with extremely high reliability for years and next to zero management overhead, in a single simple docker container. Which I then administer and upload files to via Twisted's own twisted.conch SSH server.

If that seems like an appealing project and you'd like to volunteer to help out, drop me a line and maybe I can point you at some tasks to get started. Twisted falls far short of this aspiration despite 30 years of work, and we could use all the help we can get.


Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!


  1. Yes, it was the 90s, we didn't have "texts" yet, it was all "IMs". ↩

  2. - with varying degrees of success - ↩

05 Nov 2026 10:51pm GMT

09 Oct 2026

feedPlanet Twisted

Glyph Lefkowitz: Programming Isn’t Special

Creative Work

Writers went on strike to get protections against "AI". Thousands of artists have signed open letters in protest of "AI". There are so many copyright lawsuits from creative industry groups against AI that there's a whole dedicated website for it. Popular YouTubers absolutely hate it. If they're also musicians, they REALLY hate it. Across all creative industries, there is a concerted push to reject this technology.

Yet, almost unique among creative fields, many experienced programmers remain convinced that it's fine to use "AI" for programming. We do seem to hate it, and what it's doing to our industry. It's making us all miserable, but we are using it anyway.

A lot of the justification of this resignation seems to be to be because programming is not Art. If the tool can get the job done, and the job is just functional, then why does it matter?

It does matter, though. It matters because we shouldn't be using AI to produce Art, and programming is Art.

Art can be Mundane

Some people will say that programs cannot be art because programs are functional, rather than being expressive. Programs are mundane whereas art is transcendent.

This is based on a distorted understanding of what Art actually is.

In John Berger's "Ways of Seeing", he names this type of distortion "mystification". His example of this process is both amusing and illustrative. I encourage you to read it in its entirety.

In summary, though: Berger critiques the florid prose of an art historian describing a commissioned group portrait, including phrases like "subtle modulations of the deep, glowing blacks" and "harmonious fusion". The portrait is described as sublime, in nearly ecstatic terms.

Berger reveals that the reality of this portrait is that a poor old painter needed some work, and some officials probably thought it might be nice to have an official portrait. So they paid some money to the poor old man, and he painted it, and then they had a painting. It's a well-executed portrait of a group of people. Beautiful, even. But it was work commissioned for a fairly mundane purpose and it suited that purpose just fine. It was not, and is not, a divine relic.

Culturally, we are prone to mystifying painting, and sculpture, and film, and music. We imbue them with "subtle modulations". We ignore their functional aspects - we desire decoration, amusement, and distraction - and focus on their emotional impact.

Don't get me wrong: I love me some good aesthetic philosophy. I think it's great to really examine our reactions to artwork and to try and gain a deeper understanding of our culture and our selves through media analysis. If anything we really need to do more of it.

This does not mean that the creation of such works is mystical or that it should be venerated beyond any other sort of labor.

Not least of which other types of labor that are adjacent to, but not as culturally venerated, as fine art. We tend to mystify the work of a novelist, but to denigrate the work of a journalist. In reality, the functional prose of the journalist is no less important and deserves no less respect.

Although they might be far below the ethereal realm that novelists inhabit in our collective imagination, even journalists receive more respect and thus more mystification than lowly copywriters. Yet, there is no transcendental distinction between "novelist" and "copywriter"; the many of the skills are the same, and the distinction is merely an accident of commerce and opportunity.

In fact, many famous writers have famously inhabited both roles. This is not an accident! Working with words professionally, even (perhaps especially) mundane words, is excellent practice for working with words in a more purely artistic context, because even mundane creativity is still artistic.

Code can be Beautiful

One thousand Internet years ago, when I was in my late teens, I would describe myself as a "code poet". I was relentlessly mocked for this as what the Youth would today call "being cringe", and at the time was referred to as "pretentious".

I succumbed to the peer pressure, removed it from my email signature and my bio. While I still believed strongly in the parallels, I accepted that - socially, at least - comparing code to poetry, or indeed to Art, was a silly thing to do.

However, I never abandoned the idea, in my heart.

The thing that I am most well-known for, the invention of Deferred, was specifically an aesthetic reaction to the tedium of passing callback and errback parameters to every single remote procedure call in an RPC client/server application. Those two callbacks got the job done just fine. But they were ugly, and annoying to work with.

Deferred is an intentional poem about asynchronous task execution, with a deliberate eye to the aesthetics of the problem and the experience of using it. It was influential because of its focus on aesthetics.

I do not want to overstate the beauty or profundity of this minor contribution, or indeed its durability. That a poem exists does not mean it is a great poem, merely that it is a poem.

Our aesthetic culture around programs is more like folk epic poetry than fine art, so the influence of this contribution is less about its specific enduring power than it is about its influence on what came next; from MochiKit.Async to JQuery Deferred to JavaScript Promises and eventually to async/await; a long chain of different artisans each adding something of their own until the original has all but dissolved. (And I wasn't the "original" here, either, as I drew heavily from the E language's Promises, among other things.)

In order to make code into a deliberate artistic expression, one must have spent quite a bit of time contemplating the problem domain. Without having experienced the tedium of manually passing a thousand callback parameters, I would have had neither the skill, nor indeed the motivation, to bother creating such a thing.

Now, most code does not have to be like this. Most code does not get to be like this. Most code is functional, workday code. Most code could not make a lady weep. It's just copy-writing, if you will.

As I explained previously, most writing couldn't do that either. Most writing is just copy-writing, too. Most visual art is advertising. Most live music performance is background music in bars that will go largely ignored.

However, code that is intentionally aesthetically designed tends to be important, both socially and technologically.

We do have some tradition of self-mystification in software. As Abelson memorably put it, "Programs must be written for people to read, and only incidentally for machines to execute.", so we have long had some conception of programs as highly expressive, even if we can't always agree on what they're expressing or to whom. We will occasionally wax poetical about the philosophical implications of a particular piece of software. This is not unique to a single piece of software, either; more than one community has indulged in similar philosophizing.

The expressive and aesthetic qualities of software are not limited to reading source code or interacting with other programmers via APIs, either. For example, every year, Federico Viticci does a review of Apple's new operating system, which is (among other things) an aesthetic critique. Such a project would not be possible if the software did not have an aesthetic impact on its users.

Not to mention that every video game review is also a software review.

A Brief Aside about Software Literacy

It does make me a bit sad that we don't have much of a critical reading tradition in the software community. Literate Programming is often praised, but rarely practiced.

Moreover, it makes me sad that users have a pretty jumbled idea of what goes into making software, that programming literacy is pretty low, and that modern programming practices often deliberately produce a bad mental model of what the software is doing so it's even harder for the user to understand. The aesthetic experience of software is often wildly detached from its internal state.

While all of these problems predate AI by years or indeed decades, that's no reason to enthusiastically make them worse.

Defend The Mundane

If we use AI to erase all the copy-writing, all the graphic design, all the boring mundane art, and yes, all the boring custom WordPress theme development, then we will be removing all the practical opportunities for the vast amounts of practice and contemplation required for people to elevate their craft to eventually achieve great things. Education is great, but the majority of true skill development happens on the job and always has.

This doesn't mean that we can't use abstractions, or automation, to make our work easier. Programming is the art of abstraction, of understanding how to compose smaller ideas into bigger ones, of how to understand the automation of a larger system by understanding the rules that automate smaller ones and understanding how to combine them.

When we use "AI" to eliminate that understanding rather than raise it up to a higher level, to entirely destroy that creative decision-making process, we do a disservice both to ourselves as programmers and to our users. We would be doing a disservice to our users and our downstream fellow developers in the same way that a visual artist would be doing a disservice to their viewers or a musician would be doing a disservice to their listeners if they served them auto-generated filler instead of their own creative output.

Slop is slop, no matter the medium.

Each mundane project has some tiny chance - let's say, something like 0.1% - of achieving greatness. If we do a single project with AI, then sure, whatever, there's almost no chance that that project was going to be the one hit to create that career-defining moment for an engineer working on it. If we make a habit of doing all projects that way, though, we take the total likelihood of those moments of greatness from "definitely sometimes" to "never".

The precisely appropriate ways in which to resist AI encroachment on all software development lie well beyond the margins of this one short post. How much you can resist and which specific uses you should resist are up to you. But it is worth resisting in software just as much as it would be worth resisting in any creative medium.

Programming isn't special. It's just Art, and Art is the most human - and thus, the most universal - thing that there is.


Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!

09 Oct 2026 7:00am GMT

28 Sep 2026

feedPlanet Twisted

Glyph Lefkowitz: What Would A Serious AI Product Look Like?

One of the issues that I have with the current generation of "AI" products is that they do not appear to take their own premises seriously. I look at a plethora of obsequious chatbots claiming to be serious tools for problem solving, and I think, this is not what a problem-solving tool would look like.

Even before we get to the tremendous ethical problems with the frontier labs, it is this impression of their composition as a product that makes me feel, constantly, whenever I am interacting with them, that they are less a software product than that they are a grift, a scam designed to make me feel like I am interacting with a product that has capabilities that it simply does not, to try to lull me into a false sense of security that I can trust it.

The frontier labs are of course the worst offenders, but every criticism here applies just as much to Ollama, which (if anything, due to the obviously poorer quality of the available models themselves) needs these features even more than the frontier labs do.

Here, I will set down a few features that might convince me that an LLM-based product, particularly one focused on research or software development, was actually serious about helping me do useful things with it.

Make "Checking For Mistakes" A First-Class Feature

This is the biggest issue, and the major reason that I was inspired to write this post.

It is a truth universally acknowledged, that AIs cannot reliably provide information.

I could cite a ton of news articles and studies about this fact, but there is no need. Every single chatbot admits this, up front, in a fine-print disclaimer as a core part of their user interface. Gemini says "AI can make mistakes, so double-check responses", Claude says "Claude is AI and can make mistakes. Please double-check responses.1" ChatGPT says "ChatGPT can make mistakes. Check important info.".

Every time I see that last one, I wonder how I'm supposed to know what "info" is supposed to be "important".

All of these warnings are all small, gray text, painfully obviously included as legalese to push responsibility back onto the user rather than to help with anything. This is a core limitation of all these products. Checking their output is a part of the workflow for using them that:

  1. you absolutely cannot skip or skimp on without creating risks to yourself and whoever you are conveying its output to, and,
  2. it is very easy to skip or skimp on and you are encouraged at every turn to do so, because "just trust the output" is one of the quickest ways to save time.

A chatbot product that took this weakness seriously, as an actual consideration for using it, would put a checkbox next to every claim in its output. It would be a 2-column worksheet, where you've got the LLM output in the first column, and next to it, human notes in the second column, explaining what work went into checking this claim, and a big checkbox that you would only check off after you believe you'd checked its claims thoroughly enough.

Coding assistants would need to have some version of this as well. Right now, this is pushed off into code review, which means it is a dark pattern which subtly encourages the "author"2 to offload this work to their code reviewer without ever looking. Once again, "it's probably fine, I don't need to check" is the quickest way to save time and churn out those PRs faster.

It might even be useful for coding harnesses to have some affordance for checking code before it even runs tests. As the vendors themselves have admitted, it's not just expensive to burn tokens on your "AI", you also end up burning far more compute on the AI. Being able to check your diffs before sending them over to uselessly exhaust your testing compute cluster would be useful.

If your product tells me that it makes mistakes and I must be the one to check for the mistakes, but then gives me zero tools to check for mistakes, I cannot take it seriously.

More Citations to Check, And More Details

Most chatbots prefer to give an answer, rather than a citation. In my own personal use, I find that when asked to provide a list of citations with clearly marked sources for each one, they will appear to "get bored" halfway through the list and simply stop including citations at some point.

When the bots include citations at all, present them as inline annotations that say nothing but the domain name of the search result, in a font so small that it's barely legible, and an equally indecipherable icon that is fewer than 16 pixels on a side.

This is backwards.

Now, I am aware that these citations do come from somewhere, and in an attempt to reduce hallucinations, all of the major providers support some form of "grounding"3, and that those little barely-readable citation links are referencing actual structures in the RAG pipeline and not just potentially-hallucinated tokens, but I'm not talking about the underlying machinery in the model, I'm talking about the presentation to the user.

Plus, regardless of whether a snippet of text came from a RAG query, we know that LLMs can never provide an authoritative result; it's a fundamental limitation of the technology. They can still garble the results of RAG as much as they can misrepresent any other training data. This means that it must never present its results as authoritative.

If you ask an AI to do research queries, every result should be presented as a list of citations. Moreover, the presentation should display each citation as a large object of in its own right, with clearly identified metadata, including not just the site where it was found but its publication date and, if possible, the name of the author. The literal, unmodified quotation (not from RAG, not a summary: a quotation extracted with a regular program and not an LLM) should be front-and-center, larger than any AI-generated text.

If the AI product wants to editorialize or summarize (which should not always be necessary!), the AI-generated text should be presented as small text underneath the citation that has been found, de-emphasized as much as the disclaimer is right now, at the very least until the user has verified that the summary is accurate. Perhaps, for a research project, a "did you read the citation" checkbox might even be helpful.

If your product openly tells me that it will scramble, misrepresent, or omit its citations in its summaries, and I must read the original human-authored citations to be sure, but then gives me no tools to track my reading of those citations or even any way to find them, I cannot take it seriously.

No First-Person Output, No Apologies

There is no reason for a software development or research tool to use first-person language to describe itself. They should not do so. In fact they should not be allowed to do so.

There is also no reason that they should ever apologize. It is a waste of everyone's time; it's a waste for the chatbot to generate the apology, it's a waste for the user to read the apology, and it's a waste for the user to respond to the apology. Yet they unfailingly do this upon every correction.

The vendors of these tools know that they are routinely causing mental-health crises. In response, they have added non-functional "guard rails" that can still, in 2026, easily be bypassed.4

A product seriously interested in helping with productivity would correct this glaringly obvious flaw, focus on the task at hand, and stop emitting useless verbiage.

In the previous two sections, I tried to focus on ways in which the harness would be constructed differently even if the LLM technology is fundamentally impossible to improve; in this case, I have to assume that the labs have some control over the model itself. But unless they are truly incapable of influencing their output (and all their "benchmarks" and "capabilities" seem to indicate that they can control it very tightly) they ought to be building models that are much less verbose.

More Non-Natural-Language User Interfaces

Although natural language could hypothetically be a powerful interface for interacting with a computer system, the practical upshot of LLM natural language interfaces is that these interfaces are imprecise and repetitive, full of superstitions masquerading as "best practices". The inputs are a mess and the resulting outputs are a mess.

The general way of addressing this unstructured mess is to allow the chatbot to directly take action in response to the user's input; in other words to supply it with "tools" via an MCP server. But again, this is backwards. If we cannot even express our intent clearly in the first place, why are we trusting this system to take potentially destructive and harmful actions on our behalf?

Instead, I would expect a product that was seriously invested in helping me accomplish specific tasks, to have user interfaces specific to those tasks. Is it supposed to be able to be a security scanner that can discover OWASP top 10 bugs in a codebase? Have a button for that. Build that functionality into your harness, train it directly into the model, use smaller models that can satisfy that functionality more effectively than throwing it at the planet-sized brain of Fable or whatever.

I'm aware that there are small software startups that do something like this, but they are bolted on to the side of the main model providers' APIs, not integrated into the core of the product and not using their own models and AI systems to achieve consistent and repeatable results.

Strong Data Provenance Indicators

Chatbots produce data tables pulled from websites, from APIs, from MCP tools or from summarizing and scrambling the user's input. In order to provide the illusion of a seamless interface, this data is presented in-line regardless of where it comes from. But some of these outputs are produced mechanically via regular old API calls, for example, from the result of calling a tool or querying a website, but presented uniformly.

But there is a huge difference between an authoritative data source being inlined as part of a chatbot conversation, being treated as input by the chatbot, and some ad-hoc hallucinated data being treated as output of the chatbot.

If a product is trying to help me make accurate, empirically-grounded, data-driven decisions, the source of the data is critical.

Integrated into the "check for mistakes" and "verify citations" workflow I described above, there's a necessary "verify data programmatically" pass as well; to have tools that will treat portions of the output as a regular spreadsheet, allowing regular-old computer arithmetic to verify things and showing where such arithmetic was used, and how.

Better User Control of Reproducibility

Anyone familiar with the technical specifics of LLMs will know that they have a variable called "temperature" which controls the degree of randomness that the LLM uses to produce its outputs. But most users don't know this, because it isn't exposed as part of the user interface by default.

This leads to a subjective impression that you asked ChatGPT, and you got ChatGPT's authoritative answer.

You can't just set the temperature to zero and still get useful results - I am aware that it does more than just scramble the output at random, and there are perhaps good reasons that simply exposing just a temperature setting would not be that useful to users. But if we followed some more of my earlier recommendations for making more structured UI elements to solve specific problems rather than having long back-and-forth chats where each refinement depends on the previous response, perhaps those elements could also re-play the process so that users can see how reliable the bot is at a particular task and develop a sense of how the stochastic nature of the process actually affects it.

Similarly, if a user is trying to solve the same problem repeatedly with a chatbot, and the chatbot product has numerous computational tools that don't really have anything to do with the LLM, such as deterministic data-processing tools, then having a way to freeze the non-deterministic parts of the transcript but re-populate a particular data frame with updated information and fork / continue the conversation from there would be a way to avoid introducing pointless additional randomness when you already know what tool you're trying to use.

The fact that every conversation is presented as this flat chat prompt that doesn't let me interact with any of the widgets that were previously produced except through more chatting, really makes me feel like the whole product is just doing predatory social-media style "increase time on site" optimization, just trying to lure me into further repetitive and unreliable chats, rather than letting me get in, solve my problem, and get out.

Context Visibility

Managing the LLM context is the ongoing challenge facing organizations that are trying to use "agentic" workflows. Filling up the context with too much information causes well-known problems. In response, advanced LLM users attempting to solve larger problems must break up very long prompts into "skills", give access to lengthy information via "tools", and delegating sub-problems to "sub-agents" rather than simply extending a single prompt indefinitely.

All of these strategies have flaws, because even on the largest models, compared to the breadth and depth of knowledge-work problems, LLM contexts are quite small.

And yet, none of these products will show the context to the user by default. There are third-party addons that can show you a simple progress bar but for addressing the premier engineering difficulty with this technology, that is below the bare minimum.

This lack of visibility means that almost all of the tools for extending the context are flying blind. Rather than responding meaningfully to a full context, everyone just kind of guesses how much state they need by guessing and trying over and over again with progressively more elaborate skill and sub-agent layouts. Even managing context compaction ends up being an advanced API-driven workflow5.

A serious product that was trying to help the user understand would not only show "available context" but explain the impact of context compactions, make it easier to see harness-generated prompts, and so on. This would be a first-class feature, combined with the aforementioned reproducibility / replay tools, would allow users to do real experiments to develop an understanding about how to make good use of the context window.

A Sandbox That Actually Works

I've been focused on the chatbot interface here because it is the most immediately egregious upon looking at the UI. But the "agentic loop" tools used for coding are equally dangerous, if not more so. Coding tools keep destroying everyone's data, over the course of years.

These catastrophic incidents that become front-page news are relatively rare compared to the amount of coding-agent use out there. But they also aren't the only kind of sandbox violation. Coding models will so routinely edit test code instead of the system under test that there are "pro tips" articles all over the web giving you the flawed advice to simply ask the agent not to cheat. News write-ups of the catastrophic incidents themselves will also offer glib and wrong advice, like "use a docker container". That might prevent it from literally deleting your operating system, but it won't prevent it from destroying all the local work you have in your codebase (it needs access to a checkout, after all!)

There is a flurry of activity in the infosec space where people are rushing to plug the gaps left by these coding harnesses. Everyone's got their own version of an MCP approval gateway where you can optionally place a proxy between your agent and your production infrastructure.

In the best case, though, all these mitigations and proxies and prompts simply turn the user into an auto-approval automaton, hitting Y, Y, Y, Y over and over again, until you finally are driven mad and hit "yes to all", turn on full-auto mode and submit yourself to the void. With nothing between your personal vigilance and disaster, there are no workflows left beyond decrementing your own vigilance until there's nothing left and then hoping the disaster never arrives.

The fact that some mitigations exist that can be deployed by extra-cautious users does not change the fact that "agentic coding" is an unsafe-by-default technology deployed without concern or guidance. Every frontier lab has tied a spring-loaded shotgun to a dog; the fact that dog owners can publish thoughtful blog posts explaining how you can teach your dog the basics of gun safety or how you can have your dogs play in a bullet-proof room does not mitigate the fact that the product should not have been allowed in the first place, nor should it continue to exist without VERY strong security controls.

I might believe that a frontier lab were seriously interested in providing developers with a useful tool if they shipped something that had safety built-in.

That means tools in the harness, detached from any LLM, independent of the prompt, that could:

In the same way that I suggested above that research-based tasks should have a way of re-issuing prompts to determine how reproducible a result is, or whether other sources might be found, agent-based tasks should have a way of being executed against mock services for popular APIs, so that the verification can match both on the front-end (review the plan for making the API calls before they're executed) and the back end (review the API calls that were issued to the mock service and verify that they matched).

Instead, the frontier labs provide us products that are disasters out of the box, give us "best practices" to build massive and elaborate, as well as incomplete and error-prone, security perimeters of our own design. Then they blame "operator error" when it inevitably goes wrong. I cannot believe that these design choices are intended to help us be productive.

Bonus: Human Processes

Organizations deploying AI also frequently come across as unserious, for similar reasons. In 2023, naive exuberance could perhaps be forgiven. But today, as we near the close of 2026, there are several well-known problems, that have been extremely well-covered in the press. None of these things should be surprising, but most orgs deploying these tools are still just letting them rip and hoping it all works out.

Organizations deploying these tools would need at least three kinds of major modifications to their internal processes, if they wanted to be serious about using them safely:

1. Shift Rotations to Prevent Vigilance Decrement

There have been several high-profile incidents where software developers' gradual acquiescence to accepting LLM output have lead to serious economic consequences for the companies deploying them, perhaps best typified by Amazon's "millions of lost orders" due to a gradual decay of their engineering processes from LLM use.

These outages, and other AI-related failures, are due to the difficulty of maintaining focus on the same problems. In other words, as I described above, vigilance decrement is a constant problem, because AI outputs are most often correct, but continue to be incorrect in surprising and non-intuitive ways. As I have previously written, you cannot trust yourself to catch every bug with code review, and LLM output.

Aviation, for example, has very strict rules around rest requirements. There is also a specific rule that "No certificate holder may operate an aircraft without a second in command if that aircraft has a passenger seating configuration, excluding any pilot seat, of ten seats or more.". Other safety-critical professions have similar rules.

And yet, even in the age of the supposed "AI revolution", most software teams are still assigning every engineer a full feature load, not planning for any rest, and telling people to review code whenever they happen to have some "free time".

Maintenance of vigilance has to be your top priority. Regular, scheduled, inviolable rest periods where people do work without AI assistance, and are not exposed to any AI output for review or otherwise, would be crucial in order to stay mentally sharp enough.

The tools themselves should have this sort of thing built in. The mistake-review process described above should have a periodic spot-check mode where a second reviewer periodically reviews a chatbot log, doing their own independent verification of claims, to see if they spot the same errors. This could provide a feedback loop to determine how much rest is necessary to maintain continuous attention and actually spot hallucinations.

2. Skill Practice To Prevent Skill Loss

It is also well-known that AI use leads to AI reliance, and AI reliance leads to skill loss.

I like to use the analogy to dockworkers at a seaport6 adopting automation.

If you employ dockworkers to load and unload ships all day long, they are going to be getting tons of exercise. They will be able to lift heavy objects on demand, whenever. They might have plenty of health problems and injuries from this type of work, but "lack of exercise" will not be a problem.

With the development of standardized container ships and mechanized cranes, you are going to be changing their job description substantially: now they mostly spend all day sitting in a small cubicle moving a control lever back and forth, not lifting heavy stuff. They will get worse at lifting heavy objects.

In this analogy however, the cranes are not all that reliable. We know they break, and they drop their payloads sometimes, and the stuff needs to be manually moved. But this only happens a few times a week, at most. If you need whoever is driving the crane to be able to jump out at any moment and still move stuff around manually, then you need to make an affordance for that. You need to give them time to go to the gym and do some lifting for practice, or every crane failure is going to be a major emergency.

An organization doing an AI transformation would also need a massive increase to learning & development budget, both in terms of resources and in terms of schedule. If your people are going to lose skills because they've lost regular practice in the incidental course of doing their duties, then they are going to need deliberate, intentional, non-incidental practice of those skills to keep them sharp.

But rather than trying to accommodate new workflows and give time for people to adjust, most AI mandates are simply dropped on workers like a ton of bricks, with no time to adapt and no affordance for maintaining their skills. Operate the crane and stay fit and healthy and ready to switch back to manual lifting at any time and then get back in the crane cockpit right afterwards. Don't mess up.

Then an accident happens and everyone is surprised, as if this process weren't practically designed to produce a terrible result.

3. Mental Health Resources to Deal with Mental Health Risks

AI psychosis often begins with practical problem-solving, and beyond that, it can start specifically at work. Not to mention the more pedestrian condition of "AI brain fry".

If you are mandating your employees to use a hazardous tool that may seriously and directly damage their mental health, you need trainings and resources. You need in-house therapists and you need to be making sure to check in with people actively to make sure that this is not happening.

Again, the tool itself ought to have some way of dealing with this. An occasional "take a break" popup is easily dismissed; they need a user-visible AI personal dosimeter so you can see your cumulative usage over time.

I don't even know if "usage over time" is a sufficient metric to gauge risk. Maybe if your work chatbot start to talk about resonance too much, unless you literally work as an acoustic engineer, that should be flagged for someone.

We are, again, years into dealing with these tools, and we know these risks exist. Yet no serious mitigations are provided. Not even any way of measuring the risk exposure.

And More

There are also many other risks associated with the technology. There are intellectual property risks with the foundation models, due to recklessness with their training data. There are existential financial risks associated with the infrastructure build-out. The extent to which most "open" models are simply derivatives of frontier models is an open question.

What I Think

If any one of these things were regularly overlooked by AI vendors or users, that would be a totally normal product oversight. Room for improvement for the next version, but nothing catastrophic.

Shipping without any of them doesn't seem like lean product management, it seems like a careless attitude towards risk and a product design philosophy oriented entirely towards short-term demos, with no regard for how to realize actual productivity gains.

Furthermore, being available for years without anything like these features, despite hundreds of incidents demonstrating the risks, with hundreds of billions of dollars of funding, makes it seem to me like if they were to add all the features that would make their product actually safe and hypothetically useful, these features would reveal that it is actually not an improvement to productivity.

In the year since I first wrote about measuring the cost/benefit ratio of AI, I have heard from numerous people who have shown this to management to try to illustrate why their AI initiatives - like almost all AI initiatives - were either failing or burning out their engineers.

I've also heard from lots of people that have told me that it's obviously useful and they don't need to measure so carefully, because they are getting lots of work done that they couldn't have otherwise.7

I have yet to hear from a single person who has said "yeah, we measured according to your methodology8, and it turns out that our AI work is going great and that our ratio is 0.75".

Obviously, I cannot say for sure why this is; absence of evidence is not evidence of absence. But at this point I think the null hypothesis is that AI tools provide, in aggregate, zero value. They make mistakes too often, and the externalities they produce are so bad and so difficult to control that even before we get to the places where they are just physically poisoning people, even the negative effects on their direct users end up cancelling out whatever benefit to they provide to their organizations.

If I were wrong, then including tools to measure an AI's effectiveness at the tasks their users are actually trying to accomplish, rather than meaningless benchmarks, would show big productivity gains. The frontier labs would be champing at the bit to add such features, and crowing about their fantastic results.

I think the labs know that if they did that, it would present a grim picture to their users. Such tools would let their users see that it's making mistakes much more often than they realized, that they're spending much more time with it than they want to be, and that it's just generally not fit for purpose.

If they prove me wrong by adding in all of these safety mechanisms, and in the process, they make all of their AI technology less harmful, I'll be thrilled to be debunked.


Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!


  1. It is also interesting that for the next section, sometimes it seems that Claude's disclaimer is "Please double-check cited sources." instead. ↩

  2. ... by which I mean the "prompter", since authorship is not what's happening here. ↩

  3. Claude has the "citations API", Google has various different kinds of "grounding" against its own APIs, and I guess Microsoft can check OpenAI's homework if you want. ↩

  4. Given the relatively slow speed of the justice system and the mainstream press around the world, we probably will not hear about whether people are managing to incidentally break through these guard rails to self harm right now, but there are no shortage of stories still being reported right now where people were still doing just that, such as in this story where the effect of the much vaunted "guard rails" in 2025 was that if you wanted it to write you a suicide note, it would refuse twice but acquiesce on the third try. I don't see any reason to believe this fundamental issue has been addressed in the meanwhile, since it had been happening for years at that point. ↩

  5. In this tutorial we can also see an incredibly rosy scenario presented, where a long-running workflow effortlessly compresses all of the necessary information into the new context, even if it uses a lower-fidelity model to do so, rather than the tangled and gnarly problem of problems which really are too big to fit in the context, which is to say, "most real-world problems". This presents the context limit instead as a minor speedbump to be worked around rather than the fundamental flaw in LLM tooling. ↩

  6. A heavily fictionalized seaport. This is not how actual dockworkers work. This is a simplistic metaphor about incidental benefits of instrumental tasks, it is not supposed to delve deeply into the mechanics of maritime shipping. In particular I know that cranes are more reliable than this and this is not actually how you would respond to a crane malfunction anyway. Feel free to share fun facts about maritime shipping if that is your special interest but please do not @ me to correct this metaphor. ↩

  7. To my knowledge, none of their publicly-traded employers have posted a measurable improvement to efficiency outside the margin of error. ↩

  8. Or any similar methodology. I don't need people to adopt the exact practice that I proposed there. ↩

28 Sep 2026 4:27am GMT