I was writing some code where I was using httpx.get() and its params parameter. I decided to use a TypedDict for the dictionary I was passing as the argument since it was for a REST API, where the potential keys were fully known. I then ran Pyrefly over my code and got an unexpected error about how "object" is not a subtype of "str". I had no object in my TypedDict, so I didn&apost understand what was going on. I tried Pyright and it also failed. I then tried ty and it passed! What?! I know ty takes a less strict approach to typing to support a more gradual approach, so I figured there was a strict typing thing I was doing wrong. I did some digging and I found out that a new feature of TypedDict solves the issue for me, and so I figured I would share what I learned.

Starting in Python 3.15 and typing-extensions today, there are two dimensions to TypedDict and how keys and their existence are treated. The first dimension is whether the specified keys in a TypedDict are all required or not (controlled by the total argument or Required and NotRequired on a per-key basis). This represents whether every key specified in your TypedDict must be in the dictionary or not. So if you have a TypedDict of:

class OptionalOpen(typing_extensions.TypedDict, total=False):
    spam: str

it means the "spam" key is optional. To make it required you just set total=True or spam: Required[str]:

class RequiredOpen(typing_extensions.TypedDict, total=True):
    spam: str

This concept has been around since Python 3.8 when TypedDict was introduced, with Required and NotRequired added in Python 3.11.

But starting in Python 3.15, a second dimension has been introduced that affects whether the TypedDict is closed. By default, a dictionary that is typed to a TypedDict can have any optional keys that it wants. So with either of our example TypedDict above, you could have any number of extra keys, each with any value. So what is a type checker to do if you reference some key that isn&apost defined by the TypedDict? Since the arbitrary keys are legal, you assume the "worst", and that the value for the key is object as that&aposs the base class of everything.

So, let&aposs say you have a function that takes a Mapping of str keys and str values:

def func(data: collections.abc.Mapping[str, str]) -> None:
    print(data["spam"])

It turns out that if you try to pass in a dictionary that is typed to either of our TypedDict examples you get a type failure like this (this is from Pyright):

/home/brett/py/typeddict_typing.py
  /home/brett/py/typeddict_typing.py:26:6 - error: Argument of type "OptionalOpen" cannot be assigned to parameter "data" of type "Mapping[str, str]" in function "func"
    "OptionalOpen" is not assignable to "Mapping[str, str]"
      Type parameter "_VT_co@Mapping" is covariant, but "object" is not a subtype of "str"
        "object" is not assignable to "str" (reportArgumentType)

This happens because Mapping[str, str] only accepts values of str, but with our TypedDict there is the possibility of some unspecified key having a value of object. As such, e.g. Pyright complains that you can&apost use an object where str is expected, since you can&apost substitute anything that inherits from object for a str (that&aposs what the variance bit is all about in that error message).

So how do you solve this? You say the TypedDict cannot have any keys that are not specified; it&aposs closed via the closed argument introduced in PEP 728 (currently, there are no docs for this in Python 3.15 even though it&aposs implemented):

class OptionalClosed(typing_extensions.TypedDict, total=False, closed=True):
    spam: str
    

With that argument you tell the type checkers that unless a key is specified in the TypedDict, the key isn&apost allowed to exist. That means our example TypedDict will only ever have keys that have a str value since we only have one possible key and its type is str. As such, that makes it a Mapping[str, str] since the only key it can ever have has a value type of str.

Another way to make this work is with the extra_items parameter that also came from PEP 728. What that parameter lets you do is specify the value type for any keys that are not defined by the TypedDict:

class RequiredOpen(typing_extensions.TypedDict, extra_items=str):
    spam: str

So now any dictionary that is typed to this TypedDict will be presumed to have str be the type for any keys that aren&apost spam. That then means our TypedDict supports the Mapping[str, str] type as the only defined key is str and we have said any other key will have a value type of str.

20 Nov 2025 9:18pm GMT

Here are some Django-related deals for this year's Black Friday (28th November) and Cyber Monday (1st December), including my own.

I'll keep this post up to date with any new deals I learn about. If you are also a creator, email me with details of your offer and I'll add it here.

My books

My four books have a 50% discount, for both individual and team licenses, until the end of Cyber Monday (1st December), including Boost Your GitHub DX, which I released last week. This deal stacks with the bundle offers and purchasing power parity discount for those in lower-income countries.

Individual books:

• Boost Your GitHub DX - $22 instead of $44
• Boost Your Git DX - $22 instead of $44
• Boost Your Django DX - $22 instead of $44
• Speed Up Your Django Tests - $24.50 instead of $49

Bundles:

• Boost Your Git* DX Bundle - the Git and GitHub "Boost Your DX" books, $39 instead of $78
• Boost Your DX Bundle - all three "Boost Your DX" books, $58.50 instead of $117

Aidas Bendoraitis' paid packages

Aidas Bendoraitis of djangotricks.com has created three paid packages. Use the links below for a 20% discount, available until the end of the 1st December.

• Django GDPR Cookie Consent - a customizable, self-hosted cookie consent screen. This package takes the pain out of setting up legally-mandated cookie banners and settings, without using an expensive or inflexible vendor.

  Buy it on Gumroad

• Django Paddle Subscriptions - an integration with Paddle's billing API for subscriptions. This package allows you to collect SaaS payments internationally with a reliable payment processor.

  Buy it on Gumroad

• Django Messaging - ready-to-use, real-time solution for Django that saves months of development. It offers private and group chats, embeddable widgets, flexible settings, a modern UI, and supports both WebSocket and polling.

  Buy it on Gumroad

Appliku

Appliku is a deployment tool designed for Django. It can deploy your project to AWS, DigitalOcean, Hetzner, and other cloud servers.

They're offering 30% off all annual plans, for one year, until the 1st December. Use code BLACKFRIDAY2025 at checkout.

Buy on the Appliku site

Async Patterns in Django

This book is a tour through the advanced topic of asynchronous programming in Django. It covers the range of tools and protocols available for asynchronous behaviour in your web application. It's written by Paul Bailey, an experienced Principal Engineer.

Paul is offering ~50% off the book with the coupon link, discounting the book from $39.95 to $21. This is available until the 1st December.

Buy on Leanpub

SaaS Pegasus

Cory Zue's SaaS Pegasus is a configurable Django project template with many preset niceties, including teams, Stripe subscriptions, a front end pipeline, and built-in AI tooling. It can massively accelerate setting up a SaaS in Django.

The "unlimited license" is discounted 50%, from $999 to $499.50. This deal is available from the 21st November until the 3rd December.

Buy it on saaspegasus.com

Bonus: Django itself

Django is maintained by the Django Software Foundation (DSF), a non-profit organization that relies on donations to fund its work. So while it cannot run sales, supporting it is definitely a good deal!

If your organization relies on Django, please push to sign it up as a corporate sponsor. For $1,000 or more, you can get your logo featured on the Fundraising page, and you can feature a badge on your own site.

Otherwise, you can also donate as an individual on:

• GitHub Sponsors - adds to your GitHub bill.
• Threadless - grab some custom-printed merchandise.
• The Fundraising Page - bills from a credit card with Stripe.

Your money will go towards:

• Paying the Django Fellows, Natalia, Sarah, and Jacob, who merge respond to tickets, merge code, and make releases.
• Helping organize DjangoCons in Africa, America, and Europe, and other events.
• Hosting the documentation, source code, ticketing system, and CI system.
• A very long tail of activities that keep the framework alive and thriving.

At the time of writing, Django is only 59% towards its annual funding goal of $300,000:

Let's fill up that heart! 💚💚💚

20 Nov 2025 6:00am GMT

On November 16th 2005, Django co-creator Adrian Holovaty announced the first ever Django release, Django 0.90. Twenty years later, today here we are shipping the first release candidate of Django 6.0 🚀.

Since we're celebrating Django's 20th birthday this year, here are a few release-related numbers that represent Django's history:

• 447 releases over 20 years. That's about 22 per year on average. We're at 38 so far for 2025. Fun fact: 33 of those releases predate PyPI, and were published via the Django website only!
• 131 security vulnerabilities addressed in those Django releases. Our security issues archive is a testament to our stellar track-record.
• 262,203 releases of Django-related packages. Django's community ecosystem is gigantic. There's tens of releases of Django packages per day as of 2025. There were 52 just today. With the caveat this depends a lot on what you classify as a "Django" package.

This is what decades' worth of a stable framework looks like. Expect more gradual improvements and bug fixes over the next twenty years' worth of releases. And if you like this kind of data, check out the State of Django 2025 report by JetBrains, with lots of statistics on our ecosystem (and there's a few hours left on their Get PyCharm Pro with 30 % Off & Support Django offer).

Support Django

If you or your employer counts on Django's 20 years of stability, consider whether you can support the project via donations to our non-profit Django Software Foundation.

• ⚠️ today only - Get PyCharm Pro for 30% off - all the revenue goes to our Foundation.
• Donate on the Django website
• Donate on GitHub sponsors
• Check out how to become a Corporate Member

Once you've done it, post with #DjangoBirthday and tag us on Mastodon / on Bluesky / on X / on LinkedIn so we can say thank you!

59%

Of our US $300,000.00 goal for 2025, as of November 19th, 2025, we are at:

• 58.7% funded
• $176,098.60 donated

Donate to support Django

19 Nov 2025 3:27pm GMT

Building an MCP client in Python can be a good option when you're coding MCP servers and want a quick way to test them. In this step-by-step project, you'll build a minimal MCP client for the command line. It'll be able to connect to an MCP server through the standard input/output (stdio) transport, list the server's capabilities, and use the server's tools to feed an AI-powered chat.

By the end of this tutorial, you'll understand that:

• You can build an MCP client app for the command line using the MCP Python SDK and argparse.
• You can list a server's capabilities by calling .list_tools(), .list_prompts(), and .list_resources() on a ClientSession instance.
• You can use the OpenAI Python SDK to integrate MCP tool responses into an AI-powered chat session.

Next, you'll move through setup, client implementation, capability discovery, chat handling, and packaging to test MCP servers from your terminal.

Prerequisites

To get the most out of this coding project, you should have some previous knowledge of how to manage a Python project with uv. You should also know the basics of working with the asyncio and argparse libraries from the standard library.

To satisfy these knowledge requirements, you can take a look at the following resources:

• Managing Python Projects With uv: An All-in-One Solution
• Python's asyncio: A Hands-On Walkthrough
• Build Command-Line Interfaces With Python's argparse

Familiarity with OpenAI's Python API, openai, will also be helpful because you'll use this library to power the chat functionality of your MCP client. You'll also use the Model Context Protocol (MCP) Python SDK.

Don't worry if you don't have all of the prerequisite knowledge before starting this tutorial-that's completely okay! You'll learn through the process of getting your hands dirty as you build the project. If you get stuck, then take some time to review the resources linked above. Then, get back to the code.

You'll also need an MCP server to try your client as you build it. Don't worry if you don't have one available-you can use the server provided in step 2.

In this tutorial, you won't get into the details of creating MCP servers. To learn more about this topic, check out the Python MCP Server: Connect LLMs to Your Data tutorial. Finally, you can download the project's source code and related files by clicking the link below.

Get Your Code: Click here to download the free sample code you'll use to build a Python MCP client to test servers from your terminal.

Take the Quiz: Test your knowledge with our interactive "Build a Python MCP Client to Test Servers From Your Terminal" quiz. You'll receive a score upon completion to help you track your learning progress:

---

Interactive Quiz

Build a Python MCP Client to Test Servers From Your Terminal

Learn how to create a Python MCP client, start an AI-powered chat session, and run it from the command line. Check your understanding.

Step 1: Set Up the Project and the Environment

To manage your MCP client project, you'll use uv, a command-line tool for Python project management. If you don't have this tool on your current system, then it's worth checking out the Managing Python Projects With uv: An All-in-One Solution tutorial.

Note: If you prefer not to use uv, then you can use a combination of alternative tools such as pyenv, venv, pip, or poetry.

Once you have uv or another tool set up, go ahead and open a terminal window. Then, move to a directory where you typically store your projects. From there, run the following commands to scaffold and initialize a new mcp-client/ project:

Shell

$ uv init mcp-client
$ cd mcp-client/
$ uv add mcp openai

The first command creates a new Python project in an mcp-client/ directory. The resulting directory will have the following structure:

mcp-client/
├── .git/
├── .gitignore
├── .python-version
├── README.md
├── main.py
└── pyproject.toml

First, you have the .git/ directory and the .gitignore file, which will help you version-control the project.

The .python-version file contains the default Python version for the current project. This file tells uv which Python version to use when creating a dedicated virtual environment for the project. This file will contain the version number of the Python interpreter you're currently using.

Next, you have an empty README.md file that you can use to provide basic documentation for your project. The main.py file provides a Python script that you can optionally use as the project's entry point. You won't use this file in this tutorial, so feel free to remove it.

Finally, you have the pyproject.toml file, which you'll use to prepare your project for building and distribution.

Read the full article at https://realpython.com/python-mcp-client/ »

[ Improve Your Python With 🐍 Python Tricks 💌 - Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

19 Nov 2025 2:00pm GMT

ModuleNotFoundError: No module named 'ansible.parsing.yaml.constructor'

[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/astral-sh/ruff-pre-commit.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/ansible-community/ansible-lint.git.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command:
  ('/opt/hostedtoolcache/Python/3.12.12/x64/bin/python', '-mvirtualenv',
  '/home/runner/.cache/pre-commit/repomm4m0yuo/py_env-python3.13', '-p', 'python3.13')
return code: 1
stdout:
    RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.13'
stderr: (none)
Check the log at /home/runner/.cache/pre-commit/pre-commit.log
Error: Process completed with exit code 3.

- repo: https://github.com/ansible-community/ansible-lint.git
  hooks:
    - id: ansible-lint
      language_version: python3  # or python3.12 or so

19 Nov 2025 5:00am GMT

After more than 20 years of building for the web, this topic somehow kept slipping past me. It always felt obvious, so I never looked deeper. Recently I finally took the time to explore it properly, did some quick research, and now I'm sharing the results. Here's a simple walkthrough of the different content types you can send in POST requests.

Standard Form Data

When you submit a basic HTML form like <form method="post" action=""></form>, for example a login form, the browser sends the data using the application/x-www-form-urlencoded content type. The body of the request looks like a URL-encoded query string, the same format typically used in GET requests.

Example: username=john_doe&password=pass123.

A POST request with this content type using the fetch API looks like this:

async function sendURLEncoded() {
    const params = new URLSearchParams();
    params.append('username', 'john_doe');
    params.append('email', 'john@example.com');
    params.append('password', 'Secret123');
    params.append('bio', 'This is a multi-line\nbio text.\nIt supports newlines!');

    const response = await fetch('/api/urlencoded/', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/x-www-form-urlencoded',
            'X-CSRFToken': getCSRFToken()
        },
        body: params
    });
    const result = await response.json();
    console.log(result);
}

On the Django side, validation with a Django form is the usual approach:

from django import forms
from django.http import JsonResponse
from django.views.decorators.http import require_http_methods
from .forms import URLEncodedForm


class URLEncodedForm(forms.Form):
    username = forms.CharField(
        max_length=150,
        required=True,
        min_length=3,
    )
    email = forms.EmailField(
        required=True,
        error_messages={
            "required": "Email is required",
            "invalid": "Please enter a valid email address",
        },
    )
    password = forms.CharField(
        required=True,
        min_length=8,
    )
    bio = forms.CharField(
        required=False,
        max_length=500,
        widget=forms.Textarea,
        error_messages={"max_length": "Bio cannot exceed 500 characters"},
    )


@require_http_methods(["POST"])
def handle_urlencoded(request):
    """Handle application/x-www-form-urlencoded requests"""
    form = URLEncodedForm(request.POST)

    if not form.is_valid():
        return JsonResponse(
            {
                "status": "error",
                "errors": form.errors,
                "content_type": request.content_type,
            },
            status=400,
        )

    return JsonResponse(
        {
            "status": "success",
            "form_data": form.cleaned_data,
            "content_type": request.content_type,
        }
    )

Form Data with Files

If your form includes file uploads, the browser switches to multipart/form-data. You enable it in HTML like this:

<form method="post" action="" enctype="multipart/form-data">

A fetch-based upload looks like:

async function sendMultipart() {
    const formData = new FormData();
    formData.append('username', 'john_doe');
    formData.append('email', 'john@example.com');

    const fileInput = document.getElementById('file-input');
    if (fileInput.files[0]) {
        formData.append('avatar', fileInput.files[0]);
    }

    const response = await fetch('/api/multipart/', {
        method: 'POST',
        headers: {
            'X-CSRFToken': getCSRFToken()
        },
        body: formData
    });
    const result = await response.json();
    console.log(result);
}

The Django view works the same way, just with request.FILES added.

from django import forms
from django.http import JsonResponse
from django.views.decorators.http import require_http_methods


class MultipartForm(forms.Form):
    username = forms.CharField(
        max_length=150,
        required=True,
        min_length=3,
    )
    email = forms.EmailField(
        required=True,
    )
    avatar = forms.FileField(
        required=False,
    )

    def clean_avatar(self):
        avatar = self.cleaned_data.get("avatar")
        if avatar:
            if avatar.size > 5 * 1024 * 1024:
                raise forms.ValidationError("File size cannot exceed 5MB")

            allowed_types = ["image/jpeg", "image/png", "image/gif", "image/webp"]
            if avatar.content_type not in allowed_types:
                raise forms.ValidationError(
                    f'Invalid file type. Allowed types: {", ".join(allowed_types)}'
                )
        return avatar



@require_http_methods(["POST"])
def handle_multipart(request):
    """Handle multipart/form-data requests (forms with files)"""
    form = MultipartForm(request.POST, request.FILES)

    if not form.is_valid():
        return JsonResponse(
            {
                "status": "error",
                "errors": form.errors,
                "content_type": request.content_type,
            },
            status=400,
        )

    form_data = {}

    for key, value in form.cleaned_data.items():
        if hasattr(value, "read"):
            form_data[key] = {
                "name": value.name,
                "size": value.size,
                "content_type": value.content_type,
            }
        else:
            form_data[key] = value

    return JsonResponse(
        {
            "status": "success",
            "form_data": form_data,
            "content_type": request.content_type,
        }
    )        

JSON String

A very common modern content type is application/json. Single-Page Applications and most JavaScript-heavy frontends rely on it.

Frontend example:

async function sendJSON() {
    const data = { 
        name: "John Doe", 
        email: "john@example.com", 
        age: 30
    };
    const response = await fetch('/api/json/', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'X-CSRFToken': getCSRFToken()
        },
        body: JSON.stringify(data)
    });
    const result = await response.json();
    console.log(result)
}

For validation, you can use Pydantic, which is a nice alternative to Django-REST-Framework serializers.

from django.http import JsonResponse
from django.views.decorators.http import require_http_methods
from pydantic import BaseModel, EmailStr, Field, ConfigDict, ValidationError


class JSONDataSchema(BaseModel):
    model_config = ConfigDict(
        json_schema_extra={
            "example": {
                "name": "John Doe", 
                "email": "john@example.com", 
                "age": 30,
            }
        }
    )

    name: str = Field(min_length=2, max_length=100, description="User's full name")
    email: EmailStr = Field(description="Valid email address")
    age: int = Field(ge=0, le=150, description="User's age")


@require_http_methods(["POST"])
def handle_json(request):
    """Handle application/json requests"""
    try:
        data = json.loads(request.body)
        validated = JSONDataSchema(**data)

        return JsonResponse(
            {
                "status": "success",
                "received": validated.model_dump(),
                "content_type": request.content_type,
            }
        )
    except json.JSONDecodeError:
        return JsonResponse(
            {
                "status": "error",
                "error": "Invalid JSON",
                "content_type": request.content_type,
            },
            status=400,
        )
    except ValidationError as e:
        return JsonResponse(
            {
                "status": "error",
                "errors": e.errors(),
                "content_type": request.content_type,
            },
            status=400,
        )

Newline-Delimited JSON

application/x-ndjson (or simply NDJSON) is an experimental but handy format where each line is a separate JSON object. It's useful for bulk imports - logs, analytics, and other large datasets.

async function sendNDJSON() {
    const ndjsonData = (
`{"name":"John","age":30}
{"name":"Jane","age":25}
{"name":"Bob","age":35}`
);
    const response = await fetch('/api/ndjson/', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/x-ndjson',
            'X-CSRFToken': getCSRFToken()
        },
        body: ndjsonData
    });
    const result = await response.json();
    console.log(result);
}

The processing logic is standard: split, parse, validate.

import json
from django.http import JsonResponse
from django.views.decorators.http import require_http_methods


@require_http_methods(["POST"])
def handle_ndjson(request):
    """Handle application/x-ndjson requests"""
    try:
        ndjson_content = request.body.decode("utf-8")
        lines = [
            json.loads(line) for line in ndjson_content.strip().split("\n") if line
        ]

        return JsonResponse(
            {
                "status": "success",
                "lines_count": len(lines),
                "data": lines,
                "content_type": request.content_type,
            }
        )
    except json.JSONDecodeError:
        return JsonResponse({"error": "Invalid NDJSON"}, status=400)

Plain Text

Some systems accept plain text via text/plain, especially object storage services or endpoints meant for raw logs or unstructured content.

async function sendTextPlain() {
    const textData = `This is plain text content.
It can span multiple lines.
Line 3 of the content.`;

    const response = await fetch('/api/text/', {
        method: 'POST',
        headers: {
            'Content-Type': 'text/plain',
            'X-CSRFToken': getCSRFToken()
        },
        body: textData
    });
    const result = await response.json();
    console.log(result);
}

The Django view is minimal and functional:

from django.http import JsonResponse
from django.views.decorators.http import require_http_methods


@require_http_methods(["POST"])
def handle_text_plain(request):
    """Handle text/plain requests"""
    text_content = request.body.decode("utf-8")
    return JsonResponse(
        {
            "status": "success",
            "received": text_content,
            "length": len(text_content),
            "content_type": request.content_type,
        }
    )

HTML Text

You can also POST HTML content using text/html, which can be useful for wiki-style pages or HTML editors that save full documents.

async function sendHTML() {
    const htmlData = `<!DOCTYPE html>
<html>
<head><title>Example</title></head>
<body><h1>Hello World!</h1></body>
</html>`;
    const response = await fetch('/api/html/', {
        method: 'POST',
        headers: {
            'Content-Type': 'text/html',
            'X-CSRFToken': getCSRFToken()
        },
        body: htmlData
    });
    const result = await response.json();
    console.log(result);
}

The Django view would accept the data like this (add your own validation):

from django.http import JsonResponse
from django.views.decorators.http import require_http_methods


@require_http_methods(["POST"])
def handle_html(request):
    """Handle text/html requests"""
    html_content = request.body.decode("utf-8")
    return JsonResponse(
        {
            "status": "success",
            "received": html_content,
            "length": len(html_content),
            "content_type": request.content_type,
        }
    )

XML Data

Older or legacy integrations (especially SOAP-based ones) still rely on application/xml.

Here's a JavaScript example:

async function sendXML() {
    const xmlData = `<?xml version="1.0" encoding="UTF-8"?>
<user>
<name>John Doe</name>
<email>john@example.com</email>
<age>30</age>
</user>`;

    const response = await fetch('/api/xml/', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/xml',
            'X-CSRFToken': getCSRFToken()
        },
        body: xmlData
    });
    const result = await response.json();
    console.log(result);
}

And the Django view can parse the XML data into a dictionary like this:

import xml.etree.ElementTree as ET
from django.http import JsonResponse
from django.views.decorators.http import require_http_methods


@require_http_methods(["POST"])
def handle_xml(request):
    """Handle application/xml requests"""
    try:
        xml_content = request.body.decode("utf-8")
        root = ET.fromstring(xml_content)

        data = {
            child.tag: child.text 
            for child in root
        }

        return JsonResponse(
            {
                "status": "success",
                "root_tag": root.tag,
                "data": data,
                "content_type": request.content_type,
            }
        )
    except ET.ParseError:
        return JsonResponse({"error": "Invalid XML"}, status=400)

SVG Image

SVG graphics can be sent as image/svg+xml, since they are XML-based.

async function sendSVG() {
    const svgData = `<svg xmlns="http://www.w3.org/2000/svg" width="100" height="100">
<circle cx="50" cy="50" r="40" stroke="black" stroke-width="3" fill="red" />
</svg>`;
    const response = await fetch('/api/svg/', {
        method: 'POST',
        headers: {
            'Content-Type': 'image/svg+xml',
            'X-CSRFToken': getCSRFToken()
        },
        body: svgData
    });
    const result = await response.json();
    console.log(result);
}

The Django view (add your own validation):

from django.http import JsonResponse
from django.views.decorators.http import require_http_methods


@require_http_methods(["POST"])
def handle_svg(request):
    """Handle image/svg+xml requests"""
    svg_content = request.body.decode("utf-8")
    return JsonResponse(
        {
            "status": "success",
            "received": svg_content,
            "length": len(svg_content),
            "content_type": request.content_type,
        }
    )

Binary Data

For raw binary streams - images, audio, video, PDF, or any unknown byte sequence - the fallback is application/octet-stream.

Here is a JavaScript example of how to post it:

async function sendBinary() {
    const binaryData = new Uint8Array([72, 101, 108, 108, 111]);
    const response = await fetch('/api/binary/', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/octet-stream',
            'X-CSRFToken': getCSRFToken()
        },
        body: binaryData
    });
    const result = await response.json();
    console.log(result);
}

The Django view would be the most straightforward, because request.body is already coming as bytestring:

from django.http import JsonResponse
from django.views.decorators.http import require_http_methods


@require_http_methods(["POST"])
def handle_binary(request):
    """Handle application/octet-stream requests (raw binary data)"""
    binary_data = request.body
    return JsonResponse(
        {
            "status": "success",
            "size": len(binary_data),
            "content_type": request.content_type,
            "first_bytes": list(binary_data[:10]),
        }
    )

Protocol Buffers

Protocol Buffers use application/x-protobuf, and compared to JSON they're usually around 50% smaller and faster to parse. I won't cover the code in this article, but do your research if you need speed.

Code to Play Around with

The rough distribution of content-type usage in real-world APIs is this:

• JSON: ~85-90%
• Multipart: ~10-15%
• Everything else: <5%

If you want to experiment with all the content types, and see the code in context, here's a repo as a great playground: https://github.com/archatas/django-post-content-types

16 Nov 2025 6:00pm GMT

Tell me if you've heard this one before.

You're working on an application. Let's call it "FooApp". FooApp has a dependency on an open source library, let's call it "LibBar". You find a bug in LibBar that affects FooApp.

To envisage the best possible version of this scenario, let's say you actively like LibBar, both technically and socially. You've contributed to it in the past. But this bug is causing production issues in FooApp today, and LibBar's release schedule is quarterly. FooApp is your job; LibBar is (at best) your hobby. Blocking on the full upstream contribution cycle and waiting for a release is an absolute non-starter.

What do you do?

There are a few common reactions to this type of scenario, all of which are bad options.

I will enumerate them specifically here, because I suspect that some of them may resonate with many readers:

1. Find an alternative to LibBar, and switch to it.

   This is a bad idea because a transition to a core infrastructure component could be extremely expensive.

2. Vendor LibBar into your codebase and fix your vendored version.

   This is a bad idea because carrying this one fix now requires you to maintain all the tooling associated with a monorepo¹: you have to be able to start pulling in new versions from LibBar regularly, reconcile your changes even though you now have a separate version history on your imported version, and so on.

3. Monkey-patch LibBar to include your fix.

   This is a bad idea because you are now extremely tightly coupled to a specific version of LibBar. By modifying LibBar internally like this, you're inherently violating its compatibility contract, in a way which is going to be extremely difficult to test. You can test this change, of course, but as LibBar changes, you will need to replicate any relevant portions of its test suite (which may be its entire test suite) in FooApp. Lots of potential duplication of effort there.

4. Implement a workaround in your own code, rather than fixing it.

   This is a bad idea because you are distorting the responsibility for correct behavior. LibBar is supposed to do LibBar's job, and unless you have a full wrapper for it in your own codebase, other engineers (including "yourself, personally") might later forget to go through the alternate, workaround codepath, and invoke the buggy LibBar behavior again in some new place.

5. Implement the fix upstream in LibBar anyway, because that's the Right Thing To Do, and burn credibility with management while you anxiously wait for a release with the bug in production.

   This is a bad idea because you are betraying your users - by allowing the buggy behavior to persist - for the workflow convenience of your dependency providers. Your users are probably giving you money, and trusting you with their data. This means you have both ethical and economic obligations to consider their interests.

   As much as it's nice to participate in the open source community and take on an appropriate level of burden to maintain the commons, this cannot sustainably be at the explicit expense of the population you serve directly.

   Even if we only care about the open source maintainers here, there's still a problem: as you are likely to come under immediate pressure to ship your changes, you will inevitably relay at least a bit of that stress to the maintainers. Even if you try to be exceedingly polite, the maintainers will know that you are coming under fire for not having shipped the fix yet, and are likely to feel an even greater burden of obligation to ship your code fast.

   Much as it's good to contribute the fix, it's not great to put this on the maintainers.

The respective incentive structures of software development - specifically, of corporate application development and open source infrastructure development - make options 1-4 very common.

On the corporate / application side, these issues are:

• it's difficult for corporate developers to get clearance to spend even small amounts of their work hours on upstream open source projects, but clearance to spend time on the project they actually work on is implicit. If it takes 3 hours of wrangling with Legal² and 3 hours of implementation work to fix the issue in LibBar, but 0 hours of wrangling with Legal and 40 hours of implementation work in FooApp, a FooApp developer will often perceive it as "easier" to fix the issue downstream.

• it's difficult for corporate developers to get clearance from management to spend even small amounts of money sponsoring upstream reviewers, so even if they can find the time to contribute the fix, chances are high that it will remain stuck in review unless they are personally well-integrated members of the LibBar development team already.

• even assuming there's zero pressure whatsoever to avoid open sourcing the upstream changes, there's still the fact inherent to any development team that FooApp's developers will be more familiar with FooApp's codebase and development processes than they are with LibBar's. It's just easier to work there, even if all other things are equal.

• systems for tracking risk from open source dependencies often lack visibility into vendoring, particularly if you're doing a hybrid approach and only vendoring a few things to address work in progress, rather than a comprehensive and disciplined approach to a monorepo. If you fully absorb a vendored dependency and then modify it, Dependabot isn't going to tell you that a new version is available any more, because it won't be present in your dependency list. Organizationally this is bad of course but from the perspective of an individual developer this manifests mostly as fewer annoying emails.

But there are problems on the open source side as well. Those problems are all derived from one big issue: because we're often working with relatively small sums of money, it's hard for upstream open source developers to consume either money or patches from application developers. It's nice to say that you should contribute money to your dependencies, and you absolutely should, but the cost-benefit function is discontinuous. Before a project reaches the fiscal threshold where it can be at least one person's full-time job to worry about this stuff, there's often no-one responsible in the first place. Developers will therefore gravitate to the issues that are either fun, or relevant to their own job.

These mutually-reinforcing incentive structures are a big reason that users of open source infrastructure, even teams who work at corporate users with zillions of dollars, don't reliably contribute back.

The Answer We Want

All those options are bad. If we had a good option, what would it look like?

It is both practically necessary³ and morally required⁴ for you to have a way to temporarily rely on a modified version of an open source dependency, without permanently diverging.

Below, I will describe a desirable abstract workflow for achieving this goal.

Step 0: Report the Problem

Before you get started with any of these other steps, write up a clear description of the problem and report it to the project as an issue; specifically, in contrast to writing it up as a pull request. Describe the problem before submitting a solution.

You may not be able to wait for a volunteer-run open source project to respond to your request, but you should at least tell the project what you're planning on doing.

If you don't hear back from them at all, you will have at least made sure to comprehensively describe your issue and strategy beforehand, which will provide some clarity and focus to your changes.

If you do hear back from them, in the worst case scenario, you may discover that a hard fork will be necessary because they don't consider your issue valid, but even that information will save you time, if you know it before you get started. In the best case, you may get a reply from the project telling you that you've misunderstood its functionality and that there is already a configuration parameter or usage pattern that will resolve your problems with no new code. But in all cases, you will benefit from early coordination on what needs fixing before you get to how to fix it.

Step 1: Source Code and CI Setup

Fork the source code for your upstream dependency to a writable location where it can live at least for the duration of this one bug-fix, and possibly for the duration of your application's use of the dependency. After all, you might want to fix more than one bug in LibBar.

You want to have a place where you can put your edits, that will be version controlled and code reviewed according to your normal development process. This probably means you'll need to have your own main branch that diverges from your upstream's main branch.

Remember: you're going to need to deploy this to your production, so testing gates that your upstream only applies to final releases of LibBar will need to be applied to every commit here.

Depending on your LibBar's own development process, this may result in slightly unusual configurations where, for example, your fixes are written against the last LibBar release tag, rather than its current⁵ main; if the project has a branch-freshness requirement, you might need two branches, one for your upstream PR (based on main) and one for your own use (based on the release branch with your changes).

Ideally for projects with really good CI and a strong "keep main release-ready at all times" policy, you can deploy straight from a development branch, but it's good to take a moment to consider this before you get started. It's usually easier to rebase changes from an older HEAD onto a newer one than it is to go backwards.

Speaking of CI, you will want to have your own CI system. The fact that GitHub Actions has become a de-facto lingua franca of continuous integration means that this step may be quite simple, and your forked repo can just run its own instance.

Optional Bonus Step 1a: Artifact Management

If you have an in-house artifact repository, you should set that up for your dependency too, and upload your own build artifacts to it. You can often treat your modified dependency as an extension of your own source tree and install from a GitHub URL, but if you've already gone to the trouble of having an in-house package repository, you can pretend you've taken over maintenance of the upstream package temporarily (which you kind of have) and leverage those workflows for caching and build-time savings as you would with any other internal repo.

Step 2: Do The Fix

Now that you've got somewhere to edit LibBar's code, you will want to actually fix the bug.

Step 2a: Local Filesystem Setup

Before you have a production version on your own deployed branch, you'll want to test locally, which means having both repositories in a single integrated development environment.

At this point, you will want to have a local filesystem reference to your LibBar dependency, so that you can make real-time edits, without going through a slow cycle of pushing to a branch in your LibBar fork, pushing to a FooApp branch, and waiting for all of CI to run on both.

This is useful in both directions: as you prepare the FooApp branch that makes any necessary updates on that end, you'll want to make sure that FooApp can exercise the LibBar fix in any integration tests. As you work on the LibBar fix itself, you'll also want to be able to use FooApp to exercise the code and see if you've missed anything - and this, you wouldn't get in CI, since LibBar can't depend on FooApp itself.

In short, you want to be able to treat both projects as an integrated development environment, with support from your usual testing and debugging tools, just as much as you want your deployment output to be an integrated artifact.

Step 2b: Branch Setup for PR

However, for continuous integration to work, you will also need to have a remote resource reference of some kind from FooApp's branch to LibBar. You will need 2 pull requests: the first to land your LibBar changes to your internal LibBar fork and make sure it's passing its own tests, and then a second PR to switch your LibBar dependency from the public repository to your internal fork.

At this step it is very important to ensure that there is an issue filed on your own internal backlog to drop your LibBar fork. You do not want to lose track of this work; it is technical debt that must be addressed.

Until it's addressed, automated tools like Dependabot will not be able to apply security updates to LibBar for you; you're going to need to manually integrate every upstream change. This type of work is itself very easy to drop or lose track of, so you might just end up stuck on a vulnerable version.

Step 3: Deploy Internally

Now that you're confident that the fix will work, and that your temporarily-internally-maintained version of LibBar isn't going to break anything on your site, it's time to deploy.

Some deployment heritage should help to provide some evidence that your fix is ready to land in LibBar, but at the next step, please remember that your production environment isn't necessarily emblematic of that of all LibBar users.

Step 4: Propose Externally

You've got the fix, you've tested the fix, you've got the fix in your own production, you've told upstream you want to send them some changes. Now, it's time to make the pull request.

You're likely going to get some feedback on the PR, even if you think it's already ready to go; as I said, despite having been proven in your production environment, you may get feedback about additional concerns from other users that you'll need to address before LibBar's maintainers can land it.

As you process the feedback, make sure that each new iteration of your branch gets re-deployed to your own production. It would be a huge bummer to go through all this trouble, and then end up unable to deploy the next publicly released version of LibBar within FooApp because you forgot to test that your responses to feedback still worked on your own environment.

Step 4a: Hurry Up And Wait

If you're lucky, upstream will land your changes to LibBar. But, there's still no release version available. Here, you'll have to stay in a holding pattern until upstream can finalize the release on their end.

Depending on some particulars, it might make sense at this point to archive your internal LibBar repository and move your pinned release version to a git hash of the LibBar version where your fix landed, in their repository.

Before you do this, check in with the LibBar core team and make sure that they understand that's what you're doing and they don't have any wacky workflows which may involve rebasing or eliding that commit as part of their release process.

Step 5: Unwind Everything

Finally, you eventually want to stop carrying any patches and move back to an official released version that integrates your fix.

You want to do this because this is what the upstream will expect when you are reporting bugs. Part of the benefit of using open source is benefiting from the collective work to do bug-fixes and such, so you don't want to be stuck off on a pinned git hash that the developers do not support for anyone else.

As I said in step 2b⁶, make sure to maintain a tracking task for doing this work, because leaving this sort of relatively easy-to-clean-up technical debt lying around is something that can potentially create a lot of aggravation for no particular benefit. Make sure to put your internal LibBar repository into an appropriate state at this point as well.

Up Next

This is part 1 of a 2-part series. In part 2, I will explore in depth how to execute this workflow specifically for Python packages, using some popular tools. I'll discuss my own workflow, standards like PEP 517 and pyproject.toml, and of course, by the popular demand that I just know will come, uv.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!

11 Nov 2025 1:44am GMT

15 Oct 2025 3:44pm GMT

15 Oct 2025 3:41pm GMT

15 Oct 2025 3:39pm GMT

The most optimistic vision of generative AI¹ is that it will relieve us of the tedious, repetitive elements of knowledge work so that we can get to work on the really interesting problems that such tedium stands in the way of. Even if you fully believe in this vision, it's hard to deny that today, some tedium is associated with the process of using generative AI itself.

Generative AI also isn't free, and so, as responsible consumers, we need to ask: is it worth it? What's the ROI of genAI, and how can we tell? In this post, I'd like to explore a logical framework for evaluating genAI expenditures, to determine if your organization is getting its money's worth.

Perpetually Proffering Permuted Prompts

I think most LLM users would agree with me that a typical workflow with an LLM rarely involves prompting it only one time and getting a perfectly useful answer that solves the whole problem.

Generative AI best practices, even from the most optimistic vendors all suggest that you should continuously evaluate everything. ChatGPT, which is really the only genAI product with significantly scaled adoption, still says at the bottom of every interaction:

ChatGPT can make mistakes. Check important info.

If we have to "check important info" on every interaction, it stands to reason that even if we think it's useful, some of those checks will find an error. Again, if we think it's useful, presumably the next thing to do is to perturb our prompt somehow, and issue it again, in the hopes that the next invocation will, by dint of either:

1. better luck this time with the stochastic aspect of the inference process,
2. enhanced application of our skill to engineer a better prompt based on the deficiencies of the current inference, or
3. better performance of the model by populating additional context in subsequent chained prompts.

Unfortunately, given the relative lack of reliable methods to re-generate the prompt and receive a better answer², checking the output and re-prompting the model can feel like just kinda futzing around with it. You try, you get a wrong answer, you try a few more times, eventually you get the right answer that you wanted in the first place. It's a somewhat unsatisfying process, but if you get the right answer eventually, it does feel like progress, and you didn't need to use up another human's time.

In fact, the hottest buzzword of the last hype cycle is "agentic". While I have my own feelings about this particular word³, its current practical definition is "a generative AI system which automates the process of re-prompting itself, by having a deterministic program evaluate its outputs for correctness".

A better term for an "agentic" system would be a "self-futzing system".

However, the ability to automate some level of checking and re-prompting does not mean that you can fully delegate tasks to an agentic tool, either. It is, plainly put, not safe. If you leave the AI on its own, you will get terrible results that will at best make for a funny story⁴⁵ and at worst might end up causing serious damage⁶⁷.

Taken together, this all means that for any consequential task that you want to accomplish with genAI, you need an expert human in the loop. The human must be capable of independently doing the job that the genAI system is being asked to accomplish.

When the genAI guesses correctly and produces usable output, some of the human's time will be saved. When the genAI guesses wrong and produces hallucinatory gibberish or even "correct" output that nevertheless fails to account for some unstated but necessary property such as security or scale, some of the human's time will be wasted evaluating it and re-trying it.

Income from Investment in Inference

Let's evaluate an abstract, hypothetical genAI system that can automate some work for our organization. To avoid implicating any specific vendor, let's call the system "Mallory".

Is Mallory worth the money? How can we know?

Logically, there are only two outcomes that might result from using Mallory to do our work.

1. We prompt Mallory to do some work; we check its work, it is correct, and some time is saved.
2. We prompt Mallory to do some work; we check its work, it fails, and we futz around with the result; this time is wasted.

As a logical framework, this makes sense, but ROI is an arithmetical concept, not a logical one. So let's translate this into some terms.

In order to evaluate Mallory, let's define the Futzing Fraction, " FF ", in terms of the following variables:

H

the average amount of time a Human worker would take to do a task, unaided by Mallory

I

the amount of time that Mallory takes to run one Inference⁸

C

the amount of time that a human has to spend Checking Mallory's output for each inference

P

the Probability that Mallory will produce a correct inference for each prompt

W

the average amount of time that it takes for a human to Write one prompt for Mallory

E

since we are normalizing everything to time, rather than money, we do also have to account for the dollar of Mallory as as a product, so we will include the Equivalent amount of human time we could purchase for the marginal cost of one⁹ inference.

As in last week's example of simple ROI arithmetic, we will put our costs in the numerator, and our benefits in the denominator.

The idea here is that for each prompt, the minimum amount of time-equivalent cost possible is W+I+C+E. The user must, at least once, write a prompt, wait for inference to run, then check the output; and, of course, pay any costs to Mallory's vendor.

If the probability of a correct answer is P=13, then they will do this entire process 3 times¹⁰, so we put P in the denominator. Finally, we divide everything by H, because we are trying to determine if we are actually saving any time or money, versus just letting our existing human, who has to be driving this process anyway, do the whole thing.

If the Futzing Fraction evaluates to a number greater than 1, as previously discussed, you are a bozo; you're spending more time futzing with Mallory than getting value out of it.

Figuring out the Fraction is Frustrating

In order to even evaluate the value of the Futzing Fraction though, you have to have a sound method to even get a vague sense of all the terms.

If you are a business leader, a lot of this is relatively easy to measure. You vaguely know what H is, because you know what your payroll costs, and similarly, you can figure out E with some pretty trivial arithmetic based on Mallory's pricing table. There are endless YouTube channels, spec sheets and benchmarks to give you I. W is probably going to be so small compared to H that it hardly merits consideration¹¹.

But, are you measuring C? If your employees are not checking the outputs of the AI, you're on a path to catastrophe that no ROI calculation can capture, so it had better be greater than zero.

Are you measuring P? How often does the AI get it right on the first try?

Challenges to Computing Checking Costs

In the fraction defined above, the term C is going to be large. Larger than you think.

Measuring P and C with a high degree of precision is probably going to be very hard; possibly unreasonably so, or too expensive¹² to bother with in practice. So you will undoubtedly need to work with estimates and proxy metrics. But you have to be aware that this is a problem domain where your normal method of estimating is going to be extremely vulnerable to inherent cognitive bias, and find ways to measure.

Margins, Money, and Metacognition

First let's discuss cognitive and metacognitive bias.

My favorite cognitive bias is the availability heuristic and a close second is its cousin salience bias. Humans are empirically predisposed towards noticing and remembering things that are more striking, and to overestimate their frequency.

If you are estimating the variables above based on the vibe that you're getting from the experience of using an LLM, you may be overestimating its utility.

Consider a slot machine.

If you put a dollar in to a slot machine, and you lose that dollar, this is an unremarkable event. Expected, even. It doesn't seem interesting. You can repeat this over and over again, a thousand times, and each time it will seem equally unremarkable. If you do it a thousand times, you will probably get gradually more anxious as your sense of your dwindling bank account becomes slowly more salient, but losing one more dollar still seems unremarkable.

If you put a dollar in a slot machine and it gives you a thousand dollars, that will probably seem pretty cool. Interesting. Memorable. You might tell a story about this happening, but you definitely wouldn't really remember any particular time you lost one dollar.

Luckily, when you arrive at a casino with slot machines, you probably know well enough to set a hard budget in the form of some amount of physical currency you will have available to you. The odds are against you, you'll probably lose it all, but any responsible gambler will have an immediate, physical representation of their balance in front of them, so when they have lost it all, they can see that their hands are empty, and can try to resist the "just one more pull" temptation, after hitting that limit.

Now, consider Mallory.

If you put ten minutes into writing a prompt, and Mallory gives a completely off-the-rails, useless answer, and you lose ten minutes, well, that's just what using a computer is like sometimes. Mallory malfunctioned, or hallucinated, but it does that sometimes, everybody knows that. You only wasted ten minutes. It's fine. Not a big deal. Let's try it a few more times. Just ten more minutes. It'll probably work this time.

If you put ten minutes into writing a prompt, and it completes a task that would have otherwise taken you 4 hours, that feels amazing. Like the computer is magic! An absolute endorphin rush.

Very memorable. When it happens, it feels like P=1.

But... did you have a time budget before you started? Did you have a specified N such that "I will give up on Mallory as soon as I have spent N minutes attempting to solve this problem with it"? When the jackpot finally pays out that 4 hours, did you notice that you put 6 hours worth of 10-minute prompt coins into it in?

If you are attempting to use the same sort of heuristic intuition that probably works pretty well for other business leadership decisions, Mallory's slot-machine chat-prompt user interface is practically designed to subvert those sensibilities. Most business activities do not have nearly such an emotionally variable, intermittent reward schedule. They're not going to trick you with this sort of cognitive illusion.

Thus far we have been talking about cognitive bias, but there is a metacognitive bias at play too: while Dunning-Kruger, everybody's favorite metacognitive bias does have some problems with it, the main underlying metacognitive bias is that we tend to believe our own thoughts and perceptions, and it requires active effort to distance ourselves from them, even if we know they might be wrong.

This means you must assume any intuitive estimate of C is going to be biased low; similarly P is going to be biased high. You will forget the time you spent checking, and you will underestimate the number of times you had to re-check.

To avoid this, you will need to decide on a Ulysses pact to provide some inputs to a calculation for these factors that you will not be able to able to fudge if they seem wrong to you.

Problematically Plausible Presentation

Another nasty little cognitive-bias landmine for you to watch out for is the authority bias, for two reasons:

1. People will tend to see Mallory as an unbiased, external authority, and thereby see it as more of an authority than a similarly-situated human¹³.
2. Being an LLM, Mallory will be overconfident in its answers¹⁴.

The nature of LLM training is also such that commonly co-occurring tokens in the training corpus produce higher likelihood of co-occurring in the output; they're just going to be closer together in the vector-space of the weights; that's, like, what training a model is, establishing those relationships.

If you've ever used an heuristic to informally evaluate someone's credibility by listening for industry-specific shibboleths or ways of describing a particular issue, that skill is now useless. Having ingested every industry's expert literature, commonly-occurring phrases will always be present in Mallory's output. Mallory will usually sound like an expert, but then make mistakes at random.¹⁵.

While you might intuitively estimate C by thinking "well, if I asked a person, how could I check that they were correct, and how long would that take?" that estimate will be extremely optimistic, because the heuristic techniques you would use to quickly evaluate incorrect information from other humans will fail with Mallory. You need to go all the way back to primary sources and actually fully verify the output every time, or you will likely fall into one of these traps.

Mallory Mangling Mentorship

So far, I've been describing the effect Mallory will have in the context of an individual attempting to get some work done. If we are considering organization-wide adoption of Mallory, however, we must also consider the impact on team dynamics. There are a number of possible potential side effects that one might consider when looking at, but here I will focus on just one that I have observed.

I have a cohort of friends in the software industry, most of whom are individual contributors. I'm a programmer who likes programming, so are most of my friends, and we are also (sigh), charitably, pretty solidly middle-aged at this point, so we tend to have a lot of experience.

As such, we are often the folks that the team - or, in my case, the community - goes to when less-experienced folks need answers.

On its own, this is actually pretty great. Answering questions from more junior folks is one of the best parts of a software development job. It's an opportunity to be helpful, mostly just by knowing a thing we already knew. And it's an opportunity to help someone else improve their own agency by giving them knowledge that they can use in the future.

However, generative AI throws a bit of a wrench into the mix.

Let's imagine a scenario where we have 2 developers: Alice, a staff engineer who has a good understanding of the system being built, and Bob, a relatively junior engineer who is still onboarding.

The traditional interaction between Alice and Bob, when Bob has a question, goes like this:

1. Bob gets confused about something in the system being developed, because Bob's understanding of the system is incorrect.
2. Bob formulates a question based on this confusion.
3. Bob asks Alice that question.
4. Alice knows the system, so she gives an answer which accurately reflects the state of the system to Bob.
5. Bob's understanding of the system improves, and thus he will have fewer and better-informed questions going forward.

You can imagine how repeating this simple 5-step process will eventually transform Bob into a senior developer, and then he can start answering questions on his own. Making sufficient time for regularly iterating this loop is the heart of any good mentorship process.

Now, though, with Mallory in the mix, the process now has a new decision point, changing it from a linear sequence to a flow chart.

We begin the same way, with steps 1 and 2. Bob's confused, Bob formulates a question, but then:

3. Bob asks Mallory that question.

Here, our path then diverges into a "happy" path, a "meh" path, and a "sad" path.

The "happy" path proceeds like so:

4. Mallory happens to formulate a correct answer.
5. Bob's understanding of the system improves, and thus he will have fewer and better-informed questions going forward.

Great. Problem solved. We just saved some of Alice's time. But as we learned earlier,

Mallory can make mistakes. When that happens, we will need to check important info. So let's get checking:

4. Mallory happens to formulate an incorrect answer.
5. Bob investigates this answer.
6. Bob realizes that this answer is incorrect because it is inconsistent with some of his prior, correct knowledge of the system, or his investigation.
7. Bob asks Alice the same question; GOTO traditional interaction step 4.

On this path, Bob spent a while futzing around with Mallory, to no particular benefit. This wastes some of Bob's time, but then again, Bob could have ended up on the happy path, so perhaps it was worth the risk; at least Bob wasn't wasting any of Alice's much more valuable time in the process.¹⁶

Notice that beginning at the start of step 4, we must begin allocating all of Bob's time to C, so C already starts getting a bit bigger than if it were just Bob checking Mallory's output specifically on tasks that Bob is doing.

That brings us to the "sad" path.

4. Mallory happens to formulate an incorrect answer.
5. Bob investigates this answer.
6. Bob does not realize that this answer is incorrect because he is unable to recognize any inconsistencies with his existing, incomplete knowledge of the system.
7. Bob integrates Mallory's incorrect information of the system into his mental model.
8. Bob proceeds to make a larger and larger mess of his work, based on an incorrect mental model.
9. Eventually, Bob asks Alice a new, worse question, based on this incorrect understanding.
10. Sadly we cannot return to the happy path at this point, because now Alice must unravel the complex series of confusing misunderstandings that Mallory has unfortunately conveyed to Bob at this point. In the really sad case, Bob actually doesn't believe Alice for a while, because Mallory seems unbiased¹⁷, and Alice has to waste even more time convincing Bob before she can simply explain to him.

Now, we have wasted some of Bob's time, and some of Alice's time. Everything from step 5-10 is C, and as soon as Alice gets involved, we are now adding to C at double real-time. If more team members are pulled in to the investigation, you are now multiplying C by the number of investigators, potentially running at triple or quadruple real time.

But That's Not All

Here I've presented a brief selection reasons why C will be both large, and larger than you expect. To review:

1. Gambling-style mechanics of the user interface will interfere with your own self-monitoring and developing a good estimate.
2. You can't use human heuristics for quickly spotting bad answers.
3. Wrong answers given to junior people who can't evaluate them will waste more time from your more senior employees.

But this is a small selection of ways that Mallory's output can cost you money and time. It's harder to simplistically model second-order effects like this, but there's also a broad range of possibilities for ways that, rather than simply checking and catching errors, an error slips through and starts doing damage. Or ways in which the output isn't exactly wrong, but still sub-optimal in ways which can be difficult to notice in the short term.

For example, you might successfully vibe-code your way to launch a series of applications, successfully "checking" the output along the way, but then discover that the resulting code is unmaintainable garbage that prevents future feature delivery, and needs to be re-written¹⁸. But this kind of intellectual debt isn't even specific to technical debt while coding; it can even affect such apparently genAI-amenable fields as LinkedIn content marketing¹⁹.

Problems with the Prediction of P

C isn't the only challenging term though. P, is just as, if not more important, and just as hard to measure.

LLM marketing materials love to phrase their accuracy in terms of a percentage. Accuracy claims for LLMs in general tend to hover around 70%²⁰. But these scores vary per field, and when you aggregate them across multiple topic areas, they start to trend down. This is exactly why "agentic" approaches for more immediately-verifiable LLM outputs (with checks like "did the code work") got popular in the first place: you need to try more than once.

Independently measured claims about accuracy tend to be quite a bit lower²¹. The field of AI benchmarks is exploding, but it probably goes without saying that LLM vendors game those benchmarks²², because of course every incentive would encourage them to do that. Regardless of what their arbitrary scoring on some benchmark might say, all that matters to your business is whether it is accurate for the problems you are solving, for the way that you use it. Which is not necessarily going to correspond to any benchmark. You will need to measure it for yourself.

With that goal in mind, our formulation of P must be a somewhat harsher standard than "accuracy". It's not merely "was the factual information contained in any generated output accurate", but, "is the output good enough that some given real knowledge-work task is done and the human does not need to issue another prompt"?

Surprisingly Small Space for Slip-Ups

The problem with reporting these things as percentages at all, however, is that our actual definition for P is 1attempts, where attempts for any given attempt, at least, must be an integer greater than or equal to 1.

Taken in aggregate, if we succeed on the first prompt more often than not, we could end up with a P>12, but combined with the previous observation that you almost always have to prompt it more than once, the practical reality is that P will start at 50% and go down from there.

If we plug in some numbers, trying to be as extremely optimistic as we can, and say that we have a uniform stream of tasks, every one of which can be addressed by Mallory, every one of which:

• we can measure perfectly, with no overhead
• would take a human 45 minutes
• takes Mallory only a single minute to generate a response
• Mallory will require only 1 re-prompt, so "good enough" half the time
• takes a human only 5 minutes to write a prompt for
• takes a human only 5 minutes to check the result of
• has a per-prompt cost of the equivalent of a single second of a human's time

Thought experiments are a dicey basis for reasoning in the face of disagreements, so I have tried to formulate something here that is absolutely, comically, over-the-top stacked in favor of the AI optimist here.

Would that be a profitable? It sure seems like it, given that we are trading off 45 minutes of human time for 1 minute of Mallory-time and 10 minutes of human time. If we ask Python:

1
2
3
4
5

>>> def FF(H, I, C, P, W, E):
...     return (W + I + C + E) / (P * H)
... FF(H=45.0, I=1.0, C=5.0, P=1/2, W=5.0, E=0.01)
...
0.48933333333333334

We get a futzing fraction of about 0.4896. Not bad! Sounds like, at least under these conditions, it would indeed be cost-effective to deploy Mallory. But… realistically, do you reliably get useful, done-with-the-task quality output on the second prompt? Let's bump up the denominator on P just a little bit there, and see how we fare:

1
2

>>> FF(H=45.0, I=1.0, C=5.0, P=1/3, W=5.0, E=0.01)
0.734

Oof. Still cost-effective at 0.734, but not quite as good. Where do we cap out, exactly?

1
2
3
4
5
6
7
8
9

>>> from itertools import count
... for A in count(start=4):
...     print(A, result := FF(H=45.0, I=1.0, C=5.0, P=1 / A, W=5.0, E=1/60.))
...     if result > 1:
...         break
...
4 0.9792592592592594
5 1.224074074074074
>>>

With this little test, we can see that at our next iteration we are already at 0.9792, and by 5 tries per prompt, even in this absolute fever-dream of an over-optimistic scenario, with a futzing fraction of 1.2240, Mallory is now a net detriment to our bottom line.

Harm to the Humans

We are treating H as functionally constant so far, an average around some hypothetical Gaussian distribution, but the distribution itself can also change over time.

Formally speaking, an increase to H would be good for our fraction. Maybe it would even be a good thing; it could mean we're taking on harder and harder tasks due to the superpowers that Mallory has given us.

But an observed increase to H would probably not be good. An increase could also mean your humans are getting worse at solving problems, because using Mallory has atrophied their skills²³ and sabotaged learning opportunities²⁴²⁵. It could also go up because your senior, experienced people now hate their jobs²⁶.

For some more vulnerable folks, Mallory might just take a shortcut to all these complex interactions and drive them completely insane²⁷ directly. Employees experiencing an intense psychotic episode are famously less productive than those who are not.

This could all be very bad, if our futzing fraction eventually does head north of 1 and you need to reconsider introducing human-only workflows, without Mallory.

Abridging the Artificial Arithmetic (Alliteratively)

To reiterate, I have proposed this fraction:

which shows us positive ROI when FF is less than 1, and negative ROI when it is more than 1.

This model is heavily simplified. A comprehensive measurement program that tests the efficacy of any technology, let alone one as complex and rapidly changing as LLMs, is more complex than could be captured in a single blog post.

Real-world work might be insufficiently uniform to fit into a closed-form solution like this. Perhaps an iterated simulation with variables based on the range of values seem from your team's metrics would give better results.

However, in this post, I want to illustrate that if you are going to try to evaluate an LLM-based tool, you need to at least include some representation of each of these terms somewhere. They are all fundamental to the way the technology works, and if you're not measuring them somehow, then you are flying blind into the genAI storm.

I also hope to show that a lot of existing assumptions about how benefits might be demonstrated, for example with user surveys about general impressions, or by evaluating artificial benchmark scores, are deeply flawed.

Even making what I consider to be wildly, unrealistically optimistic assumptions about these measurements, I hope I've shown:

1. in the numerator, C might be a lot higher than you expect,
2. in the denominator, P might be a lot lower than you expect,
3. repeated use of an LLM might make H go up, but despite the fact that it's in the denominator, that will ultimately be quite bad for your business.

Personally, I don't have all that many concerns about E and I. E is still seeing significant loss-leader pricing, and I might not be coming down as fast as vendors would like us to believe, if the other numbers work out I don't think they make a huge difference. However, there might still be surprises lurking in there, and if you want to rationally evaluate the effectiveness of a model, you need to be able to measure them and incorporate them as well.

In particular, I really want to stress the importance of the influence of LLMs on your team dynamic, as that can cause massive, hidden increases to C. LLMs present opportunities for junior employees to generate an endless stream of chaff that will simultaneously:

• wreck your performance review process by making them look much more productive than they are,
• increase stress and load on senior employees who need to clean up unforeseen messes created by their LLM output,
• and ruin their own opportunities for career development by skipping over learning opportunities.

If you've already deployed LLM tooling without measuring these things and without updating your performance management processes to account for the strange distortions that these tools make possible, your Futzing Fraction may be much, much greater than 1, creating hidden costs and technical debt that your organization will not notice until a lot of damage has already been done.

If you got all the way here, particularly if you're someone who is enthusiastic about these technologies, thank you for reading. I appreciate your attention and I am hopeful that if we can start paying attention to these details, perhaps we can all stop futzing around so much with this stuff and get back to doing real work.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!

15 Aug 2025 7:51am GMT

My father, also known as "R0ML" once described a methodology for evaluating volume purchases that I think needs to be more popular.

If you are a hardcore fan, you might know that he has already described this concept publicly in a talk at OSCON in 2005, among other places, but it has never found its way to the public Internet, so I'm giving it a home here, and in the process, appropriating some of his words.¹

Let's say you're running a circus. The circus has many clowns. Ten thousand clowns, to be precise. They require bright red clown noses. Therefore, you must acquire a significant volume of clown noses. An enterprise licensing agreement for clown noses, if you will.

If the nose plays, it can really make the act. In order to make sure you're getting quality noses, you go with a quality vendor. You select a vendor who can supply noses for $100 each, at retail.

Do you want to buy retail? Ten thousand clowns, ten thousand noses, one hundred dollars: that's a million bucks worth of noses, so it's worth your while to get a good deal.

As a conscientious executive, you go to the golf course with your favorite clown accessories vendor and negotiate yourself a 50% discount, with a commitment to buy all ten thousand noses.

Is this a good deal? Should you take it?

To determine this, we will use an analytical tool called R0ML's Ratio (RR).

The ratio has 2 terms:

1. the Full Undiscounted Retail List Price of Units Used (FURLPoUU), which can of course be computed by the individual retail list price of a single unit (in our case, $100) multiplied by the number of units used
2. the Total Price of the Entire Enterprise Volume Licensing Agreement (TPotEEVLA), which in our case is $500,000.

It is expressed as:

RR = TPotEEVLA FURLPoUU

Crucially, you must be able to compute the number of units used in order to complete this ratio. If, as expected, every single clown wears their nose at least once during the period of the license agreement, then our Units Used is 10,000, our FURLPoUU is $1,000,000 and our TPotEEVLA is $500,000, which makes our RR 0.5.

Congratulations. If R0ML's Ratio is less than 1, it's a good deal. Proceed.

But… maybe the nose doesn't play. Not every clown's costume is an exact clone of the traditional, stereotypical image of a clown. Many are avant-garde. Perhaps this plentiful proboscis pledge was premature. Here, I must quote the originator of this theoretical framework directly:

What if the wheeze doesn't please?

What if the schnozz gives some pause?

In other words: what if some clowns don't wear their noses?

If we were to do this deal, and then ask around afterwards to find out that only 200 of our 10,000 clowns were to use their noses, then FURLPoUU comes out to 200 * $100, for a total of $20,000. In that scenario, RR is 25, which you may observe is substantially greater than 1.

If you do a deal where R0ML's ratio is greater than 1, then you are the bozo.

I apologize if I have belabored this point. As R0ML expressed in the email we exchanged about this many years ago,

I do not mind if you blog about it - and I don't mind getting the credit - although one would think it would be obvious.

And yeah, one would think this would be obvious? But I have belabored it because many discounted enterprise volume purchasing agreements still fail the R0ML's Ratio Bozo Test.²

In the case of clown noses, if you pay the discounted price, at least you get to keep the nose; maybe lightly-used clown noses have some resale value. But in software licensing or SaaS deals, once you've purchased the "discounted" software or service, once you have provisioned the "seats", the money is gone, and if your employees don't use it, then no value for your organization will ever result.

Measuring number of units used is very important. Without this number, you have no idea if you are a bozo or not.

It is often better to give your individual employees a corporate card and allow them to make arbitrary individual purchases of software licenses and SaaS tools, with minimal expense-reporting overhead; this will always keep R0ML's Ratio at 1.0, and thus, you will never be a bozo.

It is always better to do that the first time you are purchasing a new software tool, because the first time making such a purchase you (almost by definition) have no information about "units used" yet. You have no idea - you cannot have any idea - if you are a bozo or not.

If you don't know who the bozo is, it's probably you.

Acknowledgments

Thank you for reading, and especially thank you to my patrons who are supporting my writing on this blog. Of course, extra thanks to dad for, like, having this idea and doing most of the work here beyond my transcription. If you like my dad's ideas and you'd like to post more of them, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!

09 Aug 2025 4:41am GMT