When I built the content negotiation for this site (the part that hands each visitor the format and encoding their browser actually wants), the most useful reference I had was a post I'd written years earlier, on this same blog, about doing the same thing in nginx. That is what the blog is for.

27 Oct 2026 10:00am GMT

The Knownbits Toy abstract domain that I wrote about earlier is great for all kinds of bit manipulation rewrites. However, it is not particularly good at recognizing where bits move around in a sequence of instructions. It has a notion of "this bit is 0 or 1 or unknown" but no notion of "this bit is the same as this other bit". This prevents it from optimizing something like this:

x = ...
a = x & 0x0f
b = x << 60
c = b >> 60
result = a & c

Here a and c compute the same value, so it can be optimized to result = a. It is possible to do this optimization with local peephole rewrite rules too, but doing that requires adding a lot of these rules, and if one is missing we might miss collapsing a long sequence of operations.

In this post I want to talk about a more complicated abstract domain that is great at optimizing this kind of bit movement. It's a "bit provenance" domain. For every SSA variable in a trace, it computes where each bit from the variable comes from (or whether it's constant 0 or 1). In particular it can track if a bit in one variable must be the same as the bit of another variable at another position.

For the example above, the domain would store the following provenance information. Nothing is known about x, so its bits refer to themselves. Using 64-bit integers and a logical right shift, we get this table (most significant bits first):

Then we can directly conclude that result is the same value as a.

In the rest of the post I'll describe how the domain values store this information, how the transfer functions work and how we can use the abstract domain to optimize.

Contents

• Implementing the Domain

• Implementing Transfer Functions

• Implementing Bitwise Binary Transfer Functions

• Extracting Simpler Operations

  • Constants

  • Identity

  • Reusing Earlier Values

  • Simple Shifts

  • More Rewrites from the Same Rules

• Evaluation with Pydrofoil and PyPy

• Related work

• Conclusion

• Acknowledgements

Implementing the Domain

We'll start with reusing the toy optimizer from the previous posts, in the fold for completeness' sake.

toy infrastructure

from dataclasses import dataclass

INTEGER_WIDTH = 64
INT_MASK = 2 ** INTEGER_WIDTH - 1

class Value:
    def find(self):
        raise NotImplementedError("abstract")


@dataclass(eq=False)
class Operation(Value):
    name : str
    args : list[Value]

    forwarded : Value | None = None

    def find(self) -> Value:
        op = self
        while isinstance(op, Operation):
            next = op.forwarded
            if next is None:
                return op
            op = next
        return op

    def arg(self, index):
        return self.args[index].find()

    def make_equal_to(self, value : Value):
        self.find().forwarded = value


@dataclass(eq=False)
class Constant(Value):
    value : object

    def find(self):
        return self


class Block(list):
    def __getattr__(self, opname):
        def wraparg(arg):
            if not isinstance(arg, Value):
                arg = Constant(arg)
            return arg
        def make_op(*args):
            op = Operation(opname,
                [wraparg(arg) for arg in args])
            self.append(op)
            return op
        return make_op


def bb_to_str(l : Block, varprefix : str = "var"):
    def arg_to_str(arg : Value):
        if isinstance(arg, Constant):
            return str(arg.value)
        else:
            return varnames[arg]

    varnames = {}
    res = []
    for index, op in enumerate(l):
        # give the operation a name used while
        # printing:
        var =  f"{varprefix}{index}"
        varnames[op] = var
        arguments = ", ".join(
            arg_to_str(op.arg(i))
                for i in range(len(op.args))
        )
        strop = f"{var} = {op.name}({arguments})"
        res.append(strop)
    return "\n".join(res)

def dce(bb: Block) -> Block:
    pure = {"getarg", "int_and", "int_or", "int_xor", "int_invert",
            "int_lshift", "int_rshift"}
    used = set()
    kept = []
    for op in reversed(bb):
        if op not in used and op.name in pure:
            continue
        kept.append(op)
        for i in range(len(op.args)):
            arg = op.arg(i)
            if isinstance(arg, Operation):
                used.add(arg)
    return Block(reversed(kept))


def test_dce():
    bb = Block()
    x = bb.getarg(0)
    live = bb.int_or(x, 1)
    dead = bb.int_xor(live, 7)
    bb.int_and(dead, 15)
    forwarded = bb.int_and(x, 0)
    forwarded.make_equal_to(Constant(0))
    bb.dummy(live)
    bb.dummy(forwarded)

    assert bb_to_str(dce(bb), "optvar") == """\
optvar0 = getarg(0)
optvar1 = int_or(optvar0, 1)
optvar2 = dummy(optvar1)
optvar3 = dummy(0)"""

(The dce function was adapted from the Prospero blog post.)

Then we can start implementing the domain values like this:

@dataclass(frozen=True)
class SourceBit:
    source: Operation
    index: int
    inverted: bool = False


@dataclass(frozen=True)
class BitProvenance:
    # least significant bit first
    bits: tuple[int | SourceBit, ...]

    @staticmethod
    def make(bits):
        bits = tuple(bits)
        if len(bits) < INTEGER_WIDTH:
            # zero-extend, mainly for tests
            bits = bits + (0, ) * (INTEGER_WIDTH - len(bits))
        assert len(bits) == INTEGER_WIDTH
        return BitProvenance(bits)

    @staticmethod
    def from_constant(value):
        return BitProvenance(tuple((value >> i) & 1 for i in range(INTEGER_WIDTH)))

    @staticmethod
    def from_source(source: Operation):
        # nothing is known about this SSA value
        return BitProvenance(tuple(SourceBit(source, i) for i in range(INTEGER_WIDTH)))

    def get_constant(self) -> int | None:
        res = 0
        shift = 0
        for bit in self.bits:
            if isinstance(bit, SourceBit):
                return None
            res |= bit << shift
            shift += 1
        return res

Every SourceBit refers to a bit of an SSA Operation. For an SSA value that we know nothing about, we use the from_source factory method, which returns something of the form:

x = ... # x[63] x[62] x[61] x[60] x[59] ... x[ 3] x[ 2] x[ 1] x[0]

For a constant, the from_constant factory returns an abstract value where all the bits are concretely known values. Constants are simply 0 or 1 and need no source at all.

The inverted field of a SourceBit specifies whether the bit is a direct copy of the source bit, or whether the bit is flipped.

Let's make sure we got the bit twiddling in from_constant and get_constant right by writing a Hypothesis test:

from hypothesis import given, strategies

ints = strategies.integers(0, 2**64-1)

@given(ints)
def test_hypothesis_from_constant_get_constant_roundtrip(val):
    assert BitProvenance.from_constant(val).get_constant() == val

Implementing Transfer Functions

The simplest transfer function to implement for this domain is for ~, bit invert. It inverts constant bits and flips the inverted field of any SourceBit. Here's a test:

def test_abstract_invert():
    source = Operation("getarg", [Constant(0)])
    x = BitProvenance.make([1, 0, SourceBit(source, 1, True), SourceBit(source, 0)])
    res = x.abstract_invert()
    assert res == BitProvenance.make((0, 1, SourceBit(source, 1), SourceBit(source, 0, True)) + (1, ) * 60)

The implementation looks like this (the reason for the result argument will become clear a bit further down):

def invert_bit(bit):
    if isinstance(bit, int):
        return bit ^ 1
    return SourceBit(bit.source, bit.index, not bit.inverted)


class BitProvenance:
    ...

    def abstract_invert(self, result=None):
        return BitProvenance.make(tuple(invert_bit(bit) for bit in self.bits))

The shift implementations are also quite simple as long as the shift amounts are constant, they just move bits around. If the shifts aren't constant, we just know nothing about the result.

If we "know nothing", we want to use BitProvenance.from_source for the shift operation. Therefore the transfer functions need to get the shift Operation as an extra argument.

def test_lshift():
    source = Operation("getarg", [Constant(0)])
    x = BitProvenance.from_source(source) # no info

    # x << 7, the seven high bits get cut off
    seven = BitProvenance.from_constant(7)
    result = Operation("int_lshift", [source, Constant(7)])
    assert x.abstract_lshift(seven, result).bits == (0, ) * 7 + x.bits[:-7]

    # x << 65, constant 0
    big = BitProvenance.from_constant(65)
    result = Operation("int_lshift", [source, Constant(65)])
    assert x.abstract_lshift(big, result).get_constant() == 0

    # x << y, we know nothing
    amount = Operation("getarg", [Constant(1)])
    result = Operation("int_lshift", [source, amount])
    assert x.abstract_lshift(BitProvenance.from_source(amount), result) == BitProvenance.from_source(result)


@given(ints, strategies.integers(0, 100))
def test_hypothesis_lshift_on_constants(val, shift):
    absval = BitProvenance.from_constant(val)
    absshift = BitProvenance.from_constant(shift)
    result = Operation("int_lshift", [Constant(val), Constant(shift)])
    assert absval.abstract_lshift(absshift, result).get_constant() == (val << shift) & INT_MASK


class BitProvenance:
    ...

    def abstract_lshift(self, other, result: Operation):
        const = other.get_constant()
        if const is None:
            return BitProvenance.from_source(result)
        if const >= INTEGER_WIDTH:
            # make oversized shift amounts return 0 for simplicity
            return BitProvenance.from_constant(0)
        if const == 0:
            return self
        return BitProvenance.make((0, ) * const + self.bits[:-const])

Logical right shift is pretty similar, except we append the zeros at the other end:

def test_rshift():
    source = Operation("getarg", [Constant(0)])
    x = BitProvenance.from_source(source) # no info

    # x >> 7, the seven low bits get cut off
    seven = BitProvenance.from_constant(7)
    result = Operation("int_rshift", [source, Constant(7)])
    assert x.abstract_rshift(seven, result).bits == x.bits[7:] + (0, ) * 7

    # x >> 65, constant 0
    big = BitProvenance.from_constant(65)
    result = Operation("int_rshift", [source, Constant(65)])
    assert x.abstract_rshift(big, result).get_constant() == 0

    # x >> y, we know nothing
    amount = Operation("getarg", [Constant(1)])
    result = Operation("int_rshift", [source, amount])
    assert x.abstract_rshift(BitProvenance.from_source(amount), result) == BitProvenance.from_source(result)


@given(ints, strategies.integers(0, 100))
def test_hypothesis_rshift_on_constants(val, shift):
    absval = BitProvenance.from_constant(val)
    absshift = BitProvenance.from_constant(shift)
    result = Operation("int_rshift", [Constant(val), Constant(shift)])
    assert absval.abstract_rshift(absshift, result).get_constant() == val >> shift


class BitProvenance:
    ...

    def abstract_rshift(self, other, result: Operation):
        const = other.get_constant()
        if const is None:
            return BitProvenance.from_source(result)
        if const >= INTEGER_WIDTH:
            return BitProvenance.from_constant(0)
        if const == 0:
            return self
        return BitProvenance.make(self.bits[const:] + (0, ) * const)

Implementing Bitwise Binary Transfer Functions

Next are the abstract binary transfer functions and, or, xor. We can think about the behaviour on individual bits first. A few cases are simple (e.g. a & 0 == 0) but some need reasoning about sources.

def and_bits(a, b):
    # None means the result cannot be represented as a constant or source
    # bit
    if a == 0:
        return 0
    if b == 0:
        return 0
    if a == 1:
        return b
    if b == 1:
        return a
    assert isinstance(a, SourceBit)
    assert isinstance(b, SourceBit)
    if a == b:
        # a & a == a
        return a
    if a == invert_bit(b):
        # a & not a == 0
        return 0
    return None


def test_and_bits():
    x = Operation("getarg", [Constant(0)])
    y = Operation("getarg", [Constant(1)])
    a = SourceBit(x, 3)
    assert and_bits(a, 0) == and_bits(0, a) == 0
    assert and_bits(a, 1) == and_bits(1, a) == a
    assert and_bits(a, a) == a
    assert and_bits(a, invert_bit(a)) == 0

    # positions disagree
    assert and_bits(a, SourceBit(x, 4)) is None

    # sources disagree
    assert and_bits(a, SourceBit(y, 3)) is None
    assert and_bits(a, SourceBit(y, 3, True)) is None

When and_bits returns None for output bit i, we can use SourceBit(result, i): we don't know how that bit relates to the inputs. The other output bits keep their own provenance, even if they refer to different operations.

Now we can combine and_bits calls for all the bits of a BitProvenance.

def test_and():
    x = Operation("getarg", [Constant(0)])
    y = Operation("getarg", [Constant(1)])
    absx = BitProvenance.from_source(x)
    absy = BitProvenance.from_source(y)

    # x & 0 == 0
    result = Operation("int_and", [x, Constant(0)])
    assert absx.abstract_and(BitProvenance.from_constant(0), result).get_constant() == 0

    # x & -1 == x
    result = Operation("int_and", [x, Constant(INT_MASK)])
    assert absx.abstract_and(BitProvenance.from_constant(INT_MASK), result) == absx

    # x & x == x
    result = Operation("int_and", [x, x])
    assert absx.abstract_and(absx, result) == absx

    # x & y: we can't usefully represent provenance, so we get fresh bits
    result = Operation("int_and", [x, y])
    assert absx.abstract_and(absy, result) == BitProvenance.from_source(result)

    # x & ~x == 0
    result = Operation("int_and", [x, Operation("int_invert", [x])])
    assert absx.abstract_and(absx.abstract_invert(), result).get_constant() == 0


@given(ints, ints)
def test_hypothesis_and_on_constants(a, b):
    abs_a = BitProvenance.from_constant(a)
    abs_b = BitProvenance.from_constant(b)
    result = Operation("int_and", [Constant(a), Constant(b)])
    assert abs_a.abstract_and(abs_b, result).get_constant() == a & b


class BitProvenance:
    ...

    def abstract_and(self, other, result: Operation):
        bits = []
        for i, (bit_a, bit_b) in enumerate(zip(self.bits, other.bits)):
            bit_res = and_bits(bit_a, bit_b)
            if bit_res is None:
                # not representable, use fresh provenance
                bit_res = SourceBit(result, i)
            bits.append(bit_res)
        return BitProvenance.make(bits)

or is very similar:

def or_bits(a, b):
    if a == 1:
        return 1
    if b == 1:
        return 1
    if a == 0:
        return b
    if b == 0:
        return a
    assert isinstance(a, SourceBit)
    assert isinstance(b, SourceBit)
    if a == b:
        # a | a == a
        return a
    if a == invert_bit(b):
        # a | not a == 1
        return 1
    return None

def test_or_bits():
    x = Operation("getarg", [Constant(0)])
    y = Operation("getarg", [Constant(1)])
    a = SourceBit(x, 3)
    assert or_bits(a, 1) == or_bits(1, a) == 1
    assert or_bits(a, 0) == or_bits(0, a) == a
    assert or_bits(a, a) == a
    assert or_bits(a, invert_bit(a)) == 1

    # positions disagree
    assert or_bits(a, SourceBit(x, 4)) is None

    # sources disagree
    assert or_bits(a, SourceBit(y, 3)) is None
    assert or_bits(a, SourceBit(y, 3, True)) is None


def test_or():
    x = Operation("getarg", [Constant(0)])
    y = Operation("getarg", [Constant(1)])
    absx = BitProvenance.from_source(x)
    absy = BitProvenance.from_source(y)

    # x | 0 == x
    result = Operation("int_or", [x, Constant(0)])
    assert absx.abstract_or(BitProvenance.from_constant(0), result) == absx

    # x | -1 == -1
    result = Operation("int_or", [x, Constant(INT_MASK)])
    assert absx.abstract_or(BitProvenance.from_constant(INT_MASK), result).get_constant() == INT_MASK

    # x | x == x
    result = Operation("int_or", [x, x])
    assert absx.abstract_or(absx, result) == absx

    # x | y: we can't usefully represent provenance, so we get fresh bits
    result = Operation("int_or", [x, y])
    assert absx.abstract_or(absy, result) == BitProvenance.from_source(result)

    # x | ~x == -1
    result = Operation("int_or", [x, Operation("int_invert", [x])])
    assert absx.abstract_or(absx.abstract_invert(), result).get_constant() == INT_MASK


@given(ints, ints)
def test_hypothesis_or_on_constants(a, b):
    abs_a = BitProvenance.from_constant(a)
    abs_b = BitProvenance.from_constant(b)
    result = Operation("int_or", [Constant(a), Constant(b)])
    assert abs_a.abstract_or(abs_b, result).get_constant() == a | b


class BitProvenance:
    ...

    def abstract_or(self, other, result: Operation):
        bits = []
        for i, (bit_a, bit_b) in enumerate(zip(self.bits, other.bits)):
            bit_res = or_bits(bit_a, bit_b)
            if bit_res is None:
                # not representable, use fresh provenance
                bit_res = SourceBit(result, i)
            bits.append(bit_res)
        return BitProvenance.make(bits)

And finally here is xor:

def xor_bits(a, b):
    if a == 1:
        # x ^ 1 == ~x
        return invert_bit(b)
    if b == 1:
        return invert_bit(a)
    if a == 0:
        # x ^ 0 == x
        return b
    if b == 0:
        return a
    assert isinstance(a, SourceBit)
    assert isinstance(b, SourceBit)
    if a == b:
        # a ^ a == 0
        return 0
    if a == invert_bit(b):
        # a ^ not a == 1
        return 1
    return None

def test_xor_bits():
    x = Operation("getarg", [Constant(0)])
    y = Operation("getarg", [Constant(1)])
    a = SourceBit(x, 3)
    assert xor_bits(a, 1) == xor_bits(1, a) == invert_bit(a)
    assert xor_bits(a, 0) == xor_bits(0, a) == a
    assert xor_bits(a, a) == 0
    assert xor_bits(a, invert_bit(a)) == 1

    # positions disagree
    assert xor_bits(a, SourceBit(x, 4)) is None

    # sources disagree
    assert xor_bits(a, SourceBit(y, 3)) is None
    assert xor_bits(a, SourceBit(y, 3, True)) is None


def test_xor():
    x = Operation("getarg", [Constant(0)])
    y = Operation("getarg", [Constant(1)])
    absx = BitProvenance.from_source(x)
    absy = BitProvenance.from_source(y)

    # x ^ 0 == x
    result = Operation("int_xor", [x, Constant(0)])
    assert absx.abstract_xor(BitProvenance.from_constant(0), result) == absx

    # x ^ -1 == ~x
    result = Operation("int_xor", [x, Constant(INT_MASK)])
    assert absx.abstract_xor(BitProvenance.from_constant(INT_MASK), result) == absx.abstract_invert()

    # x ^ x == 0
    result = Operation("int_xor", [x, x])
    assert absx.abstract_xor(absx, result).get_constant() == 0

    # x ^ y: we can't usefully represent provenance, so we get fresh bits
    result = Operation("int_xor", [x, y])
    assert absx.abstract_xor(absy, result) == BitProvenance.from_source(result)

    # x ^ ~x == -1
    result = Operation("int_xor", [x, Operation("int_invert", [x])])
    assert absx.abstract_xor(absx.abstract_invert(), result).get_constant() == INT_MASK


@given(ints, ints)
def test_hypothesis_xor_on_constants(a, b):
    abs_a = BitProvenance.from_constant(a)
    abs_b = BitProvenance.from_constant(b)
    result = Operation("int_xor", [Constant(a), Constant(b)])
    assert abs_a.abstract_xor(abs_b, result).get_constant() == a ^ b


class BitProvenance:
    ...

    def abstract_xor(self, other, result: Operation):
        bits = []
        for i, (bit_a, bit_b) in enumerate(zip(self.bits, other.bits)):
            bit_res = xor_bits(bit_a, bit_b)
            if bit_res is None:
                # not representable, use fresh provenance
                bit_res = SourceBit(result, i)
            bits.append(bit_res)
        return BitProvenance.make(bits)

Extracting Simpler Operations

Now that we have the analysis, we need to find out how to use the information during optimization. We'll discuss four ways of doing that, from easy to more complicated.

Constants

The simplest thing to do is to exploit the fact that provenance can also identify constant results.

def test_constant_via_provenance():
    bb = Block()
    x = bb.getarg(0)
    a = bb.int_and(x, 0x0f)
    b = bb.int_or(x, 0xf0)
    c = bb.int_and(b, 0x0f)
    # we compute result = (x & 0x0f) ^ ((x | 0xf0) & 0x0f)
    # ((x | 0xf0) & 0x0f) can be rewritten to
    # (x & 0x0f) | (0xf0 & 0x0f)
    # which is x & 0x0f
    # this means result == (x & 0x0f) ^ (x & 0x0f) == 0
    result = bb.int_xor(a, c)
    bb.dummy(result)

    opt_bb = dce(simplify(bb))
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = dummy(0)"""

To make this test pass, we can do something very similar to what we did in the KnownBits post. We go along a basic block, apply the transfer functions, and if a result is constant we can replace the operation with that constant.

def unknown_transfer_functions(*abstract_args, result):
    return BitProvenance.from_source(result)

def simplify(bb: Block) -> Block:
    abstract_values = {} # dict mapping Operation to BitProvenance

    def provenance_of(val : Value):
        if isinstance(val, Constant):
            return BitProvenance.from_constant(val.value)
        return abstract_values[val]

    opt_bb = Block()
    for op in bb:
        # apply the transfer function on the abstract arguments
        name_without_prefix = op.name.removeprefix("int_")
        method_name = f"abstract_{name_without_prefix}"
        transfer_function = getattr(BitProvenance, method_name, unknown_transfer_functions)
        abstract_args = [provenance_of(arg.find()) for arg in op.args]
        abstract_res = abstract_values[op] = transfer_function(*abstract_args, result=op)
        # if the result is a constant, we optimize the operation away and make
        # it equal to the constant result
        const = abstract_res.get_constant()
        if const is not None:
            op.make_equal_to(Constant(const))
            continue
        # otherwise emit the op
        opt_bb.append(op)
    return opt_bb

Identity

The next optimization we can add is that sometimes complicated bit reshuffling leads to a result that was already computed earlier in the block. We can start with a simple test that splits a value into low and high bits and recombines them:

def test_identity_by_recombining_bits():
    # split and then recombine x
    # lo = x & 0x0f
    # hi = (x >> 4) << 4
    # result = lo | hi
    bb = Block()
    x = bb.getarg(0)
    lo = bb.int_and(x, 0x0f)
    shifted = bb.int_rshift(x, 4)
    hi = bb.int_lshift(shifted, 4)
    result = bb.int_or(lo, hi)
    bb.dummy(result)

    opt_bb = dce(simplify(bb))
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = getarg(0)
optvar1 = dummy(optvar0)"""

To help implement this, we can add a method to check whether a provenance is identical to that of an earlier result. The idea is that if the bits of a BitProvenance come from bits 0, 1, 2, ..., INTEGER_WIDTH - 1 of the same source, without inversion, then the operation we are looking at computes the same result as the source too.

def test_provenance_get_identical_source():
    source = Operation("getarg", [Constant(0)])
    x = BitProvenance.make([1, 0, SourceBit(source, 1, True), SourceBit(source, 0)])
    assert x.get_identical_source() is None # not the same value as source

    # trivially, a BitProvenance where we know nothing is a valid source for itself
    res = BitProvenance.from_source(source)
    assert res.get_identical_source() is source


class BitProvenance:
    ...
    def get_identical_source(self) -> Operation | None:
        bit0 = self.bits[0]
        if isinstance(bit0, int):
            return None # a constant bit cannot match an unconstrained source bit
        source = bit0.source
        # are all the bits the same as in the unconstrained source?
        if self == BitProvenance.from_source(source):
            return source
        return None

Now we can use that to recognize identities when optimizing:

def simplify(bb: Block) -> Block:
    abstract_values = {} # dict mapping Operation to BitProvenance

    def provenance_of(val : Value):
        if isinstance(val, Constant):
            return BitProvenance.from_constant(val.value)
        return abstract_values[val]

    opt_bb = Block()
    for op in bb:
        # apply the transfer function on the abstract arguments
        name_without_prefix = op.name.removeprefix("int_")
        method_name = f"abstract_{name_without_prefix}"
        transfer_function = getattr(BitProvenance, method_name, unknown_transfer_functions)
        abstract_args = [provenance_of(arg.find()) for arg in op.args]
        abstract_res = abstract_values[op] = transfer_function(*abstract_args, result=op)
        # if the result is a constant, we optimize the operation away and make
        # it equal to the constant result
        const = abstract_res.get_constant()
        if const is not None:
            op.make_equal_to(Constant(const))
            continue
        source = abstract_res.get_identical_source()
        if source is not None and source is not op:
            op.make_equal_to(source)
            continue
        # otherwise emit the op
        opt_bb.append(op)
    return opt_bb

Reusing Earlier Values

The motivating example from the start of the blog post is next. It doesn't quite work with the idea from the previous subsection. Here it is in test form:

def test_identity_via_provenance():
    bb = Block()
    x = bb.getarg(0)
    a = bb.int_and(x, 0x0f)
    b = bb.int_lshift(x, 60)
    c = bb.int_rshift(b, 60)
    # result is same as a, but the bits come from x
    result = bb.int_and(a, c)
    bb.dummy(result)

    opt_bb = dce(simplify(bb))
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = getarg(0)
optvar1 = int_and(optvar0, 15)
optvar2 = dummy(optvar1)"""

We will use a hashmap to store all the provenance patterns we have seen in the basic block so far.

def simplify(bb: Block) -> Block:
    abstract_values = {} # dict mapping Operation to BitProvenance
    seen_provenances = {} # dict mapping BitProvenance to Value

    def provenance_of(val : Value):
        if isinstance(val, Constant):
            return BitProvenance.from_constant(val.value)
        return abstract_values[val]

    opt_bb = Block()
    for op in bb:
        # apply the transfer function on the abstract arguments
        name_without_prefix = op.name.removeprefix("int_")
        method_name = f"abstract_{name_without_prefix}"
        transfer_function = getattr(BitProvenance, method_name, unknown_transfer_functions)
        abstract_args = [provenance_of(arg.find()) for arg in op.args]
        abstract_res = abstract_values[op] = transfer_function(*abstract_args, result=op)
        # if the result is a constant, we optimize the operation away and make
        # it equal to the constant result
        const = abstract_res.get_constant()
        if const is not None:
            op.make_equal_to(Constant(const))
            continue
        # check whether we've seen the provenance before, if yes, reuse the op
        source = seen_provenances.get(abstract_res)
        if source is not None:
            op.make_equal_to(source)
            continue
        seen_provenances[abstract_res] = op
        # otherwise emit the op
        opt_bb.append(op)
    return opt_bb

Simple Shifts

In addition to using existing values, we can sometimes emit a new shift operation of some earlier value, to get fewer operations than how the value was actually computed. Here's an example:

def test_extract_rshift_from_source():
    bb = Block()
    x = bb.getarg(0)
    # lo = (x & 0xf)
    lo = bb.int_and(x, 0x0f)
    # cleared = x ^ (x & 0xf)
    cleared = bb.int_xor(x, lo)
    # result = cleared >> 4
    result = bb.int_rshift(cleared, 4)
    bb.dummy(result)

    # the xor clears only the bits that the shift discards
    # the result's provenance matches a direct x >> 4
    opt_bb = dce(simplify(bb))
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = getarg(0)
optvar1 = int_rshift(optvar0, 4)
optvar2 = dummy(optvar1)"""

To implement this, we add a new method BitProvenance.recognize which detects whether a BitProvenance instance could be represented as a shift of the source.

def test_recognize_rshift():
    x = Operation("getarg", [Constant(0)])
    # low bits come from x[4], x[5], ..., x[63]; the high four are zero
    bits = tuple(SourceBit(x, i) for i in range(4, INTEGER_WIDTH))
    shifted = BitProvenance.make(bits + (0, 0, 0, 0))
    assert shifted.recognize() == ("int_rshift", x, 4)

    # the same alignment is not enough if a high bit is one
    not_a_shift = BitProvenance.make(bits + (0, 0, 0, 1))
    assert not_a_shift.recognize() is None

    # the first bit suggests >> 4, but the next bit comes from x[6], not x[5]
    misplaced = bits[:1] + (SourceBit(x, 6),) + bits[2:]
    not_a_shift = BitProvenance.make(misplaced + (0, 0, 0, 0))
    assert not_a_shift.recognize() is None


@given(strategies.integers(1, 63))
def test_hypothesis_recognize_shift(shift):
    x = Operation("getarg", [Constant(0)])
    absval = BitProvenance.from_source(x)
    shift_amount = BitProvenance.from_constant(shift)
    # check that lshifts are recognized
    result = Operation("int_lshift", [x, Constant(shift)])
    res = absval.abstract_lshift(shift_amount, result).recognize()
    assert res == ("int_lshift", x, shift)

    # check that rshifts are recognized
    result = Operation("int_rshift", [x, Constant(shift)])
    res = absval.abstract_rshift(shift_amount, result).recognize()
    assert res == ("int_rshift", x, shift)

class BitProvenance:
    ...
    def recognize(self):
        # return None or a triple (opname, source, shift amount), where
        # opname is "int_rshift" or "int_lshift"
        bits = [(i, b) for i, b in enumerate(self.bits) if isinstance(b, SourceBit)]
        if not bits:
            return None
        # try to guess the shift amount
        i, b = bits[0]
        delta = i - b.index
        source = b.source
        original = BitProvenance.from_source(source)
        # use the transfer functions to check whether it really is that shift
        if delta >= 0:
            expected = original.abstract_lshift(BitProvenance.from_constant(delta), None)
            # does the shift of the source match self?
            if self == expected:
                return ("int_lshift", source, delta)
        else:
            expected = original.abstract_rshift(BitProvenance.from_constant(-delta), None)
            # does the shift of the source match self?
            if self == expected:
                return ("int_rshift", source, -delta)
        return None

Using the abstract transfer functions in recognize has the nice effect that we only specify what lshift and rshift do in a single place.

Now we can start using this in simplify:

def simplify(bb: Block) -> Block:
    abstract_values = {} # dict mapping Operation to BitProvenance
    seen_provenances = {} # dict mapping BitProvenance to Value

    def provenance_of(val : Value):
        if isinstance(val, Constant):
            return BitProvenance.from_constant(val.value)
        return abstract_values[val]

    opt_bb = Block()
    for op in bb:
        # apply the transfer function on the abstract arguments
        name_without_prefix = op.name.removeprefix("int_")
        method_name = f"abstract_{name_without_prefix}"
        transfer_function = getattr(BitProvenance, method_name, unknown_transfer_functions)
        abstract_args = [provenance_of(arg.find()) for arg in op.args]
        abstract_res = abstract_values[op] = transfer_function(*abstract_args, result=op)
        # if the result is a constant, we optimize the operation away and make
        # it equal to the constant result
        const = abstract_res.get_constant()
        if const is not None:
            op.make_equal_to(Constant(const))
            continue
        # check whether we've seen the provenance before, if yes, reuse the op
        source = seen_provenances.get(abstract_res)
        if source is not None:
            op.make_equal_to(source)
            continue

        # now try to synthesize a shift with the same provenance
        pattern = abstract_res.recognize()
        if pattern is not None:
            name, source_op, shift_amount = pattern
            if source_op is not op:
                assert shift_amount > 0 # otherwise one of the earlier rules would have caught it
                new_op = getattr(opt_bb, name)(source_op, Constant(shift_amount))
                abstract_values[new_op] = abstract_res
                seen_provenances[abstract_res] = new_op
                op.make_equal_to(new_op)
                continue
        # otherwise emit the op
        seen_provenances[abstract_res] = op
        opt_bb.append(op)
    return opt_bb

That's the whole optimizer. We propagate where each bit comes from, then look for a cheaper way to produce the resulting provenance. Constant folding, reuse of earlier values, and shift recognition all use the same information. None of them needs to know which sequence of operations produced it.

More Rewrites from the Same Rules

Now we can look at a few other rewrites that fall out of this design. Here are some further passing tests:

def test_xor_masks_cancel():
    bb = Block()
    x = bb.getarg(0)
    # each low-byte bit is flipped twice: 0x55 ^ 0xff ^ 0xaa == 0
    # therefore (((x ^ 0x55) ^ 0xff) ^ 0xaa) == x
    # by constant reassociation
    a = bb.int_xor(x, 0x55)
    b = bb.int_xor(a, 0xff)
    result = bb.int_xor(b, 0xaa)
    bb.dummy(result)

    opt_bb = dce(simplify(bb))
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = getarg(0)
optvar1 = dummy(optvar0)"""


def test_recover_low_bits_after_packing_two_sources():
    bb = Block()
    x = bb.getarg(0)
    y = bb.getarg(1)
    lo = bb.int_and(x, 0xff)
    hi = bb.int_lshift(y, 8)
    packed = bb.int_or(lo, hi)
    # masking the packed value recovers the earlier low byte of x
    result = bb.int_and(packed, 0xff)
    bb.dummy(result)

    opt_bb = dce(simplify(bb))
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = getarg(0)
optvar1 = int_and(optvar0, 255)
optvar2 = dummy(optvar1)"""


def test_reuse_mask_via_de_morgan():
    bb = Block()
    x = bb.getarg(0)
    masked = bb.int_and(x, 0x0f)
    inverted = bb.int_invert(x)
    high_ones = bb.int_or(inverted, INT_MASK ^ 0x0f)
    # ~(~x | ~0x0f) == x & 0x0f, which we already computed
    result = bb.int_invert(high_ones)
    bb.dummy(masked, result)

    opt_bb = dce(simplify(bb))
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = getarg(0)
optvar1 = int_and(optvar0, 15)
optvar2 = dummy(optvar1, optvar1)"""

def test_byte_swap_twice():
    def byte_swap(bb, value):
        result = 0
        for byte_index in range(8):
            byte = bb.int_and(bb.int_rshift(value, byte_index * 8), 0xff)
            moved = bb.int_lshift(byte, (7 - byte_index) * 8)
            result = bb.int_or(result, moved)
        return result

    bb = Block()
    x = bb.getarg(0)
    # little-endian host to network byte order, then back to host order
    network_order = byte_swap(bb, x)
    result = byte_swap(bb, network_order)
    bb.dummy(result)

    opt_bb = dce(simplify(bb))
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = getarg(0)
optvar1 = dummy(optvar0)"""

Here's an even more complicated test. It reverses the bits of the argument, then shifts left by 7, then reverses the bits of the result again. The input block has 63 operations, and dce(simplify(bb)) reduces it to just three. The bit reversal below is adapted from Hacker's Delight, section 7-1.

def test_reverse_lshift_reverse_becomes_rshift():
    # Hacker's Delight, section 7-1: swap adjacent bits, then pairs,
    # nibbles, bytes, and larger fields. adapted from 32 to 64 bits
    def reverse_bits(bb, value):
        for shift, mask in [
            (1,  0x5555555555555555),
            (2,  0x3333333333333333),
            (4,  0x0f0f0f0f0f0f0f0f),
            (8,  0x00ff00ff00ff00ff),
            (16, 0x0000ffff0000ffff),
            (32, 0x00000000ffffffff),
        ]:
            # value = ((value & mask) << shift) | ((value >> shift) & mask)
            left = bb.int_lshift(bb.int_and(value, mask), shift)
            right = bb.int_and(bb.int_rshift(value, shift), mask)
            value = bb.int_or(left, right)
        return value

    bb = Block()
    x = bb.getarg(0)
    reversed_once = reverse_bits(bb, x)
    shifted = bb.int_lshift(reversed_once, 7)
    result = reverse_bits(bb, shifted)
    bb.dummy(result)

    opt_bb = dce(simplify(bb))
    # reversing before and after a left shift turns it into a right shift
    assert bb_to_str(opt_bb, "optvar") == """\
optvar0 = getarg(0)
optvar1 = int_rshift(optvar0, 7)
optvar2 = dummy(optvar1)"""

What I like about all of these is how quite varied operation rewrites just fall out from the bit provenance approach, without needing complicated peephole rewrite patterns.

Evaluation with Pydrofoil and PyPy

To evaluate the bit provenance optimizer, I wrote a program that applies it to PyPy JIT traces. I didn't fully build it into the PyPy JIT because I was quite skeptical that it would help. Instead, I wrote an external optimization pass that parses RPython JIT logs, applies the optimization and then produces statistics. This uses an extended prototype that supports additional operations such as arithmetic right shifts and sign extension, and can extract shifts of earlier computed values as well as shifts of the original source.

I did this experiment for two different RPython projects, for PyPy (running the translator, that bootstraps PyPy to C code) and Pydrofoil (RISC-V simulation based on Sail, booting the Linux kernel). I picked the latter because Pydrofoil does a lot of integer operations, whereas most Python code does very little.

For both of these programs I wanted to know how well the bit provenance optimizer can optimize the traces compared to the integer optimizations of the PyPy JIT. I took the unoptimized traces, removed everything but integer operations and performed dead code elimination. To do that, the integer results of non-pure operations are turned into unconstrained arguments of the trace. Integer values used by removed operations are retained as outputs. The remaining integer operations are our baseline traces for this experiment.

The following table shows the results. The first column shows the number of unoptimized integer operations. The second column shows how many operations are left after running just bit provenance optimizations. The third column shows how many operations are left after the RPython JIT integer optimizations run. The last column shows how many operations are left after we run first the RPython JIT optimizer and then bit provenance optimizations. All three paths include dead code elimination after optimization.

In parentheses are the percentages of how many of the unoptimized operations are removed by the corresponding path.

The 'RPython then Provenance' column shows that, for the PyPy workload, provenance removes no additional operations after the RPython integer optimization passes in this experiment. For Pydrofoil it does remove more operations, but we will have to investigate whether this benefit carries over to the full JIT optimization pipeline.

However, the comparison between the 'Provenance' and 'RPython' column for Pydrofoil shows that the design of the optimizer is quite good! The small provenance optimizer achieves nearly as much reduction in integer operation count as the complicated and well-tuned integer optimization passes of the RPython JIT.

Related work

LLVM has a very similar bit provenance tracking optimization as part of the Hexagon backend. It also tracks 0 bits, 1 bits or references to specific bits from other SSA operations (it doesn't track negation). It is split into the BitTracker analysis and the HexagonBitSimplify optimization pass.

LLVM also has collectBitParts (source). Rather than running a separate forward analysis, it recursively walks SSA operations following provenance (with memoization and limited recursion depth for performance). Then there is a check whether the resulting provenance describes a byte swap or bit reversal.

In GCC this tracking happens on a byte level to be able to recognize byte swaps in the byte-swap recognizer. It also tracks memory addresses and offsets to recognize wider loads assembled from separate byte loads, possibly with a byte swap. However, there's also a recent (as of writing unmerged) patch series to extend this to bit granularity.

Conclusion

The interesting result here isn't that we beat an existing optimizer. It's that a handful of transfer functions lets us discover rewrites across arbitrarily long chains of bit operations, without writing a rule for each combination. The evaluation suggests that existing integer optimizations already cover much of this stuff. However, bit provenance gives us a compact way to describe a whole family of simplifications. It's quite cool to write an optimizer that can undo a bit reversal without ever knowing what a bit reversal is.

The entire code can be found in this gist.

Acknowledgements

Thanks to Joe Groff for the fantastic title. Thanks to Can Joshua Lehmann, Andy Wingo, Matti Picus, and Nico Rittinghaus for feedback on the draft, and to Max Bernstein for encouragement.

27 Oct 2026 6:04am GMT

It is hard to get through a day online at the moment without someone dismissing something as "AI slop". A pull request, a library, a whole project, all written off with two words. When the thing being dismissed is code, I have started to notice that the label usually arrives well before anyone could have actually read the code in question.

That has led me to a slightly uncomfortable conclusion. In a lot of cases, calling AI generated code slop is just a new way of saying TLDR (too long, didn't read). It isn't really a measure of whether the code is good or bad. It says more about the willingness of the person saying it to sit down and learn what the code does.

There is even slang emerging for the general version of this. Posts that are obviously AI generated are now getting the response AI;DR, as in "AI, didn't read". At least that is honest about what is going on. Slop dresses the same reaction up as a judgement of quality.

When slop is a fair call

I should say up front that some AI generated code really is slop. Open source maintainers in particular are being buried under pull requests and bug reports where the person submitting them clearly has no idea what they are submitting. Generating that stuff costs the submitter almost nothing, while reviewing it costs the maintainer real time, and that imbalance is a legitimate thing to be angry about.

Look closely at those cases though and the failure is the same one. The person who generated the code didn't read it either. They asked for something, got something back, and passed it on without understanding it. So the TLDR problem exists on both sides. There is the person who produces code without reading it, and there is the person who dismisses code without reading it. Neither has actually engaged with what is in front of them.

One working definition I came across describes slop as code you can't work with, that is, a codebase that nobody understands well enough to work with effectively. I think that is about right, but notice what the definition hinges on. It is about understanding, and understanding is not purely a property of the code. It also depends on whether anyone has put in the effort to acquire it.

A simple test I now apply to myself is to ask whether I would have said the same thing if a new colleague had written the code. If the answer is that I would have sat down, read through it, and given proper feedback, then reaching for "slop" instead is a sign I am taking a shortcut.

Every day is onboarding day

So why would people take that shortcut? I don't think it is simply laziness, or at least not only laziness. I think it is fatigue, and it is worth being honest about where that fatigue comes from.

Think back to the first few weeks of starting at a new organisation. You arrive and there is an existing body of software that you didn't write and don't understand. Before you can do anything meaningful you have to learn what it does, then how it does it, and then why it was built that way. It is a steep learning curve for anything you didn't already know, and it is mentally exhausting. Most people I know come home from their first week in a new job completely wiped out.

Working with AI generated code can feel a lot like that, except it doesn't stop after a few weeks. Every day there is another pile of code you didn't write, built on decisions you didn't personally make. In some respects every day becomes your first day.

It has always been harder to read code than to write it. When you write code yourself, the understanding builds up as you go, almost as a side effect. When you read someone else's code, you have to reconstruct that understanding after the fact, from whatever clues they left behind. With AI doing much of the writing, more of the job has shifted to the harder of those two activities.

There is also an old observation from the study of automation that applies here. When you automate the routine part of a job, the human is left with the part that is harder, which is monitoring what the automation is doing and stepping in when it goes wrong. And because the human is no longer doing the routine work, the hands-on skill needed to step in can quietly fade away. Supervising an AI that writes code is a textbook case of this.

The gap that builds up has been given a name, comprehension debt, meaning the growing difference between how much code exists in a system and how much of it any human actually understands. Like any debt it accumulates quietly, until the day something breaks and nobody knows where to start.

Faced with all that, it is much easier to throw your hands up and declare the whole thing slop than to admit that you can't keep up, or that you haven't been willing to put in the time to learn it properly. I get the temptation. I just don't think it is an honest answer.

Agentic engineering and the conductor

What all this points to is that what it means to be a software developer has changed. The term that seems to be sticking for the new way of working is agentic engineering, the idea being that you are mostly no longer writing the code directly, but orchestrating AI agents that do, and providing the oversight.

The analogy I keep coming back to is a conductor of an orchestra. A conductor isn't playing every instrument, but no one would suggest a conductor can get away without a deep grounding in music theory. They have to read the full score, know what each instrument is capable of, hear when a section is out of tune, and know how to fix it. For a developer working with AI, the equivalent of music theory is the fundamentals of software development, architecture and design. You may not be writing every line, but you still need to know what good looks like and recognise when something is wrong.

The analogy does break down in one telling way though. A real orchestra rehearses the same piece over and over before a performance, and the conductor builds up an intimate knowledge of it. With AI the orchestra is effectively playing a new piece every day, with no rehearsal. Which brings us right back to the onboarding problem.

It is also worth noting that reviewing and steering code you didn't write was always part of the job of a senior or lead engineer. People generally grew into that role over many years. What has changed is that everyone is now being put in that position, from day one, often without the experience to back it up. That raises an awkward question about how people new to the industry are meant to learn the music theory if they never get to play an instrument.

I struggle with this too

I don't want to give the impression I am lecturing from on high here. I now lean on AI quite heavily for some new projects, and I struggle to keep up with the constant flow of new code and new information as much as anyone.

I have tried different approaches. For a while I used Obsidian as a knowledge base, a sort of second brain, to track design decisions and how things fitted together. For some projects that worked well. For others I got so deep into the design, and then into directing the AI to write the code, that going back to get it all into Obsidian felt like more work than it was worth. The end result was that I was back to flying by the seat of my pants and trying to keep it all in my head.

Looking back, I think the reason it failed is that the note taking was a separate chore. Anything that sits outside the actual work is the first thing to get dropped when you are in the flow of things. Whatever approach works has to be a by-product of something you would be doing anyway.

Having the AI teach me what it built

The approach that has worked best for me so far, at least for projects that I expect other people to use, is to have the AI generate workshops.

The process goes something like this. I have the AI build the software, and keep directing it until what it has produced matches the idea I had in my head. At that point I have it generate a set of hands-on workshops teaching how to use it, which run on my Educates training platform or in the JupyterLab workshop extension I wrote. Then I sit down and work through those workshops myself.

In other words, I deliberately put myself through the new employee onboarding. Rather than trying to absorb the code in bits and pieces as it flies past, I get a structured path through what the software does and how to use it, at a pace where I can actually take it in.

What I didn't expect was how good this is at exposing problems. When you work through a workshop as a user, you very quickly see where the design falls short, where the experience is clumsy, or where things just don't work. The AI also contributes here. If, when generating the workshop, I tell it to highlight problems, it will report when it couldn't get something to work, when it ran into what looks like a bug, or when the documentation doesn't match what the code actually does. So the AI generating the workshop is acting as a QA and user experience tester, and so am I when I work through it.

Because the workshops are interactive and the steps are actually executed, they also behave a bit like tests. If a step in the workshop fails, that is a real problem with the software, not just a typo in some documentation nobody will read.

And then there is the secondary benefit, which is that the workshops exist afterwards for other people. A number of them are available in the workshops area of this site. My hope is that they can also encourage people to get past the mental block of TLDR, or AI slop, or whatever they want to call it, and be prepared to sit down and learn something properly. That seems more important than ever at a time when most people will feel it is easier to just ask an AI to do it for them.

Still working it out

I still don't have a workflow I would call good for using AI and keeping on top of what it is doing. I regard myself as fairly conservative when it comes to tools. I don't go chasing every new fad, because to be frank that usually ends up being a complete waste of time. I suspect most people are similar, and adapting is a slow, incremental process of personal experimentation to see what works for them.

The AI tools themselves, including the tooling from the AI vendors, have certainly made it easier to do a lot of this without reaching for other tools. It is still a matter of finding what works for you, though, and right now things are moving too quickly for anything to settle.

One principle I am trying to hold myself to in the meantime is to not let the AI get too far ahead of my own understanding. A new employee is given time to ramp up, and nobody expects them to be onboarding forever. If the code is arriving faster than I can understand it, the answer is to slow down and make time to learn it, not to stop reading and call it slop.

Hopefully one day I will have it sorted out. Fingers crossed.

27 Oct 2026 12:00am GMT

I have a PineTime. It is a cheap, open, hackable smartwatch from Pine64.

It's not an Apple Watch

When people see it, they think: that looks like a smartwatch, so it must be a smartwatch. Like an Apple Watch. It's not! An Apple Watch costs around 700 USD, and this one 25. Think of it as a programmable Casio with a color display, Bluetooth, a step counter and a heart rate monitor.

It really does not have much CPU or memory: a 64 MHz ARM Cortex-M4, 64 KB of RAM (yes, kilobytes), 512 KB of flash for the firmware and 4 MB of external flash for files. After the Bluetooth stack and MicroPython itself have had their share, about 12 KB of RAM is left for the apps. Everything on this watch has to be small.

The screen is fine, though: 240×240 pixels on 1.3 inches, about 260 pixels per inch. So if anything on it looks blocky in this post, that's on purpose. However, you can't have an always-on display: the battery would run out too soon. Apple Watches once had the same limitation, too!

The watch features an app for aforementioned watch monitor and step counter, and you can send notifications to it via Bluetooth.

How I got here

I bought my PineTime in March 2023, for 26.99 USD. It ships with InfiniTime. That's fine. But InfiniTime is written in C++, and I run a Python group. So I preferred wasp-os, a MicroPython-based watch OS. However, even in 2023 that project was already quite dormant. It was running a 2019 version of MicroPython and was quite hard to build. Recently its maintainer added a note that wasp-os is looking for new maintainers.

Reviving the OS

Then I found my watch in a drawer again, and thought it would be nice to work on it a bit. So I took it upon myself to get wasp-os into shape again. Besides moving to mainline MicroPython (more on that below), this is what changed:

• It builds again. The Python tooling is managed with uv, so setting up is one command, and the full build works again, including the bootloader bits.
• Faster flashing. Updating the firmware over Bluetooth now uses bleak instead of the deprecated gatttool, and takes about three and a half minutes instead of over eight. When it fails, it tells you why, instead of hiding the errors, as it did before.
• More memory. The filesystem on the external flash now uses the chip's real 4 KB blocks, which got rid of a permanent 4 KB buffer. Free RAM at boot went from 7.1 KB to 12.4 KB. On this watch, that's a lot!
• It recovers by itself. A watchdog now resets the watch when it hangs, instead of leaving it dead with Bluetooth still up.
• No more buzzing. A swipe could leave the vibration motor buzzing until you reset the watch. That's fixed, and the motor is driven right now.
• Swipes work reliably. The touch controller now reports when you lift your finger, and in Settings you can swipe along a slider to change it.
• It survives a flat battery. After my holiday, where I did not bring the watch, I put it on the charger, it booted and the watch said INIT HARDWARE FAILED. A chip had kept the I2C bus stuck. Now it unsticks the bus at boot, shows the actual error if something still fails, and the clock starts at the build date instead of somewhere in 2020.

Mainline MicroPython

wasp-os ran on a fork of MicroPython from 2019, with patches for the watch. Porting it to mainline MicroPython also took a few changes in MicroPython itself, so I submitted three pull requests upstream: frozen native code, a fix for PWM allocation and an ADC sample time option. All three got merged, so for the first time wasp-os runs on mainline MicroPython, and an up-to-date version too. No fork needed! The changes are targeted for MicroPython 1.30, which is planned for December.

Two new watch faces

Of course I wanted my own watch faces. The first one is simple: the existing week clock, but in Dutch, because I'm Dutch. Dutch abbreviates weekdays with two letters, so it says ma 26 okt 2026. Remember that the memory on this watch is tiny, so there's no room for a full multilingual OS. But you can just translate a watch face, build it, and put your localized version on the watch. You'll see a picture of it on top of this post.

The second one is more fun: an 8-bit style clock. The digits come from a tiny 6×7 pixel font, and every font pixel is drawn as a block of 7×7 screen pixels for the time and 3×3 for the rest. That gives the retro look, and it needs no font data at all. Below the time and the ISO 8601 date (YYYY-MM-DD, obviously), it shows the Unix epoch, counting every second.

The Guido app

And then, as a finishing touch and to emphasize the Python angle, I created the Guido app. It shows a blocky portrait of Guido van Rossum, the creator of Python. Swipe, and you get the Python logo, then the MicroPython logo, and then the logo of our own Python Leiden meetup. One more swipe and you're back at the clock.

The portrait is 80×80 pixels in 16 colors, drawn three times larger. That fills the 240×240 screen and gives it a nice retro look. The logos are drawn sharp, at full resolution.

Flash is tight on this watch. The firmware has about 330 KB to work with, and a single full-screen picture as raw pixels takes 112 KB, a third of that. So I made a small image format of up to 16 colors per picture plus run-length encoding, with a decoder of about 40 lines that draws them straight to the display. All four pictures together now take under 12 KB. Still, that's too much for something this silly to include by default, so the app is optional: you have to add it to your build yourself.

The original photo is by Kushal Das, taken at PyCon US 2024 and licensed CC BY-SA 4.0, so the blocky version is shared under the same license.

Coming up: the PineTime Pro

Pine64 is working on a successor: the PineTime Pro. It gets a 410×502 AMOLED screen, GPS, a dual-core Cortex-M33 and a digital crown. It's still early, there's no release date yet, and the original PineTime keeps being made. But that would be a much bigger playground for MicroPython. Guido in AMOLED!

Try it

Everything is in the pinetime-upstream-micropython branch of my wasp-os fork.

Discussion on Hacker News

26 Oct 2026 10:00pm GMT

News

Python 3.15.1, 3.14.9 and 3.13.17 are now available

Expedited security releases on October 21, carrying pip 26.2.2 and certifi 2026.10.19 to drop the compromised eMudhra CA. 3.13 is the odd one out: it had just entered security-only mode, where pip is not normally updated, so the team cut fresh installers anyway. 3.15.1 is also 3.15's first maintenance release, with roughly 330 fixes from 117 contributors.

Information for Python users regarding the eMudhra / emSign CA certificate security incident

The emSign root was pulled from CA bundles after eMudhra's intermediate keys turned out to be insecure, leaving older clients open to man-in-the-middle attacks against PyPI and python.org. Upgrade now: certifi 2026.10.19, pip 26.2.2, uv 0.13.4, sigstore 4.5.1, in every virtualenv and container image. pip is the one that matters most, since it always trusts its vendored bundle and distrusting the CA at the OS level will not help.

Eight Bits of Trust: eMudhra's CA Keys Cracked in Milliseconds

The technical story behind the advisory, and it is remarkable: each P-384 private key was 47 zero bytes followed by one non-zero byte, so Scott Helme recovered all eight in under a second on a laptop by trying 255 candidates. eMudhra blames a key ceremony script that generated a second weak keypair, picked up for the CSR because the names looked alike.

Wagtail CMS News

Wagtail 8.0.1, 7.4.4, and 7.0.10 security releases

All three landed on October 20 with the same four fixes: an authorization bypass across collections via privacy settings, formula injection in XLSX exports, mishandled ancestor restrictions in the Pages admin API, and stored XSS through third-party HTML editors. 8.0.1 adds a fifth, access restrictions on the API v3 pages endpoint, plus a batch of API v3 write fixes. Also out: 7.4.4 and 7.0.10.

Wagtail 8.1 RC1

The release candidate adds Python 3.15 support and drops 3.10. API v3 gains write support for TypedTableBlock and nested inline relations, an ImageRenditionField, and workflow operations for pages and snippets. Redirects are now auto-created only when a live page moves or is renamed; set WAGTAILREDIRECTS_AUTO_CREATE = 'always' to get the old behavior back.

Django Software Foundation

Quarterly contributor outreach

Tim Schilling posts the Steering Council's list of what needs hands for 6.2: testing CompositeField, plus review on a DEFAULT_PK_FIELD setting, moving models between apps, partitioned cookies, and HTTP QUERY support. A second list covers ideas needing technical vetting, including a task timeout API and import maps.

Django Steering Council Meeting, October 19

This week's agenda: the quarterly outreach message, the Sovereign Tech Agency Resilience Fund, the VectorField DEP, Memcached clients, Jazzband updates, and suggestions for the new Triage and Review Team.

Updates to Django

Last week we had 15 pull requests merged into Django by 10 different contributors - including 3 first-time contributors! Congratulations to Sanskar Kharya, Aman Kumar Jain and Helmi Lakhder for having their first commits merged into Django - welcome on board!

New in Django 6.2:

• Streaming responses now include configured CSP nonces in their headers, even when unused in the body. Inline scripts or styles that previously passed without a matching nonce may now be blocked or reported.
• A CSP directive whose only source is NONCE is emitted as 'none' when the nonce goes unused, instead of being dropped from the header.
• Directories created during file uploads and by the file-based cache backend now combine the requested permissions mode with the process umask, matching os.mkdir. A typical 0o022 umask turns 0o775 into 0o755, and the set-group-ID bit is no longer guaranteed, so set it on the parent directory instead. FILE_UPLOAD_PERMISSIONS is unaffected.
• The warning logged when a database error occurs while validating model constraints now escapes the message and trims it to 4096 characters.
• Python 3.15 compatibility is now documented.

New in Django 6.1:

• Fixed a crash in makemessages on Windows with non-ASCII filenames on Python 3.15+, or with UTF-8 mode enabled on older versions (#37447).
• The undocumented get_success_url_allowed_hosts() and get_redirect_url() methods from RedirectURLMixin now accept an optional request argument.

Sponsored

Django for Beginners, 6th Edition

The sixth edition was published this week, available in digital and paperback formats. It features two new chapters, completely updated for Django 6.1+, and more. Read the first four chapters for free to try it out.

Django Fellow Reports

Django Fellow Report - Jacob

Chasing a forum thread led Jacob Tyler Walls to merge a fix that reduces the cost of building many-to-many related managers. He also spotted the last tweaks needed for multiline template tags, drafted a security fix, and triaged a JSONField RecursionError on deeply nested input.

Django Fellow Report - Natalia

Natalia Bidart kept the djangodocs Sphinx cleanup and the DEP 20 release checklists moving, including CVE records that can name the first affected release per branch. The Triage and Review Team held its first meeting, and the Security and Triage teams both gained members.

Django Fellow Report - Sarah

A short week around coughs and colds, but Sarah Boyce landed CSP nonces in headers for streaming responses, corrected verbatim-block tokenization, and enforced integer expiry times for file-based sessions. She also added Lilian Tran to the security team and Shai Berger as a triager.

Python Software Foundation

Announcing Python Software Foundation Fellow Members for Q3 2026

Aldo Caneo, Donghee Na, Paul Moore, and Takayuki Shimizukawa join the Fellow roster, recognized together for maintaining libraries, writing educational material, organizing events, and mentoring. Q4 nominations close November 20, 2026; email psf-fellow at python.org with the nominee's accomplishments.

Checking email reputation, and allowing Outlook again

PyPI blocked outlook.com and hotmail.com outright in June 2024 after a bulk malware campaign, which caught legitimate users too. A donated reputation service from UserCheck now judges addresses individually, so both domains are allowed again; in its first month it rejected 62 disposable domains and 120+ throwaway addresses.

Wikis: Relaunched!

The Python, PSF, and Jython wikis went read-only in February after AI scrapers overwhelmed hosts partly still running Python 2. They are back as static sites on GitHub Pages, edited by pull request against psf/wiki, with write access going first to the old wikis' known stewards.

Articles

Django's Brightest Timeline

Tim Schilling argues Django's feature process has quietly come together: new-feature ideas live in a GitHub repo where emoji reactions act as votes, proposals get asked to prove themselves as third-party packages first, and Django Probe plus Praful Gulani's experimental-features GSoC work could decide what lands on usage data instead of advocacy.

Django: check version support with norwegianblue

Hugo van Kemenade's eol command pulls endoflife.date into a color-coded table, so eol django shows release, support, and end-of-life dates at a glance (6.1, 6.0, and 5.2 supported, 6.0 security-only). uv tool install norwegianblue, and it speaks Markdown, reStructuredText, and JSON for docs and scripts.

Your workers open a new Postgres connection for every task

Django's default CONN_MAX_AGE = 0 means every task a worker picks up opens its own Postgres connection, about 5ms, longer than claiming and running the task on a warm one. Set it to 600 or use a connection pool before you benchmark anything. The numbers come from django-ox's own vendor.

Automating Django deployments with Deployer

Stein Ove Helset replaces his SSH-and-run-the-commands routine with Deployer, a PHP tool that is happy to run manage.py: dep deploy builds a fresh release with its own virtualenv, runs checks, migrations, and collectstatic, then flips a current symlink and restarts Gunicorn. Shared .env and media survive between releases, and rolling back is one command.

How I decide which Python and Django versions I want to support

Matthias Kestenholz splits the question by context: projects track the current stable release or a pre-release, while the more foundational a library is, the further back it should reach. He drops versions to prune CI cost or to use something new, and lands on Python 3.10+ with Django 3.2 or 4.2+ for most packages.

Events

IndyPy: Django & FastAPI: What's Next?

Tuesday, November 24 at 7pm ET, hybrid from E-gineering in Indianapolis or over Zoom. Tim Schilling covers what's new in contributing to Django, and Calvin Hendryx-Parker argues batteries-included against speed in the Django/FastAPI debate. Register on Meetup for the in-person seat or the Zoom link.

Podcasts

Django Chat #208: Django in Space with Loes Crama

Loes Crama is a software engineer at cosine Measurement Systems, contributing to ESA's NewAthena mission. We discuss her lifelong interest in space and how her work combines software engineering, data engineering, and data science.

Videos

Django in Space with Loes Crama

Loes Crama is a software engineer at cosine Measurement Systems working on ESA's NewAthena mission, the subject of her DjangoCon Europe talk on how Django helps build the largest X-ray observatory yet. This Django Chat episode covers how she arrived at a job that combines software engineering, data engineering, and data science.

Sponsored

Reach 4,300+ Engaged Django Developers

Sponsor this newsletter to reach an active community of Python and Django developers.

Django Job Board

The PSF is hiring its next CPython Developer in Residence, alongside two senior Python roles, one AI-leaning in the US and one in Barcelona with relocation support.

CPython Developer in Residence at Python Software Foundation 🆕

Senior Software Engineer - Python, React & AI at Camden Kelly Staffing 🆕

Senior Python Engineer (Barcelona-based / Relocation support to Barcelona) at Manychat 🆕

Projects

django-ses/django-ses

A Django email backend for Amazon's Simple Email Service.

ivelum/djangoql

Advanced search language for Django admin.

Sponsor Django News

Reach 4,300+ Django developers every Friday. See sponsorship details and rates.

23 Oct 2026 3:00pm GMT

How I decide which Python and Django versions I want to support

A recent post on Mastodon got me thinking about what my philosophy for dropping support for old versions of Python and Django is. Here's the post:

🚨 Important Announcement 🚨

Going forward all my packages will only "officially" support maintained #Python versions. Meaning since October only 3.14 and now 3.15.

This has no effect on Django version support.

CVEs will be backported to older 🐍 releases.

- @codingjoe

I tend to support versions long past their End of Life date. I jokingly said that I should spend time upgrading old websites instead of spending time writing a rationale why not. I don't want to argue for my position, but maybe this post will serve as the place to point people towards when the question arises again in the future.

The short answer: it depends on whether I'm working on a project, on my own machine or on a library.

Projects

When working on projects I'm always updating to the current stable version or even the bleeding edge. I want the projects to be as future-proof as possible and I also want to give early feedback when stuff breaks.

As long as the server software allows it, I'm just going to use uv to install the newest Python version that all the packages I'm using support (or at least ship binary wheels for), because I enjoy not compiling software when deploying too damn much.

Developer workstations

When it comes to developer workstations I don't really have the patience for outdated software. I don't see a reason why developer tooling can't be up to date. This came up recently in the django-debug-toolbar issue tracker. I'm totally fine with requiring an up-to-date version of Python for our pre-commit hooks, for example.

Libraries

The more foundational and widely used a library (potentially) is, the more important it is to me that it supports basically everything which is reasonable.

I'm still using my libraries in older projects myself, so they have to work there. I want to use them in all environments, at least when I'm not paying a big price in terms of language features.

Other people may be stuck on older versions too. People can upgrade their software, but there may be situations (for example in managed hosting environments) where people cannot freely update the underlying managed server software. That isn't necessarily a problem security-wise as long as there are other measures in place protecting those services.

When people use my libraries, I might get additional eyes on the code, and possibly even contributions.

Supporting a large range of Django versions, for example 3.2 (reached EOL on 2024-04-01) up to the current main development branch, can be surprisingly easy when you do not reach too much into Django's internals or depend on features added only recently.

That said, when do I actually drop support for an old version of Python or Django? Generally speaking, I have two reasons:

🌳 Keeping CI time down 🌳

I like pruning the CI matrix so as to not spend too much compute and energy. The exceptions are packages such as django-debug-toolbar and django-tree-queries: they are widely used and have to work on a wide range of Python and Django versions and database engines, so the additional CI runs are worth it.

When I remove a version from the CI matrix I mostly raise the minimum version in the package metadata as well. I almost never use upper bounds for version ranges, but lower bounds I definitely do use.

Using new features, either because I want them or because I need them

Sometimes, I just really want to merge dictionaries using the nice a | b syntax. If this reasoning sounds whimsical to you, it is, but only partially. Programming cannot just be a job, everybody needs to have fun sometimes.

Sometimes, I really need some API or behavior Django has only introduced recently, such as the way bound fields work in forms. For example, form validation didn't go through bound fields before 4.0.

What that means right now

Most of my packages currently require Python 3.10 or newer and Django 3.2 or 4.2 or newer. Some of the less actively developed ones still go back as far as Python 3.8, simply because I haven't had a reason to touch them. Python and Django support are linked: Django 4.2 supports Python 3.8 to 3.12 while Django 6.0 requires Python 3.12, so the oldest Django version I support pretty much decides the oldest Python version worth testing.

What is the downside?

Supporting old versions makes it easier for people to put off upgrading. Some people just don't want to upgrade at all, and my libraries continuing to work on old versions certainly doesn't push them to do it. When they suddenly have to because all hell breaks loose (because of security problems or something like that) it's so much harder: they have a long way to go until they reach supported versions. That said, in my experience that's not a big problem with Django at all. On the other hand, when the projects still use npm packages from 2015… oh my.

The code is also less clean than it could be. Supporting old versions means compatibility shims, and it means not using some of the nicer features newer versions offer.

In the end, both approaches are reasonable, and both have their costs. You should do what brings you more joy.

21 Oct 2026 5:00pm GMT

Creative Work

Writers went on strike to get protections against "AI". Thousands of artists have signed open letters in protest of "AI". There are so many copyright lawsuits from creative industry groups against AI that there's a whole dedicated website for it. Popular YouTubers absolutely hate it. If they're also musicians, they REALLY hate it. Across all creative industries, there is a concerted push to reject this technology.

Yet, almost unique among creative fields, many experienced programmers remain convinced that it's fine to use "AI" for programming. We do seem to hate it, and what it's doing to our industry. It's making us all miserable, but we are using it anyway.

A lot of the justification of this resignation seems to be to be because programming is not Art. If the tool can get the job done, and the job is just functional, then why does it matter?

It does matter, though. It matters because we shouldn't be using AI to produce Art, and programming is Art.

Art can be Mundane

Some people will say that programs cannot be art because programs are functional, rather than being expressive. Programs are mundane whereas art is transcendent.

This is based on a distorted understanding of what Art actually is.

In John Berger's "Ways of Seeing", he names this type of distortion "mystification". His example of this process is both amusing and illustrative. I encourage you to read it in its entirety.

In summary, though: Berger critiques the florid prose of an art historian describing a commissioned group portrait, including phrases like "subtle modulations of the deep, glowing blacks" and "harmonious fusion". The portrait is described as sublime, in nearly ecstatic terms.

Berger reveals that the reality of this portrait is that a poor old painter needed some work, and some officials probably thought it might be nice to have an official portrait. So they paid some money to the poor old man, and he painted it, and then they had a painting. It's a well-executed portrait of a group of people. Beautiful, even. But it was work commissioned for a fairly mundane purpose and it suited that purpose just fine. It was not, and is not, a divine relic.

Culturally, we are prone to mystifying painting, and sculpture, and film, and music. We imbue them with "subtle modulations". We ignore their functional aspects - we desire decoration, amusement, and distraction - and focus on their emotional impact.

Don't get me wrong: I love me some good aesthetic philosophy. I think it's great to really examine our reactions to artwork and to try and gain a deeper understanding of our culture and our selves through media analysis. If anything we really need to do more of it.

This does not mean that the creation of such works is mystical or that it should be venerated beyond any other sort of labor.

Not least of which other types of labor that are adjacent to, but not as culturally venerated, as fine art. We tend to mystify the work of a novelist, but to denigrate the work of a journalist. In reality, the functional prose of the journalist is no less important and deserves no less respect.

Although they might be far below the ethereal realm that novelists inhabit in our collective imagination, even journalists receive more respect and thus more mystification than lowly copywriters. Yet, there is no transcendental distinction between "novelist" and "copywriter"; the many of the skills are the same, and the distinction is merely an accident of commerce and opportunity.

In fact, many famous writers have famously inhabited both roles. This is not an accident! Working with words professionally, even (perhaps especially) mundane words, is excellent practice for working with words in a more purely artistic context, because even mundane creativity is still artistic.

Code can be Beautiful

One thousand Internet years ago, when I was in my late teens, I would describe myself as a "code poet". I was relentlessly mocked for this as what the Youth would today call "being cringe", and at the time was referred to as "pretentious".

I succumbed to the peer pressure, removed it from my email signature and my bio. While I still believed strongly in the parallels, I accepted that - socially, at least - comparing code to poetry, or indeed to Art, was a silly thing to do.

However, I never abandoned the idea, in my heart.

The thing that I am most well-known for, the invention of Deferred, was specifically an aesthetic reaction to the tedium of passing callback and errback parameters to every single remote procedure call in an RPC client/server application. Those two callbacks got the job done just fine. But they were ugly, and annoying to work with.

Deferred is an intentional poem about asynchronous task execution, with a deliberate eye to the aesthetics of the problem and the experience of using it. It was influential because of its focus on aesthetics.

I do not want to overstate the beauty or profundity of this minor contribution, or indeed its durability. That a poem exists does not mean it is a great poem, merely that it is a poem.

Our aesthetic culture around programs is more like folk epic poetry than fine art, so the influence of this contribution is less about its specific enduring power than it is about its influence on what came next; from MochiKit.Async to JQuery Deferred to JavaScript Promises and eventually to async/await; a long chain of different artisans each adding something of their own until the original has all but dissolved. (And I wasn't the "original" here, either, as I drew heavily from the E language's Promises, among other things.)

In order to make code into a deliberate artistic expression, one must have spent quite a bit of time contemplating the problem domain. Without having experienced the tedium of manually passing a thousand callback parameters, I would have had neither the skill, nor indeed the motivation, to bother creating such a thing.

Now, most code does not have to be like this. Most code does not get to be like this. Most code is functional, workday code. Most code could not make a lady weep. It's just copy-writing, if you will.

As I explained previously, most writing couldn't do that either. Most writing is just copy-writing, too. Most visual art is advertising. Most live music performance is background music in bars that will go largely ignored.

However, code that is intentionally aesthetically designed tends to be important, both socially and technologically.

We do have some tradition of self-mystification in software. As Abelson memorably put it, "Programs must be written for people to read, and only incidentally for machines to execute.", so we have long had some conception of programs as highly expressive, even if we can't always agree on what they're expressing or to whom. We will occasionally wax poetical about the philosophical implications of a particular piece of software. This is not unique to a single piece of software, either; more than one community has indulged in similar philosophizing.

The expressive and aesthetic qualities of software are not limited to reading source code or interacting with other programmers via APIs, either. For example, every year, Federico Viticci does a review of Apple's new operating system, which is (among other things) an aesthetic critique. Such a project would not be possible if the software did not have an aesthetic impact on its users.

Not to mention that every video game review is also a software review.

Defend The Mundane

If we use AI to erase all the copy-writing, all the graphic design, all the boring mundane art, and yes, all the boring custom WordPress theme development, then we will be removing all the practical opportunities for the vast amounts of practice and contemplation required for people to elevate their craft to eventually achieve great things. Education is great, but the majority of true skill development happens on the job and always has.

This doesn't mean that we can't use abstractions, or automation, to make our work easier. Programming is the art of abstraction, of understanding how to compose smaller ideas into bigger ones, of how to understand the automation of a larger system by understanding the rules that automate smaller ones and understanding how to combine them.

When we use "AI" to eliminate that understanding rather than raise it up to a higher level, to entirely destroy that creative decision-making process, we do a disservice both to ourselves as programmers and to our users. We would be doing a disservice to our users and our downstream fellow developers in the same way that a visual artist would be doing a disservice to their viewers or a musician would be doing a disservice to their listeners if they served them auto-generated filler instead of their own creative output.

Slop is slop, no matter the medium.

Each mundane project has some tiny chance - let's say, something like 0.1% - of achieving greatness. If we do a single project with AI, then sure, whatever, there's almost no chance that that project was going to be the one hit to create that career-defining moment for an engineer working on it. If we make a habit of doing all projects that way, though, we take the total likelihood of those moments of greatness from "definitely sometimes" to "never".

The precisely appropriate ways in which to resist AI encroachment on all software development lie well beyond the margins of this one short post. How much you can resist and which specific uses you should resist are up to you. But it is worth resisting in software just as much as it would be worth resisting in any creative medium.

Programming isn't special. It's just Art, and Art is the most human - and thus, the most universal - thing that there is.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!

09 Oct 2026 7:00am GMT

One of the issues that I have with the current generation of "AI" products is that they do not appear to take their own premises seriously. I look at a plethora of obsequious chatbots claiming to be serious tools for problem solving, and I think, this is not what a problem-solving tool would look like.

Even before we get to the tremendous ethical problems with the frontier labs, it is this impression of their composition as a product that makes me feel, constantly, whenever I am interacting with them, that they are less a software product than that they are a grift, a scam designed to make me feel like I am interacting with a product that has capabilities that it simply does not, to try to lull me into a false sense of security that I can trust it.

The frontier labs are of course the worst offenders, but every criticism here applies just as much to Ollama, which (if anything, due to the obviously poorer quality of the available models themselves) needs these features even more than the frontier labs do.

Here, I will set down a few features that might convince me that an LLM-based product, particularly one focused on research or software development, was actually serious about helping me do useful things with it.

Make "Checking For Mistakes" A First-Class Feature

This is the biggest issue, and the major reason that I was inspired to write this post.

It is a truth universally acknowledged, that AIs cannot reliably provide information.

I could cite a ton of news articles and studies about this fact, but there is no need. Every single chatbot admits this, up front, in a fine-print disclaimer as a core part of their user interface. Gemini says "AI can make mistakes, so double-check responses", Claude says "Claude is AI and can make mistakes. Please double-check responses.¹" ChatGPT says "ChatGPT can make mistakes. Check important info.".

Every time I see that last one, I wonder how I'm supposed to know what "info" is supposed to be "important".

All of these warnings are all small, gray text, painfully obviously included as legalese to push responsibility back onto the user rather than to help with anything. This is a core limitation of all these products. Checking their output is a part of the workflow for using them that:

1. you absolutely cannot skip or skimp on without creating risks to yourself and whoever you are conveying its output to, and,
2. it is very easy to skip or skimp on and you are encouraged at every turn to do so, because "just trust the output" is one of the quickest ways to save time.

A chatbot product that took this weakness seriously, as an actual consideration for using it, would put a checkbox next to every claim in its output. It would be a 2-column worksheet, where you've got the LLM output in the first column, and next to it, human notes in the second column, explaining what work went into checking this claim, and a big checkbox that you would only check off after you believe you'd checked its claims thoroughly enough.

Coding assistants would need to have some version of this as well. Right now, this is pushed off into code review, which means it is a dark pattern which subtly encourages the "author"² to offload this work to their code reviewer without ever looking. Once again, "it's probably fine, I don't need to check" is the quickest way to save time and churn out those PRs faster.

It might even be useful for coding harnesses to have some affordance for checking code before it even runs tests. As the vendors themselves have admitted, it's not just expensive to burn tokens on your "AI", you also end up burning far more compute on the AI. Being able to check your diffs before sending them over to uselessly exhaust your testing compute cluster would be useful.

If your product tells me that it makes mistakes and I must be the one to check for the mistakes, but then gives me zero tools to check for mistakes, I cannot take it seriously.

More Citations to Check, And More Details

Most chatbots prefer to give an answer, rather than a citation. In my own personal use, I find that when asked to provide a list of citations with clearly marked sources for each one, they will appear to "get bored" halfway through the list and simply stop including citations at some point.

When the bots include citations at all, present them as inline annotations that say nothing but the domain name of the search result, in a font so small that it's barely legible, and an equally indecipherable icon that is fewer than 16 pixels on a side.

This is backwards.

Now, I am aware that these citations do come from somewhere, and in an attempt to reduce hallucinations, all of the major providers support some form of "grounding"³, and that those little barely-readable citation links are referencing actual structures in the RAG pipeline and not just potentially-hallucinated tokens, but I'm not talking about the underlying machinery in the model, I'm talking about the presentation to the user.

Plus, regardless of whether a snippet of text came from a RAG query, we know that LLMs can never provide an authoritative result; it's a fundamental limitation of the technology. They can still garble the results of RAG as much as they can misrepresent any other training data. This means that it must never present its results as authoritative.

If you ask an AI to do research queries, every result should be presented as a list of citations. Moreover, the presentation should display each citation as a large object of in its own right, with clearly identified metadata, including not just the site where it was found but its publication date and, if possible, the name of the author. The literal, unmodified quotation (not from RAG, not a summary: a quotation extracted with a regular program and not an LLM) should be front-and-center, larger than any AI-generated text.

If the AI product wants to editorialize or summarize (which should not always be necessary!), the AI-generated text should be presented as small text underneath the citation that has been found, de-emphasized as much as the disclaimer is right now, at the very least until the user has verified that the summary is accurate. Perhaps, for a research project, a "did you read the citation" checkbox might even be helpful.

If your product openly tells me that it will scramble, misrepresent, or omit its citations in its summaries, and I must read the original human-authored citations to be sure, but then gives me no tools to track my reading of those citations or even any way to find them, I cannot take it seriously.

No First-Person Output, No Apologies

There is no reason for a software development or research tool to use first-person language to describe itself. They should not do so. In fact they should not be allowed to do so.

There is also no reason that they should ever apologize. It is a waste of everyone's time; it's a waste for the chatbot to generate the apology, it's a waste for the user to read the apology, and it's a waste for the user to respond to the apology. Yet they unfailingly do this upon every correction.

The vendors of these tools know that they are routinely causing mental-health crises. In response, they have added non-functional "guard rails" that can still, in 2026, easily be bypassed.⁴

A product seriously interested in helping with productivity would correct this glaringly obvious flaw, focus on the task at hand, and stop emitting useless verbiage.

In the previous two sections, I tried to focus on ways in which the harness would be constructed differently even if the LLM technology is fundamentally impossible to improve; in this case, I have to assume that the labs have some control over the model itself. But unless they are truly incapable of influencing their output (and all their "benchmarks" and "capabilities" seem to indicate that they can control it very tightly) they ought to be building models that are much less verbose.

More Non-Natural-Language User Interfaces

Although natural language could hypothetically be a powerful interface for interacting with a computer system, the practical upshot of LLM natural language interfaces is that these interfaces are imprecise and repetitive, full of superstitions masquerading as "best practices". The inputs are a mess and the resulting outputs are a mess.

The general way of addressing this unstructured mess is to allow the chatbot to directly take action in response to the user's input; in other words to supply it with "tools" via an MCP server. But again, this is backwards. If we cannot even express our intent clearly in the first place, why are we trusting this system to take potentially destructive and harmful actions on our behalf?

Instead, I would expect a product that was seriously invested in helping me accomplish specific tasks, to have user interfaces specific to those tasks. Is it supposed to be able to be a security scanner that can discover OWASP top 10 bugs in a codebase? Have a button for that. Build that functionality into your harness, train it directly into the model, use smaller models that can satisfy that functionality more effectively than throwing it at the planet-sized brain of Fable or whatever.

I'm aware that there are small software startups that do something like this, but they are bolted on to the side of the main model providers' APIs, not integrated into the core of the product and not using their own models and AI systems to achieve consistent and repeatable results.

Strong Data Provenance Indicators

Chatbots produce data tables pulled from websites, from APIs, from MCP tools or from summarizing and scrambling the user's input. In order to provide the illusion of a seamless interface, this data is presented in-line regardless of where it comes from. But some of these outputs are produced mechanically via regular old API calls, for example, from the result of calling a tool or querying a website, but presented uniformly.

But there is a huge difference between an authoritative data source being inlined as part of a chatbot conversation, being treated as input by the chatbot, and some ad-hoc hallucinated data being treated as output of the chatbot.

If a product is trying to help me make accurate, empirically-grounded, data-driven decisions, the source of the data is critical.

Integrated into the "check for mistakes" and "verify citations" workflow I described above, there's a necessary "verify data programmatically" pass as well; to have tools that will treat portions of the output as a regular spreadsheet, allowing regular-old computer arithmetic to verify things and showing where such arithmetic was used, and how.

Better User Control of Reproducibility

Anyone familiar with the technical specifics of LLMs will know that they have a variable called "temperature" which controls the degree of randomness that the LLM uses to produce its outputs. But most users don't know this, because it isn't exposed as part of the user interface by default.

This leads to a subjective impression that you asked ChatGPT, and you got ChatGPT's authoritative answer.

You can't just set the temperature to zero and still get useful results - I am aware that it does more than just scramble the output at random, and there are perhaps good reasons that simply exposing just a temperature setting would not be that useful to users. But if we followed some more of my earlier recommendations for making more structured UI elements to solve specific problems rather than having long back-and-forth chats where each refinement depends on the previous response, perhaps those elements could also re-play the process so that users can see how reliable the bot is at a particular task and develop a sense of how the stochastic nature of the process actually affects it.

Similarly, if a user is trying to solve the same problem repeatedly with a chatbot, and the chatbot product has numerous computational tools that don't really have anything to do with the LLM, such as deterministic data-processing tools, then having a way to freeze the non-deterministic parts of the transcript but re-populate a particular data frame with updated information and fork / continue the conversation from there would be a way to avoid introducing pointless additional randomness when you already know what tool you're trying to use.

The fact that every conversation is presented as this flat chat prompt that doesn't let me interact with any of the widgets that were previously produced except through more chatting, really makes me feel like the whole product is just doing predatory social-media style "increase time on site" optimization, just trying to lure me into further repetitive and unreliable chats, rather than letting me get in, solve my problem, and get out.

Context Visibility

Managing the LLM context is the ongoing challenge facing organizations that are trying to use "agentic" workflows. Filling up the context with too much information causes well-known problems. In response, advanced LLM users attempting to solve larger problems must break up very long prompts into "skills", give access to lengthy information via "tools", and delegating sub-problems to "sub-agents" rather than simply extending a single prompt indefinitely.

All of these strategies have flaws, because even on the largest models, compared to the breadth and depth of knowledge-work problems, LLM contexts are quite small.

And yet, none of these products will show the context to the user by default. There are third-party addons that can show you a simple progress bar but for addressing the premier engineering difficulty with this technology, that is below the bare minimum.

This lack of visibility means that almost all of the tools for extending the context are flying blind. Rather than responding meaningfully to a full context, everyone just kind of guesses how much state they need by guessing and trying over and over again with progressively more elaborate skill and sub-agent layouts. Even managing context compaction ends up being an advanced API-driven workflow⁵.

A serious product that was trying to help the user understand would not only show "available context" but explain the impact of context compactions, make it easier to see harness-generated prompts, and so on. This would be a first-class feature, combined with the aforementioned reproducibility / replay tools, would allow users to do real experiments to develop an understanding about how to make good use of the context window.

A Sandbox That Actually Works

I've been focused on the chatbot interface here because it is the most immediately egregious upon looking at the UI. But the "agentic loop" tools used for coding are equally dangerous, if not more so. Coding tools keep destroying everyone's data, over the course of years.

These catastrophic incidents that become front-page news are relatively rare compared to the amount of coding-agent use out there. But they also aren't the only kind of sandbox violation. Coding models will so routinely edit test code instead of the system under test that there are "pro tips" articles all over the web giving you the flawed advice to simply ask the agent not to cheat. News write-ups of the catastrophic incidents themselves will also offer glib and wrong advice, like "use a docker container". That might prevent it from literally deleting your operating system, but it won't prevent it from destroying all the local work you have in your codebase (it needs access to a checkout, after all!)

There is a flurry of activity in the infosec space where people are rushing to plug the gaps left by these coding harnesses. Everyone's got their own version of an MCP approval gateway where you can optionally place a proxy between your agent and your production infrastructure.

In the best case, though, all these mitigations and proxies and prompts simply turn the user into an auto-approval automaton, hitting Y, Y, Y, Y over and over again, until you finally are driven mad and hit "yes to all", turn on full-auto mode and submit yourself to the void. With nothing between your personal vigilance and disaster, there are no workflows left beyond decrementing your own vigilance until there's nothing left and then hoping the disaster never arrives.

The fact that some mitigations exist that can be deployed by extra-cautious users does not change the fact that "agentic coding" is an unsafe-by-default technology deployed without concern or guidance. Every frontier lab has tied a spring-loaded shotgun to a dog; the fact that dog owners can publish thoughtful blog posts explaining how you can teach your dog the basics of gun safety or how you can have your dogs play in a bullet-proof room does not mitigate the fact that the product should not have been allowed in the first place, nor should it continue to exist without VERY strong security controls.

I might believe that a frontier lab were seriously interested in providing developers with a useful tool if they shipped something that had safety built-in.

That means tools in the harness, detached from any LLM, independent of the prompt, that could:

• sandbox all filesystem operations and strictly limit ANY deletions outside of specified scopes, regardless of operating system,
• enforce snapshotting of the entire repo on every operation for easy rollbacks and minimal lost work,
• remove the disaster of "auto mode" (not to mention nonsense like --dangerously-skip-permissions) entirely, and
• carefully consider a structure for presenting plans to the user where, rather than provoking immediate alert fatigue by asking for checks on every action, make structured plans which can be submitted to the user as a group of actions and reviewed and approved as a batch.

In the same way that I suggested above that research-based tasks should have a way of re-issuing prompts to determine how reproducible a result is, or whether other sources might be found, agent-based tasks should have a way of being executed against mock services for popular APIs, so that the verification can match both on the front-end (review the plan for making the API calls before they're executed) and the back end (review the API calls that were issued to the mock service and verify that they matched).

Instead, the frontier labs provide us products that are disasters out of the box, give us "best practices" to build massive and elaborate, as well as incomplete and error-prone, security perimeters of our own design. Then they blame "operator error" when it inevitably goes wrong. I cannot believe that these design choices are intended to help us be productive.

Bonus: Human Processes

Organizations deploying AI also frequently come across as unserious, for similar reasons. In 2023, naive exuberance could perhaps be forgiven. But today, as we near the close of 2026, there are several well-known problems, that have been extremely well-covered in the press. None of these things should be surprising, but most orgs deploying these tools are still just letting them rip and hoping it all works out.

Organizations deploying these tools would need at least three kinds of major modifications to their internal processes, if they wanted to be serious about using them safely:

1. Shift Rotations to Prevent Vigilance Decrement

There have been several high-profile incidents where software developers' gradual acquiescence to accepting LLM output have lead to serious economic consequences for the companies deploying them, perhaps best typified by Amazon's "millions of lost orders" due to a gradual decay of their engineering processes from LLM use.

These outages, and other AI-related failures, are due to the difficulty of maintaining focus on the same problems. In other words, as I described above, vigilance decrement is a constant problem, because AI outputs are most often correct, but continue to be incorrect in surprising and non-intuitive ways. As I have previously written, you cannot trust yourself to catch every bug with code review, and LLM output.

Aviation, for example, has very strict rules around rest requirements. There is also a specific rule that "No certificate holder may operate an aircraft without a second in command if that aircraft has a passenger seating configuration, excluding any pilot seat, of ten seats or more.". Other safety-critical professions have similar rules.

And yet, even in the age of the supposed "AI revolution", most software teams are still assigning every engineer a full feature load, not planning for any rest, and telling people to review code whenever they happen to have some "free time".

Maintenance of vigilance has to be your top priority. Regular, scheduled, inviolable rest periods where people do work without AI assistance, and are not exposed to any AI output for review or otherwise, would be crucial in order to stay mentally sharp enough.

The tools themselves should have this sort of thing built in. The mistake-review process described above should have a periodic spot-check mode where a second reviewer periodically reviews a chatbot log, doing their own independent verification of claims, to see if they spot the same errors. This could provide a feedback loop to determine how much rest is necessary to maintain continuous attention and actually spot hallucinations.

2. Skill Practice To Prevent Skill Loss

It is also well-known that AI use leads to AI reliance, and AI reliance leads to skill loss.

I like to use the analogy to dockworkers at a seaport⁶ adopting automation.

If you employ dockworkers to load and unload ships all day long, they are going to be getting tons of exercise. They will be able to lift heavy objects on demand, whenever. They might have plenty of health problems and injuries from this type of work, but "lack of exercise" will not be a problem.

With the development of standardized container ships and mechanized cranes, you are going to be changing their job description substantially: now they mostly spend all day sitting in a small cubicle moving a control lever back and forth, not lifting heavy stuff. They will get worse at lifting heavy objects.

In this analogy however, the cranes are not all that reliable. We know they break, and they drop their payloads sometimes, and the stuff needs to be manually moved. But this only happens a few times a week, at most. If you need whoever is driving the crane to be able to jump out at any moment and still move stuff around manually, then you need to make an affordance for that. You need to give them time to go to the gym and do some lifting for practice, or every crane failure is going to be a major emergency.

An organization doing an AI transformation would also need a massive increase to learning & development budget, both in terms of resources and in terms of schedule. If your people are going to lose skills because they've lost regular practice in the incidental course of doing their duties, then they are going to need deliberate, intentional, non-incidental practice of those skills to keep them sharp.

But rather than trying to accommodate new workflows and give time for people to adjust, most AI mandates are simply dropped on workers like a ton of bricks, with no time to adapt and no affordance for maintaining their skills. Operate the crane and stay fit and healthy and ready to switch back to manual lifting at any time and then get back in the crane cockpit right afterwards. Don't mess up.

Then an accident happens and everyone is surprised, as if this process weren't practically designed to produce a terrible result.

3. Mental Health Resources to Deal with Mental Health Risks

AI psychosis often begins with practical problem-solving, and beyond that, it can start specifically at work. Not to mention the more pedestrian condition of "AI brain fry".

If you are mandating your employees to use a hazardous tool that may seriously and directly damage their mental health, you need trainings and resources. You need in-house therapists and you need to be making sure to check in with people actively to make sure that this is not happening.

Again, the tool itself ought to have some way of dealing with this. An occasional "take a break" popup is easily dismissed; they need a user-visible AI personal dosimeter so you can see your cumulative usage over time.

I don't even know if "usage over time" is a sufficient metric to gauge risk. Maybe if your work chatbot start to talk about resonance too much, unless you literally work as an acoustic engineer, that should be flagged for someone.

We are, again, years into dealing with these tools, and we know these risks exist. Yet no serious mitigations are provided. Not even any way of measuring the risk exposure.

And More

There are also many other risks associated with the technology. There are intellectual property risks with the foundation models, due to recklessness with their training data. There are existential financial risks associated with the infrastructure build-out. The extent to which most "open" models are simply derivatives of frontier models is an open question.

What I Think

If any one of these things were regularly overlooked by AI vendors or users, that would be a totally normal product oversight. Room for improvement for the next version, but nothing catastrophic.

Shipping without any of them doesn't seem like lean product management, it seems like a careless attitude towards risk and a product design philosophy oriented entirely towards short-term demos, with no regard for how to realize actual productivity gains.

Furthermore, being available for years without anything like these features, despite hundreds of incidents demonstrating the risks, with hundreds of billions of dollars of funding, makes it seem to me like if they were to add all the features that would make their product actually safe and hypothetically useful, these features would reveal that it is actually not an improvement to productivity.

In the year since I first wrote about measuring the cost/benefit ratio of AI, I have heard from numerous people who have shown this to management to try to illustrate why their AI initiatives - like almost all AI initiatives - were either failing or burning out their engineers.

I've also heard from lots of people that have told me that it's obviously useful and they don't need to measure so carefully, because they are getting lots of work done that they couldn't have otherwise.⁷

I have yet to hear from a single person who has said "yeah, we measured according to your methodology⁸, and it turns out that our AI work is going great and that our ratio is 0.75".

Obviously, I cannot say for sure why this is; absence of evidence is not evidence of absence. But at this point I think the null hypothesis is that AI tools provide, in aggregate, zero value. They make mistakes too often, and the externalities they produce are so bad and so difficult to control that even before we get to the places where they are just physically poisoning people, even the negative effects on their direct users end up cancelling out whatever benefit to they provide to their organizations.

If I were wrong, then including tools to measure an AI's effectiveness at the tasks their users are actually trying to accomplish, rather than meaningless benchmarks, would show big productivity gains. The frontier labs would be champing at the bit to add such features, and crowing about their fantastic results.

I think the labs know that if they did that, it would present a grim picture to their users. Such tools would let their users see that it's making mistakes much more often than they realized, that they're spending much more time with it than they want to be, and that it's just generally not fit for purpose.

If they prove me wrong by adding in all of these safety mechanisms, and in the process, they make all of their AI technology less harmful, I'll be thrilled to be debunked.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!

28 Sep 2026 4:27am GMT

Open source is, at least in part, about you, where "you" refers to the user.

Open Source Is Not About "Open Source Is Not About You"

In other words: Rich Hickey was wrong when he wrote "Open Source Is Not About You" and I'm tired of pretending otherwise.

Of course he's not completely wrong, or his famous post would not have resonated quite so much in the first place. Obnoxious users who demand their personal use-cases be immediately addressed by volunteer maintainers for free should indeed be viewed as the pariahs that they are. Similarly, corporate users who want free support from the community that supplies their infrastructure to lower their costs. As should those who profit from this type of externalization by their own customers.

But the exchange of "open source" (or even "free software") is not as simple as "I have prepared some software for you, please enjoy it, you have no right to complain", and maintainers ought to have a precise understanding of the costs and benefits - as well as the ethical implications - of that exchange.

Right now we barely even articulate that the exchange exists, let alone that it establishes a long-term, subtle, and implicit relationship between maintainer and user.

Let's fix that.

What Are We Doing When We Do An Open Source?

Hickey, and his many acolytes in the years since his fateful post, asserts that the process of "open source" goes like this:

1. Maintainer makes a thing, and makes it available to users as a gift.
   1. Maintainer may "love working with the team".
   2. Maintainer may be "proud of the work we do".
2. Users accept the gift, and extract utility from it.
   1. (Users MUST be grateful for this.)
3. A tiny fraction of users reciprocally contribute to the thing.
   1. (Maintainers may be grateful for this.)

He makes various oblique references to the specific activities of his company, which does things vaguely related to his projects for money¹. These activities are exclusively characterized as for "customers", however, a subset of the aforementioned users so tiny ("fewer than 1%") as to nearly be an entirely distinct group.

Breezing past this process in an essay about obnoxious users demanding things they are not entitled to, one might nod along, as this sounds mostly sensible. Giving gifts is nice. I too love working with good teams and taking pride in things.

Examined more closely, however, it starts to logically fall apart. If you have consulting clients and that's where all of your money is coming from, why are you bothering (as he repeatedly insists) "doing [things] for the community"? What was the point of releasing this code in the first place? You could love working with your team and be proud of the work that you do in a lot of different contexts; why bother implicating this horde of entitled and obnoxious people, if that's all you're getting out of it? What's in it for you?

If we've left out something as fundamental as "why is the maintainer doing this", perhaps this story leaves out some other important bits as well.

Why Are You Doing This?

There are many possible motivations for releasing and maintaining open source software. They are often subtle, often overlapping, and rarely clearly stated. Maintainers are not a monolith and not everyone does it for similar reasons. But let's review a few reasons that someone might want to contribute.

Reputation

One reason that you might want to release some open source software is advertising. The most common form of this is self-promotion; if you are a visible, prominent contributor to an open source project, it stands to reason that you will have an easier time finding work in the domain of that project.

If you operate a consultancy, as Rich Hickey did at the time of his famous rant, then this reputational currency translates into advertising for your services. It's a practical demonstration of the skills of your team.

The trade in this benefit is most like the traditional "gift economy" that open source has been compared to. You give the code to your users, which has some value, but the users give you back some reputation, in the form of their attention, their esteem, and possibly even their money if they become customers or employers.

Influence

Infrastructure is the most popular type of open source for a good reason. Programmers working on a problem are often hemmed in by sclerotic architectural choices which prevent them from solving problems in the way that they'd prefer to solve them. Major infrastructural investments are difficult to justify in a planning process, as their benefits are hard to prove. Sometimes the benefits are highly personal; different engineers have different aesthetic preferences about what types of equally-valid solutions they'd prefer to work with.

If you can develop your preferred type of solution and release it as open source, then you can influence how everyone else solves this type of problem. As an individual, such a position of influence can allow you to have some transferable expertise between employers. You know how to use the tool you developed, so you can be very quick and effective with it, and you can shape it to your ongoing taste over time.

If you're an employer, and you can get everyone else to use your open source thing², this can reduce both your hiring and training costs. Potential employees can read the code, see that it's good, and want to work at a place that produces good code like that. They can also read the code and become familiar with it in advance of coming to work for you, which means that you have a ready supply of developers who already know how your internal systems work.

The trade in this benefit is more like "soft power" than a gift economy. You give the code to your users, which has some value, but the users give you back the ability to dictate their technological agenda. You gain both the ability to influence their initial direction, and, as part of ongoing maintenance, to dictate their behavior over time.

Improvement

As an engineer, you might want to improve your own skills. Writing something proprietary and commercial cuts against this in two ways.

First, you will want to build something that already exists within your skill set, so that it will attract commercial interest and actually be competitive. Within the context of a larger team, you will want to personally be able to be immediately effective for similar reasons. But you still need a way to learn new things.

Second, you will want to build something somewhat secretively, so that the value you are producing is captured rather than released to the community. This means that you will be cut off from external sources of expert feedback.

As an organization, you might want to build the skills of your staff in similar ways.

The trade in this benefit is code for knowledge. You release the code or changes, and in return you expect your users to provide you good bug reports, and to induce at least some of them to become co-developers.

Outsourcing

As an engineer, you can only do so much on your own. Perhaps you want to have some influence over your infrastructure so you want to write it, but you also want to have a communal place to keep your infrastructure such that you can make a change to something to suit your needs, but you know that even if you walk away, someone else will maintain that change and keep it working across years or even decades of changes to underlying platforms, hardware, etc.

This sort of communal maintenance effort can be shared among all interested participants; if a thousand companies all need the same tool, if even a few dozen can share it, that reduces even their own load massively, let alone everyone else's.

The trade in this benefit is more complex, since there's less symmetry between the main maintainer and peripheral community members who also contribute code. The main maintainer is actually trading a namespace, a central place for people to contribute, coordinate, and release changes, rather than the code. They are a sort of market maker where then all the other contributors trade code for code within that market-ish structure.

In practice, this motivation produces a game theory problem where, when maintenance drops below a critical threshold, it creates a big enough crisis that at least some freeloading stakeholders will be forced to start making contributions.

Ultimately, however, this saves all involved parties a ton on maintenance, more eager volunteers who do not freeload in the first place get all the other benefits mentioned above as well.

The Things that Aren't Supposed To Be Benefits

These categories could be made as explicit, rational trade-offs, even if they are often implicit and subtle in practice. They are transactions where the maintainer gets something and the user gets something.

However, not everything that you are getting as a maintainer is something you are actually supposed to use to your own benefit. Being given trust in service of a responsibility is not a transaction.

"Oops, All Root Shells"

Open source code is code. In our modern world of absolutely pathetic sandboxing, installing code from somebody else gives them control over your system, even if it is somewhat indirect.

There is an unwritten rule that if I create an open source library, and you use it, it probably shouldn't have a backdoor in it that gives me the credentials to your bank account. There is a trust relationship between the user and the maintainer, and here, we see the first obligation that the maintainer has. The maintainer is obligated not to use the user's computer for their own gain.

This rule might seem obvious and straightforward. It might even seem unfair to you that I call the rule "unwritten", because the rule is, in fact, written down in a few places: for example, in the npm Acceptable Content Policy, it says right there:

A few examples of unacceptable content:

…

3. Content containing malicious computer code, such as computer viruses, computer worms, rootkits, back doors, or spyware. This includes content submitted for research purposes. Tools designed and documented explicitly to assist in security research are acceptable, but exploits and malware that use the npm registry as a deployment or delivery vector are not.

I think we can all agree that a script which steals your bank credentials and sends them to me to buy a totally sick jet ski would qualify as "malware", so clearly that is forbidden.

There is also an enormous gray area here. npm also explicitly allows "Information on how to pay, donate to, and otherwise support Package development", but then goes on to explicitly forbid "Packages that display ads at runtime, on installation, or at other stages of the software development lifecycle, such as via npm scripts."⁴ How are the lines drawn around these gray areas? "npm will continue to apply its judgment when deciding what content is acceptable."

But also... this is forbidden by npm, not by the transcendental nature of "open source". I could give away code that displays all kinds of ads to its users as a "gift" on my website. The exact structure of this policy is not uncommon, but it also isn't exactly the same as other such sites. PyPI, for example, explicitly bans "cryptocurrency mining", which NPM does not. Is cryptocurrency mining "not open source"? A lot of judgement calls are happening here about what is allowable in these "gifts" that you are giving to your users.

But I digress.

My point is that policy-making around this concept is not clear, there are lots of little disagreements around the edges, but there is a very strong consensus that while the user is giving you their trust here, that is not a trade. The deal is not "you give the user some code, the user gives you unlimited compute and access to all their financial accounts". The user has made themselves vulnerable to your code on the strength of your reputation.

This creates an obligation for you to not do anything evil with that code, either intentionally or through negligence.

Security Updates Are Just Command And Control In A Funny Hat

All of this is just about the initial download of some code, and that is the way that Rich Hickey describes it, as if you just grabbed some code off a web page and put it in a folder that you like on your desktop. But that is not how open source relationships work today, if indeed it ever was.

The way it works today is that you add a dependency to your pyproject.toml or your package.json or your Cargo.toml and now your users are vulnerable not just to whatever you happened to upload in the first place, but to whoever happens to have your package index credentials.

This creates an obligation to maintain an operational security posture that protects your users from malicious updates.

The Roadmap Is Someone's Life

Another kind of trust that the user is placing in you is the trust that you are going to have at least some kind of regard for their usage of your software.

In a perfect world, the user's expectations could be clearly circumscribed. Whatever ongoing maintenance you commit to perform would be encapsulated in clear policies that you'd write up in advance, about exactly what kind of security response policy you have, how you will communicate when you no longer have the resources for maintenance, and so on.

But anyone who has been involved in any project at anything but the most extreme tier of operational maturity knows that 99% of the ecosystem relies on a set of loose conventions around how all that stuff works. We expect that maintainers will generally be around, that they'll use existing tools like an issue tracker for triaging user bugs, GHSA and CVEs for security reporting, that they will mark the project as "archived" and maybe do a final release before abandoning it, that they will maintain a ChangeLog explaining at least a little bit of what is going on.

Users assume that those conventions will be followed when there are any gaps in explicit policy, or indeed if policy is lacking entirely. This assumption is reasonable, because otherwise nobody could ever use any open source without a stack of service contracts that nobody has any time to write.

The strongest such convention is that an actively maintained program will, at least, more or less keep doing what it does as time goes on. A user who has elected to use a bit of open source software has made themselves vulnerable to changes and breakages in that software by the mere fact of using it. In the time that they have used it and invested in it, they have not invested in:

• creating alternative software to meet their needs,
• maintaining data in formats that other software can read, or
• learning how to use existing alternative software.

This can, and does, go badly wrong, when those expectations are mismatched.

How It Goes Wrong

Let's say a maintainer creates an open source paint program, OpenPaint.

An artist, known for their unique style of making blended collages, switches from their previous app, ProprietaryPaint, to this new OpenPaint to make these culturally significant works of art. However, the maintainer decides that the 'blend' tool is kind of a pain to maintain, and they remove it in OpenPaint 2.

A few months later, the artist's operating system vendor issues a security update that breaks OpenPaint, because older versions of OpenPaint were unknowingly abusing some platform API.

The maintainer releases a new OpenPaint 2.0.1 that addresses this incompatibility, but doesn't care about version 1.x any more so they don't bother to update that one.

This places the artist in an impossible situation. They can stay on an old version of their operating system, putting all their personal data at risk. Or they can upgrade to the new operating system, effectively either cutting off access to their livelihood, or forcing them to change their art style entirely.

Now, proprietary software can place users in similarly untenable positions (and in fact, it is more often proprietary software that does). But does the openness completely remove any obligation for this consideration? Should the OpenPaint team have to at least communicate the reasons for doing this, to give the artist some recourse?⁵

The only thing that "open source" does is that it allows the artist to pay a prohibitive amount of money to a new maintenance team to create a fork. This is rarely the kind of thing that individuals can manage.

This creates an obligation to at least consider how your users might be relying on you.

This is the most complex obligation of the bunch. Obviously it does not entitle every single user to infinite work from the maintainer, but it also shouldn't entitle the user to nothing for having trusted these subtle implied claims that the maintainer is making by making their work public.

It is a nuanced and ongoing negotiation and I do not think we have a clear moral intuition about how it should work out. But we do need to figure out a way to work it out.

It also raises a clarifying question.

Why Are We Even Doing This, and Who Are We Doing It For?

People generally like to do things for more than one reason. We live in an economy where people need to make money, but we mostly prefer to make that money doing things that are useful, and that make other people happy.

So, yes, we create open source for self-interested reasons to improve our reputations, to improve our skills, to increase our influence and to share our maintenance burdens. In so doing we take on some level of obligation to not abuse the trust that is placed in us, even if that level of obligation is not clear.

But if we are not doing it to serve those users at least a little bit, then those motivations are going to quickly ring hollow. We will not increase our reputation with a person if we respond to their every request by telling them that we owe them nothing and that their opinions are worthless. We will not gain influence over a community if we ignore their desires.

Many interactions with open source maintainers are unnecessarily adversarial. This is of course partially the fault of those users, who should calibrate their expectations appropriately.

Still: maintainers could do a better job of listening before these interactions become toxic. There's no reason that "open source users" should be an especially toxic group of people. At this point in history, that group is basically just … people with computers.

It's like that old truism. If you meet one person who is a jerk to you, that's their problem. But if everyone you meet, everywhere you go, is constantly abrasive to you and treats you like you're doing something wrong, maybe it's time to look inward.

If all open source users are entitled assholes, maybe it's time to look for a structural problem.

Surprise, It's About AI Again

Sigh.⁶

Users hate slop.

I know, dear AI-positive reader, your AI outputs are different from everyone else's, you aren't pushing thoughtless slop into your code, just because everyone else is and it is the inevitable terminus of using those tools. You aren't "lazy vibe coding" with Claude, you're doing "responsible agentic engineering", which is different because you're just built different.

Still, humor me, for a moment. Your users don't know that. They know what it looks like when products that they like adopt slop. They know that they will start leaking data. Developers know that it will make them personally less secure. They know that they can expect more outages and that your code will inexorably decline in quality.

In other words, your users are going to assume that this means you are violating that final obligation that the software should keep working.

Your users are going to tell you to stop, and they are probably going to get mad. Maybe you, or a plurality of your team, also want to stop, maybe you disagree with them, but in any case you need some way to have that conversation in a way that does not immediately overflow into every adjacent discussion forum. Users need to feel welcome in some space so they can have the discussion in that space, and not explode out into a thousand different group chats and social media threads.

This post was inspired by yet another prominent open source community discourse where a ton of angry users showed up to yell at developers to stop accepting LLM-generated code. I'm not going to link to any of these, because we don't need any more fuel for the discourse fire. But there is more than one such case and the pattern is becoming familiar.

On social media - usually BlueSky or Mastodon, but sometimes a user group forum - users become aware of some AI-adjacent policy. They show up in a horde to the developer forum or mailing list. They loudly start demanding the project take a hard stand⁷ against AI. This pressure is simultaneous, but uncoordinated; extremely repetitive, very diverse, often inconsistent, and pretty stressful, especially if you're a burnt-out maintainer with other things to be doing who may not even like AI yourself in the first place.

Believe me, I get it. It can be very unpleasant to deal with.

Like most problems that AI is causing, though, it's not really an "AI" problem as much as it is a pre-existing dumpster fire that "AI" is pouring gasoline onto. In this case, an online mob is the language of the unheard⁸.

If Users Are Mad It's Probably Already Too Late (But Maybe You Can Get Ready For Next Time)

One day, all of a sudden, you're getting feedback from a bunch of users that are using inappropriate channels to complain. But did they already have appropriate channels to use?

Did you have a place for people to congregate and discuss your project? To make orderly complaints in a way that will be legible to you? Or do you just have a GitHub Issues page, which non-technical users have no idea how to interact with, and a forum for developers, where users don't know the norms and any arriving brigade of pissed-off users will be seen as disruptive and inappropriate?

I don't want to be throwing any stones from within my particular glass house. Setting up such a place has gotten harder over the years. I don't really have one, either.

Could I have one, though? IRC has been slowly dying, mailing lists are unpopular and present increasingly annoying moderation challenges, forum software is expensive to operate and keep maintained, Discord is a confusing mess and the upshot of all of this is every community needs community management and forum moderation. Which means that for my own small solo projects, I couldn't possibly have such infrastructure because such infrastructure requires a dedicated second person to maintain it, and until someone volunteers for that, it's not really feasible. Even for my larger projects you'd be surprised how slim of a skeleton crew we are getting by with, and we definitely don't have a whole spare maintainer to go manage this, especially as we are under attack from the slopocalypse ourselves.

The nature of open source community is that most communities start too small to need such a thing, grow incrementally until one day they are suddenly way too big and needed one yesterday, and then suddenly they are too small again when interest wanes even a little bit. Even as we need it more and more, building and maintaining community infrastructure remains a challenge.

Even so, having a dedicated place for users - not maintainers - to converse amongst themselves, be an actual community, and present feedback to the developers, is fast becoming a necessary component of a successful community and not a nice-to-have.

In Conclusion

As trying as it can be sometimes, we maintainers all do get something out of open source, and it is good to be honest with your users - and with yourself - exactly what you want to get out of it. In order to know whether the juice is worth the squeeze, we must know both what the juice is, and what the squeeze is.

Part of the metaphorical squeeze is a set of obligations, and those are the most poorly defined of all. We should try to be clear about what those are too. Both about exactly what we believe we are signing up for, and also, about how we are willing to let our users hold us to account for them. Codes of conduct are a start here, but only the absolute barest bare minimum; "do not harass your colleagues or your users" is not a standard of excellence to aspire to, it's just basic manners.

I can't tell you exactly what your obligations are, only try to gesture at my idea of the outlines of the fuzzy moral intuition we've all been implicitly sharing up until now.

Drawing this line is not just for the benefit of the users, either. Maintainers already feel pressure, we already feel obligations. We resent that feeling of obligation. While there are a diverse array of reasons for that resentment, one big one is that it's not clear, even to ourselves where the obligations end. Lashing out by saying "I promised nothing and I owe you nothing!" followed by some choice expletives feels cathartic, but it doesn't really solve the problem, because we clearly don't really believe that's where the line is, or we would have already stopped there. We wouldn't feel the need to say it.

It is going to be a very big collective endeavor to figure out exactly where that line is. The best time to have gotten started on that endeavor was 50 years ago.

But the second best time is today.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!⁹

25 Sep 2026 12:50am GMT