02 Jul 2026
We watch websites for a living at Oh Dear , which puts us in a unique position: we have data on a lot of different outages.
02 Jul 2026 10:15pm GMT
During the last MariaDB Foundation Board Meeting (24 June 2026), Barry shared how it can be difficult to deploy an upgrade immediately and that they sometimes have to wait for one that fixes security bugs. Wait for the validation, wait for the fix, and the release. Even if the MariaDB engineers are doing incredible work, […]
02 Jul 2026 10:15pm GMT
Back in 2019, I wrote that Open Source is not a meritocracy. Meritocracy says talent is the only thing that counts, but that is not true. To contribute, you also need time, a steady income, and a flexible schedule. Plenty of people lack one or more of these.
Some people can give their nights and weekends to learning a codebase, clearing the issue queue, or reviewing patches. Some are paid to do it on the clock. A lot of people can't do either. Their hours go to a second job, caring for family, or simply making it through the week.
That doesn't make these people less talented. It means they have less opportunity.
AI changes the math. It can compress the time it takes to understand enough to act. A contributor might have the skill to fix a bug, but not the time to learn an unfamiliar codebase. AI can help them understand the codebase faster.
On paper, that should be great news for Open Source. In practice, AI will only help if access and skill become shared, not private advantages.
AI access is not equal. The most capable models and coding agents cost real money, and using them well takes real skill. I pay hundreds of dollars a month for these tools and have spent countless hours learning when to trust them, when to doubt them, and how to turn their output into useful work. Many contributors do not have that money or that time.
We learned once that "anyone can contribute" is not the same as "everyone has the same opportunity to contribute". AI can repeat that mistake in a new form.
Powerful technologies rarely share their benefits evenly at first. Electricity did not create equal opportunity the moment it was invented. It only changed lives broadly when people built the infrastructure to make it widely available. The internet followed a similar path: it started with privileged access, then became useful to millions more people as access became cheaper and easier.
AI is no different. If we want AI to reduce privilege in Open Source instead of reinforcing it, Open Source projects can do their part by helping close two gaps.
The first is cost. Contributors should be able to do meaningful work without paying for the most expensive AI tools. As lower-cost models, including open-weight models, improve, Open Source projects should make them practical for contribution work.
The second is skill. Knowing how to use AI well should become shared knowledge within Open Source projects so more people can learn faster and make better contributions.
Contributing with AI should come down to talent, not to who can afford the best tools or who has the time to learn them.
Open Source already moves many things from private advantage to shared infrastructure: code, documentation, best practices, and more. We make all of these public so more people can participate and build on each other's work. The ability to use AI well for contribution should move in the same direction.
Publicly sharing AI best practices is an important start, but not enough. If we want AI to reduce the privilege of free time, those practices need to be embedded in the project and the contributor experience, not live on the side. If potential contributors have to hunt down the tools, prompts, skill files, and know-how themselves, the people short on time are the first to give up, even though they stand to benefit the most.
But more contribution is not automatically progress. As I wrote in AI creates asymmetric pressure on Open Source, AI can make it cheaper to contribute without making it cheaper to review.
The test is whether AI helps more people move from issue to tested patch while making the result easier for maintainers to trust and merge.
If we do this well, AI can make contribution less dependent on free time. If we do it poorly, it will widen the gap for contributors and increase the burden on maintainers. If we ignore AI or discourage its use, it will still show up in contributions, just without shared norms or shared accountability.
In 2019, I argued that Open Source communities should create opportunity by paying contributors. I still believe that. Paying contributors gives people time. But AI gives us another way to reduce the privilege of free time: it helps people do more with the time they have.
I want Drupal to help explore this in practice: not because we have all the answers, but because this is the kind of problem Open Source should help solve.
02 Jul 2026 10:15pm GMT
I've spent about 100 hours of work over the past month to make sure git-annex can build without dependencies that contain LLM generated code. At least so far.
https://git-annex.branchable.com/no_llm_code/
Needing to review a program's whole dependency tree on an ongoing basis is apparently what programming has come to?
I've found some real stinkers. Large LLM generated changes being reverted in the next release without any explanation. An incoherent 1489 line commit message with 10,000 lines of changes to a 26,000 LOC code base. A LLM prompt to copy code from another project that seems to have only avoided being copyright infringement due to luck.
I now have additional information about the quality of dependencies which will surely influence future decisions. As far as I can see, that's the only positive benefit of this work.
I realize that I am probably trying to hold back the tide at this point. That appears to be why Software Freedom Conservancy punted, and I doubt that the FSF will do any better.
As these dominos fall, I am reconsidering my participation in these communities. But I continue my work and support my users.
It may seem easy to prompt a LLM with
Add fourmolu config and restyled
neat
format a module
And commit the result and call yourself a 10xer. But please consider the broader impact of your actions. (In the above case, that project lost my further collaboration on it.)
02 Jul 2026 2:06pm GMT
When you log into a service you're given an authentication token. Each further request to the site includes that token, allowing the server to figure out who you are and ensuring that you have access to your data. Depending on site policy, this token may either be stored in memory (and so vanish if you restart your browser) or disk. The token is the proof of your identity. As far as the site is concerned, anyone with your token is you. These tokens may be traditional browser cookies, but they may also be stored in either site local storage or (if you're not using a browser) in some other storage location.
In recent years we've seen infostealer malware (like LummaC2) gain the ability to exfiltrate user tokens, allowing attackers to gain access to the user's data without needing to retain access to the user's machine. This attack is viable even if the site has strong MFA requirements, so passkeys don't help. Encrypting the tokens on disk doesn't prevent the malware from scraping them out of the browser's RAM or obtaining whatever key is used to encrypt them. This feels like a pretty hard problem to solve.
But that hasn't stopped people from trying! Dirk Balfanz wrote an IETF draft describing a mechanism for using self-signed certificates for TLS authentication. This uses the mutual authentication feature of the TLS protocol that requires both sides prove their identity to each other. In regular TLS, the remote site presents a signed certificate that tells you who it is. When performing mutual authentication, you then present a certificate to the remote site telling it who you are. These client certificates are largely unused outside enterprise environments because they're a huge pain to deploy. It's not so much that this has sharp edges, it's that it's entirely made of sharp edges. Managing certificate deployment to your devices is hard. Browsers get confused if the certificates change under them. You have one certificate and it lives forever, so sites you present it to can track your identity. Users are prompted to choose a certificate to authenticate with, and if they pick the wrong one everything breaks and is hard to recover. I've deployed this and I did not have a good time.
But Balfanz's idea was simple. Rather than require certificates to be deployed, browsers would simply generate a certificate on the fly. The goal wasn't to prove the device or user's identity in any global way - but it would associate a TLS session with a specific certificate. You could then, for example, include a hash of the certificate in the cookie, and if someone tried to use that cookie without presenting that certificate then the cookie could be rejected. If the browser used a hardware-backed private key for the certificate then it would be impossible for an attacker to steal it. Sure, you could still steal cookies, but you wouldn't be able to use them.
This was written almost 15 years ago, and seems simple, elegant, and functional. It didn't happen. Part of the reason for that is that, well, it wasn't quite so simple. One problem was privacy related. Cookies are only sent after the TLS session is established, so anyone monitoring the network doesn't know anything about the user identity. A naive implementation of this approach would have meant the client certificate being sent before session establishment, and now user identity can be tracked (no longer an issue if this was implemented on top of TLS 1.3, but this was a log time ago). This was avoided by reordering the client handshake, but that meant having to modify the TLS specification and implementations would have to be updated to support this. Another was that figuring out the granularity of the certificates was difficult. You'd want to use different certificates for every site to avoid them effectively becoming tracking cookies, but you need to provide the certificate before cookies are set, and you don't know what origin the site is going to set in its cookies. If you generate a certificate for a.example.com and a different one for b.example.com, and a.example.com sets a cookie for *.example.com and includes the certificate you used for a.example.com, that cookie isn't going to work on b.example.com and things are broken. This meant supporting it wasn't as straightforward as it seemed - you'd need to ensure that your cookie scope was compatible with the certificate scope. You could probably make this work well enough by aligning it with the Public Suffix List, but there was still some risk of expectations not being aligned.
And, perhaps most importantly, TLS session resumption (replaced by pre-shared keys in TLS 1.3) somewhat defeats the purpose of the exercise - clients store state that allows them to re-establish a TLS connection without performing certificate exchange (this reduces overhead if a connection gets interrupted or you switch to a new network or anything along those lines), and anyone in a position to steal cookies could steal that state as well.
The followup attempt was channel IDs. This simplified the implementation somewhat - rather than certificates, a raw public key would be sent, along with proof of possession of the private key in the form of a signature over a portion of the TLS handshake. This was required even in the event of session resumption, which avoided having to worry about theft of session secrets. The timing of the exchange was after the encrypted session had been established, so user identity couldn't be leaked that way either. Cookies could then be bound to this identifier. Unfortunately it didn't really deal with the problem of scoping keys in a way that would match cookie requirements, and the spec suggests that the right way of handling this is to scope keys to TLDs, which would enable user tracking across sites (Chrome's implementation apparently restricted it to eTLD+1, which would match the third party cookie policy and avoid the tracking risk).
Chrome added support for this, but it was removed in early 2018. The discussion of some of the pain points in that message is interesting, explicitly calling out problems with connection coalescing across domains and the incompatibility with zero-RTT TLS1.3. The overall consensus at the time seems to be that trying to solve this entirely at the TLS layer has too many rough edges, and a different approach should be taken.
And so almost 7 years after the initial draft for origin bound certificates, we come to token binding. This ended up being a rather more complex endeavour, covering 3 different RFCs describing how it impacts TLS, how to incorporate it into HTTP, and how to manage all the various parties involved in the process. The short version is that it's pretty similar to channel ID, except that there's also a documented mechanism for allowing tokens to be bound to one party and consumed by another, avoiding any need for widely scoped keys. Token binding effectively solved all the issues in the original proposal, but at the cost of somewhat more complexity.
The RFC was finalised in October 2018. Chrome removed its (incomplete, draft) support for token binding in November 2018. Edge carried support until late 2024. Despite getting all the way through the RFC process, it's functionally dead.
The process up until this point had been largely initiated by Google, with Microsoft contributing significantly to the token binding standards. The work had been focused on identifying a generic solution to the problem rather than tying it to any specific authentication flow. The next step was in a different direction - rather than trying to fix this for the entire internet, how about we try to fix it for OAuth?
RFC 8705 is titled "OAuth 2.0 Mutual-TLS Client Authentication and Certificate-Bound Access Tokens". This is basically the 2011 approach, but (a) with an explicit definition of how the certificate should be incorporated into issued auth cookies, and (b) with a proviso that well uh if you're going to use tokens issued by your IdP to authenticate to someone else then well you're going to need to use the same cert for both. This is probably fine for the company-owned-laptop case where you're actually fine with multiple sites being able to tie identities together (that's kind of the point here!), and also works for "I am using an app and not a browser", but doesn't work for more generic scenarios. It also doesn't seem to take the session resumption case into account at all? Support for RFC8705 seems poor, as far as I can tell of the big players only Auth0 implements it. In theory it works fine with self-signed client certs but in reality that's going to be almost as difficult to support across multiple platforms as just issuing proper client certs in the first place, so deployment is going to be kind of a pain. But the good news is it doesn't rely on any TLS extensions or custom browser behaviour, so at the client side it works fine with any browser.
Which brings us on to RFC 9449, "Demonstrating Proof of Possession". This goes even further than RFC8705 in terms of reducing the burden of deployment - it works fine with existing browsers, and it doesn't even require any certs. The client generates a keypair and provides the pubkey when requesting the cookie. The cookie contains the pubkey. Every request to the service now provides the cookie with the pubkey and also provides a signature over the URI and HTTP method. If the signature matches the pubkey in the token then clearly the signature came from the machine the token was issued to, and everything is good.
This does come with some downsides, though. The first is that it uses browser interfaces to generate the keys (typically crypto.subtle.generatekey()) and as far as I can tell there are no browsers that guarantee that that key is going to be generated in hardware even if it's marked non-exportable, so anyone able to steal the cookies can also steal the keys. The second is that the signature only covers the URI and HTTP method, and not the message content or any other headers, so anyone able to exfiltrate a valid signature can replay it against the same URI with different message content. The recommended way to handle this is to reject any signatures that weren't generated within the last few seconds, which is a wonderful additional way to allow clock skew to give you a Bad Day. And the third is that every single request has to be separately signed, which is not intrinsically a problem because computers are fast and have multiple cores, but if you're trying to solve the first problem by sticking the key in a TPM then you're dealing with something that's slow and single threaded and that's maybe acceptable if you're using client certificates (because there's going to be one signature per session and you can use the same session for multiple requests) but probably not if you're dealing with a user opening a browser that restores previous tabs and each of those is a webapp that fires off 100 requests in parallel.
In case it wasn't clear, I don't like DPoP. It doesn't feel like it actually solves the underlying problem that we see in the real world (malware running in a context where if it can grab the tokens it can grab the keys), it adds a massive amount of overhead, and it has baked in replay vulnerabilities. I don't know why it exists and I'm incredibly suspicious of vendors telling me that it fixes my problems, because if they're telling me that then I'm going to end up assuming that they either don't understand my problems or they don't understand their technology, and neither of those is good.
Still. Then we get to the thing that prompted me to write this - Chrome's announcement that they had launched device-bound session credentials. This is interesting because it's a Chrome feature that's explicitly intended to counter on-device malware, which was one of the things that was out of scope in 2018 when token binding was being removed. Since this is entire web level it doesn't have to be an RFC, and so is instead defined by W3C. I'm going to handwave all the complexity and say that it's basically a way to register a public key when a cookie is issued, and then prove possession of the private key when it's time to renew the cookie. By making the cookies shortlived and having support for rotating them in the background, user impact is basically zero and while it's still possible for an attacker to exfiltrate and use a cookie they'll only be able to do so for a short window before it needs to be refreshed - something the attacker can't do, since they don't have the private key. This avoids the DPoP overhead because you only need to do signing once per cookie per cookie lifetime, and not on every single request. I don't like this due to the window where exfiltrated tokens can be used, but it feels like a strict improvement over the status quo. An extension called device-bound session credentials for enterprise allows pre-enrollment of device keys, so even though the actual runtime DBCE flow doesn't involve certificates, certificates can be used for device registration in enterprise environments and you can make sure that auth cookies only go to trusted devices. Unfortunately this is Chrome-only, and so we're going to need to wait for it to be backported to all the random app frameworks for it to have widespread support on mobile or for almost everyone's desktop app that's actually three websites in an Electron wrapper. Mozilla's current position is that they're not in favour of it, so I guess we'll see where Safari lands in terms of broad uptake.
The last thing on my list is another client cert/OAuth binding, this one still in draft state at the time of writing. This one is aimed primarily at the use of agent-driven tooling, where you have something running in the background using a whole bunch of tools that are each acting on your behalf. Authenticating to all of them separately isn't a fun time, but giving broadly scoped access tokens to a non-deterministic agent and trusting that it'll never post them somewhere public also isn't a fun time. The key distinction between it and RFC8705 is that it's aimed at connections rather than sessions, which avoids the worries about session resumption. This is done with TLS Exporters, which in TLS 1.3 should be unique to the connection even over session resumption (TLS 1.2 may reuse some of the same key material for exporters over session resumption, so it's recommended to enforce 1.3 for this). By providing a new signature alongside the cookie on every new connection, the client proves that it still has access to the private key. This is a very new spec and I haven't had much time to work through it yet, but my naive understanding is that unlike RFC8705 this would require some additional client support to be able to regenerate the client signature on every TLS reconnection.
This doesn't avoid all the problems that RFC8705 has, including how to scope certificates. For the agentic use case that probably doesn't matter - all these tools are acting on behalf of the same user, it's fine if all the sites involved know they're the same user. But it doesn't solve the general purpose user use case, and right now DBSC seems like the best we have there.
But. Part of me still wonders whether Dirk Balfanz's approach was the right one. Yes, there's risk associated with TLS session resumption, but in the worst case you could just switch that off for high risk setups. The cookie scope argument is real, and also in cases where it could violate privacy the site owner could already choose to broaden their cookie scope and violate your privacy, and in cases where it breaks things you could just not make use of it. The other problems are largely fixed by TLS 1.3, and then we're just left with "Browsers handle client certificates badly" to which my answer is "Yes, and we should fix that anyway".
Despite having a pretty good answer to this solution over a decade ago, the closest we have to actual deployment is something that offers strictly worse security guarantees. And tokens keep getting stolen, and compromises keep occurring, and for the most part people shrug and get on with things.
02 Jul 2026 2:23am GMT
Many months ago I had been ordering some furniture from IKEA and on one of those orders I got tempted by a hair towel wrap: it mostly worked as an idea, but it was too short for my hair.
On the other hand, I had two old towels I wasn't using, and a recently unpacked sewing machine.
I didn't plan too much, I just put the STJÄRNBUSKE over the towel, cut, realized that the two pieces didn't fit with right sides together, cut the second towel (I had planned to make two wraps, anyway, so it wasn't a big deal), and started sewing by machine in what seemed like a reasonable procedure, taking notes and pictures.
The result was pretty good, and I started using it every time I washed my hair, but then I started to entertain the idea of shooting myself sewing the second one by hand for a video, but never found the time to actually doing it, and the pieces remained in the Pile for months, and months, and way more than a year.
Until one day I bought a meter of cotton cheesecloth (mostly because it was almost cheaper than buying a sample) and it felt like a good material to make a nicely looking head wrapper, to keep my hair out of the way when needed.
A couple months later, it was finally time to bring this project to the top of the list, and even if I was sewing two by hand it went pretty quickly: we had a weekend when it was too hot to do anything else, and by the end of it the wraps were done.
All that remained was finish writing the instructions for my FreeSoftWear patterns website <https://sewing-patterns.trueelena.org/contemporary_unisex/headwear/hair_towel_wrap/index.html>>, and having some pictures taken, and this project was done.
And now, on to documenting a few more things I've done lately, and to start working on the other projects I have added to the queue in the meantime.
02 Jul 2026 12:00am GMT
28 Jun 2026
Lately I've been playing with writing a chatbot library in Common Lisp.
My previous gemini bindings were getting unweildy. I wanted to add the ability to run LLMs on my local machine but it turned out to be really kind of kludgy, so I decided to start from scratch with multiple back ends in mind.
I've got it to the point where in supports multiple back ends, so now I can prompt local LLMs from Lisp.
Recently I added the ability to recursively launch chatbots that can call each other. Since the chatbots do not share their contexts, this greatly reduces the context bloat of thet main chat because it can spawn off subtasks to a minion and not pollute the main context. This also allows you to create a federation of chatbots, each of which specializes in some topic and is overseen by a controlling chatbot that talks to the user.
Chatbots can be serialized and checkpointed, so if one is carrying out an agentic task and Lisp crashes, when we restart the agentic tasks are restarted as well and pick up where they left off.
IT turns out that recursive chats are a useful abstraction once you figure out how to use them. Basically any prompt you may issue may also want to be issued by an llm and this enables that to happen. It allows you to run subprocesses that would otherwise put junk in your context, for example reading the contents of a lange number of files. If you put that into a rocursive chatbot, it could slurp up the files into its context without adding tokens to the parent chat.
You can use a recursive chat as a `smart component'. The recursive chat can have a specialized system instruction and can preload its context with relevant information specific to it. It's context doesn't get diluted by the caller's context
28 Jun 2026 10:52pm GMT
25 Jun 2026
I saw that there was some argument over how much slower slot access is than struct access, so I just decided to measure it naively. I made a two slot sruct and a CLOS version of a CONS cell with car and cdr slots and I ran LTAK using regular lists, `lists' made from CLOS conses, and `lists' made from structs. Here are the results:
D:\repositories\clos-benchmark>sbcl --script run-benchmarks.lisp
Benchmark: ltak over native cons cells, CLOS my-cons nodes, and my-cons-struct nodes
Inputs: x=15 y=9 z=4 repeats=35
Scenario min-ms mean-ms max-ms ratio
--------------------------------------------------------------------
native standard 0.129 0.146 0.186
clos standard 1.346 1.365 1.475 9.37x
struct standard 0.172 0.175 0.179 1.20x
native optimized 0.068 0.069 0.073
clos optimized 0.411 0.414 0.419 6.04x
struct optimized 0.068 0.069 0.073 1.01x
In this naive use case, structs are same as native cons cells, but CLOS objects are one ninth the speed of a struct or cons cell if you just use it unoptimized, and one sixth the speed if optimizations are turned on.
But the CLOS instance is more functional than the cons cell in mimics. For instance, I could add a slot to the class and all the instances would be lazily updated with the new slot. I can also subclass the CLOS class and the selector functions will continue to work. Finally, I can redefine the CLOS closs while I'm developing it and all the instances will be uppdated. THe machinery to keep all this running is costing us our factor of 9.
But this might be worth the cost if we are running on a network where the bulk of the time will be transmitting the answer down the pipe once it is computed. Taking a few extra milliseconds to compute the answer might be worth the convenience features of CLOS.
25 Jun 2026 4:11pm GMT
18 Jun 2026
Back in the day, the US government had a program called SBIR (Small Business Innovation Research) that funded small businesses to do research and development. I recall sitting in our dorm in college, reading through a giant printed catalog of SBIR grants just to amuse ourselves by brainstorming solutions over bad pizza.
.
So, I got curious the other day: what does the SBIR landscape look like now?
I can tell you right now: do not even try to read an SBIR solicitation on your local machine. You are opening yourself up to a world of absolute, unmitigated pain.
You might think, what harm could there be in simply opening a file?
Well, in the modern compliance panopticon, any manipulation of digital information that comes from the govenment has the potential to spawn CUI (Controlled Unclassified Information). CUI is basically a digital pathogen; once you download that file, *anything whatsover* derived from it, including notes and metadata, instantly becomes CUI by association. The moment you read an SBIR on your computer, you've infected your system, rendering you subject to a nightmare of Byzantine federal regulations.
These days, the amount of beurocratic red tape surrounding CUI is insane. To even look at the file legally, you need a dedicated, air-gapped machine completely disconnected from the internet, conforming to a massive, expensive slew of NIST standards covering everything from hardware-level encryption to strict access controls. Alternatively you could contract with a cloud company that offers a pre-certified "CUI-compliant" environment.
And assuming you actually shell out the cash and jump through the hoops to set up this digital containment zone just to read a PDF, you must meticulously audit and account for every single action you take in its presence. Under current federal auditing logic, you are explicitly assumed to be attempting to defraud the government unless you can produce a mountain of paper proving otherwise. Want to bring in a partner to bounce ideas around? You can't just "know a guy." You have to navigate a labyrinth of federal subcontracting regulations.
I had intended on amusing myself by reading some SBIRs and daydreaming about solutions that might involve Lisp (an impossibility in the modern enterprise stack for entirely separate, depressing reasons). Instead, I quickly discovered I did not even own the physical hardware required to even read an SBIR without running afoul of federal regulations.
I wanted to read some clever and inspiring engineering proposals. I ended up reading a lot of very dry and boring compliance regulations.
18 Jun 2026 11:48am GMT
25 Apr 2026
All video recordings from FOSDEM 2026 that are worth publishing have been processed and released. Videos are linked from the individual schedule pages for the talks and the full schedule page. They are also available, organised by room, at video.fosdem.org/2026. While all released videos have been reviewed by a human, it remains possible that one or more issues fell through the cracks. If you notice any problem with a video you care about, please let us know as soon as possible so we can look into it before the video-processing infrastructure is shut down for this edition. To report any舰
25 Apr 2026 10:00pm GMT
29 Jan 2026
Are you ready for another challenge? We're excited to host the second yearly edition of our treasure hunt at FOSDEM! Participants must solve five sequential challenges to uncover the final answer. Update: the treasure hunt has been successfully solved by multiple participants, and the main prizes have now been claimed. But the fun doesn't stop here. If you still manage to find the correct final answer and go to Infodesk K, you will receive a small consolation prize as a reward for your effort. If you're still looking for a challenge, the 2025 treasure hunt is still unsolved, so舰
29 Jan 2026 11:00pm GMT
26 Jan 2026
With FOSDEM just a few days away, it is time for us to enlist your help. Every year, an enthusiastic band of volunteers make FOSDEM happen and make it a fun and safe place for all our attendees. We could not do this without you. This year we again need as many hands as possible, especially for heralding during the conference, during the buildup (starting Friday at noon) and teardown (Sunday evening). No need to worry about missing lunch at the weekend, food will be provided. Would you like to be part of the team that makes FOSDEM tick?舰
26 Jan 2026 11:00pm GMT
