A few days ago systemd gained tools which allow you to monitor Varlink traffic. A new Varlink service (io.systemd.VarlinkMonitor) allows you to set up a ring buffer which will get filled with packets containing the traffic which matches the filters you provide. This provides the functionality that org.freedesktop.DBus.Monitoring provides for D-Bus. A new varlinkctl monitor verb allows you to use this feature in terminals and scripts. This provides the functionality that dbus-monitor provides for D-Bus and uses io.systemd.VarlinkMonitor under the hood. If you pass --json short to varlinkctl monitor, it produces JSON-seq which can be passed into jq --seq to further process the captured data.

I presented this in the broader context of observability with Varlink at All Systems Go. Slides and video of the talk are available online.

This is a good start, but I think we should add more tooling around it. Especially Bustle would be an interesting target because it would allow us to capture D-Bus and Varlink at the same time and get a global view of what is going on in our systems again.

18 Oct 2026 1:37pm GMT

I was travelling last week, and got back to Mike rebasing the zink-video branch and using LLM powerups to fix the final bits of it. We had been locked in a death stare for a few years, and he flinched first. But it got me thinking about another stalled branch I started working on ages ago.

Adding support for avx512 to llvmpipe. The basics have been there, but running LP_NATIVE_VECTOR_WIDTH=512 piglit exploded in a bunch of fragment shading paths, mostly blending and depth handling. This should work even without actual avx512, we just make LLVM generate worse code. I'd gotten it most of the way and then fell over some 1d texture and RGB32 type failures and let it slide when more urgent work came up.

Well this week I had tiredness, and inspired by Mike I decided to dust off my old branch and let Opencode + GPT Terra Pro 5.6 have a go at closing the final gaps. It started finding a bunch of problem, and I chewed through a lot of my monthly quota, and it was generating a lot of good work, but costing a lot of tokens and taking a long time. It closed the gap on a bunch of the 512 wide work over a day or two.

I then decided to switch to Astra, and it was so much better/quicker/cheaper for this job. It started to find other problems in llvmpipe that I hadn't seen, it found some bugs in the 128-wide handling for 1D framebuffer fetch while looking at the code for 512-wide framebuffer fetch. It identified the current test gap and wrote a test filling it. It found a 128/256 difference on a misc piglit fog test and it identified the conversion path that was failing. I had it review the patch series I wrote and Terra had fixed, and it found a bunch of cleanups and cases where the 4/8/16 lane code be made common.

I fixed a niche bug in geometry shader indirect loads (without the LLM) and an egl fd = -1 problem valgrind popped up.

All the niche fixes are pushed, and the 512-wide support is in an MR, it's still all my code, I've just cleaned it up a lot.

I also don't have any avx512 hardware, so I can't answer is it faster until I either rent some AWS time or find a machine in RH, but if anyone else wants to test gears at least it should be possible now.

16 Oct 2026 9:32am GMT

I don't like LLMs.

End of post. Okay, not really. There's more to it:

I really don't like LLMs. God, they piss me off.

Iteration #9999999

LLMs are the hottest topic in software. Regrettably so, I'd say. Based on that, you'll probably already know basically every point I'll bring up to justify my position. Enough blog posts have argued about the topic from every possible angle. LLMs bäd? LLMs gud? Find a blog page that suits you. LLMs are the machine god, get on your knees and pray? Anthropic's newsroom is down that way.

Merely listing my reasons to dislike LLMs would therefore be rather boring, wouldn't it? Instead, I'll try to also comment on some observations I've made over time on how we reason about LLMs and how the discussions go (and why the discussions SUCK).

Digital Veganism

If you ask most anti-AI people on their reasoning, ethical concerns are some of the most common points you'll probably hear. Before you ask: I totally agree. All of the popular LLMs in use nowadays are wildly unethical for a plethora of reasons, from like siphoning off the entire internet for training to siphoning off huge swaths of land to build the datacenters that train and run the LLMs. The industry is inherently predatory and steals from everyone it can (sorry, I meant "collecting user data for training purposes"). And so on. You most certainly didn't hear it here first.

I'd argue this reasoning is very close to how people argue about going vegan. Few people would contest that industrial animal farming is also wildly unethical, and a logical consequence of that would be to also reject, or at least avoid, the products that result from these practices. Not entirely coincidentally, I also think there are plenty of parallels in the general discussion of LLMs and of industrially farmed animal products. At the core of both of these arguments sits the idea that even if the great evils of this world are outside of your control as an individual, you still have a responsibility to reflect on the broader implications of your choices (in what to consume, for example).

This isn't by any means a controversial idea, but people end up being rather selective which specific aspects of their life this idea is applied to. To some extent, everyone has to be: The only fully morally consistent choice is to move to a cabin in the woods, and unless your name is Ted Kaczynski you'll probably consider this too impractical. While this is a fairly horrible indictment about the society we've collectively built, different people draw different conclusions from it, and most importantly, they draw their moral red lines at different points. What I believe can serve as a takeaway from most discussions both about going vegan and about LLMs is that pointing out the immorality of a thing is fairly ineffective in changing someone else's mind, at least most of the time. This isn't an excuse for unethical behavior of course, it's still wrong. And even though pointing out that everyone's a hypocrite in some way reeks of whataboutism, I can't help but observe that arguments purely appealing to morality appear fairly weak to someone that isn't already convinced.

There are still contexts in which it absolutely makes sense to point out the ethical concerns with LLMs. For example, it's still a completely valid reason to explain why you yourself don't use LLMs. When it stops being just about you, for example in a project-wide AI policy discussion, I've observed these arguments just falling flat.

Side note: I'm also super annoyed by people arguing in favor of AI saying that "it's just a tool" to handwave any such ethical concerns away. The tech industry as a whole has enough of a problem with people dissociating themselves from the consequences of their work. One might not consider the ethical concerns a dealbreaker for their LLM use, whatever, but I find it dishonest to reject thinking about the ethical implications of LLM use entirely.

Side rant: Proxy arguments

Speaking of AI policy discussions: They're always such a mess. I'm not sure I've ever seen instances where public AI policy discussion didn't immediately turn into an utter cesspit. Usually, the only party that really got something out of it was Phoronix getting free traffic on the articles about how messy the discussion turned out.

From what I can tell, one of the easiest ways to derail such a discussion is to start "arguing by proxy". That is, you conceal the actual reasons for why you hold your opinion and instead argue based on other points (that you believe might be more likely to convince people).

The most common case of this, in my experience, has been people who will argue based on qualitative aspects of LLMs ("generated code is bad", "we're getting into licensing trouble", etc.) when they're actually not that worried about these points and instead think about things like ethical concerns. This is a really, really bad idea, and above all it's HORRIBLY annoying. In almost any case, it leads to bad-faith arguments being made when other participants respond to these proxy arguments to try and rebuke them. And before you know it, everyone's talking past each other and the entire discussion turns into hot garbage.

LLMs are annoying

So what can I do about people around me trying to use LLMs?

Not much, really. In the end, people make their own choices, and I can't just force them not to use it. However, in the same sense, I can reject LLMs being forced upon me (to a reasonable extent). I can push back on people who mistakenly think they're helping by having Chap Gibbidy™ write up the issue report for them, I can push back on slop MRs, for code I own I can tell people to submit human PRs or go away.

And honestly, there isn't much of an alternative. Because these things are SO GODDAMN ANNOYING. Genuinely. I'd rather have 100 "game crashes pls fix" bug reports than one more LLMslop "analysis" that hides actual information behind 5 pages of useless babble that only sounds good because it's peppered with smart-seeming keywords of zero substance. How did we arrive at companies automating the "confidently incorrect internet person discovers a thesaurus for the first time" cliché???????

Code contributions aren't much better, really. Most LLM contributions give a distinct smell of "I couldn't be bothered to actually write this up, but I have an LLM subscription, please put me in the changelog". In a way, it's a matter of respect to the codebase and the people who spent time writing the original code. "If you can't be bothered to write it, I can't be bothered to read it".

I honestly believe that this alone is reason enough to reject LLMs and their products. They're a hassle to deal with for everyone except those who prompt it. Why should developers accept this blatant offloading of the annoyance that is dealing with LLM slop onto them?

Developers typically believe that they do better, that their LLM code is above the pure slop churned out by random uninitiated people. That's not always wrong, but developers are not above the universal trend that the more LLMs are involved, the more annoying things get for everyone else. Unfortunately, it seems like once you start interacting with LLMs at all, it seems to take over more and more as time goes on.

Crack CocAIne

One more thing that makes me extremely uncomfortable with anything relating to LLMs is that they seem like a genuinely addictive substance. I've lost count of how many people in my immediate surroundings I've seen that "just tried AI for feedback on prototyping" and two weeks later they're slopping out code like there's no tomorrow. At every step along the way I get assured that "I definitely still know what I'm doing, I'm reviewing the code", but are you really?

Up until two years ago, reviewing code was the thing everyone mostly accepted and did in a sort of "FOSS jury duty" way, where it's definitely not the favorite part of the job but everyone does it to keep everyone else unblocked. Suddenly, reviewing the code is the main way you interact with it at all, and I just don't believe that you actually get a great understanding of what is happening just by reading code. It sounds like the review step is a temporary concession you make to those who doubt AI code, but I don't trust for a second that inconvenience won't be automated away later.

This isn't even getting into the cases of genuine delusion, where people elevate their LLMs to genuine personal friends (or even closer relationships). Not all people fall into the hole this deep, but I can't shake the impression that people are falling down a treacherous slippery slope where something about these things keeps luring people into taking over more and more, bit by bit, and I can't help but think it's deliberate design. I can't help but suspect getting people hooked if not completely reliant is the whole point.

This almost sounds conspiratorial, but the general principle of getting people subtly hooked into your service has seen a literal decade of optimization in the social media realm, and at this point there are more than enough studies backing up how well that works. LLMs look like they're like a reprise of that same playbook, but now targeted at your work time instead of your free time. I'm not sure if there are reliable studies on the addictiveness of LLMs yet, so for now I mostly rely on anecdotal evidence to back me up here.

What also doesn't help is the constant proclamation that "AI is inevitable". It's easy to claim that if you're prompting an LLM and are absolutely amazed at all the things it spits out for you, but outside of vague references to calculators¹ or compilers there really isn't much to back that claim up. Usually, it seems to rely on implicit assumptions that any possible limitation or annoyance that's present in LLMs today will be overcome by tomorrow, because they've already made so much progress up to here!

Of course, if you genuinely got addicted to LLMs, it is also hard to imagine how you'd continue programming without it. Just like you can't imagine a life without social media if you end up addicted to that. I'm just baselessly speculating here, but it would not surprise me in the slightest if this sentiment was more of an expression of people being personally unable to resume coding without an LLM to prompt.

Who knows whether that's actually true. But given how many people express this opinion, it is quite a terrifying thought to have.

(pixelcluster from the future here. To clarify: I don't mean to call every single AI user an addict. What I'm trying to point out is that commercial LLM products seem to employ a range of techniques I'd consider manipulative, and that I'd suspect that this can (sometimes) lead to addiction-like states or at least entice people to use LLMs more than they otherwise would.)

Side Note 2: I don't care about LLM "quality"

As alluded to before, the actual quality of LLM output is a popular point of contention, though in many cases I'd argue it's used as a proxy argument rather than a genuine concern. That's also why I haven't talked about output quality at all up to this point. Whether LLM usage actually accelerates you as a developer isn't as clear-cut as many proponents would claim it is (there are more than enough case studies claiming it's a massive speedup, or is at least perceived to be, yet some studies claim to not find any objective speedup after LLMs are introduced), but for my opinion on LLMs it honestly doesn't matter that much. My concerns have been talked about at length already, so I don't really care if GPT-6.7 ends up reasoning 200% better than the previous iteration or anything.

Random Ramblings on Reasonable Applications

All my concerns here are really just about LLMs as in the ones sold by <insert your favorite model provider here>. The core idea of (VERY roughly) creating a neural network that can ingest human language and extract some comprehensive meaning from it isn't really a bad one at all. Human language is super weird and messy and it makes some sense that a neural network fits the problem.

For everything else, though? Throwing LLMs at problems like reading or writing code feels like a brute-force hack more than anything. Programming languages have been designed to be structured and unambiguous for a reason, quite literally their whole point is specifying an exact behavior in a form that is easy for machines to ingest and process. And we're throwing all of that in the wind by having an LLM ingest the text, and then doing funny statistical transformations that you have neither lots of insight nor lots of control over (well, it's not really as simple anymore, but you get the point). It may be a solution, but I think it sounds horribly stupid if you think about it.

The "AI" industry seems to be increasingly moving to offloading work onto tool calls provided by whatever harness you use, to invoke programs doing things the LLM itself can't effectively do (for example,
rm -rf --no-preserve-root /). Sometimes I'm wondering if there are more reasonable use cases if you basically invert that relationship: That is, write a proper program that "Does A Thing", but some inputs/parameters etc. are determined by running a language model against some input in human language. Ideally, given that such kind of model doesn't need to predict arbitrary token sequences, you wouldn't even need to scrape the entire internet for it, and you won't really get addicted to that program, either. I probably would have way fewer issues with something like that.

Conclusion

I don't like LLMs. If you use them, kindly go away. Thanks.

Footnotes

16 Oct 2026 12:00am GMT

After my mother passed away I ended up with a big box full of old film negatives containing photos from childhood family trips and similar. So I decided to buy a negative photo scanner and found that the Plustek 8200i was supported under Linux so I bought that. Only to discover that the chipset in the scanner had changed to one that wasn't supported under Linux anymore. So I had some success making Linux configuration tools for some of my hardware using Claude code, like my Dell Camera controller app so I thought, maybe Claude can help me get a working SANE driver. I basically got stuck and got nowhere for the longest time and even basically gave up on the project for a few Months.

Then I saw a post on X from Tom Crider aka Glorious Eggroll talking about how he had managed to fix a mouse driver using AI tooling. He basically described how he set it up so that the AI agent had full control of a Windows instance to capture data needed to fix the driver. So I pasted his tweet into Claude and asked it to help me set up a windows virtual machine to do the same things Tom as doing for my scanner. I think the combination of having the ability to run scans under windows and capture the USB data while doing so worked wonders, probably combined with improvements to Claude code made a huge difference. Within a few hours I had my first ever working scan under Linux, sure with some quality issues, but 100% recognizable as the image in the negative. Some days later I had a near perfect driver, including support for Infrared scanning ready to go, only to discover that in the time between me giving up and then trying again someone had committed code to SANE to support the scanner already…..it happens sometimes. I also used Claude to write an application for doing negative scanning, which I named Negative Vibes, thought that was a cute name.

Anyway, with the success of my scanner effort I thought, I wonder if I can use this to finally fix the last issues with MIPI camera support on my laptop. So I told claude to re-use the harness I had created and try to fix the MIPI camera on my laptop by updating the kernel driver and libcamera as needed. I started the job in the morning and by the end of the day I had a much improved situation with patches for both the kernel and libcamera ready.

So I thought that this might be worthwhile sharing with the world, and while anyone could start from scratch I did ask Claude to make the harness generic and document it to make the pathway for anyone else who wants to try this on their hardware a little easier. Writing a driver for undocumented hardware has always been tedious. You capture bus traffic, stare at hex dumps, and form hypotheses about what each register or control transfer does. You then test them one at a time, usually for weeks. None of it is conceptually hard, but it takes a lot of patience and skills.

That kind of work suits an AI model well. It can read thousands of lines of USB capture, notice that a sequence repeats with one byte changing, and propose what that byte means. It can write the test to confirm it, and it doesn't get bored. The human still decides what to build and checks that it works on real hardware, but the slog shrinks a lot.

hw-harness

Be aware that I did not use hw-harness in this form for my projects as this was basically bespoke code at that time, so this project is an attempt to make the harness more generic and shareable from the scripts I was using while doing the work. What hw-harness is meant to help your Claude or Codex session with:

1. Record a reference session. Run the vendor's Windows driver in a KVM virtual machine, with the real device passed through to the guest.
2. Capture from outside the VM. Record the traffic on the Linux host with usbmon. It sits below the passthrough layer, so the guest can't see it.
3. Repeat the same operation on Linux with your own driver, and capture that the same way.
4. Diff the two traces. Both come from the same instrument, so they compare directly. The differences are your list of driver bugs.

For devices with no bus to tap, such as an internal MIPI camera, the harness uses Windows-side inspection instead. That means INF files, registry data, ACPI tables and device properties.

The approach should be fine in the sense it doesn't rely on copying any code instead it observes what the hardware does and only extract configuration data from the Windows drivers, so every constant in the new driver traces back to something recorded.

So for those interested I have 3 short descriptions of the projects I did and what was needed.

Project 1: a SANE backend for the Plustek OpticFilm 8200i

Comparing the driver output of two drivers available.

The first project is a SANE backend for the newer Plustek OpticFilm 8200i. It uses the Genesys GL128 controller (USB 07b3:1825) and had no Linux support. The older model with the same name uses a GL845 and is already handled by SANE.

The backend supports:

• Color, gray, infrared and color+infrared modes
• Resolutions from 300 to 7200 dpi, at 8 or 16 bits per channel
• Dust and scratch removal from the infrared channel
• An exposure multiplier for dense negatives

It's packaged as a Fedora RPM, built automatically on every push. I mostly tested it with my own Negative Vibes application as there are not that many options under Linux for negative scanning.

Upstream SANE's genesys backend also gained support for this scanner in September 2026, so I compared the two. Both resolve the same fine detail at 3600 dpi. Mine is faster at full resolution (34.6 s against 54.4 s) and supports infrared dust removal. Upstream offers more resolution steps. The webpage has an interactive side-by-side comparison you can check out. These are single-unit, single-frame results, so treat them as one data point. Check out the interactive page I made showing the current state of the two drivers, hopefully the discoveries I made can be used by the SANE project to fix some smaller items in the upstream driver. Anyway, I put this project up more as a proof of concept due to that since I expect everyone including myself will use the upstream driver once it reaches parity with this one.

Project 2: Negative Vibes

Full scan of negative completed including infrared cleaning of image.

A driver alone isn't much use if scanning negatives is still painful, so I built Negative Vibes. It's a GNOME app, written in Rust, for scanning colour and black-and-white negatives and slides on any film-capable SANE scanner.

The scan is kept exactly as the scanner delivered it: 16-bit, linear, with the infrared plane. Everything else is a stack of non-destructive settings on top, so you can change the look and export again. It includes:

• Density-space negative inversion with film base and dye-crosstalk correction
• Roll-wide settings with per-frame overrides
• Infrared and software dust removal
• IT8 calibration through ArgyllCMS
• Export to TIFF, JPEG, PNG and DNG, with ICC profiles and metadata

It ships as a Flatpak that includes the scanner backend. It's tested on the 8200i so far. The goal is to support any film scanner SANE supports. For that, SANE needs a standard way to expose infrared channels, which it doesn't have yet.

Project 3: Fixing the MIPI camera on my ThinkPad

Showing MIPI output before and after fixes. Colors are better than they look as I was sitting in a relatively dark room by the end of the session.

The camera in my Lenovo ThinkPad X1 Carbon Gen 14 (Aura Edition) uses a Sony IMX471 sensor behind an Intel IPU7. On Linux it produced dark images with heavy vignetting and a strong yellow-green cast. mipi-fixes-x1-aura collects the fixes. It's only relevant to this laptop, which is why it isn't one of the bigger packaged projects, but I have already submitted the kernel patches to Kate Hsuan and the libcamera patches is being reviewed by Wim Taymans for submission to Libcamera.

Kernel (IMX471 driver):

• A crop-selection bug returned an error before storing the crop rectangle, so libcamera assumed the full pixel array.
• The test-pattern register was declared with the wrong width.
• The analogue gain cap was too low, which cost about 1.8 stops of sensitivity.

libcamera software ISP:

• Lens shading correction was missing entirely, leaving the corners about 2.7 stops darker.
• White-balance statistics were summed at the sensor's native bit depth against an 8-bit black level. That skewed white balance, and the fix applies to any RAW10/RAW12 sensor.

In a controlled low-light test, frame brightness rose from 31.1 to 90.9. The red/green and blue/green ratios moved from 0.890 and 0.672 to 1.047 and 0.823, close to a USB webcam reference. Blue is still about 17% short in low light, which I put down to grey-world white balance. Fedora 44 RPMs are on the releases page.

What I learned

• Reference captures are a big step. Having the AI compare a known-good trace against a failing one turns a vague "it doesn't work" into a concrete list of differences. For example for the MIPI camera having a USB camera also connected so Claude could compare the two captures was critical.
• The AI is good at the tedium. That covers decoding captures, writing probes and cross-checking constants. I still have to be the one holding the physical hardware.
• Real hardware is the only judge. Every claim above was checked on the actual scanner or laptop.

If you have hardware with no Linux driver, try hw-harness. Thanks again to Glorious Eggroll for the inspiration.

NOTE: I am not claiming that my scans or MIPI camera is perfect now, but the scanner gives me decent looking scans and the MIPI camera looks a lot better than before. So over time I am sure we be able to tweak and polish further, but I am happy that it allowed me to move the needle despite lacking the in-depth knowledge for this type of work.

-

15 Oct 2026 1:28pm GMT

The First, Probably Not The Last

Wall of text in coming. If we get through this, we can have funny memes again next time, promise.

Everyone's having thoughts on LLM use in coding these days, and being that SGC is at the forefront of all cutting-edge software development, obviously I have thoughts of my own. Thoughts that need to be shared. Get out of my head, thoughts.

I'll start off by stating my position: I'm pro-LLM use.

There's plenty of arguments against it at a "moral" level, but the same can be said for most technologies these days. Television, for example, is primarily used to indoctrinate the masses into taking detrimental stances on political/social issues. Brawndo, The Thirst Annihilator was a less harmful product than most advertisements shill for. I'm not dismissing those arguments. As a holder of a degree in Philosophy with a specialization in bioethics, this is literally what I trained for. But I'm not going to rage against the machine here because it's just not productive or useful.

In the future, we will all use LLMs. It's inevitable. Local models are already becoming competitive at the low-end, and within a year or two they'll likely be good enough to compete with what I'm using now.

Sidebar: I applied for Anthropic's "Claude for Open Source" usage. They did not accept my application. I therefore have nothing nice to say about Claude outside of it was kinda cool to have a free 7-day trial. Shame they missed out on that sponsorship opportunity cuz I've got some bangers come up and whichever LLM I'm actually using is doing awesome/unhinged work.

The Basics: Positivity

Anyway, we'll all be using LLMs eventually. They're the new calculator. Why do long division by hand when you can just press some buttons and get the answer immediately? Sure, we learn how to do long division in school, and maybe if I stare at a piece of paper hard enough I can will myself into remembering how, but I'm never going to go back to those days of pencil and paper. I have tools, and they improve my life.

So far, my favorite use for LLMs is the branch rebase. Pull down the latest from git, slam that rebase button in the shell, and boom, a million conflicts. I've spent my hours manually resolving these, and it's not super fun. Low tier annoyance, sometimes high tier whenever I have to rebase across some giant new refactor/feature that touches all the things.

But now I can just tell my LLM "do the rebase and give me a recap". I then switch to another task (or get a snack) while it does the thing for me. This is a purely mechanical process, and I no longer have to do it.

Code review is also great. LLMs are an incredible static analyzer. I can tell it "consider XYZ. review this area of the tree. find issues related to XYZ." and it'll think for a while (maybe even an hour or two) and then come back to me with a report. I can even tell it to write new (piglit) tests. Coverage++.

I can point it at commits I'm working on and tell it to review them. Then humans have less of my shit code to wade through, and my future self has fewer regressions to bisect.

I can even point it at open MRs and tell it to review them for me. If the MR is entirely generated, I might save myself the time and just post the LLM's thoughts. If it's a human's work, I'm much more likely to read the machine's thinking, do my own analysis, and respond thoughtfully.

Best of all, I can throw it a link to a failing CI pipeline and tell it "figure this shit out", and it'll read the log, find the failing tests, build the branch, run the tests, debug them, and tell me how I fucked up.

These are all nice workflow and ecosystem improvements.

Trickier: LLM Contributions

This is where everyone gets stuck. What happens when people start submitting "slop" MRs? What do we do about that? How do we keep from getting overwhelmed?

I'm gonna break this into a few different categories:

• drive-by patch submissions by enthusiasts to fix bugs
• larger patch submissions by enthusiasts for features/performance
• existing developers using LLM

First, I want to avoid the term "slop". I know we use it when we see AI slop memes, and that's fine, and AI slop art is what it is, but just as I don't think code is art (come at me bro),

I also don't think that LLM-generated code is slop. It's all just code, and it compiles into machine stuff so the thinking sand can render triangles for me.

I also want to avoid witch-hunting the "non-developer class" of contributors using LLMs to submit patches. I like the enthusiasm. Genuinely. It brings me back to ye olden days of open source before I became a jaded husk. That's why I call them enthusiasts. That's what they are. I may or may not accept the patches they submit, but I don't think less of them for trying to contribute. I was once one of them, yearning to be able to comprehend these files full of arcane arrows and punctuation.

Drive-by Bugfix Patches From Enthusiasts

I've had a flurry of these over the past week. I can only imagine that my big-endian removal shitpost inspired some people to take their problems into their own hands.

I love this, and I think it's great.

Before, whenever we had a bug, we just had to sit and rage at our PCs, maybe open a ticket if we were upset enough, or if we were thoroughly furious we'd start our own competing project just to say fuck-you to whatever thing we were using that didn't work how we wanted. Now, we can tell our computer "figure this shit out for me" and get something that fixes the problem. Maybe it doesn't fix it the "right" way, as described in Old Testament, but it's good enough for government.

The next step is to take that fix and throw it upstream. Maybe it's good enough. Maybe it's not. We don't know, and honestly maybe we don't care that much; we're just trying to help.

That's how I view the perspective here. So when I see this sort of patch come in, I review it accordingly. Most of the time they get my stamp of approval because zink has no bugs, and when it does have bugs they're usually tiny little micro-bugs that only need a one-liner.

Saves me time, fixes real issues, awesome. Keep 'em coming. But know that you gotta pay the piglit tax any time you find a new bug.

Larger Patch Submissions By Enthusiasts

This is where it starts to get tricky. I'm not referencing any particular MR or patchset here, but if I see a big LLM-generated/assisted patchset from someone I don't know, I get antsy. And if you aren't putting the LLM trailers on your patches, then don't worry: I can still tell you're vibecoding because you're submitting a zink patch and I've spent most of a decade writing the kind of spaghetti code that no organic human can comprehend.

The reason this is tricky is that now I'm "reviewing" code that a human didn't write. So why am I bothering to review it? Just like I don't think code is "art", I don't think LLM code is "work".

I've done a lot of vibecoding the past couple weeks, so I know the actual human effort required to churn out lines of code. If an enthusiast submits a patchset they don't understand, I'm effectively just talking to a machine that has a high-latency meat proxy.

At that point, I may as well just close the MR and tell my own machine to write code for the same thing. Then I can directly interact with the output. LLM-generated code isn't "owned" by anyone (unless courts decide to start taking real action about training materials lmaohahahahahaha), so even if I take your submission, tell my agent to rework it, and slap my authorship on it, that's my prerogative.

I don't love this. It doesn't fill me with joy or enthusiasm.

If you find yourself thinking about doing this, honestly don't. Don't do it. Please. Just create an issue on the tracker requesting that I do it. I'm the human that has to review the end result anyway, and it's going to be way faster and easier for all of us if I don't burn out in the process by using some shitty gitlab-RPC to tell an LLM what I really want. If we get down to my LLM reviewing your LLMs 500-patch series over gitlab, we have failed.

Sidebar: I fucking hate gitlab now. How/why is it so bad? Should I vibecode a replacement or just send them patches to fix it?

Existing Developers Using LLM

The final frontier. What happens when we're all doing it?

I think this comes down to the human element. At the end of the day, there is a real human behind all of our hardened developer facades. If I know you and you're throwing LLM code up into the repo, that's fine. I can have some degree of confidence that your 500-patch series has been driven/steered by you, so I can have the confidence that you will understand my review comments and be able to consider how best to steer your LLM towards a solution we can all agree on. And you won't be insulted when I tell you what the actual fuck is this shit did you even read any of this?

My only ask is that you don't burn me out by chucking a shitload of huge MRs over the wall at once. Pace yourself. Keep your MRs to a reasonable, reviewable size in areas you possess organic competency. Humans are still the bottleneck here.

Closing Thoughts

I'm not going to make it official or anything, but this is roughly where I'm at as far as a policy for zink or any other stuff I maintain. I'm not a big fan of official policy. I don't trust governance or governments. They're out to get me. Fluoride in drinking water is a psyop to make us ignore how James Buchanan has been secretly running the global political system from his base on the moon since 1862. Look it up.

That said, I think it's valuable to keep the Assisted-by and Generated-by trailers, though I'm not keen on becoming a billboard for exact models. We need accountability. If, a year from now, I find out that the model I'm currently using produced a bunch of code that has bugs, having a tag on commits will let me do deeper review on suspect patches. If I see a wild drive-by patch on my dashboard, knowing up-front whether I'm interacting with a real person or a LLM's meat interface helps me determine how to handle the review. Alternatively, someone trying to conceal LLM use from me gives me a different inclination of how to handle the review.

If I see a car veering unexpectedly into my lane, I want to know that it's a dog at the steering wheel.

The steering is important. Anyone and everyone can use LLMs to vibecode whatever they want now, but that doesn't mean we should immediately yolo whatever comes out the pipe into the repo. As an example, one project I've been working on recently involved CPU optimizations related to draw calls. The initial version is in the top few commits of this branch. You can clearly see where it was going with this: eliminate as many instructions as possible by creating an entirely bespoke draw function to handle the special case. Massive instruction savings-the result looks great.

Then check out the version I actually went with. This is a radically different version which provides equivalent results, and it does so without creating any bespoke special-case functions. Moreover, it improves the general-case performance of draws, not just a single case.

I think the difference between these two cases exemplifies the difference between "good" AI use and "bad" AI use. It took me a long time to get from A to B, even though I wasn't personally writing the code (still faster than if I was tho), but with enough domain knowledge I could steer my way towards the better solution even after the LLM claimed it was done.

The goal of using any tool should be to get better results while improving QoL for the human using it.

15 Oct 2026 12:00am GMT

On September 22 we released systemd v262 into the wild.

In the weeks leading up to that release (and since then) I have posted a series of serieses of posts to Mastodon about key new features in this release, under the #systemd262 hash tag. In case you aren't using Mastodon, but would like to read up, here's a list of all 35 posts:

• Post #1: Word List Based Hostname Patterns
• Post #2: Extended Attributes on Socket Inodes
• Post #3: Varlink Socket Tagging & varlinkctl list-sockets
• Post #4: Machine Tags from udev/hwdb
• Post #5: systemd-sysupdate's installdb
• Post #6: Signed systemd-report Reports
• Post #7: CLI Introspection via --introspect-cli
• Post #8: fscrypt v2 Support in systemd-homed
• Post #9: Read-Only ESP Support in systemd-boot/systemd-stub
• Post #10: Recovery Key Enrollment on First Boot & Varlink IPC for systemd-cryptenroll
• Post #11: Rogue TPM Protection for LUKS Enrollments
• Post #12: Encrypted + Verity Protected DDIs
• Post #13: FONT_SCALE= in /etc/vconsole.conf
• Post #14: Static, Single-Binary Service Manager Builds
• Post #15: Intel TDX CoCo Support in systemd-vmspawn
• Post #16: Varlink IPC for systemd-sysinstall
• Post #17: systemd-report as Varlink Service & New Metrics
• Post #18: Sessions as Inhibitors via pam_systemd's inhibit=
• Post #19: NvPCR Rework
• Post #20: bootctl link-auto
• Post #21: systemd-sysupdate Completion Hooks
• Post #22: /etc/clonetab
• Post #23: Atomically Enqueuing Multiple Jobs
• Post #24: Following the Journal via Varlink
• Post #25: Coredumps via the Kernel's New Socket Protocol
• Post #26: Hardware-Suggested systemd-sysupdate Features/Components
• Post #27: ActivatingConcurrencyMax=
• Post #28: Restarting/Reloading Units on confext/sysext Application
• Post #29: System Credentials for CoCo VMs in systemd-vmspawn
• Post #30: Persisting the TPM EK
• Post #31: systemd.firstboot=headless
• Post #32: RestartRandomizedDelaySec=
• Post #33: systemd.credentials_boot_policy=
• Post #34: systemctl kexec --kernel-cmdline-reuse
• Post #35: PCR Signature Policyrefs

I intend to do a similar series of serieses of posts for the next systemd release (v263), hence if you haven't left tech Twitter for Mastodon yet, now is the opportunity. My series for v263 will begin in a few weeks most likely, under the #systemd263 hash tag.

In case you are interested, here is the corresponding blog story for systemd v261, here for v260, here for v259, here for v258, here for v257, and here for v256.

14 Oct 2026 10:00pm GMT

Hm

I'm no stranger to big tree-wide refactors. I've done lots of them. They suck for everyone.

That's why this time I decided to hand over this job to the new high-tech calculator, Claude.

We've had a lot of discussion about AI use in mesa, and while I can appreciate the various factions, I came out sort of vaguely in favor of allowing AI as long as it was used with some discrimination (i.e., a calculator exists to decrease the workload of humans, not increase it). But I had never actually used it other than asking the free models to find me spec references for various things.

Today, that changed.

Process

I don't know if anyone will find this useful/interesting, but I'm going to document the process I used today.

Install Claude Code VSCode plugin

It's just the normal one from Anthropic. I linked it to my Claude subscription, where I have a 7-day free trial for Claude Pro activated.

Prompt

My exact prompt was as follows:

in the mesa repo I have open:
* do not create or alter any git commits
* delete support for big endian

This took a long while to execute, and it printed out lots of in-progress stuff it was doing. Some neat things I observed along the way were how it refused to modify src/gallium/drivers/asahi due to in-tree policy documentation and how it wrote a script to mimic unifdef when it was unable to find my kernel-devel install directory. One of the times it ran the homegrown script resulted in a fuckup related to line-wrapping, which it detected and then fixed. Cool.

When it was finally finished, it started a test build to verify. The build passed.

Organize

I glanced over all the changes, and they seemed reasonable. My next prompt:

organize the big endian removal into logical git commits:
* each commit must be independently buildable
* each commit must "make sense"
* add `Assisted-by: Claude Code (Opus 5.5 Medium)` at the end of every commit log

This also took a while, but it ended up with 15 commits and refused to write commit logs for them because apparently that's against our AI policy. It then tried to kick off builds to verify the bisectability.

Unfortunately, my main workstation is still an ICL laptop from 2019, so this would've taken literal hours. I told Claude it could ssh to my big test machine with a threadripper for this step. It worked.

Document

Drunk with power, I asked Claude:

briefly describe the effects of each commit

Then I confirmed it did actually reflect the contents of each commit, massaged the output a bit, and that became the commit logs.

Fixups

Our AI policy allows individual drivers/components to ban AI-assisted changes. For this reason, pipe_cap::endianness could not be removed by Claude. I did this manually. It didn't take long.

Done?

In total, this took maybe an hour for me, an actual organic human, to get big endian completely removed from mesa. Doing it manually would've taken days, so I think this is a win.

https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44863

02 Oct 2026 12:00am GMT

Over the last few months I have been working on Torx, an ExecuTorch backend in Mesa3D that delegates to the NPU drivers Mesa already contains. Initial Torx support has been merged already, with YOLOX support being currently under review.

Torx sits next to Teflon, the existing TensorFlow Lite delegate, and both use the same Gallium ML API. Today Torx works with the ethosu driver.

Torx implements the backend API that ExecuTorch provides for running portions of a model's graph on specialized hardware such as NPUs or GPUs. It then uses the Gallium ML API to compile and execute those portions on the available hardware.

Unlike TensorFlow Lite, ExecuTorch compiles models in a step separate from execution. The model can therefore be compiled on a different machine from the target, where it is practical to apply more expensive optimizations than edge-class hardware could run.

I worked on this alongside Rob Herring from Arm, who supported the effort on the kernel side.

Below is a demo of object detection with YOLOX-Nano at 416x416, quantized to 8 bits integers, on an i.MX93 board with an Arm Ethos-U65 NPU. Inference takes 17 milliseconds per frame.

The convolutional layers of YOLOX run entirely on the U65. The final decoding steps need floating point operations, so they run on the CPU.

Once the other NPU drivers in Mesa support the ahead-of-time compilation extension to the Gallium ML API, they will be usable from ExecuTorch as well.

To try it yourself, you can follow the instructions in the documentation. If you prefer not to build ExecuTorch from sources, you can use this merge request from Meta's Anthony Shoumikhin.

01 Oct 2026 12:09pm GMT

I'm happy to announce the availability of a COPR repository which rebuilds Fedora 45 kernels with the patches from the qcom-laptops git branch added. This is a branch where various upstream pending patches with Snapdragon laptop improvements are gathered. For the exact contents of the latest kernel from this COPR see this Fedora kernel git-repo fork.

Installing the kernel from this COPR adds camera support for various X1 laptop models, adds bluetooth support for the ThinkPad T14s as well various other improvements. Most of these are expected to land in the 7.4 kernel.



comments

28 Sep 2026 1:28pm GMT

After the initial work to make Fedora live iso media boot OOTB on Snapdragon X1 laptops in Fedora 44, I've continued working on improving the Fedora experience on these laptops for Fedora 45. For Fedora 44 a long list of workarounds was necessary, see the Snapdragon laptop install instructions on the wiki.

For Fedora 45 various improvements have been made in the mainline kernel and I've been working on improvements at the initramfs generator / distro level. Together these result in a much smoother experience, see the install instructions for Fedora 45 on the wiki. A few workarounds are unfortunately still necessary for Fedora 45. For Fedora 46 I hope that the Fedora aarch64 live iso media will just work on X1 laptops.

Note that this blog post is about Snapdragon X1 (Elite,Plus) laptops. Support for Snapdragon X2 laptops at the same level as the current X1 laptop support is quickly coming together in the mainline kernel. But the 7.2 kernel used for the Fedora 45 beta and release isos is still missing some bits and Devicetrees, so Fedora 45 will not work OOTB on Snapdragon X2 laptops.



comments

28 Sep 2026 1:06pm GMT

Tessellate Your Nightmares

Way, way, way back in the day, there was EXT_shader_object, and inside Big Triangle there was a lot of debate about how things should work. By this time, I had long since infiltrated their organization. They viewed me as one of their own. Because of this, I was able to influence their sinister operation.

My plotting was even more underhanded than the Big Triangle fat-cats, and I nudged them to design tessellation shader objects in a manner which matched OpenGL. Specifically, all the spacing and vertex ordering mechanics were specified in the same shader stages as OpenGL. The D3D members of Big Triangle were asleep at the wheel. My influence went unopposed.

I was the butterfly flapping my wings to cause a hurricane.

That hurricane manifested years later. Suddenly, those sleeping giants awoke and discovered that shader objects were utterly incompatible with their chosen API. Their howls of rage reverberated across the world.

I was unprepared for the backlash. They wielded their monstrous power and upended everything. Suddenly, spacing and vertex ordering could be specified in any tessellation shader stage. It was a nightmare of epic proportions.

Draw Hell

Lavapipe, unbeknownst to many, is really just llvmpipe wearing a funny hat. This means it inherits all the llvmpipe-isms, including its deepest flaws. One flaw is that llvmpipe is primarily a driver for OpenGL rendering, and most of its internals are structured around that. Chief among them is the tessellation support, which goes through auxiliary/draw and then even deeper, into auxiliary/tessellator, a mysterious land where few have tread and even fewer have returned.

And all of this code expects tessellation parameters in the shader stages required by OpenGL.

This was fine due to my initial machinations, but it was no longer fine once the more fiendish parts of Big Triangle awoke. The tessellator was broken. Tests were failing. A new crisis had emerged.

Compatibility

Historically, this mismatch was handled by a function called merge_tess_info. It's still present in a number of drivers. And it did work, propagating that info to the right place in lavapipe before being sent to llvmpipe, except for one wrinkle.

Dynamic domain origin.

OpenGL hardcodes this value to lower-left, but in Vulkan it can be either lower-left or upper-left. Lavapipe worked around this by treating upper-left as equivalent to toggling vertex ordering CCW: for the dynamic state, two versions of the shader were compiled, and lavapipe would run the one which corresponded to (shader_ccw ^ dynamic_domain_ccw). This was fine since it was all restricted to the tessellation evaluation shader.

It was no longer fine once vertex ordering could also be specified in tessellation control.

Another Battle

I had two options: add even more hacks into lavapipe to work around this, or go spelunking deep into gallium to make all the weird bits support setting params in either shader stage. Naturally I went spelunking. This essentially meant shoving merge_tess_info into the depths of auxiliary/draw.

I won't claim it was pretty or easy, but the battle was won. The forces of good have once again triumphed over the evils of Big Triangle's tessellation monster.

25 Sep 2026 12:00am GMT

21 Sep 2026 8:17pm GMT

It's been more than 5 years since I started working on Turnip, the open source Mesa 3D driver for Adreno GPUs, with Igalia's graphics team. Looking back, it's amazing how much we have achieved together, and how much the driver has improved since then. Here's my take on the evolution through those years, some of the challenges we faced and how we overcame them. Before I start, I want to thank all those great people from Igalia, Valve, and the Mesa community I've had the pleasure to work with.

This post is a bit one-sided as it contains my particular view of the evolution of Turnip through those years and it doesn't touch on a lot of important features implemented, issues debugged, and improvements made by others.

Challenges

There are several overarching challenges that the Turnip driver has had and still has to this day:

• There was no hardware documentation, we had to reverse engineer everything, with the most painful part being hardware bugs that necessitate workarounds. I cannot say I became as good as I would have wanted in all that, but I'm grateful that in Turnip we had people with absolutely amazing HW reverse-engineering skills!
• Hardware wasn't really built to run desktop games, at least at first. It became much better in A7XX generation, but there are still some bumpy parts. Running desktop games is specially important for hardware like the Steam Frame and others.

2021

Way back, when I started to contribute to the Turnip driver, we were fixing CTS tests, running "trivial", by today's standards, applications like "Genshin Impact", "TauCeti Vulkan Technology Benchmark", "3DMark". I have a few posts describing some issues debugged back then:

• Turnips in the wild (Part 1) - Fixing "Genshin Impact";
• Turnips in the wild (Part 2) - Fixing "Genshin Impact" and "TauCeti Benchmark".

If you'd asked me back then about running AAA PC games, I would have nervously laughed at best =)

DXVK Time!

But soon, we found a way to at least test how PC games render on Adreno GPUs with Turnip; we weren't able to run games on our development boards, but the driver, after a bit of massaging, had enough features to work with DXVK (Vulkan-based translation layer for Direct3D 8/9/10/11). So the solution at that time was to run the game on the PC, record Vulkan API calls with GFXReconstruct and a Vulkan profile that constrained the desktop GPU's capabilities to those of the Adreno GPU.

A bit later I was able to "play" simple DirectX games on the development board by playing the game on a PC and in real-time replaying translated Vulkan API calls on the board. You can read more on this in "Testing Vulkan drivers with games that cannot run on the target device".

That was a great start; we weren't able to run games directly on our development boards, but we were now able to test them and begin to implement more features necessary for DXVK.

Measuring Performance

In parallel with trying to support DXVK, fixing issues, and reverse engineering hardware features, we found ourselves needing to better understand performance. We chose to support Mesa's u_trace framework and integrate with Perfetto. At that time the performance measurement support in Mesa was rough, with only Freedreno (OpenGL driver for Adreno GPUs) supporting u_trace. Spoiler: as of now, most Mesa drivers either support or are in the process of merging support for u_trace/Perfetto integration.

2022

At the end of 2021 Turnip became Vulkan 1.1 conformant. We also started testing lots of single frame D3D11 captures of games we had, which uncovered plenty of new issues.

With more testing came more issues!

At that point I met our three two main adversaries over the years, in full force:

• Low-Resolution-Z (LRZ) depth optimization;
• GPU hangs, with the worst of them completely shutting down the SoC.

Low Resolution Z (LRZ)

Low-Resolution-Z is an extremely important optimization to get right in order to get reasonable performance out of our GPU (especially for VR games), but I believe it's also one of the most complicated (at least from a software POV) depth-related hardware optimizations across various GPUs.

Conceptually it's relatively simple - create a low resolution depth buffer during primitive binning pre-pass, throw out primitives that completely fail LRZ tests during binning. While during tiling, prevent a lot of overdraw by testing against this already formed low resolution depth buffer. But, in practice, there are lots of implicit restrictions on what can be done without disabling LRZ (to prevent correctness issues), and being too pessimistic when disabling LRZ can lead to unacceptable performance.

GPU Hangs

GPU hangs, on the other hand, are a plague that every graphics driver developer is intimately familiar with. However, at that time there were also unrecoverable hangs, which, due to certain issues in firmware or GPU HW itself, caused the entire SoC to shut down. They were an extreme pain to debug.

I tried many methods over time to debug them:

• At first I tried Vulkan-level breadcrumbs via "Graphics Flight Recorder", it was useful at that time but far from perfect and abandoned by Google;
• Later, I implemented driver-level breadcrumbs that allowed finding the source of hangs at a finer level, and more importantly, synchronously going through breadcrumbs to debug unrecoverable hangs;
• Prototyped editing a captured submission to the GPU.

That made debugging somewhat bearable.

Turnip supports Vulkan 1.3

Meanwhile Turnip gained Vulkan 1.3 support! It was necessary for DXVK and VKD3D-Proton.

Reviewing VK_EXT_fragment_density_map

The third Harbinger of the Apocalypse has arrived: VK_EXT_fragment_density_map, the first stepping stone of the extensions essential for VR, was implemented by Connor Abbott. Since then I have been on the hook reviewing increasingly complicated interactions between VK_EXT_fragment_density_map and the growing host of extensions.

The fragment density map (Foveated Rendering) explanation can be found in the following blog posts from Qualcomm and Meta:

• Eye Tracked Foveated Rendering
• Improving Foveated Rendering with the Fragment Density Map Offset Extension for Vulkan

2023

Turnip began to support Adreno 7XX GPUs; previously we only supported the single 6XX generation, albeit with several sub-generations. The proprietary driver supported Vulkan since the 4XX generation, but it was only Vulkan 1.0, and 4XX/5XX generations were not powerful enough and didn't have enough features to support anything with Vulkan.

New generation means more issues to fix!

To help with that I implemented:

• rddecompiler which makes it possible to decompile captured raw submissions to the GPU into editable C code, coupled with the ability to replay them, print from shaders, and print from the command stream - resulted in a much faster debug loop;
• A debug option that helps find where we use stale register values TU_DEBUG_STALE_REGS_RANGE.

/* pkt4: GRAS_SC_SCREEN_SCISSOR[0].TL = { X = 0 | Y = 0 } */
pkt4(cs, REG_A6XX_GRAS_SC_SCREEN_SCISSOR_TL(0), (2), 0);
/* pkt4: GRAS_SC_SCREEN_SCISSOR[0].BR = { X = 32767 | Y = 32767 } */
pkt(cs, 2147450879);
/* pkt4: VFD_INDEX_OFFSET = 0 */
pkt4(cs, REG_A6XX_VFD_INDEX_OFFSET, (2), 0);
/* pkt4: VFD_INSTANCE_START_OFFSET = 0 */
pkt(cs, 0);
/* pkt4: SP_FS_OUTPUT[0].REG = { REGID = r0.x } */
pkt4(cs, REG_A6XX_SP_FS_OUTPUT_REG(0), (1), 0);

After a lot of command stream and shader reverse-engineering - at the end of the year the Adreno 7XX generation was in decent shape in Turnip.

2024

Work continued with Adreno 750 now being the main target. We had a lot of issues to debug and fix:

A Hat In Time

Farming Simulator

Inspecting Dark Souls 3

And many, many more games.

Android (Waste)Lands

While Turnip, as far as I can remember, was officially only used on some Google Chromebooks, there apparently was a dedicated community of people who tried and actually ran desktop games on Android via Termux. The community has grown since then, but it was great to have a real user already using the driver to play games and having better outcomes than with the proprietary driver, which often doesn't get updated after a phone's release.

While fascinating, those Android setups were hard to debug, so we used them only a few times.

Turnip Is Vulkan 1.4 Conformant and Preemption Support

Not my achievements at all, but two major milestones for Turnip were:

• We caught up with the Vulkan releases and were day 1 conformant to Vulkan 1.4 on A7XX.
• Preemption support - this is a crucial feature for VR and needed by the Steam Frame. The VR compositor has to be able to preempt games that are being rendered at that moment, otherwise we can miss a frame or several, which feels extremely bad for a user moving their head in the VR environment. An Adreno GPU can be preempted at two boundaries: at the drawcall/dispatch boundary in direct (sysmem) rendering, and at the tile boundary in tiling (gmem) rendering.

2025

More VR Extensions

Even more extensions came in from Connor: VK_QCOM_multiview_per_view_viewports, VK_QCOM_multiview_per_view_render_areas, VK_VALVE_fragment_density_map_layered, VK_QCOM_fragment_density_map_offset, VK_QCOM_subpass_shader_resolve, VK_EXT_custom_resolve. Those are essential for pushing VR performance to its limit. The biggest issue with FDM related extensions is that they don't have good CTS tests; the complexity comes from the fact that a conforming driver implementation may simply choose not to reduce quality in regions specified by the density map. Even the size of those regions is not known to a game using those extensions!

As a result I wrote lots of tests that exercise various combinations of those extensions and require visual inspection for them to pass 🫠🫠🫠.

Result of one of FDM tests (Notice lower resolution at the edges of both eyes)

Those tests also measured performance, which helped us fix disparities with the proprietary driver.

Half-Life: Alyx

What is going to use the above extensions to push Steam Frame to its limits? Of course, Half-Life: Alyx.

Even with previously mentioned tests in place, Half-Life: Alyx found plenty of both rendering and performance issues.

That's how I felt when a new issue was found

Are We Performant Yet?

We already had feedback from Android users running games that Turnip performance is sometimes better than the proprietary driver, but sometimes noticeably worse. But how to compare them? Turnip has Perfetto support, the proprietary driver has Snapdragon Profiler, but it's hard to use and even then - we'd just see that some particular renderpass is faster or not. There could be hundreds or thousands of draw calls in a renderpass!

Staring at the command stream from the proprietary driver stopped yielding any insights, so the next step was to take a game trace that can run both on Turnip and Qualcomm's driver and compare them draw by draw, measuring all performance counters along the way. It was done by lots of command stream patching, but the end result was an ability to compare key registers at every draw and every performance counter in existence.

Turnip VS Qualcomm's driver draw by draw comparison

The resulting table was huge:

This helped us to close several gaps in performance, however the most useful part of the comparison was not the counters, but the register comparison. The counters, aside from execution time and LRZ stats, were surprisingly hard to convert into any useful insight.

Preventing Regressions

With the driver gaining capabilities, but not gaining many more users, we needed a way to prevent regressions while introducing more and more complex features. And while Vulkan Conformance Test Suite is being tirelessly improved upon year after year, it's still far from enough.

It was time to introduce a CI system that would be able to detect visual regressions in rendering and performance regressions. We already had a number of d3d11 and d3d9 captures to start working with, and so it came to be:

Above, you can see one of the results where the rendering regressed. In most cases we run only a single frame capture instead of longer multi-frame traces. This was an explicit choice due to the observation that it's better to have a wider selection of captures, than trying to cover any single game better (VR games are an exception here). In many cases only one or two captures out of hundreds regressed due to some issue; the regression was caused by some very specific pattern the game had, which wouldn't appear in others.

Every night we test several driver configurations:

• Fixed DXVK/VKD3D versions + forced direct (sysmem) rendering;
• Fixed DXVK/VKD3D versions + forced tiling (gmem) rendering;
• Upstream DXVK/VKD3D versions;
• Turnip compiled with ubsan (undefined behaviour sanitizer).

We also run important MRs through that CI to find regressions early on.

Every nightly run generates a performance datapoint, so we can see the line going down day by day. Don't mind the bump where we had concurrent binning "optimization" enabled 🫠 (that's a sad tale of an incredibly complicated HW feature which failed to deliver performance gains).

At the moment we are testing more than 600 different game frames per driver configuration; the APIs span D3D8-D3D12, Vulkan, and OpenGL. A single configuration runs in under an hour on just two Adreno 750 devices.

With CI in place we became increasingly confident in making changes to the driver.

See more in my XDC 2025 talk:

2026

Steam Frame was announced at the end of 2025; now we still had plenty of things to polish.

Performance

We were now working much more on performance, and the main culprit in bad performance is often Low-Resolution-Z being fully or partially disabled, especially in VR. We improved Perfetto tracepoints a lot: added new ones, fixed tracepoints with complex renderpass suspend-resume setups, and added performance warnings:

Now it was much nicer to work with, and something external developers could reasonably use.

As a result, while working on the performance of Half-Life: Alyx and some other VR games, we improved LRZ support a lot.

We've also compared Turnip against Qualcomm's driver on a new GPU Performance Microbenchmark (gpu-ratemeter) to squash the rest of the performance differences.

There was also a lot of compiler work, done by great compiler engineers working on Turnip.

D3D12 Woes

As we have been testing more D3D12 games running through VKD3D-Proton, we started to find interesting issues. Those generally are: implicit D3D12 features/requirements, or some behaviour out of the D3D12 spec but which all desktop GPUs do, or in one case Turnip having a higher limit than desktop drivers. We saw at least:

• UE5 not working correctly with wave128 (only Adreno has such wide waves) up until a few months ago: vkd3d-proton PR #3265
• D3D12 not having a proper query/limit for number of elements in buffers, implicitly supporting more than Turnip advertises: mesa MR !41477
• One game relying on "fair" execution of dispatches when implementing its own spin locks in compute shaders: mesa MR !41562
• UE5 relying on higher memory allocation alignment than Turnip has: vkd3d-proton PR #3231
• Games not checking for D3D12_FEATURE_DATA_D3D12_OPTIONS21::ExecuteIndirectTier before using D3D12_EXECUTE_INDIRECT_TIER_1_1 commands

Alyx ☆ Rare GPU Hangs

It's "good" when the GPU hangs at a predictable place, it's bad when the GPU hangs randomly, and it's even worse when the GPU stops hanging when you try to isolate the issue in any way to debug it.

One of such hangs happened in Half-Life: Alyx, when moving through a specific location the GPU hung, sometimes, and sometimes it didn't for a long while. I've tried:

• At first it seemed to happen only on the stable Turnip branch, so I've tried to bisect the issue;
• After a while I found out that it happened in any branch;
• I tried to stare at GPU coredumps - no luck;
• Tried to get a gfxreconstruct trace to reproduce the hang; it might hang once or twice out of many replays.

What is almost impossible to do in such cases is test whether disabling a certain driver feature helps; you always have doubts - it didn't hang this time because of the feature I disabled, or I'm just unlucky. This happened several times during the investigation, the hang would disappear for an hour, and then reproduce several times in a row.

Previously I wrote that we have a mechanism to capture and replay raw submissions that are sent to the GPU. Yes, I tried that too, we can capture the submission that hanged the GPU. Guess what? The captured submission executed absolutely normally and didn't hang, not on the first execution, nor on thousands….

At that point I still didn't have a single clue what's going wrong, aside from some kind of hardware errata being involved. It was time to improve the debug tooling even further. Before, the captured .rd submission could be replayed once with one replay invocation, but the issue at hand demanded lots of iterations, and more ergonomic editing of the command stream than we previously had. So I've made improvements (they still are work-in-progress) to:

• Loop specific submission any number of times;
• Quickly disable specific draw call ranges in the submission;
• Automatically bisect which draw/dispatch causes hang/fault.

Only with each bisection step doing 50000 iterations was I able to narrow things down to a few draw calls. Looking at them closely still yielded only more head scratches though. However, with things narrowed down that far, I had something to poke other, more knowledgeable people with.

After some back and forth, it appeared that I hadn't thoroughly checked all shader debug options we had, or rather, I checked the option, but due to the rarity of the hang I misidentified it as not helpful!

In the end it appeared that there were two hardware errata that needed to be implemented in our shader compiler. And we got "lucky" that they were revealed by one of the most important games to run on the headset.

Present

Driver work never ends, there are still games to debug, features to implement, and VR performance to improve. The fact that Steam Frame runs Linux, is based on open-source software, and isn't locked down means that Steam Frame would be used in many ways unforeseen by us. I hope that Steam Frame release would bring improvements to the VR ecosystem and spearhead PC gaming on Linux running on ARM platforms.

20 Sep 2026 10:00pm GMT

Q3: Big Updates

Hi.

Long time no see.

I've been in Big Triangle jail for the past several months, but now I'm out and "free" once again. I can feel the news sites trembling. I can hear the RSS readers dinging.

That's right. SGC is back.

Future Posts

I had a lot of plans going into 2026. I promised cool stuff. I promised a new level of insanity.

I'll give you a couple weeks to prepare yourselves, but it's coming, and you are not prepared.

Here's a preview of some topics I'll be covering before 2027:

• Zink technical updates
• Zink non-technical updates
• That time I fell into auxiliary/tessellator and barely survived
• Top-secret project(s) that Big Triangle doesn't want you to know about

The Teaser

I'm still easing back into blogging, so today will just be a short warmup post. Nothing major. We're basically done already.

One more small detail:

On both Android and native, it's zink all the way down.

18 Sep 2026 12:00am GMT

At the end of July, I left Collabora after 8 great years there. This makes it my longest employment to date, which is… something. I'm super grateful for the opportunities Collabora has given me, and I'm leaving a great team filled with lots of talented and passionate people. It's been an honor.

Similarly, at the start of August I started working at Arm. The same company I left over 17 years ago.

Why I left

The main reason I left was that after many years of working at Collabora it had become more and more clear that the top leadership and I had some pretty fundamental disagreements on how the company should be run. I spent a lot of time and energy trying to nudge things in what I think would have been the right direction, but that didn't lead to anything but burnout on my part. In the end, it seemed best to just part ways.

But also, one thing that I always disliked while working at Collabora, was working as a contractor. It's just a lot of paperwork and makes a lot of things that should be simple much harder. This was mostly things between me and the Norwegian government, like tax filing, sick leave and parental issues. This wasn't the straw that broke the camel's back, but it certainly contributed.

I was also kinda done working from home. We had moved closer to downtown, and our new place didn't really have space for a dedicated office. For the last year, I worked out of a co-working space.

An easy choice

One of the major reasons why I left Arm back in 2009 was that I wanted to move back to Oslo, the city I am from. Trondheim, while a very nice city had started to feel a bit small. It's very much a student city, and that's fun when you're in your early twenties. But when you're getting close to 30, you've found that a lot of your non-work friends have finished their studies and moved elsewhere.

For the last 3 or 4 years at Collabora, my work mostly revolved around Panfrost, which was partially funded by Arm. I got to work with some old colleagues again, and it was fun.

So when Arm opened an Oslo office in 2023, that piqued my interest. Several of my friends (like Jake "ferris" Taylor, among others) ended up taking jobs at that office as well.

This checked a few boxes for me, as it would let me continue a lot of my work pretty much uninterrupted, would provide regular employment, and would let me connect with old and new friends. So when it became clear to me that Collabora wasn't the right place for me any more, I reached out to an old friend at Arm, and the rest is history.

So what's the job?

I'm working in the Mesa team at Arm, which means I pretty much work on the same things that I did before, just with a different employer.

It's been less than 1.5 months so far, and Arm has a pretty extensive onboarding process, so a lot of the details haven't materialized yet. But it's clear that I'll still be working on Panfrost and PanVK. I'll also still be serving on the X.Org BoD.

In the short term, I'm preparing for XDC. I have a lightning talk and a workshop to prepare. Plus a bunch of code to write to support the lightning talk, phew.

My longer term goal is to make sure Arm is as good as we can reasonably be at working with upstream. There are certainly some challenges in this area, but I hope that I can work with the stakeholders to make sure we get there.

Closing

So yeah, I'm back at Arm after 17 years away. I'm happy with my choice so far, and it's nice to be back working with a lot of new and familiar faces. I'm looking forward to finding out how I can be of most use, and…

See you at XDC?

11 Sep 2026 10:30am GMT

Two weeks ago, I released AppStream 1.2.0. This release contains a lot of great changes, but one of the most important ones concerns how media are being handled, and AppStream's default image export format.

AppStream is a Freedesktop metadata standard to describe software components. That can be anything from system services over fonts to console and graphical applications. AppStream metadata is supposed to give users enough information to decide whether they want to install a piece of software, to represent that piece of software, and to give the operating system enough information to decide whether a software component should be installed automatically and (to some extent) what capabilities and relations it has, to provide the user with sensible options.

Especially for the first two goals, and especially for GUI applications, AppStream supports icons and screenshots, which are used to showcase applications. Today, AppStream is used by all kinds of services, from Linux distributions over firmware updates to Flatpak and desktops directly. AppStream's original design however comes from the perspective of Linux distributions in 2011, where you may want to browse the software catalog offline, without delay, and without pinging an external server (which could be a privacy concern).

Therefore, a common way to deploy an AppStream-enabled software repository is to ship all icons of all applications in the repository to the user as part of the repository metadata download. AppStream does support remote icon downloads nowadays, and for a while I thought that this would become the default eventually. However, especially in today's world, having a bandwidth-saving, instantly responsive, privacy-protecting application browsing experience seems more important that ever.

PNG images are great!

The only format that AppStream supports for icons and screenshots (which are downloaded on-demand from your distributor's CDN) has always been exclusively PNG. PNG images are perfect for icons, because they compress well (especially for common icon shapes), are fast and simple to load, and can be loaded anywhere, by any toolkit or webbrowser. They also ensure we deliver faithful screenshot images, even though we may have scaled or re-rendered them. Still though, PNG images are less great for screenshots, as they are not very efficient, which puts strain on any CDN that has to deliver them, as well as on people's internet connections when browsing screenshots. Having smaller thumbnails alleviates that problem a little, but does not fully solve it.

But even for icons, PNG could be improved upon: In many cases, icons are re-downloaded with the repository metadata again and again, so having a large icon tarball adds up to the data transferred during metadata refreshes. AppStream also now supports large 128x128px icons, which nobody in 2012 expected we would need, adding even more data that will be re-downloaded. Saving some space here translates directly to lower bandwidth costs as well as faster downloads for users.

To improve PNG file sizes, the AppStream Compose library, which handles all image processing and metadata catalog composition, was running optipng on all generated PNG images. That does create smaller PNG images, but they were still relatively large compared to other image formats.

For a long time though, there was no alternative to PNG images for icons: There was no lossless image compression format that could give us the same quality as PNG images and that was also widely supported.

JPEG-XL vs PNG in AppStream

Since 2021 we have JPEG-XL (JXL), which offers a true lossless mode with often better compression than PNG. The issue was that JPEG-XL wasn't widely supported. Then, in 2025, the PDF Association selected JPEG-XL as the preferred image format for HDR images in PDFs, and now we are finally getting browser support and more ubiquitous availability of the format (you can try it right now in Firefox!).

For screenshots, using JXL's lossy mode, it has obvious and extreme size advantages over PNG, so supporting JXL or WebP for screenshot images was an obvious choice. If JXL would support the lossless case very well as well though, we could serve many use cases with the same exported image format, which is very attractive to me.

So, the obvious next question was whether it was worth the pain of switching the icon format, so I did some measurements on real icons. For that I used the AppStream component icon pool that Debian Unstable ships, which is almost 5000 application icons of various sizes, and converted them to PNG:

PNG images saved with libpng at effort=4, compression=9, then optimized using optipng -o2, JXL images encoded using vips jxlsave lossless=1 effort=7 strip=1 via VIPS/libjxl.

As the table shows, using lossless JXL images over size-optimized PNG images (using optipng's default settings) provides a roughly 20% gain. This does not look like much, until you consider how often these files are downloaded: A 20% file size reduction may only save 1-2 MiB of disk space, but if they are downloaded over and over again by many clients, it will save a lot of bandwidth.

Interesting JXL encoding findings

As a sidequest, I was curious why some images were larger than their PNG counterparts when encoded with JXL, and what the ones that were significantly smaller were.

In short, the biggest size reductions for JXL existed on images that were already small as PNG, and contained large, flat color surfaces with hard edges and simple shapes. They were not very interesting, and much of JXL's wins come from accumulating smaller gains across all files, which compound the bigger icons get (especially at 128x128px, where JXL truly shines).

The events were JXL loses to PNG are more interesting: For example, it does quite poorly with pixel-art images that have a lot of repeating patterns. Those are encoded well by PNG, but less efficiently by JXL. Take for example Vonsh:

Icon of Vonsh, an SDL-based snake game, which PNG compresses better than JXL

My guess is that while PNG can exploit the repeating pixel patterns for compression, JXL's predicts surrounding pixels from its neighbours, which fails too often and makes it pay almost full entropy per pixel. In this single rare case, the PNG is at 5.4 KiB, while the JXL is almost 8 KiB in size.

Other cases I looked at were arguably buggy input data, where color channels were hidden under the alpha channel of the input image. PNG could probably again exploit repeats, while we were forcing JXL to encode pixels that were invisible in the final image. This is arguably a problem with the original input data. Currently, AppStream does not make any changes to icons at all, but in future we might add a filter that removes invisible colors from images to solve this pathological case (it was only two icons out of 5000 though, so it is not a high priority).

The third case I found where JXL loses to PNG were icons with checkerboard-like patterns:

Icon of x3270, an IBM 3270 Terminal Emulator

For those, PNG can likely again exploit the repeating patterns, while a checkerboard layout is pretty bad for left/top predictors like JXL's. However, in this case the size difference (and loss for JXL) is only 450 bytes, so even though JXL loses to PNG, it does so not by much.

JXL in AppStream

Given these findings, JPEG-XL is the default image format starting with AppStream 1.2.0. AppStream Compose will encode all images losslessly as JXL, while screenshots are encoded in lossy mode at Q=90 effort=7. Since the optipng step does not happen for JXL images, this comes at no speed penalty and is even a bit faster on modern x86_64 CPUs (where libjxl can use SIMD). PNG is still available, and Compose can be told to switch between the two formats.

Upsides of JXL in AppStream right now

If you use JXL in Compose or the recent release of appstream-generator, you will get much smaller images and, for screenshots, will benefit from other JPEG-XL features such as progressive decoding, providing a far nicer user experience. libAppStream has supported JXL icons since version 1.1.3, so your clients will need that version or a newer one, and all software centers will have to support loading JXL images (which all of them do, provided the right plugins are installed).

Downsides of switching to JXL too quickly

JXL is a very new format, so web browsers might not yet display it if you are serving webpages. Your clients may also have bugs in processing JXL images, as the format is still "new". For example, switching on JXL in Debian sent KDE Discover into an infinite loop on startup while trying to load the icons (an issue which has been fixed, but clients will need that patch first before JXL is switched on).

This currently makes JXL enablement only possible when you know that your clients can support it. This is the case for me in Debian Unstable and Debian 14, which are using JXL images for a few weeks now, but not for any older releases. Platforms like Flatpak have it even harder, because they do know even less about their clients. So, even though it has big advantages, you may want to hold off on using JXL right away, and force PNG by setting the ImageFormat key to png in appstream-generator's configuration, or passing --image-format=png to appstreamcli compose.

It is also worth mentioning that JPEG-XL is much, much slower on systems that do not have SIMD instructions or for which the libjxl/jxl-rs library does not have them (such as apparently riscv64 right now). If this is a concern, you might not want to switch to JXL right away.

Media pipeline improvements

Besides the JXL default change, AppStream 1.2.0 also comes with a complete overhaul of its media processing pipeline. While libappstream, AppStream's main library, does not do any media processing and comes with very minimal dependencies to be embedded in client applications and used on servers, the same can not be said about libappstream-compose, AppStream's library to build metadata generating applications (the server-side part, usually).

The compose library has to render fonts into font specimen cards, inspect translation files, render SVG images, decode all kinds of raster images, inspect video files, etc. Especially the fonts, and the fact that fonts can appear in SVG images, has caused issues in the past, as libappstream-compose is a heavily threaded library and most font libraries can only work from a single thread. This forced the library to essentially go into single-thread mode anytime anything that could touch a font was being processed.

AppStream also originally was created for a "safe world" where applications were vetted by the distributors before their metadata was processed. This is increasingly not the case, so it made sense to put at least a few guardrails on the most complex part of the pipeline: The media processing. As part of the change, media processing was split out into a separate worker process. This solved two problems at once: Font handling was isolated in a single-threaded binary - if we wanted to handle fonts in parallel, we could simply spawn more workers. And, being in a separate process, the media processing could now be sandboxed.

As part of the multiprocess changes, Compose also switched from using GdkPixbuf to VIPS for image processing. The latter allows for much more fine-grained control over the image output and encoding, and comes with a lot of well-maintained filters and operations, which made it possible to eliminate a fair chunk of AppStream's hand-rolled image processing operations. As part of this transition, we unfortunately lost the ability to read XPM images, which dropped about 20-30 applications from the pool at Debian. But in the name of security, this is a sensible choice, especially since most XPM icons were very small and low-resolution, and applications using them could benefit from adding a high-quality PNG icon anyway. With VIPS, we also now restrict the amount of image formats we can load to a sensible set, so extremely niche or unexpected formats will be outright rejected (this includes sane-but-unusual formats for screenshots and icons, such as TIFF images).

The Compose library, with all of these changes, will now just request high-level operations (e.g. "render a font card for this font to a JXL image") from the worker, and provide it with input data in sealed memfds and output locations as FDs as well. On Linux systems, the worker will use Landlock if available, to block all write access to the filesystem, deny device access and deny TCP and UDP as well. The sandbox can certainly be tightened a fair bit in future, but this was a good and safe start to gain some experience with it without having things break too easily, given the many places Compose is used in (also, Landlock's API is surprisingly nice to use, so it was easier than I thought to add in this early version).

With all of these changes, the libappstream-compose library is now also officially marked API-stable, so you should be able to rely on it in future to build new things (its API has barely changed in the past, and now with the new media API and defaults change in place, it was time to declare it stable).

I want to see / try this!

Currently, the easiest way to have a look at the new data is to check out Debian Unstable. If you have a JXL-enabled browser, you can also see the icons in AppStream Generator's HTML pages for Debian Sid. If you are using appstream-generator for your distribution, you will also get much more pleasant statistics and HTML pages, as well as fully deterministic media output and a whole bunch of security updates, so, update to its recent 1.0 release.

Please keep in mind that if you switch to JXL, the client tools receiving the image data have to support it. Support varies depending on the Linux distribution, so, test it first and switch the default back to PNG in case you encounter any issues.

What's next?

With so many features and changes landed, the next changes in AppStream will focus on improving what already exists and fixing any issues (there will be more blogposts about the other features 1.2.x delivers!). Testing with the entire Debian archive as data source makes me fairly confident though that there will not be many problems. In the longer term, tightening the media processing sandbox will also be something we might want to do, e.g. by hiding parts of the filesystem tree or filtering syscalls.

For JPEG-XL, one obvious question is "Will you add support for it to the Freedesktop icon-theme specification as supported format alongside PNG, SVG(Z), and XPM?". For on-disk icon repositories, JXL's space-savings are less compelling, and it being HDR-capable is also not necessarily a killer feature (PNG can go a long way!). However, JPEG-XL's ability to immediately decode larger images at reduced resolution without resampling could legitimately be very powerful here, as applications could ship a single large image and quickly decode it at 1/2, 1/4 or 1/8 the size for different purposes in their UI. JPEG-XL also supports spot-color extra channels, which applications could use as masks to recolor raster icons at render time. This could be incredibly nice to color symbolic icons on-the-fly without any SVG and CSS. JXL also provides richer metadata, which might be neat for (license/author) documentation. So, the answer here is: Maybe it makes sense to allow another format, but this will have to be discussed first, as it would force JXL into every toolkit and desktop, which is a much bigger ask than supporting it only in AppStream.

As always, let me know what you think and please report any issues or bugs directly against AppStream or AppStream Generator if you encounter problems that are with the tools, and not with a project's metadata.

10 Sep 2026 5:48pm GMT