21 Sep 2026 8:17pm GMT

It's been more than 5 years since I started working on Turnip, the open source Mesa 3D driver for Adreno GPUs, with Igalia's graphics team. Looking back, it's amazing how much we have achieved together, and how much the driver has improved since then. Here's my take on the evolution through those years, some of the challenges we faced and how we overcame them. Before I start, I want to thank all those great people from Igalia, Valve, and the Mesa community I've had the pleasure to work with.

This post is a bit one-sided as it contains my particular view of the evolution of Turnip through those years and it doesn't touch on a lot of important features implemented, issues debugged, and improvements made by others.

Challenges

There are several overarching challenges that the Turnip driver has had and still has to this day:

• There was no hardware documentation, we had to reverse engineer everything, with the most painful part being hardware bugs that necessitate workarounds. I cannot say I became as good as I would have wanted in all that, but I'm grateful that in Turnip we had people with absolutely amazing HW reverse-engineering skills!
• Hardware wasn't really built to run desktop games, at least at first. It became much better in A7XX generation, but there are still some bumpy parts. Running desktop games is specially important for hardware like the Steam Frame and others.

2021

Way back, when I started to contribute to the Turnip driver, we were fixing CTS tests, running "trivial", by today's standards, applications like "Genshin Impact", "TauCeti Vulkan Technology Benchmark", "3DMark". I have a few posts describing some issues debugged back then:

• Turnips in the wild (Part 1) - Fixing "Genshin Impact";
• Turnips in the wild (Part 2) - Fixing "Genshin Impact" and "TauCeti Benchmark".

If you'd asked me back then about running AAA PC games, I would have nervously laughed at best =)

DXVK Time!

But soon, we found a way to at least test how PC games render on Adreno GPUs with Turnip; we weren't able to run games on our development boards, but the driver, after a bit of massaging, had enough features to work with DXVK (Vulkan-based translation layer for Direct3D 8/9/10/11). So the solution at that time was to run the game on the PC, record Vulkan API calls with GFXReconstruct and a Vulkan profile that constrained the desktop GPU's capabilities to those of the Adreno GPU.

A bit later I was able to "play" simple DirectX games on the development board by playing the game on a PC and in real-time replaying translated Vulkan API calls on the board. You can read more on this in "Testing Vulkan drivers with games that cannot run on the target device".

That was a great start; we weren't able to run games directly on our development boards, but we were now able to test them and begin to implement more features necessary for DXVK.

Measuring Performance

In parallel with trying to support DXVK, fixing issues, and reverse engineering hardware features, we found ourselves needing to better understand performance. We chose to support Mesa's u_trace framework and integrate with Perfetto. At that time the performance measurement support in Mesa was rough, with only Freedreno (OpenGL driver for Adreno GPUs) supporting u_trace. Spoiler: as of now, most Mesa drivers either support or are in the process of merging support for u_trace/Perfetto integration.

2022

At the end of 2021 Turnip became Vulkan 1.1 conformant. We also started testing lots of single frame D3D11 captures of games we had, which uncovered plenty of new issues.

With more testing came more issues!

At that point I met our three two main adversaries over the years, in full force:

• Low-Resolution-Z (LRZ) depth optimization;
• GPU hangs, with the worst of them completely shutting down the SoC.

Low Resolution Z (LRZ)

Low-Resolution-Z is an extremely important optimization to get right in order to get reasonable performance out of our GPU (especially for VR games), but I believe it's also one of the most complicated (at least from a software POV) depth-related hardware optimizations across various GPUs.

Conceptually it's relatively simple - create a low resolution depth buffer during primitive binning pre-pass, throw out primitives that completely fail LRZ tests during binning. While during tiling, prevent a lot of overdraw by testing against this already formed low resolution depth buffer. But, in practice, there are lots of implicit restrictions on what can be done without disabling LRZ (to prevent correctness issues), and being too pessimistic when disabling LRZ can lead to unacceptable performance.

GPU Hangs

GPU hangs, on the other hand, are a plague that every graphics driver developer is intimately familiar with. However, at that time there were also unrecoverable hangs, which, due to certain issues in firmware or GPU HW itself, caused the entire SoC to shut down. They were an extreme pain to debug.

I tried many methods over time to debug them:

• At first I tried Vulkan-level breadcrumbs via "Graphics Flight Recorder", it was useful at that time but far from perfect and abandoned by Google;
• Later, I implemented driver-level breadcrumbs that allowed finding the source of hangs at a finer level, and more importantly, synchronously going through breadcrumbs to debug unrecoverable hangs;
• Prototyped editing a captured submission to the GPU.

That made debugging somewhat bearable.

Turnip supports Vulkan 1.3

Meanwhile Turnip gained Vulkan 1.3 support! It was necessary for DXVK and VKD3D-Proton.

Reviewing VK_EXT_fragment_density_map

The third Harbinger of the Apocalypse has arrived: VK_EXT_fragment_density_map, the first stepping stone of the extensions essential for VR, was implemented by Connor Abbott. Since then I have been on the hook reviewing increasingly complicated interactions between VK_EXT_fragment_density_map and the growing host of extensions.

The fragment density map (Foveated Rendering) explanation can be found in the following blog posts from Qualcomm and Meta:

• Eye Tracked Foveated Rendering
• Improving Foveated Rendering with the Fragment Density Map Offset Extension for Vulkan

2023

Turnip began to support Adreno 7XX GPUs; previously we only supported the single 6XX generation, albeit with several sub-generations. The proprietary driver supported Vulkan since the 4XX generation, but it was only Vulkan 1.0, and 4XX/5XX generations were not powerful enough and didn't have enough features to support anything with Vulkan.

New generation means more issues to fix!

To help with that I implemented:

• rddecompiler which makes it possible to decompile captured raw submissions to the GPU into editable C code, coupled with the ability to replay them, print from shaders, and print from the command stream - resulted in a much faster debug loop;
• A debug option that helps find where we use stale register values TU_DEBUG_STALE_REGS_RANGE.

/* pkt4: GRAS_SC_SCREEN_SCISSOR[0].TL = { X = 0 | Y = 0 } */
pkt4(cs, REG_A6XX_GRAS_SC_SCREEN_SCISSOR_TL(0), (2), 0);
/* pkt4: GRAS_SC_SCREEN_SCISSOR[0].BR = { X = 32767 | Y = 32767 } */
pkt(cs, 2147450879);
/* pkt4: VFD_INDEX_OFFSET = 0 */
pkt4(cs, REG_A6XX_VFD_INDEX_OFFSET, (2), 0);
/* pkt4: VFD_INSTANCE_START_OFFSET = 0 */
pkt(cs, 0);
/* pkt4: SP_FS_OUTPUT[0].REG = { REGID = r0.x } */
pkt4(cs, REG_A6XX_SP_FS_OUTPUT_REG(0), (1), 0);

After a lot of command stream and shader reverse-engineering - at the end of the year the Adreno 7XX generation was in decent shape in Turnip.

2024

Work continued with Adreno 750 now being the main target. We had a lot of issues to debug and fix:

A Hat In Time

Farming Simulator

Inspecting Dark Souls 3

And many, many more games.

Android (Waste)Lands

While Turnip, as far as I can remember, was officially only used on some Google Chromebooks, there apparently was a dedicated community of people who tried and actually ran desktop games on Android via Termux. The community has grown since then, but it was great to have a real user already using the driver to play games and having better outcomes than with the proprietary driver, which often doesn't get updated after a phone's release.

While fascinating, those Android setups were hard to debug, so we used them only a few times.

Turnip Is Vulkan 1.4 Conformant and Preemption Support

Not my achievements at all, but two major milestones for Turnip were:

• We caught up with the Vulkan releases and were day 1 conformant to Vulkan 1.4 on A7XX.
• Preemption support - this is a crucial feature for VR and needed by the Steam Frame. The VR compositor has to be able to preempt games that are being rendered at that moment, otherwise we can miss a frame or several, which feels extremely bad for a user moving their head in the VR environment. An Adreno GPU can be preempted at two boundaries: at the drawcall/dispatch boundary in direct (sysmem) rendering, and at the tile boundary in tiling (gmem) rendering.

2025

More VR Extensions

Even more extensions came in from Connor: VK_QCOM_multiview_per_view_viewports, VK_QCOM_multiview_per_view_render_areas, VK_VALVE_fragment_density_map_layered, VK_QCOM_fragment_density_map_offset, VK_QCOM_subpass_shader_resolve, VK_EXT_custom_resolve. Those are essential for pushing VR performance to its limit. The biggest issue with FDM related extensions is that they don't have good CTS tests; the complexity comes from the fact that a conforming driver implementation may simply choose not to reduce quality in regions specified by the density map. Even the size of those regions is not known to a game using those extensions!

As a result I wrote lots of tests that exercise various combinations of those extensions and require visual inspection for them to pass 🫠🫠🫠.

Result of one of FDM tests (Notice lower resolution at the edges of both eyes)

Those tests also measured performance, which helped us fix disparities with the proprietary driver.

Half-Life: Alyx

What is going to use the above extensions to push Steam Frame to its limits? Of course, Half-Life: Alyx.

Even with previously mentioned tests in place, Half-Life: Alyx found plenty of both rendering and performance issues.

That's how I felt when a new issue was found

Are We Performant Yet?

We already had feedback from Android users running games that Turnip performance is sometimes better than the proprietary driver, but sometimes noticeably worse. But how to compare them? Turnip has Perfetto support, the proprietary driver has Snapdragon Profiler, but it's hard to use and even then - we'd just see that some particular renderpass is faster or not. There could be hundreds or thousands of draw calls in a renderpass!

Staring at the command stream from the proprietary driver stopped yielding any insights, so the next step was to take a game trace that can run both on Turnip and Qualcomm's driver and compare them draw by draw, measuring all performance counters along the way. It was done by lots of command stream patching, but the end result was an ability to compare key registers at every draw and every performance counter in existence.

Turnip VS Qualcomm's driver draw by draw comparison

The resulting table was huge:

This helped us to close several gaps in performance, however the most useful part of the comparison was not the counters, but the register comparison. The counters, aside from execution time and LRZ stats, were surprisingly hard to convert into any useful insight.

Preventing Regressions

With the driver gaining capabilities, but not gaining many more users, we needed a way to prevent regressions while introducing more and more complex features. And while Vulkan Conformance Test Suite is being tirelessly improved upon year after year, it's still far from enough.

It was time to introduce a CI system that would be able to detect visual regressions in rendering and performance regressions. We already had a number of d3d11 and d3d9 captures to start working with, and so it came to be:

Above, you can see one of the results where the rendering regressed. In most cases we run only a single frame capture instead of longer multi-frame traces. This was an explicit choice due to the observation that it's better to have a wider selection of captures, than trying to cover any single game better (VR games are an exception here). In many cases only one or two captures out of hundreds regressed due to some issue; the regression was caused by some very specific pattern the game had, which wouldn't appear in others.

Every night we test several driver configurations:

• Fixed DXVK/VKD3D versions + forced direct (sysmem) rendering;
• Fixed DXVK/VKD3D versions + forced tiling (gmem) rendering;
• Upstream DXVK/VKD3D versions;
• Turnip compiled with ubsan (undefined behaviour sanitizer).

We also run important MRs through that CI to find regressions early on.

Every nightly run generates a performance datapoint, so we can see the line going down day by day. Don't mind the bump where we had concurrent binning "optimization" enabled 🫠 (that's a sad tale of an incredibly complicated HW feature which failed to deliver performance gains).

At the moment we are testing more than 600 different game frames per driver configuration; the APIs span D3D8-D3D12, Vulkan, and OpenGL. A single configuration runs in under an hour on just two Adreno 750 devices.

With CI in place we became increasingly confident in making changes to the driver.

See more in my XDC 2025 talk:

2026

Steam Frame was announced at the end of 2025; now we still had plenty of things to polish.

Performance

We were now working much more on performance, and the main culprit in bad performance is often Low-Resolution-Z being fully or partially disabled, especially in VR. We improved Perfetto tracepoints a lot: added new ones, fixed tracepoints with complex renderpass suspend-resume setups, and added performance warnings:

Now it was much nicer to work with, and something external developers could reasonably use.

As a result, while working on the performance of Half-Life: Alyx and some other VR games, we improved LRZ support a lot.

We've also compared Turnip against Qualcomm's driver on a new GPU Performance Microbenchmark (gpu-ratemeter) to squash the rest of the performance differences.

There was also a lot of compiler work, done by great compiler engineers working on Turnip.

D3D12 Woes

As we have been testing more D3D12 games running through VKD3D-Proton, we started to find interesting issues. Those generally are: implicit D3D12 features/requirements, or some behaviour out of the D3D12 spec but which all desktop GPUs do, or in one case Turnip having a higher limit than desktop drivers. We saw at least:

• UE5 not working correctly with wave128 (only Adreno has such wide waves) up until a few months ago: vkd3d-proton PR #3265
• D3D12 not having a proper query/limit for number of elements in buffers, implicitly supporting more than Turnip advertises: mesa MR !41477
• One game relying on "fair" execution of dispatches when implementing its own spin locks in compute shaders: mesa MR !41562
• UE5 relying on higher memory allocation alignment than Turnip has: vkd3d-proton PR #3231
• Games not checking for D3D12_FEATURE_DATA_D3D12_OPTIONS21::ExecuteIndirectTier before using D3D12_EXECUTE_INDIRECT_TIER_1_1 commands

Alyx ☆ Rare GPU Hangs

It's "good" when the GPU hangs at a predictable place, it's bad when the GPU hangs randomly, and it's even worse when the GPU stops hanging when you try to isolate the issue in any way to debug it.

One of such hangs happened in Half-Life: Alyx, when moving through a specific location the GPU hung, sometimes, and sometimes it didn't for a long while. I've tried:

• At first it seemed to happen only on the stable Turnip branch, so I've tried to bisect the issue;
• After a while I found out that it happened in any branch;
• I tried to stare at GPU coredumps - no luck;
• Tried to get a gfxreconstruct trace to reproduce the hang; it might hang once or twice out of many replays.

What is almost impossible to do in such cases is test whether disabling a certain driver feature helps; you always have doubts - it didn't hang this time because of the feature I disabled, or I'm just unlucky. This happened several times during the investigation, the hang would disappear for an hour, and then reproduce several times in a row.

Previously I wrote that we have a mechanism to capture and replay raw submissions that are sent to the GPU. Yes, I tried that too, we can capture the submission that hanged the GPU. Guess what? The captured submission executed absolutely normally and didn't hang, not on the first execution, nor on thousands….

At that point I still didn't have a single clue what's going wrong, aside from some kind of hardware errata being involved. It was time to improve the debug tooling even further. Before, the captured .rd submission could be replayed once with one replay invocation, but the issue at hand demanded lots of iterations, and more ergonomic editing of the command stream than we previously had. So I've made improvements (they still are work-in-progress) to:

• Loop specific submission any number of times;
• Quickly disable specific draw call ranges in the submission;
• Automatically bisect which draw/dispatch causes hang/fault.

Only with each bisection step doing 50000 iterations was I able to narrow things down to a few draw calls. Looking at them closely still yielded only more head scratches though. However, with things narrowed down that far, I had something to poke other, more knowledgeable people with.

After some back and forth, it appeared that I hadn't thoroughly checked all shader debug options we had, or rather, I checked the option, but due to the rarity of the hang I misidentified it as not helpful!

In the end it appeared that there were two hardware errata that needed to be implemented in our shader compiler. And we got "lucky" that they were revealed by one of the most important games to run on the headset.

Present

Driver work never ends, there are still games to debug, features to implement, and VR performance to improve. The fact that Steam Frame runs Linux, is based on open-source software, and isn't locked down means that Steam Frame would be used in many ways unforeseen by us. I hope that Steam Frame release would bring improvements to the VR ecosystem and spearhead PC gaming on Linux running on ARM platforms.

20 Sep 2026 10:00pm GMT

Q3: Big Updates

Hi.

Long time no see.

I've been in Big Triangle jail for the past several months, but now I'm out and "free" once again. I can feel the news sites trembling. I can hear the RSS readers dinging.

That's right. SGC is back.

Future Posts

I had a lot of plans going into 2026. I promised cool stuff. I promised a new level of insanity.

I'll give you a couple weeks to prepare yourselves, but it's coming, and you are not prepared.

Here's a preview of some topics I'll be covering before 2027:

• Zink technical updates
• Zink non-technical updates
• That time I fell into auxiliary/tessellator and barely survived
• Top-secret project(s) that Big Triangle doesn't want you to know about

The Teaser

I'm still easing back into blogging, so today will just be a short warmup post. Nothing major. We're basically done already.

One more small detail:

On both Android and native, it's zink all the way down.

18 Sep 2026 12:00am GMT

At the end of July, I left Collabora after 8 great years there. This makes it my longest employment to date, which is… something. I'm super grateful for the opportunities Collabora has given me, and I'm leaving a great team filled with lots of talented and passionate people. It's been an honor.

Similarly, at the start of August I started working at Arm. The same company I left over 17 years ago.

Why I left

The main reason I left was that after many years of working at Collabora it had become more and more clear that the top leadership and I had some pretty fundamental disagreements on how the company should be run. I spent a lot of time and energy trying to nudge things in what I think would have been the right direction, but that didn't lead to anything but burnout on my part. In the end, it seemed best to just part ways.

But also, one thing that I always disliked while working at Collabora, was working as a contractor. It's just a lot of paperwork and makes a lot of things that should be simple much harder. This was mostly things between me and the Norwegian government, like tax filing, sick leave and parental issues. This wasn't the straw that broke the camel's back, but it certainly contributed.

I was also kinda done working from home. We had moved closer to downtown, and our new place didn't really have space for a dedicated office. For the last year, I worked out of a co-working space.

An easy choice

One of the major reasons why I left Arm back in 2009 was that I wanted to move back to Oslo, the city I am from. Trondheim, while a very nice city had started to feel a bit small. It's very much a student city, and that's fun when you're in your early twenties. But when you're getting close to 30, you've found that a lot of your non-work friends have finished their studies and moved elsewhere.

For the last 3 or 4 years at Collabora, my work mostly revolved around Panfrost, which was partially funded by Arm. I got to work with some old colleagues again, and it was fun.

So when Arm opened an Oslo office in 2023, that piqued my interest. Several of my friends (like Jake "ferris" Taylor, among others) ended up taking jobs at that office as well.

This checked a few boxes for me, as it would let me continue a lot of my work pretty much uninterrupted, would provide regular employment, and would let me connect with old and new friends. So when it became clear to me that Collabora wasn't the right place for me any more, I reached out to an old friend at Arm, and the rest is history.

So what's the job?

I'm working in the Mesa team at Arm, which means I pretty much work on the same things that I did before, just with a different employer.

It's been less than 1.5 months so far, and Arm has a pretty extensive onboarding process, so a lot of the details haven't materialized yet. But it's clear that I'll still be working on Panfrost and PanVK. I'll also still be serving on the X.Org BoD.

In the short term, I'm preparing for XDC. I have a lightning talk and a workshop to prepare. Plus a bunch of code to write to support the lightning talk, phew.

My longer term goal is to make sure Arm is as good as we can reasonably be at working with upstream. There are certainly some challenges in this area, but I hope that I can work with the stakeholders to make sure we get there.

Closing

So yeah, I'm back at Arm after 17 years away. I'm happy with my choice so far, and it's nice to be back working with a lot of new and familiar faces. I'm looking forward to finding out how I can be of most use, and…

See you at XDC?

11 Sep 2026 10:30am GMT

Two weeks ago, I released AppStream 1.2.0. This release contains a lot of great changes, but one of the most important ones concerns how media are being handled, and AppStream's default image export format.

AppStream is a Freedesktop metadata standard to describe software components. That can be anything from system services over fonts to console and graphical applications. AppStream metadata is supposed to give users enough information to decide whether they want to install a piece of software, to represent that piece of software, and to give the operating system enough information to decide whether a software component should be installed automatically and (to some extent) what capabilities and relations it has, to provide the user with sensible options.

Especially for the first two goals, and especially for GUI applications, AppStream supports icons and screenshots, which are used to showcase applications. Today, AppStream is used by all kinds of services, from Linux distributions over firmware updates to Flatpak and desktops directly. AppStream's original design however comes from the perspective of Linux distributions in 2011, where you may want to browse the software catalog offline, without delay, and without pinging an external server (which could be a privacy concern).

Therefore, a common way to deploy an AppStream-enabled software repository is to ship all icons of all applications in the repository to the user as part of the repository metadata download. AppStream does support remote icon downloads nowadays, and for a while I thought that this would become the default eventually. However, especially in today's world, having a bandwidth-saving, instantly responsive, privacy-protecting application browsing experience seems more important that ever.

PNG images are great!

The only format that AppStream supports for icons and screenshots (which are downloaded on-demand from your distributor's CDN) has always been exclusively PNG. PNG images are perfect for icons, because they compress well (especially for common icon shapes), are fast and simple to load, and can be loaded anywhere, by any toolkit or webbrowser. They also ensure we deliver faithful screenshot images, even though we may have scaled or re-rendered them. Still though, PNG images are less great for screenshots, as they are not very efficient, which puts strain on any CDN that has to deliver them, as well as on people's internet connections when browsing screenshots. Having smaller thumbnails alleviates that problem a little, but does not fully solve it.

But even for icons, PNG could be improved upon: In many cases, icons are re-downloaded with the repository metadata again and again, so having a large icon tarball adds up to the data transferred during metadata refreshes. AppStream also now supports large 128x128px icons, which nobody in 2012 expected we would need, adding even more data that will be re-downloaded. Saving some space here translates directly to lower bandwidth costs as well as faster downloads for users.

To improve PNG file sizes, the AppStream Compose library, which handles all image processing and metadata catalog composition, was running optipng on all generated PNG images. That does create smaller PNG images, but they were still relatively large compared to other image formats.

For a long time though, there was no alternative to PNG images for icons: There was no lossless image compression format that could give us the same quality as PNG images and that was also widely supported.

JPEG-XL vs PNG in AppStream

Since 2021 we have JPEG-XL (JXL), which offers a true lossless mode with often better compression than PNG. The issue was that JPEG-XL wasn't widely supported. Then, in 2025, the PDF Association selected JPEG-XL as the preferred image format for HDR images in PDFs, and now we are finally getting browser support and more ubiquitous availability of the format (you can try it right now in Firefox!).

For screenshots, using JXL's lossy mode, it has obvious and extreme size advantages over PNG, so supporting JXL or WebP for screenshot images was an obvious choice. If JXL would support the lossless case very well as well though, we could serve many use cases with the same exported image format, which is very attractive to me.

So, the obvious next question was whether it was worth the pain of switching the icon format, so I did some measurements on real icons. For that I used the AppStream component icon pool that Debian Unstable ships, which is almost 5000 application icons of various sizes, and converted them to PNG:

PNG images saved with libpng at effort=4, compression=9, then optimized using optipng -o2, JXL images encoded using vips jxlsave lossless=1 effort=7 strip=1 via VIPS/libjxl.

As the table shows, using lossless JXL images over size-optimized PNG images (using optipng's default settings) provides a roughly 20% gain. This does not look like much, until you consider how often these files are downloaded: A 20% file size reduction may only save 1-2 MiB of disk space, but if they are downloaded over and over again by many clients, it will save a lot of bandwidth.

Interesting JXL encoding findings

As a sidequest, I was curious why some images were larger than their PNG counterparts when encoded with JXL, and what the ones that were significantly smaller were.

In short, the biggest size reductions for JXL existed on images that were already small as PNG, and contained large, flat color surfaces with hard edges and simple shapes. They were not very interesting, and much of JXL's wins come from accumulating smaller gains across all files, which compound the bigger icons get (especially at 128x128px, where JXL truly shines).

The events were JXL loses to PNG are more interesting: For example, it does quite poorly with pixel-art images that have a lot of repeating patterns. Those are encoded well by PNG, but less efficiently by JXL. Take for example Vonsh:

Icon of Vonsh, an SDL-based snake game, which PNG compresses better than JXL

My guess is that while PNG can exploit the repeating pixel patterns for compression, JXL's predicts surrounding pixels from its neighbours, which fails too often and makes it pay almost full entropy per pixel. In this single rare case, the PNG is at 5.4 KiB, while the JXL is almost 8 KiB in size.

Other cases I looked at were arguably buggy input data, where color channels were hidden under the alpha channel of the input image. PNG could probably again exploit repeats, while we were forcing JXL to encode pixels that were invisible in the final image. This is arguably a problem with the original input data. Currently, AppStream does not make any changes to icons at all, but in future we might add a filter that removes invisible colors from images to solve this pathological case (it was only two icons out of 5000 though, so it is not a high priority).

The third case I found where JXL loses to PNG were icons with checkerboard-like patterns:

Icon of x3270, an IBM 3270 Terminal Emulator

For those, PNG can likely again exploit the repeating patterns, while a checkerboard layout is pretty bad for left/top predictors like JXL's. However, in this case the size difference (and loss for JXL) is only 450 bytes, so even though JXL loses to PNG, it does so not by much.

JXL in AppStream

Given these findings, JPEG-XL is the default image format starting with AppStream 1.2.0. AppStream Compose will encode all images losslessly as JXL, while screenshots are encoded in lossy mode at Q=90 effort=7. Since the optipng step does not happen for JXL images, this comes at no speed penalty and is even a bit faster on modern x86_64 CPUs (where libjxl can use SIMD). PNG is still available, and Compose can be told to switch between the two formats.

Upsides of JXL in AppStream right now

If you use JXL in Compose or the recent release of appstream-generator, you will get much smaller images and, for screenshots, will benefit from other JPEG-XL features such as progressive decoding, providing a far nicer user experience. libAppStream has supported JXL icons since version 1.1.3, so your clients will need that version or a newer one, and all software centers will have to support loading JXL images (which all of them do, provided the right plugins are installed).

Downsides of switching to JXL too quickly

JXL is a very new format, so web browsers might not yet display it if you are serving webpages. Your clients may also have bugs in processing JXL images, as the format is still "new". For example, switching on JXL in Debian sent KDE Discover into an infinite loop on startup while trying to load the icons (an issue which has been fixed, but clients will need that patch first before JXL is switched on).

This currently makes JXL enablement only possible when you know that your clients can support it. This is the case for me in Debian Unstable and Debian 14, which are using JXL images for a few weeks now, but not for any older releases. Platforms like Flatpak have it even harder, because they do know even less about their clients. So, even though it has big advantages, you may want to hold off on using JXL right away, and force PNG by setting the ImageFormat key to png in appstream-generator's configuration, or passing --image-format=png to appstreamcli compose.

It is also worth mentioning that JPEG-XL is much, much slower on systems that do not have SIMD instructions or for which the libjxl/jxl-rs library does not have them (such as apparently riscv64 right now). If this is a concern, you might not want to switch to JXL right away.

Media pipeline improvements

Besides the JXL default change, AppStream 1.2.0 also comes with a complete overhaul of its media processing pipeline. While libappstream, AppStream's main library, does not do any media processing and comes with very minimal dependencies to be embedded in client applications and used on servers, the same can not be said about libappstream-compose, AppStream's library to build metadata generating applications (the server-side part, usually).

The compose library has to render fonts into font specimen cards, inspect translation files, render SVG images, decode all kinds of raster images, inspect video files, etc. Especially the fonts, and the fact that fonts can appear in SVG images, has caused issues in the past, as libappstream-compose is a heavily threaded library and most font libraries can only work from a single thread. This forced the library to essentially go into single-thread mode anytime anything that could touch a font was being processed.

AppStream also originally was created for a "safe world" where applications were vetted by the distributors before their metadata was processed. This is increasingly not the case, so it made sense to put at least a few guardrails on the most complex part of the pipeline: The media processing. As part of the change, media processing was split out into a separate worker process. This solved two problems at once: Font handling was isolated in a single-threaded binary - if we wanted to handle fonts in parallel, we could simply spawn more workers. And, being in a separate process, the media processing could now be sandboxed.

As part of the multiprocess changes, Compose also switched from using GdkPixbuf to VIPS for image processing. The latter allows for much more fine-grained control over the image output and encoding, and comes with a lot of well-maintained filters and operations, which made it possible to eliminate a fair chunk of AppStream's hand-rolled image processing operations. As part of this transition, we unfortunately lost the ability to read XPM images, which dropped about 20-30 applications from the pool at Debian. But in the name of security, this is a sensible choice, especially since most XPM icons were very small and low-resolution, and applications using them could benefit from adding a high-quality PNG icon anyway. With VIPS, we also now restrict the amount of image formats we can load to a sensible set, so extremely niche or unexpected formats will be outright rejected (this includes sane-but-unusual formats for screenshots and icons, such as TIFF images).

The Compose library, with all of these changes, will now just request high-level operations (e.g. "render a font card for this font to a JXL image") from the worker, and provide it with input data in sealed memfds and output locations as FDs as well. On Linux systems, the worker will use Landlock if available, to block all write access to the filesystem, deny device access and deny TCP and UDP as well. The sandbox can certainly be tightened a fair bit in future, but this was a good and safe start to gain some experience with it without having things break too easily, given the many places Compose is used in (also, Landlock's API is surprisingly nice to use, so it was easier than I thought to add in this early version).

With all of these changes, the libappstream-compose library is now also officially marked API-stable, so you should be able to rely on it in future to build new things (its API has barely changed in the past, and now with the new media API and defaults change in place, it was time to declare it stable).

I want to see / try this!

Currently, the easiest way to have a look at the new data is to check out Debian Unstable. If you have a JXL-enabled browser, you can also see the icons in AppStream Generator's HTML pages for Debian Sid. If you are using appstream-generator for your distribution, you will also get much more pleasant statistics and HTML pages, as well as fully deterministic media output and a whole bunch of security updates, so, update to its recent 1.0 release.

Please keep in mind that if you switch to JXL, the client tools receiving the image data have to support it. Support varies depending on the Linux distribution, so, test it first and switch the default back to PNG in case you encounter any issues.

What's next?

With so many features and changes landed, the next changes in AppStream will focus on improving what already exists and fixing any issues (there will be more blogposts about the other features 1.2.x delivers!). Testing with the entire Debian archive as data source makes me fairly confident though that there will not be many problems. In the longer term, tightening the media processing sandbox will also be something we might want to do, e.g. by hiding parts of the filesystem tree or filtering syscalls.

For JPEG-XL, one obvious question is "Will you add support for it to the Freedesktop icon-theme specification as supported format alongside PNG, SVG(Z), and XPM?". For on-disk icon repositories, JXL's space-savings are less compelling, and it being HDR-capable is also not necessarily a killer feature (PNG can go a long way!). However, JPEG-XL's ability to immediately decode larger images at reduced resolution without resampling could legitimately be very powerful here, as applications could ship a single large image and quickly decode it at 1/2, 1/4 or 1/8 the size for different purposes in their UI. JPEG-XL also supports spot-color extra channels, which applications could use as masks to recolor raster icons at render time. This could be incredibly nice to color symbolic icons on-the-fly without any SVG and CSS. JXL also provides richer metadata, which might be neat for (license/author) documentation. So, the answer here is: Maybe it makes sense to allow another format, but this will have to be discussed first, as it would force JXL into every toolkit and desktop, which is a much bigger ask than supporting it only in AppStream.

As always, let me know what you think and please report any issues or bugs directly against AppStream or AppStream Generator if you encounter problems that are with the tools, and not with a project's metadata.

10 Sep 2026 5:48pm GMT

After much back and forth and hoops jumping, I can finally reveal nouveau/nvk running on a NVIDIA Spark box.

This is running on a version of nouveau[1] that has

a) ported to the 610 NVIDIA firmware

b) a bunch of display rework from Moham

c) a bunch of 0 VRAM and L2 cache handling fixes

d) spark boot support

e) spark display support

and NVK[2] with patches to handle gb10 depth/stencil differences and 0 VRAM support.

I'm not sure how best to upstream it all, it's 100 patches and a new firmware which might mean it's a wait for nova type situation, but I just wanted to see it work.

[1] https://gitlab.freedesktop.org/nouvelles/kernel/-/commits/nouveau-610-wip-spark

[2] https://gitlab.freedesktop.org/airlied/mesa/-/commits/nvk-spark-wip

09 Sep 2026 7:56pm GMT

Together with Modal, I'm happy to announce that the Sovereign Tech Agency is investing nearly €510k into Flatpak development. The focus is on closing gaps in Flatpak's sandboxing story: new portals for audio, networking, VPNs, and spell checking, plus infrastructure work on entitlements and intents.

I'll be leading the technical side alongside Adrian, with organizational support from Kateryna and Cade. We've brought on a great team and the project will ramp up over the coming months through the end of 2027.

Read the full announcement on the Modal blog.

27 Aug 2026 10:30pm GMT

AMD is known to be Linux friendly and has had an open source driver stack for their GPUs for more than a decade. However, there was one GPU that has always been an outcast on Linux because it has never worked: the Radeon HD 7870 XT. This post is about how I fixed that so that Linux gamers can enjoy this GPU being fully functional now.

The Radeon HD 7870 XT is built on the GCN 1 architecture (also known as Southern Islands or SI, or GFX6). It has a chip called "Tahiti LE" which is a variant of the Tahiti chip that was quite high-end by 2012 standards. Tahiti and other GCN 1 chips have been supported on Linux for more than a decade, first by the radeon kernel driver and more recently by the amdgpu kernel driver.

We have every reason to assume that the 7870 XT should "just work", so why doesn't it?

Motivation

Why care whether a 15 years old GPU works today or not?

• If the driver stack claims to support GCN 1, we should indeed support all GCN 1 chips without exceptions.
• If there is a serious bug that prevents a GPU from working, there is a chance the bug also affects other GPUs. It is worth an investigation.
• Most importantly, it's a good challenge to see if I can figure out a problem like this.

Story time: what is a harvested (cut down) chip?

If you look at any GPU manufacturer, they have a lot of different products every generation, and those products use different variants of the same few different chips. How is that possible?

Due to economies of scale, there is a common practice among chip makers: they prefer to manufacture massive quantities of the same few chip instead of a small quantity of many different kind of chips. However, in order to have many different products, they can configure the same chip in different ways and sell those variants under different product names. Today we are focusing on AMD's old Tahiti chip, so let's use that as an example. How many different products did they launch that use the Tahiti chip and its different variants or refreshes? We can use Wikipedia to check that:

• Radeon HD 7870 XT, 7950, 7970, 7990, 8950, 8970, 8990
• Radeon R9 280, 280X
• FirePro W8000, W9000, D500, D700, S9000, S9050, S10000

All of those use the Tahiti chip. But what are the differences between those products?

• Different target audience, eg. workstation vs. consumer
• Different memory configuration (memory size and bus width)
• Different shader and memory clock speed
• Different amount of compute units, render output units (aka. render backend or RB), etc.

How is this achieved?

At the factory, each chip is examined automatically. Due to variance in the chip manufacturing process, not all chips end up the same, even if we do everything to make them the same. In practice that means there may be defects on the chip, or maybe not all units perform up to spec. This is the so-called "silicon lottery". The chips are then sorted according to how well they ended up performing and that's when the manufacturer decides what to do with them and what product they can be sold as.

There really are no bad GPUs, just incorrectly priced GPUs. Those that are still usable but ended up less than ideal will still be put to use: some parts (eg. compute units) are fused off and disabled, and the GPU is overall still functional as a weaker, cheaper GPU. That's how we end up with products like the Radeon HD 7870 XT.

Starting point for investigating the problem

Initial testing

To start with this work, open source enthusiast Leonardo Frassetto helped me to buy a used 7870 XT in good condition from an Italian used hardware site. After plugging in this GPU and booting my system, I noticed the following:

• Firmware (BIOS) can recognize the GPU, show a logo and boot grub
• When amdgpu loads, the picture disappears and becomes a colorful mess
• Looking at the logs, it seems the GFX block immediately hangs and goes into a GPU reset loop, as the kernel attempts to reset the GPU to fix the hang, which is expected

Information from Wikipedia

From Wikipedia, we got a list of AMD GPUs and an article about GCN to give us some basic info.

We can see that Tahiti LE is a harvested (cut down) version of the full Tahiti chip as the 7870 XT has lower specs than the 7970 (or R9 280X). There are plenty of GPUs with disabled CUs supported already, so I went with the assumption that the disabled CUs aren't the issue.

Information from Freedesktop Bugzilla

Someone opened a bug report on the old Freedesktop Bugzilla in 2013 complaining that the 7870 XT didn't work. Although the issue has never been solved, we can still glean some interesting information from that bug report:

• The display (ie. modesetting) should work, and the issue is "only" with 3D acceleration.
• One commenter was able to get the 7870 XT to work with basic compute shaders but not much else (definitely not a full desktop).
• The register dumps attached to the bug only contain the DCE (display engine) registers, so are not really conclusive.
• There were suggestions to change the CGTS_TCC_DISABLE register in the kernel driver, which didn't help.
• The developers already corrected the register programming for harvested (cut down) RBs, and the 7870 XT doesn't have those anyway, so that isn't the issue.

Booting a working system

How do we even begin to diagnose what the problem is if the GPU hangs immediately at boot?

Booting in runlevel 3

I started by booting to runlevel 3 (basically just a terminal and nothing else). In this mode, the amdgpu kernel driver can initialize all blocks in the GPU correctly and the display works. We are in a terminal only environment so there is nothing submitting jobs to the GPU.

Booting a desktop with software rendering

I configured the system to use software rendering for both OpenGL and Vulkan by setting the following environment variables temporarily:

# Ask OpenGL loader for software rendering
LIBGL_ALWAYS_SOFTWARE=1
# Force using lavapipe as a Vulkan driver
VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.x86_64.json
# Force using swrast as a Gallium driver (for OpenGL)
MESA_LOADER_DRIVER_OVERRIDE=swrast

With that, the system can indeed boot with the 7870 XT, although the graphical user interface is slow because we are not using any hardware acceleration (obviously). I would not recommend anyone to regularly use their system this way.

Testing simple shaders

Even though all the system uses software rendering, we can still set different environment variables for specific apps and have just one test application run with "real" drivers for the hardware. I decided to use the vkrunner suite and wrote a very simple test case with a very simple compute shader. The test case allocates two buffers (SSBO). The shader has only one invocation that reads a small piece of data from the values_in buffer and writes it to the values_out buffer.

[compute shader]
#version 450

layout(local_size_x = 1, local_size_y = 1, local_size_z = 1) in;
layout(binding = 1) buffer block_out { uint values_out[]; };
layout(binding = 2) buffer block_in { uint values_in[]; };

void main()
{
  values_out[gl_LocalInvocationIndex] = gl_WorkGroupID.x;
}

[test]
ssbo 1 1048576
ssbo 2 1048576

compute 1 1 1

That indeed works correctly, confirming what we found in the Freedesktop bugzilla, that simple compute shaders indeed work. What happens if we slightly complicate it though? The first thing we can do is to increase the number of invocations so that more than one invocation reads and writes memory.

layout(local_size_x = 96, local_size_y = 1, local_size_z = 1) in;

Whoops! This test case now hangs the GPU. I guess the way we changed the memory access must have caused the GPU to hang. With some trial and error, we can find that local_size_x = 64 still works but local_size_x = 65 and higher hangs. After some further trial and error I noticed that I can still use a larger workgroup if I skip memory accesses to certain address ranges. This definitely confirms that the issue is with memory access somehow.

Let's look at registers with umr

One of the developers who commented on the old bugzilla thread were mentioning the CGTS_TCC_DISABLE register. Although the suggested patch didn't help, I thought okay, let's see what is actually the value of this register using umr:

$ sudo umr -O bits -r tahiti.gfx600.mmCGTS_TCC_DISABLE
gfx600.mmCGTS_TCC_DISABLE => 0x09240001
        .TCC_DISABLE[16:31] == 2340 (0x00000924)

Each bit represents a disabled TCC. We can see that some TCC units are disabled by default: 2, 5, 8, 11. Why are they disabled and what does that mean exactly? We saw from Wikipedia that the 7870 XT only has a 256-bit memory bus, although other Tahiti based GPUs have a 384-bit bus. This means that only two-thirds of the memory channels on this GPU are enabled and the rest are disabled.

I started to think, is it possible the GPU is trying to read from those disabled memory channels or use some disabled TCC units, or something like that? Technically, the disabled parts should be fused off and not accessible, but maybe it still tries to interact with them?

I went to take a look at the registers and register fields of GFX 6 to see if there is anything that stands out. I basically searched for "TCC" and looked at the results hoping for a clue. I found two interesting things:

• The TCP_ADDR_CONFIG register has a field called NUM_TCC_BANKS
• There are TCP_CHAN_STEER_LO and TCP_CHAN_STEER_HI registers which have fields for channels

Let's see the value of these registers:

$ sudo umr -O bits -r tahiti.gfx600.mmTCP_ADDR_CONFIG
gfx600.mmTCP_ADDR_CONFIG => 0x000002fb
        .COLHI_WIDTH[6:8]    ==  3 (0x00000003)
        .NUM_BANKS[4:5]      ==  3 (0x00000003)
        .NUM_TCC_BANKS[0:3]  == 11 (0x0000000b)
        .RB_SPLIT_COLHI[9:9] ==  1 (0x00000001)

$ sudo umr -O bits -r tahiti.gfx600.mmTCP_CHAN_STEER_LO
gfx600.mmTCP_CHAN_STEER_LO => 0xa9210876
        .CHAN0[0:3]   ==  6 (0x00000006)
        .CHAN1[4:7]   ==  7 (0x00000007)
        .CHAN2[8:11]  ==  8 (0x00000008)
        .CHAN3[12:15] ==  0 (0x00000000)
        .CHAN4[16:19] ==  1 (0x00000001)
        .CHAN5[20:23] ==  2 (0x00000002)
        .CHAN6[24:27] ==  9 (0x00000009)
        .CHAN7[28:31] == 10 (0x0000000a)

$ sudo umr -O bits -r tahiti.gfx600.mmTCP_CHAN_STEER_HI
gfx600.mmTCP_CHAN_STEER_HI => 0x0000543b
        .CHAN8[0:3]   == 11 (0x0000000b)
        .CHAN9[4:7]   ==  3 (0x00000003)
        .CHANA[8:11]  ==  4 (0x00000004)
        .CHANB[12:15] ==  5 (0x00000005)
        .CHANC[16:19] ==  0 (0x00000000)
        .CHAND[20:23] ==  0 (0x00000000)
        .CHANE[24:27] ==  0 (0x00000000)
        .CHANF[28:31] ==  0 (0x00000000)

Whoah!!! It seems like the registers indeed refer to the disabled memory channels:

• TCP_ADDR_CONFIG.NUM_TCC_BANKS indicates 12 channels (value of 11) even though the GPU only has 8 active memory channels.
• TCP_CHAN_STEER_LO/HI refer to channels 2, 5, 8, 11 even though those memory channels are disabled.

Let's try our luck and see what happens if we remove those. We simply need to edit the bit pattern for these registers, I typically use Gnome Calculator in programming mode to toggle the bits by hand.

# Change NUM_TCC_BANKS to 7
$ sudo umr -w tahiti.gfx600.mmTCP_ADDR_CONFIG 0x000002f7
# Use channels 0, 1, 3, 4, 6, 7, 9, 10
$ sudo umr -w tahiti.gfx600.mmTCP_CHAN_STEER_LO 0x43a91076
# We don't have any more channels so clear this to zeroes
$ sudo umr -w tahiti.gfx600.mmTCP_CHAN_STEER_HI 0x00000000

And… Yesss!!! with that, the test doesn't hang anymore, and the GPU starts working well.

What's actually happening on this GPU?

I had a chat with the lead developer of amdgpu Alex Deucher and shared my findings. He helped me understand what's what:

• TCC (texture cache per channel) is the L2 cache that is attached to each memory channel. For disabled memory channels, the corresponding TCC should be also disabled and should not be used (obviously).
• TCP (texture cache per pipe) is the L1 cache that is part of each CU.
• The TCP_ADDR_CONFIG and TCP_CHAN_STEER registers can tell the TCPs which TCCs they are allowed to use. These registers were programmed from the "golden" registers without regard to the fact that some channels may be disabled.

We can now see that the GPU was trying to use the L2 cache that is attached to the disabled memory channels, which (obviously) didn't work and caused the GPU to hang. With that understanding, I wrote a patch. With that patch, the system can boot normally (without a need to force software rendering) and the 7870 XT works fine now.

The patch has been backported to stable kernels too. If you are using a 7870 XT, you can now enjoy it fully working on Linux!

Perf testing

I also did some perf testing to see if it makes any difference how it is configured. I used Rise of the Tomb Raider at 1080p with lowest settings for these tests. This shows that the values mentioned above are indeed optimal for this GPU, so we only need to skip the disabled TCCs but otherwise can keep the order they are in.

What have we learned from this?

It turns out this is one of those rare bugs which only affect a specific variant of a chip and nothing else. The issue with harvested TCCs is a problem unique to Tahiti LE, which was only ever used in the 7870 XT (and the FirePro D500 according to Wikipedia), so the fix doesn't affect other GPUs. There have been no more GCN GPUs with this kind of harvesting (although RDNA1 has variants with harvested TCC as well).

On the bright side, it wasn't a useless exercise for me.

• I learned a lot about how the cache hierarchy works
• It was interesting to see how the cache configuration affects performance
• I feel proud that I was able to solve the mystery

27 Aug 2026 12:00pm GMT

An inconvenient part of GPU driver development is the fact that GPUs can crash and hang for various reasons, just like CPUs. Really, this can happen on any operating system to GPUs from any manufacturer. We've seen many such issues on the open source Linux driver stack for AMD GPUs. We've been tackling many bugs that lead to GPU hangs. We need to deal with the fact that hangs can happen and when they happen, we need to make sure that the system can recover and the user can keep using their Linux desktop. The purpose of this blog post is to give an overview of what the problem is and what steps we are taking to improve it.

Why does a GPU hang or freeze in the first place?

Many users are referring to this topic as "the bug", as if it was just one problem. If only we lazy driver devs fixed just that one problem, life would be better, wouldn't it?

In reality however, there are many different reasons why a GPU can hang, in fact there are so many possible reasons for hangs that it would be impossible to list all of them. We have fixed so many of these kinds of bugs over the years that it's really impossible to even give a list of those bug fixes.

I'll try to come up with some examples:

• Issues with shader instructions, eg. incorrect opcodes, unmitigated hazards, infinite loops, etc.
• On some GPUs, accessing unmapped memory (ie. a page fault) will also lead to a GPU hang
• Invalid commands submitted to the GPU
• Deadlock, ie. waiting for something that never happens (sometimes also caused by coherency issues)
• Missed interrupt which leads to a deadlock
• And many others…

Setting expectations

When talking about this problem space, it's often difficult to judge what we are talking about, what is possible and what isn't, so first let's start with setting some expectations about what we can do.

Firstly, we need to acknowledge that every GPU has different IP blocks (parts such as graphics, compute, DMA engines, display, video decoder, etc.) and each IP block may offer a very different uAPI and programming model. Therefore, each of them can fail in different ways and need to be recovered in different ways, too. It is impossible to handle every conceiveable issue with every IP block the same way. For example:

• In order to use graphics and compute, we have a command submission uAPI which userspace applications can use to submit jobs. Under the hood, user-mode drivers (UMD) use this uAPI to execute Vulkan, OpenGL commands.
• The display engine offers the modesetting uAPI, which is largely shared between all GPU vendors and works on very different principles.
• Some IP blocks like MC (memory controller) and others are not directly exposed to userspace, but are essential for correct functionality. When something goes wrong with one of these, that requires separate consideration.

This blog post focuses on GPU hangs caused by jobs submitted by userspace applications.
The other topic are out of scope for this post.
So, what can we reasonably expect about job submissions?

• When an app (or game) submits commands that crash or hang the GPU, the graphics context of that app is considered "guilty". We need to accept that it won't be able to continue and will crash (unless it was specifically designed to handle GPU failure). In some cases (such as VM faults, known hazards etc.) we can make an effort to try to stop it from crashing, but not always.
• We should do our best to make sure we don't freeze or crash other "non-guilty" applications or the whole desktop. However, sadly, trade-offs need to be made for the sake of performance.

Why is it difficult to deal with GPU hangs?

Let's start with a quick recap of how graphics drivers work. The way the graphics stack handles GPU jobs is the following:

• A userspace driver generates commands for the GPU (and compiles shaders, etc.)
• A job with those commands is submitted to the kernel driver through a uAPI (userspace API)
• The kernel resolves job dependencies, BO (buffer object) state etc. and writes the job to a hardware ring buffer (which is shared accross many processes)
• The GPU executes the commands from the job and when they are completed, signals a fence

The above model has barely any room for detecting and handling errors. The only thing we can detect is that a GPU job didn't complete within a timeout (eg. 2 seconds). We don't even know if that's because the commands are taking too long (and would complete in a longer timeout) or the GPU is stuck somewhere and isn't making any progress. Furthermore, job execution may overlap, so when it hangs, we can't always know for sure which job was responsible.

Actually, it's even worse than that. Modern GPUs have multiple job queues that can execute in parallel and all share the same resources (eg. compute units for running shaders). That means it is not even possible to be sure which queue is really responsible for a timeout: If two different jobs are executing on two queues in parallel, it can happen that the "guilty" job hogs all compute units, causing the other queue to "starve" and time out.

So, where does that leave us?

• We can detect when a job times out
• We don't know which job is really responsible
• We don't know which queue is really responsible
• We do know which jobs were in flight when a timeout happened
• We can't know why the timeout happened

How do we deal with GPU hangs, anyway?

Thanks to the excellent work of Alex Deucher and his team at AMD on the amdgpu kernel driver, various strategies were developed over the years to recover GPUs from a hung state and to make the problems less severe.

Enforcing isolation

Enforcing isolation means that we try to reduce how much an application using the GPU can affect another, effectively disabling parallel execution from different contexts. This greatly limits the scope of what jobs are potentially affected when there is a crash or hang, so makes it less likely that a "guilty" context can crash others.

• Currently, amdgpu does not allow jobs from different contexts to overlap on the same queue. This means when only one queue is active, we can 100% identify the "guilty" context correctly.
• There is a kernel parameter amdgpu.enforce_isolation which will additionally isolate different contexts that are running on different queues. This is currently disabled by default for performance reasons, you can enable it for better stability.

ASIC reset

ASIC reset is the simplest, and most destructive type of reset:

• All pending or in-flight jobs are killed
• The GPU is completely reinitialized
• On dedicated GPUs, VRAM is erased

That means that all processes that used the GPU will lose all resources they may have had in VRAM (except on APUs). The user sees that the screen turns black briefly, then every application and the desktop just crash (unless the compositor and apps were robust). This reset strategy means that just one misbehaving application can cause all other applications and the entire desktop to crash. It is better than watching a completely frozen screen, but not by much.

Soft recovery

Soft recovery was the first attempt at making resets less destructive. What it does is it kills all currently running shaders on a specific queue and hopes that the GPU can then move on and complete the job.

This can solve quite a few issues such as infinite loops in shaders, but it is somewhat dangerous because the kernel doesn't really have any knowledge if it worked, and can only judge by seeing whether the job now completes within a certain time or not. There is no way to know if the current job or subsequent jobs will actually work. So, soft recovery should be avoided when better recovery methods are available.

Queue reset

Queue reset (as its name suggests) attempts to reset just one specific queue without affecting others. There are mainly two ways this can be implemented for different hardware blocks:

• Graphics and compute queues on newer GPUs have a firmware-assisted queue reset, which means it's a feature of the CP (command processor) firmware. It basically terminates all operations (including shaders) that are executing or pending on the specific queue, and then moves on.
• SDMA, VCN and other queues can be reset by simply resetting the whole hardware IP block. These blocks usually only have one single queue, so the reset doesn't perturb anything else.

In my opinion, queue reset is the best way to handle GPU recovery. The only issue with queue reset was that in itself, it would still kill all currently executing or pending jobs from the given queue, even those jobs that haven't started yet. From a user perspective, that means it could still crash other apps and the desktop (unless you enabled enforcing isolation too).

Starting from Linux 6.18, an important improvement was made to queue resets: the kernel can now re-emit pending jobs from other contexts after the queue reset is complete, so in practice it is very likely that only the "guilty" misbehaved app is killed and everything else can continue. This is not 100% guaranteed though, because it is still possible that a job from a different queue can starve other jobs. For the safest user experience, you should turn on enforcing isolation too.

IP block soft reset

I introduced IP block soft reset as a GPU recovery method very recently. This method is more blunt than the queue reset, because IP block soft reset will reset an entire IP block including every queue it has. For the graphics/compute block this means it will practically reset all graphics and compute queues at the same time. The reason why I added this is because it works on GPUs that don't have firmware support for queue reset, or in situations where the firmware failed to do the reset. This method uses the same re-emit code that was added for queue reset so it is very likely to be able to keep your system running after a hang.

The main benefit of this reset method is that it's markedly better than a full ASIC reset because it doesn't erase the contents of VRAM, and it can be used on old APUs where ASIC reset is not available at all.

Starting from Linux 7.3 you can benefit from this new recovery method on GCN 1-4.

But… what about the page flip timeout and other issues?

Page flip timeouts are a different beast entirely, because they are caused by issues with the display engine which has a different programming model and userspace interacts with it using a different uAPI. So this is out of scope for the current post.

However, I need to mention that Leo Li has done some excellent work tracking down the root cause for many page flip timeout related issues.

Recommendations

As you can see, a lot of improvements have been made to GPU recovery recently, so if you experience a lot of GPU hangs, my recommendation is to upgrade your kernel if possible. If that's not possible, consider enabling enforcing isolation.

Here is some simple advice in a nutshell:

• For good GPU recovery
  • Queue resets with re-emit on RDNA ― use Linux 6.18 or newer
  • Queue resets with re-emit on Vega ― use Linux 7.0 or newer
  • IP block soft reset with re-emit on GCN 1-4 ― use Linux 7.3 or newer
  • For the safest experience, use amdgpu.enforce_isolation=1
• If you need to use older kernels
  • On RDNA, use amdgpu.enforce_isolation=1 which will make queue resets work much better
  • You need to accept that Vega and older don't have any decent recovery options on those kernels
• If you use newer kernels but still experience problems
  • Try amdgpu.enforce_isolation=1
  • If that didn't help, open an issue here and please don't forget to mention your system specs, upload a dmesg log and write down the steps to reproduce

Hope this helps!

26 Aug 2026 5:19pm GMT

In 2025 I was honored to be selected for the first cohort of Sovereign Tech Fellows, a program by Germany's Sovereign Tech Agency to improve the resilience of the open source ecosystem by supporting maintainers directly (complementing their existing support for larger FOSS organizations). Back in 2025, I was only working very limited hours - however, this has changed in 2026.

For the second half of 2026, I am working again as a Sovereign Tech Fellow, but this time with significantly increased hours. After finishing my PhD, I do have time now for new tasks (and new jobs!), and the fellowship presents an amazing opportunity to really advance projects that I maintain or am part of. This also has a very nice effect on contributors and bug reporters, as their feedback gets addressed a lot faster. With some luck, this ultimately will help finding new (co)maintainers for projects as well (although in the age of AI, a lot of how open source used to work is much more uncertain, but that is a matter for a different blog post).

The fellowship is time-limited, so I am intending to make the time I currently have count!

So, what's planned?

I am involved in many projects, but three of them will be getting attention as part of the fellowship. I know I am notoriously slow at blogging, but expect more details on each of them very soon. Here's an overview:

Freedesktop.org, Specifications and Organization

I maintain the Freedesktop Specifications, which is an area of Freedesktop that has traditionally been a bit chaotic. This "worked" in the past, because Freedesktop was never intended to be a formal standards body, but more a shared space where people could throw a lot of code and ideas over the wall and see what sticks and what people can collaborate on.

While I very much love the spirit of this and want to keep it in some form, we definitely would benefit not just from more formalization and better procedures, but also from better organization of the specifications in general. A lot of conflicts can be avoided by that. I will work on improving procedures, crunching through the (lots!) of pending bug reports and MRs, and to make the specifications site better searchable and accessible (similar to how Mozilla's MDN presents information, but I am not sure if we will get quite that far). I also intent to add a compatibility matrix for specifications, so if a desktop opts out of any one of them (or does not implement them yet) that fact is documented and authors of applications know what they can expect. This will allow us to move a lot faster and avoid a lot of conflict, because there is no implicit assumption that "everybody will implement everything" anymore (which has never been quite true anyway).

Hopefully, this will ultimately result in a Freedesktop that is both a lot more useful for application authors who want to bring their project to Linux, as well as developers of desktop environments who need to see which specifications are available and which ones are current.

In addition to that, I have also worked on a Freedesktop.org website refresh, which is pretty much done in its first iteration (pending sysadmin action). The aim there is to have a more official website, separate from user-contributed wiki content, that showcases what Freedesktop is and which projects are using it for hosting. Once the new website is live, I will also review every page again, archive dead projects in their own section and reorganize the software and specifications directory. Those sections are severely outdated and are missing recent efforts from the community, while still containing long-dead old projects (remember HAL? ).

AppStream

A lot of extra maintenance work will be (has been!) done on it. This includes things such as JPEG-XL support (blog post soon), sandboxed media processing, support for newer specification additions, better OARS integration (and potentially migrating it to fd.o infrastructure), improvements and API stabilization for libappstream-compose and a lot of bugfixing and resolution of issues found by AI code review.

AppStream was originally designed to parse only trusted data from vetted Linux distribution sources - this is no longer the case in today's world and in the way Flatpak uses it, so we need to increase resilience of the project.

I am also exploring a project that could vastly improve search accuracy for AppStream. Stay tuned for that.

PackageKit & System Upgrades

Many years ago, people thought we would all migrate to atomic Linux distributions and slowly not need PackageKit anymore. This has not turned out to be the case, and there are still plenty of reasons to use a package-based OS, especially in development environments. At the same time, PackageKit has been basically the same for years, and its older architecture is beginning to show. It being a daemon who's literal job it is to modify the entire system also makes it one of the most security-sensitive components that a Linux system can have, while simultaneously making it near-impossible to sandbox.

My plan is to create PackageKit 2.0 by building on the great foundation of PackageKit 1.0, but modernizing it. This will include simplifying its code and removing a bunch of features that have no more use in modern desktops, while also adding some features that PackageKit never had but that would be useful to expose to frontends (still no to interactivity an terminal-progress forwarding though!). PK 2.0 will also allow me to solve a few design issues that have been worked around in the past, by replacing them with better solutions. This will be a painful transition, as PackageKit 2.0 will break all interfaces PackageKit has - and those interfaces have been frozen for more than a decade. However, I do fully expect this change to be worth the effort.

In addition to that, I intend to look into the offline-update procedure again and improve it. The current multi-reboot operation comes with downsides, that newer systemd features such as soft-reboot can alleviate. The end result should be a much smoother, less annoying offline-update experience for users (I especially want to get rid of updates running on system startup, which I consider quite bad from a usability perspective). The new behavior is in the early drafting stages and may need direct support from systemd. I will share more about it once I can.

That's a lot of tasks!

Yes! I will see how far I get. I am moving project-by-project though, to allow me to focus on one project at a time, rather than scattering my attention continuously. Amazingly, this means that the major tasks for AppStream are already almost done, and we are nearing the 1.2.0 release. AppStream got priority, because the new Freedesktop Flatpak runtime will be released soon, and because I want FlatHub/Flatpak to have access to the new AppStream release sooner. Freedesktop and PackageKit are next on the task list.

Either way, a lot of progress is coming - if you have any feedback or want to help out, please don't hesitate to reach out! All work is happening fully in the open, so you can also chime in on the respective GitHub/GitLab tasks .

You can also expect blog posts about key features or interesting changes, so stay tuned!

24 Aug 2026 9:00pm GMT

We have expanded our open-source NPU support for object detection: YOLOX is now running on the Etnaviv driver.

This brings high-performance, open-source AI acceleration to the Vivante VIP line of NPUs, including those found in the NXP i.MX 8M Plus and the Amlogic A311D. Adding YOLOX gives users more options when balancing detection accuracy against available hardware resources.

YOLOX is an open-source object detection model developed by Megvii (Apache 2.0). It is particularly well-suited for edge NPUs, but it is significantly more complex than SSDLite MobileDet (our previously supported model). Getting it running required implementing several new operations in the driver.

As part of this work, we landed support for:

• FullyConnected: A new operation that runs directly on the NN (convolution) cores.
• Reshape, Split, and Concatenate: Handled via metadata changes; these do not execute on the hardware, saving cycles.
• Fused ReLU: Enables activation function hardware on the output.
• Absolute and Logistic: Implemented as lookup table operations on the TP (tensor processing) cores.
• Subtract: Lowered to a convolution, similar to our approach for Add.
• Transpose: Either fused into the next operation or handled as a TP operation.

Additionally, we added support for feature maps in signed 8-bit integers. For certain models, this provides increased accuracy at the exact same computational cost.

This work was performed in partnership with Ideas On Board.

YOLOX is an open-source project developed by Megvii and licensed under the Apache License 2.0.

22 Aug 2026 8:40am GMT

Earlier this year, I blogged about work I did to improve VRAM management for games. Now, after many months of floating around in mailing lists, the kernel patches are finally merged upstream and queued for Linux 7.3! Hooray!

To celebrate, let's look a bit deeper at one sentence I wrote in my previous post:

[Games] should perform much more stable - as long as the game itself doesn't use more VRAM than you actually have.

So, one may ask: What if they do, in fact, use more VRAM than you actually have?

Typical expectations for this seem to be that once this happens you're pretty much screwed. Games will start crashing left and right, performance plummets to unplayable levels, a good gaming experience becomes impossible.

But is that really just an unavoidable fact of life? What really makes running out of VRAM suck so hard? And, most importantly: How can we make it suck as little as possible?

Setting expectations

In theory, running out of VRAM should exclusively be a performance issue, not a stability one. Support for overcommitting VRAM has existed for as long as GPU drivers have: If the driver overcommits VRAM, you are generally allowed to request as much VRAM as you'd like, and you'll get as much as the kernel driver decides it can fit into the physical memory that exists on GPU.

On the performance side, the big-picture reason for bad performance when you run out of VRAM is fairly simple. As soon as the game requests more VRAM than is physically present, some of the game's memory will have to be moved/evicted to CPU RAM instead. For the GPU, accessing CPU RAM is much slower than VRAM: Not only is CPU RAM slower than a dedicated GPU's VRAM in general, all memory accesses also have to go over the PCI bus. The PCI bus adds latency and is typically also the limiting factor in bandwidth when fetching from CPU memory.

Due to PCI speed limitations, there are some truly unavoidable performance constraints when overcommitting VRAM. Assuming the GPU is hooked up via a PCIe 4.0x16 connection, you get a little less than 32GiB/s of bandwidth. Each millisecond, that PCIe bus can transfer ~32.2MiB of data. For a minimum framerate of 30 frames per second (33.3ms per frame), the absolute maximum amount of data the GPU is able to access is ~1,075.5MiB, a tiny bit over 1GiB of data. In other words, if so much memory gets evicted that the GPU needs to fetch more than 1GiB from evicted memory in one single frame, it is simply impossible to still hit 30 FPS.

Not all memory is equal

At the same time, just reading a little bit of CPU memory on the GPU is not immediately a death sentence for performance. In fact, GPU drivers sometimes decide to let things like command buffer data and related allocations live in CPU RAM even when there's plenty of VRAM available! Whenever the GPU executes these commands, it has to access CPU memory, and yet in these cases everything runs completely fine. So what makes these accesses different - why are they fine and yet running out of VRAM seems catastrophic?¹

One thing that influences the calculus significantly is caching. Since the access latency in case of a cache hit is the same regardless of whether the cached memory lives on CPU or GPU, the high initial cost of fetching over the PCI bus can be amortized by cache hits (to some extent). We can estimate latency differences between fetching CPU RAM and VRAM by writing microbenchmarks that measure access latency for different buffer sizes (using an adversarial access pattern to minimize cache hitrates as far as possible). The result you get may look something like this (captured on RDNA3):

As expected, if the buffer fits into L2 (or any higher-level cache), access latencies are exactly the same for memory backed by CPU RAM and memory backed by VRAM, because the data gets fetched directly from cache in either case. At a size of 6MB (the L2 cache size on RDNA3), CPU memory latencies go up to about 2400 cycles per access, while device memory latencies stay within the same rough ballpark. Note that VRAM accesses also go through the Infinity Cache, but CPU memory accesses do not (they hit PCIe directly on an L2 miss). I suspect this is because the Infinity Cache sits directly on top of VRAM, so any access that doesn't hit VRAM also doesn't reach the Infinity Cache.

Obviously, memory doesn't start off with being cached anywhere, so the first access will still have considerably higher latency. Also, losing the Infinity Cache definitely hurts as well: PCIe fetches seem to have somewhere around 7.3x as much latency than an Infinity Cache hit, and around 4.6x as much latency as a fetch from VRAM. This increased latency needs really high cache hitrates to fully amortize the cost of going over PCIe. That means there is only a small set of use cases where using CPU memory has such minuscule slowdowns that you'd actively decide to use it in favor of VRAM when you have the choice. When you're evicting memory from VRAM, there will almost unavoidably be at least some degree of slower performance.

Still, even though slowdown is unavoidable, there is going to be memory where eviction matters more and memory where eviction has a lesser effect on overall perf. Memory that is accessed in very cache-friendly ways is not affected by the slowdown of CPU RAM as much. If the access patterns aren't cache-friendly but the memory isn't accessed very often, things may also still be fine since the GPU only rarely needs to actually fetch data from CPU RAM. There might be many memory allocations where the GPU will only access a small part of the total allocation size, and never even read the rest. If these allocations were to be evicted, you might evict multiple GiBs of data, but still remain well below the 1GiB hard limit of data that is actually accessed per frame.

All of these variables make it surprisingly hard to predict how performance actually pans out in practice when memory is being evicted. But in short: Depending on how much the evicted memory gets accessed and how well these accesses cache, you might just be able to run out of VRAM without (completely) ruining performance!

Confronting reality

We've theorycrafted ourselves all the way towards having performant VRAM overcommitment now. Great! Let's just boot up SteamOS, start some game and crank up the setti-
radv/amdgpu: Not enough memory for command submission.

oh.

As it turns out, running out of VRAM in practice does carry plenty of stability issues with it.

This error isn't quite like a regular "couldn't allocate, out of memory" error, though. Note that the message specifically complains about command submission: RADV prints this message when the kernel returns -ENOMEM when trying to submit commands², but merely submitting commands does not allocate any new resources! All the command buffers were allocated in advance, and clearly their allocation succeeded. Even though all memory was successfully allocated, using it in a GPU submission suddenly results in "out of memory" errors being thrown.

It's time for another kernel adventure! Surely getting the kernel to accept the submission can't be that hard - after all, the kernel already accepted all the allocations³!

The horrors of kernel locking

One thing the amdgpu driver has to do on every submission, before it can direct the GPU to start executing commands, is to make sure that all memory that may potentially be referenced by the GPU commands is accessible. With more modern bindless graphics APIs, you have to assume all allocated memory may at some point get referenced. Therefore, amdgpu will try to make sure all allocated memory is also accessible.

Each memory allocation carries information about which type of memory (for our purposes here, system RAM or GPU VRAM) it can be properly accessed from. Most allocations can be accessed from either CPU RAM or VRAM, and amdgpu will be happy with the memory allocation being in either of these memory types. Some allocations, however, have to be placed in VRAM and VRAM only. If these memory allocations have been evicted to system RAM because some other application allocated VRAM in the meantime, amdgpu will have to move them back into VRAM. Because there is no free VRAM available at all, moving the allocation back requires evicting something else. For some reason, that failed and the kernel reported an out-of-memory condition.

In order to explain why evicting something randomly fails, we'll have to take a small detour to look at how the kernel handles (CPU-side) locking for GPU allocations. In order to evict a memory allocation, you have to acquire a lock associated with that allocation. However, during a submission, you also have to lock every allocation that's referenced in a submission, to prevent some other application from moving the allocation somewhere else while you're busy preparing GPU work. But if another GPU submission is doing the same thing concurrently, you can end up in a situation like this:

If one submit wants to evict an allocation that another submit has already locked, but that other submit also needs to lock an allocation from the first one to make progress, we have a textbook ABBA deadlock condition.

But fear not, the kernel knows how to detect and resolve deadlocks! The details about how deadlock detection works are explained in this kernel documentation page, but in very broad strokes, the kernel associates locking operations with a "transaction" (which basically just keeps track of which locks were acquired). If two transactions would deadlock, one of the transactions is marked as "wounded", and the next time it tries to acquire a lock, the -EDEADLCK error is returned. This error requests the transaction to be aborted: All locks acquired during the transaction should be released, and the transaction is restarted from scratch. In the context of command submission, this just means the driver will restart the process of going over all memory allocations and making sure they're accessible.

So where's the catch? There isn't one. This approach is rock solid and works really well.

At least as long as it's actually implemented everywhere.

In the graphics subsystem, the gritty internals of the wound-abort-retry loop are abstracted using a small helper library called drm_exec. Instead of having to manually track which allocations are locked, and release the locks once you run into -EDEADLCK, you simply use the drm_exec_lock_obj helper. If you study the locking code in TTM, the shared Linux GPU memory management layer, you will notice a profound lack of usage of drm_exec.

Instead, there even is a comment noting that -EDEADLCK will cause eviction to fail. There we go, we found our issue! As soon as this deadlock condition is encountered because of intense memory pressure during command submission, the kernel bails out and rejects the submission instead of retrying.

There already are some patchsets to hook up the drm_exec helper in TTM, sent all the way back in 2024, but those never made it in for a few reasons, among which were some remaining bugs that hadn't been figured out. My work had been cut out for me here: Rebase the patchset on top of my kernel version and figure out what those remaining bugs are.

Rebasing the patchset wasn't too much of a hassle, and figuring out the bugs only took one single week of intense suffering with games randomly hanging 3 minutes into heavy VRAM contention. Not the worst!

I tried resending the patchset with fixes for all bugs I found in the hopes it would get in this time, but there's going to be more work needing to be done with it before it can be merged.

Now that running out of VRAM at least won't crash your apps at random, we can at least properly crank up the settings and look at perf. The initial result gave me an absolutely glorious performance graph like this:

Hold On Where Did All The Perf Go

Figuring out why performance is so garbage requires figuring out what the system is actually doing that's this slow. For broad "what's the kernel driver doing??" questions like that, I like using gpuvis. gpuvis uses kernel tracepoints to build a timeline of things that happened (including "GPU work submission started/stopped", from which the time taken for each submission can be inferred).

Booting up gpuvis with a trace taken while the system is running out of VRAM, the timeline shows a situation like this:

Turns out, most of that time isn't actually spent on handling the submission (that's the gfx_0.0.0 activity), but instead moving around memory in preparation for that submission (sdma0 activity)!

The reason why there are so many buffer moves all the time becomes more obvious if you use gpuvis's event list, together with a filter to show only captured move events for a particular buffer object (I chose one at random here, most buffer objects have a similar pattern):

The list shows quite clearly that contending processes (in this case, gamescope and the game itself) will constantly take turns evicting and moving back the same piece of memory, over and over. That's really bad! And it's very reminiscent of something I wrote in my first blogpost:

Generally, two competing applications can be expected to roughly take turns executing GPU work - first one application submits work, then the other, then the first again, and so on. With that approach, memory would keep being moved back and forth after every single submission. One application gets kicked out and immediately moved back in, kicking the other out (which moves memory back in the next step). All this moving ended up with worse performance than if the memory had never been moved in the first place.

This described an old issue where overly aggressive VRAM allocation would lead to ping-pong-like moves happening constantly. But that issue had since been fixed by simply not trying to claim VRAM when there isn't any free VRAM left, and the kernel only started being somewhat aggressive when I implemented VRAM protection with dmem cgroups. Obviously, this must have reintroduced the ping-ponging somehow.

Conceptually, the design of the dmem cgroup VRAM protection should never result in ping-pong moves, because the kernel is only supposed to evict memory that does not have any cgroup VRAM protection associated with it. Without any VRAM protection, you should typically not be allowed to evict protected VRAM.

The single exception to this rule is memory that absolutely has to live in VRAM for things to work properly. These kinds of memory allocations are always allowed to be moved to VRAM to ensure system stability. Typically, almost nothing coming from an application is really required to live in VRAM for correct operation, but there is one buffer object coming from an application that does: The buffer containing image data to be scanned out to the display⁴.

Display hardware is funky

Not only does the display hardware like scanned-out images to be in VRAM, it also completely skips past the GPU's virtual memory architecture and works with physical addresses exclusively. In consequence, scanned-out images also have to be contiguous in physical memory.

With virtual memory and the power of page tables, typical application buffers are only contiguous in virtual memory, and may be scattered around all over physical memory⁵. The first page of a buffer at virtual address 0x5000 may be mapped in the page tables to point to physical address 0x1234000, but the second page at virtual address 0x6000 might point to physical address 0x4321000, somewhere completely different!

Here is a diagram visualizing the mapping of virtual allocations to physical ones in case where there is a lot of fragmentation (which typically is the case when you're very low on VRAM):

The arrows show page table mappings to physical memory segments for the different segments of the first allocation. They're left out for all other allocations for readability.

If you're allocating display scanout data, this fragmentation is not an option as the physical memory has to be contiguous. This has very, very unfortunate interactions with eviction of other data specifically. Let's assume the scanout data has already been evicted, but now it's time for that data to be scanned out, so it has to be moved back into VRAM.

Simply evicting one buffer won't be sufficient, even if that buffer is the same size as the display scanout data, because evicting it does not result in enough contiguous physical space to place the scanout data in! To make matters worse, the eviction algorithm does not take into account physical memory constraints at all. It is a very simplistic loop along the lines of

while (true) {
   evict(getLeastRecentlyUsedBuffer())
   if (tryAllocate(newBuffer) == SUCCESS)
      break;
}

Using this algorithm (assuming the allocations are arranged in LRU order), even if you evict the first 3 allocations (green, blue, and red), there won't be a large enough space to hold the scanout buffer! Even the largest possible free space is ever so slightly too small, as is visible in this updated diagram:

To find a large enough physically contiguous memory region in our example, every single allocation in VRAM would end up being evicted! In real-world scenarios, I observed up to 4GiB of VRAM being nuked just to make space for scanout images (which are ~32MiB of pixel data per image for a R11G11B10 pixel format). That's going to hurt real hard! Simply the act of moving all that data out from VRAM would already cost at least ~130ms, according to the PCIe transfer rate estimated earlier.

Throwing heuristics at the problem

While scanout is definitely the most egregious failure case here, this issue is more general: There are always going to be certain memory allocations that will be moved to VRAM over and over, potentially kicking out some memory that an application might prefer to stay in VRAM. Resisting this and trying to move the evicted memory back in will most likely backfire.

Even though dmem cgroup protection is not a complete solution to this problem, it does reduce the problem scope by a lot. With cgroup protection, you can be sure that any random app won't try to kick out important game resources willy-nilly. Any memory that does get moved back into VRAM by force probably has a good reason to be in VRAM. Therefore, even with dmem cgroup protection, we should be careful and not try to reclaim evicted memory back by force.

With some iterative testing, I think I've arrived at a set of heuristics that work reasonably well for most cases a game would encounter in the wild (not being too aggressive when stuff gets evicted by important system allocations is one thing, but it also needs to be reasonably quick at reclaiming evicted memory if e.g. the game is paused and the Steam menu runs instead, evicting lots of game memory, and then the game is resumed).

The heuristics work something like this:

1. When the kernel detects an application's memory is being evicted, it enters a "hard throttle" phase for a few milliseconds. During this phase, it does not try moving any memory for that app back into VRAM whatsoever (as long as all memory can be properly accessed, of course).
2. After this period, it switches a "soft throttle" phase, during which it may reclaim free space by moving things back into VRAM, but does not try evicting any memory that other apps have allocated. This period may last up to a few seconds, to make extra sure everything reached a stable state.
3. If the "soft throttle" phase has completed without any further memory being evicted again, the system is assumed to have reached a fairly stable state and restrictions on evicting other applications' memory are removed.

IME, this achieves an acceptable balance between not shooting oneself in the foot with overaggressive eviction of other apps, while still recovering reasonably fast when lots of your memory was suddenly evicted, for example because the game was paused and the user browsed around on Steam instead of playing.

Getting somewhere

With those heuristics in place, let's finally try cranking up the settings for real this time.

I ended up going with Indiana Jones: The Great Circle, since it conveniently exposes a setting for streaming pool sizes that you can mess with to modify VRAM consumption pretty much directly.

Lo and behold, even if the settings are turned up to a somewhat ridiculous point, where the game requests 9GiB of 8GiB VRAM (aka. a whole 1GiB of overcommitted game resources living in CPU memory), performance isn't cratering into oblivion anymore! A 19.6ms per frame average is what I'd still call perfectly playable.

I can also bump the settings to even more ridiculous levels and double the amount of overcommitted memory, with the game requesting 10GiB of VRAM on this 8GiB system (and thus 2GiB of resources being overcommitted). Frametime variance goes up quite a lot at this point, with spikes reaching above 33.3ms happening frequently. The overall average is around 29.8ms which isn't the worst, but especially paired with the variance, this would start being noticeable in gameplay.

While this is already a huge step forward, we aren't quite there yet. The experience under VRAM overcommit can sometimes still be a bit hit-or-miss, and frametimes may noticeably vary depending on which objects in the game you're looking at.

Remember that for actually good eviction performance, it matters a lot how the evicted memory is used by the GPU. Right now, this isn't taken into account at all! If we were able to base our eviction decisions more on how well the application's accesses work with CPU memory, a lot of this variance might simply disappear.

Handing over the controls

The complicated thing about the application's memory access patterns is that they are only really known to the application. Therefore, the driver isn't really able to take them into account as-is. Ideally there would be some API where the application can supply hints to the driver about how well a particular memory allocation is suited to being evicted.

Something exactly like vkSetDeviceMemoryPriorityEXT! The VK_EXT_pageable_device_local_memory extension provides precisely what we need here, by allowing applications to communicate any priority they want for any piece of device memory they want. As long as applications provide reasonable hints through this extension, implementing prioritization in the kernel and then utilizing app-provided priorities has the potential to stabilize things by a lot!

Hooking up priorities in the kernel turns out to be a lot less of an issue than you might expect. The kernel already maintains a Least-Recently-Used list of memory allocations that, on eviction, are traversed in order. For each entry on that LRU list, eviction is attempted until there is enough free space for whatever the eviction was for.

This LRU list provides a good heuristic for which application's memory should be evicted first. Applications that haven't submitted anything in a long while are unlikely to need the memory soon, and since their memory is Not Recently Used, it will appear early in the LRU list and be evicted first.

When an application uses a set of buffers, that set of buffers is moved to the very end of the LRU list in one bulk. However, the order of allocations within that bulk is not explicitly controlled at all. That means once the kernel closes in on some application to evict its memory, which specific pieces of memory get evicted is more or less undefined⁶. A simplified visualization could look something like this:

If the kernel walks the LRU list like this, it would evict the buffer with a priority value of 2 first, even though there are much lower-priority buffers elsewhere in the LRU list. If only the first buffer of priority 2 gets evicted, things might be okay, but if the highly important buffer with priority 4 ends up evicted as well, there are likely going to be problems.

Given that we already know specific priorities for the individual allocations, this LRU list is a very simple place to integrate them. It's as simple as ordering the list entries within a single application by their priority⁷:

Now, when the kernel goes over the LRU list to find something to evict, the very first thing it will find and try to evict are the lowest-priority buffers. The highest-priority buffers are last in the list, and thus only get evicted when evicting all the lower-priority buffers was not enough.

Memory priority adoption in apps

Unfortunately, not all applications actually set priorities via VK_EXT_pageable_device_local_memory. As for native Vulkan applications, I haven't observed any idTech game using the extension directly, at least :/

The D3D side looks a lot better, because vkd3d-proton already uses VK_EXT_pageable_device_local_memory when available, and translates both the ID3D12Device::MakeResident/ID3D12Device::Evict API calls as well as priorities set via ID3D12Device1::SetResidencyPriority to priority values set using the Vulkan vkSetDeviceMemoryPriority command. Lots of D3D12 games utilize at least one of these APIs, so the hints these games provide will now be utilized.

I don't have super solid numbers for how much memory exactly is overcommitted by most D3D12 apps, as they don't typically expose the total amount of VRAM they request in an easy-to-access way like idTech's performance overlay does. However, properly honoring memory priorities generally seems to have a good chance to improve the experience. Performance generally appears more stable over time (because you're not relying on luck with which buffers the kernel evicts as much). In some spots I had a good comparison point at, I suspect it increased performance compared to the kernel evicting random things by up to 30% in the very best case - but again, take this number with a mountain of salt as it depends almost entirely on luck with regards to eviction.

Conclusion

When all is said and done, how well does running out of VRAM hold up?

I'd say it's quite alright! In many cases, you may be surprised how much performance you can retain even when evicting a gigabyte or more of memory! Then again, that's of course a rather optimistic case, and the wrong thing ending up in CPU RAM can very quickly cause very significant slowdowns. Eviction is tricky to get just right, and to an extent, performance will always be dragged down. If a game is struggling to hit 30fps even with everything in VRAM, needing to evict something on top of all that could sometimes just unavoidably result in that 30fps target being missed.

Regardless, what I hope this blogpost can demonstrate is that even if you end up with some memory evicted to system RAM, the slowdown can be manageable. There's measures that drivers (particularly, the kernel driver) can take to make overcommit work as fast as possible, and even applications can do their part in coordinating with the driver stack to mitigate the effects of their memory being evicted. With everything in place, VRAM overcommit isn't really as big of a deal as one may think it is at first sight.

All the work I described here has already been released in SteamOS for some time now (it's both in Stable and Preview. As long as your system is up-to-date, it's good to go!).

A note on upstreaming

Of course, I'm already working on upstreaming all this work so it's available to everyone! However, there's a lot of moving parts and a lot of deep refactors of some pretty core concepts at play here, so it will likely need time to cook before everything is merged upstream.

At the same time, I don't want to put up a blogpost talking about lots of cool code just to finish it with "actually you can't see for yourself, go wait until it's all upstream lol", either.

As a middle ground, I have rebased the kernel work onto a recent upstream version of the kernel and published a git branch here. While it should theoretically yield similar effects, it did not go through as rigorous testing the SteamOS kernel did. There will likely be bugs and instabilities that weren't there in the SteamOS version. Use at your own risk, basically. I don't expect to be maintaining this branch in any significant capacity, as I'd rather focus on getting the patches into upstream properly.

In order to pass through application priority hints to the kernel, you will also need a custom Mesa branch I pushed here. Similar considerations as the kernel branch apply here, as well.

Questions of my own

While I would claim to have a fairly good overview of the driver side of memory management at this point, I am not very familiar with how applications decide on supplying memory management heuristics internally, at all. I would suspect optimizing cases where you've already run out of VRAM isn't exactly the top item on developer TODOs (who knows, maybe the memory scarcity is changing that? :P), so maybe there's some unexplored room for performance improvements there?

If you, dear reader, happen to know about VRAM management for larger games/engines (especially when running out), I'd love to chat! I have a hunch that there's still perf to be gained by making apps and drivers coordinate better, but I'm also plainly interested in how things look from an application developer's point of view.

Footnotes

17 Aug 2026 12:00am GMT

While you (yes, you! no, not you, the one behind you) have been sweltering in the heatwaves of the northern hemispheres (Assisted-by: AI), I've been busy adding graphics tablet support to libei. This is scheduled for the soon to be released libei 1.7.0.

The initial work was done by Jason Gerecke and Josh Dickens from Wacom, I've been extending, polishing and testing it for the last few weeks.

Also, upfront: this only covers the stylus part of a tablet, we do not yet have an implementation for the "pad" part (the buttons, dials, rings, strips).

libei is, of course, the library for Emulated Input, a good-enough transport layer for sending logical input events between processes. We're already using libei as part of the XDG Portal Remote Desktop and Input Capture portals where we've been busy hurtling key and pointer events between the participating parties (and soon gesture events and text).

In the next release of libei, we will now also have "ei stylus" capabilities, i.e. the ability to send tablet stylus events. Getting pointer, keyboard and touch events supported was a long undertaking, everything was new and shiny and needed to be added everywhere in the stack. Now that all this is in place, scuffed and scratched, adding tablet events will be quite simple.

The ei stylus interface

Here's a short outline of how libei handles tablet events because it is, of course, different to how libinput handles them. Logical events are much nicer after all than physical hardware events.

First: we have a new interface: "ei_stylus". An EIS implementation (e.g. your compositor) may provide you, the libei client, with a device that supports this interface and one or more associated regions (typically representing the available screen areas). Typically this will be a separate device to the pointer devices or the keyboard devices but it's not a requirement. The ei_stylus interface comes with a bunch of capabilities you'd expect from a stylus (tilt, pressure, distance, ...) that you can selectively enable to emulate the stylus you want to. So basically, EIS will say "here's a stylus device, I support pressure, tilt, rotation, ..." and then the libei client says "This stylus should have pressure and tilt but nothing else". And then you do the normal thing: send proximity events, send tip down/up events, send data for the various capabilities you've enabled.

Happily for the EIS implementation, libei forces the client to take the guesswork out of everything: if you select the pressure capability, you must send a pressure value when coming into proximity. Where libei is used to forward data from a physical stylus (e.g. via some remoting protocol) it is up to the client to deal with firmware bugs that e.g. won't send data until a few frames in.

Note that there is no "tablet" anywhere. The tablet is represented by the region that the device may interact with. So in some ways every tablet is an on-screen tablet (which makes sense since we have logical events).

Multiple styli

The only quirky thing is how to request multiple styli[1]: libei 1.5.0 has added a "request device" request that allows a client to say "hey, EIS, I want a new device with capabilities pointer, keyboard, ...". And, if you've been a nice client, minding your own business, the EIS implementation may just create such a device for you.

So for the case of multiple styli: if the default stylus (if any) isn't good enough, you can now tell EIS that you want a(nother) device with stylus capability, configure the stylus capabilities once the device shows up and voila, you now have a normal pen, an art pen and maybe even an airbrush represented as logical device in libei. And since they're all separate devices in the protocol, they can be individually tracked and used, much like libinput tracks individual styli.

[1] For the "lots" of users that actually use multiple styli...

22 Jul 2026 4:46am GMT

/me gestures vaguely at everything

Oh, hey, this works now? Great!

libei 1.7.0 (to be released soon) comes with a new interface: "ei_gestures" which, creatively, will allow for gestures to be sent between a libei client and an EIS implementation (typically: a Wayland compositor).

I'm not going to go too deeply into how pinch, swipe and hold gestures work, suffice to say we've had those in libinput (for touchpads) for years now so compositors and toolkits should already support those. And since libei and libinput have vaguely equivalent API layers integrating gestures for libei devices in compositors should be fairly straightforward.

The plumbing layers in the portals exist already too, so adding gestures to libei means that - once the compositors support it - we can have gestures support in remote desktop and input capture implementations without needing to update anything else. Hooray! Join in with me. Hooray! Louder! HOORAY!

For testing I had a (vibe-coded and thus immediately abandoned once testing was complete) gesturemouse utility which translates input events from a mouse into gesture events (depending which button is down). But don't let my lack of be a limit to your imagination, I'm sure you can come up with good use-cases for this.

22 Jul 2026 4:22am GMT

If you've been paying attention (and I know you have, because it'd be embarrassing for you if you didn't) you'd have noticed that libei 1.6 (May 2026) added support for keysym and text events.

libei sends logical events between a libei client and an EIS implementation (typically: a Wayland compositor) but the keyboard interface it had was designed like real keyboards: key codes together with an (XKB) key map. You press one key, the keymap decides what that key means on the compositor side and off we go. This is easy but not always useful.

As of 1.6.0 libei now also supports an "ei_text" interface. A compositor may choose to provide you[0] with a device that supports this interface and that gives you two really nice opportunities.

First, you can now send a key sym. Instead of sending the KEY_Q key code and hoping it actually translates to 'q' (and if there's e.g. a frenchman^Wfrenchperson lurking behind the keyboard it may mean 'a'), you can now send 'q' as actual keysym. Or 'Q' instead of sending shift+q and hoping for no french influence in the process. It becomes the EIS implementation's job to handle that keysym - if it's a shortcut it may handle it directly, otherwise it may pass it on via Wayland to an application[1]. This centralises the keysym to keycode handling in the EIS implementation which is a pain for compositor authors (though they likely have that code already for e.g. RDP support) but reduces the variety of differently-wrong implementations in clients and of course makes it so much simpler to write clients.

Second, a client can send UTF-8 text to the compositor. So instead of emulating shift, keycodes, etc. you can literally send "Hello World" and expect the EIS implementation to pass that one. Again, makes a bunch of utilities a lot simpler to write and I mostly leave it up to your imagination to figure out what to do with that.

Notably for both cases: libei is about logical events that have a specific meaning that do not need further interpretation. If a client sends 'Q' that means it is supposed to be an uppercase Q. Sending keysym Shift_L and Q makes little sense. And for the utf8 text events: how the text comes to be matters doesn't matter for libei so you may use an IM to make up the text to begin with and send it, once committed, to EIS. It's not for sending partial strings.

As mentioned in the previous post: the plumbing for this is already in place so both clients and compositors can add support for this new interface without having to bother the rest of the stack (e.g. portals). So, hooray I guess.

The text/keysym support is relatively recent so expect this to hit the next compositor version (or the one after that).

[0]: the EIS implementation decides which devices are available and arguing about this is even less useful than arguing with a world cup ref
[1]: after converting it to a key code with possible keymap changes... but hey, such is life

22 Jul 2026 4:16am GMT

Turns out it's been years since I've talked about eggs, so let's change this. libei is, of course, the library for Emulated Input[1].

This post is mostly a refresher because it's been so long and a short summary of some of the work we've done so far, in preparation for some more posts that come soon.

libei is a transport layer for logical input events, unlike libinput which is a hardware abstraction layer. In libinput's case the device's firmare/kernel pass events that are somewhere on the sanity spectrum, libinput tries to make sense of those and then we convert those to logical events to be consumed by the next layer (typically the Wayland compositor or Xorg). This is how e.g. "touch down at position x1/y1, touch up at position x1/y2" is converted into a button click event if touchpad tapping is enabled. Or maybe into nothing if we find it was an accidental palm touch.

libei works purely on the logical level - you as the libei client pass logical events to the EIS (Emulated Input Server) implementation (typically the compositor). No guesswork, you say button click, EIS gets a button click. libei supports a "sender" and "receiver" mode, depending on whether events are sent to the EIS implementation (input emulation) or receive from the EIS implementation (input capture). libei is designed for the Wayland stack but there are zero requirements for Wayland on either the client or the EIS implementation.

Core to libei's design is that the EIS implementation is in control of virtually everything, it decides which devices are available to the client, when those devices can send events, etc. Much like the compositor is in charge when it comes to physical devices - if a compositor decides a physical device doesn't exist, a Wayland client cannot get events from it.

Since the original proposal (again, [1]!) we've been busy bees and libei is now a part of the XDG Remote Desktop portal and the XDG Input Capture (both since version 1.17, mid 2023). In both cases the portal is for the negotiation and initial agreement of what should happen, libei is then used as the transport layer between the two processes [2].

More recently we also added session persistence support so you don't have to allow access on every connecton. Much of the work enabling this was done by Jonas Ådahl, it is now in the portals since version 1.21.0 and should be in the major compositors in the current or next versions.

Plumbing the Pipes

Getting all this into place was a huge amount of work across several pieces of the stack. This isn't exciting in the same way as laying plumbing pipes isn't particularly exciting but much like regular plumbing: once it's in place you can change your diet without severely impacting everyone again. Try get that analogy out of your head now. You're welcome.

In libei's case this means three things:

• if you have a client that uses the XDG portals to send/receive events they will now work with any compositor that implements the portal. No need for GNOME/KDE/... specific APIs.
• if you have a compositor that implements EIS you have all the infrastructure in place to talk to libei clients from somewhere else, if need be. The use-cases for this aren't fully scoped yet (assisitive technologies, virtual keyboards, touchpads, etc?) but the piping is there and ready to be (ab)used .
• since the actual events back and forth don't affect the layers in between, we can now add new events to libei without having to change everything else again.

Let's look at how this works in practice.

The XWayland XTEST use-case

An example for such a case where we can now abuse the piping is Xwayland support for XTEST. XTEST is the protocol that everyone uses to emulate input under X but in Wayland it's not hooked up to anything so those APIs simply won't work.

But what we can do in Xwayland is translate XTEST to libei events and facilitate the portal interaction. This means our stack looks roughly like this:

    +--------------------+             +------------------+
    | Wayland compositor |---wayland---| Wayland client B |
    +--------------------+\            +------------------+
    | libinput |   EIS   | \_wayland______
    +----------+---------+                \
        |          |           +-------+------------------+
 /dev/input/       +-----------| libei |     XWayland     |
                               +-------+------------------+
                                                |
                                                | XTEST
                                                |
                                         +-----------+
                                         |  X client |
                                         +-----------+

And if said X client uses XTEST to try to emulate devices, Xwayland will ask the Remote Desktop portal for permission and set up the session, then pass the XTEST events on as libei events and voila - your 20 year old X client can send pointer and keyboard events through an XDG Portal without knowing about it (and the user can prohibit this and even gets some information on who is sending events which is not possible with normal XTEST at all). This has now been supported since Xwayland 23.2.0. Compositors don't need extra support for this.

What's next

So we have a lot of the plumbing in place, or in another anology: we have a hammer, let's go looking for nails. And right now the nails we can see are sending text, gestures, and tablet support. And those will be the subject of the next few posts.

[1]: 6 years ago?! whoah...
[2]: in Remote Desktop's case replacing the DBus emulation APIs which were a Newton's Cradle of wakeups for at least 4 processes per event

22 Jul 2026 4:07am GMT