fidzu : gentoo

In November 2025, I've started working on steve, the jobserver for Gentoo. I wrote about it already, in the "One jobserver to rule them all" post. Back then, my main focus was on the motivation for a system-wide jobserver, and the technical details of getting it working. I have also mentioned a few client bugs we've discovered along the way.

Since then, we've found a few more bugs, as well as problematic design patterns. I think they're kind of interesting, so I've decided to dedicate this post specifically to them. As a disclaimer, my aim is not to pick on specific projects; I'm bringing them up as real examples of what we've hit, and how that impacts jobserver operation.

Missing token release on error

As I've mentioned in the previous post already, the POSIX jobserver implementation relies on clients fully accounting for tokens. Clients "acquire" (read) tokens when they need to start a job, and they "release" them (write them back) when the job's done; and they must eventually release all the tokens, or they're going to be lost and other clients won't be able to acquire them. The documentation is pretty explicit on that:

Your tool should be sure to write back the tokens it read, even under error conditions. This includes not only errors in your tool but also outside influences such as interrupts (SIGINT), etc. You may want to install signal handlers to manage this write-back.

POSIX jobserver (GNU make)

Is this hard? Well, it depends. For a start, there's a lot of signals to cover. There are the more obvious ones, like SIGINT and SIGTERM. There are the less obvious ones, like SIGSEGV and SIGILL; I mean, do you really need to catch them? I'd say a robust implementation has to. And then there is SIGKILL that you really can't catch.

Given that you can't catch SIGKILL, an indiscriminate OOM-kill could easily result in tokens being lost. Hence, the whole architecture of steve is built upon the idea that we can't rely on clients releasing tokens reliably and we need to work around that. Still, as a feature it can log whenever a client exits without releasing all the tokens. That way, I was quite surprised to learn that GNU make itself does not return jobserver tokens on SIGINT. Well, bugs happen.

But then, there are harder cases. For example, jobserver-rs can't install signal handlers, because it's a library. Any program using it needs to take care of that, and they do not necessarily do.

Missing error handling

Missing handling for signals is one thing. However, GCC did not implement error handling in jobserver code. It just assumes that the named pipe will open successfully, that reads will succeed and so on. If anything bad happens, GCC will crash with an ICE somewhere later on.

If this could be bad with a plain named pipe, having it backed by steve makes things even worse. Steve can literally crash (or be stopped by the user), and then suddenly all the clients connected to it can explode in a variety of ways. Admittedly, this is a corner case and I find it hard to argue what's the best behavior here; though I suppose, say, disabling jobserver support and proceeding with the implicit slot is still better than suddenly crashing.

Overzealous file type checks

The jobserver protocol was originally built using named pipes. However, to the best of my knowledge, it is impossible to build proper token accounting on top of that, and therefore avoid the risk of losing tokens. For this reason, steve is using a character device governed by CUSE. Some other implementations are using files governed by FUSE.

While GNU make itself never checked the underlying file type, many implementations do: we've removed the named pipe requirement in LLVM and in pytest-jobserver. This is not strictly a bug; rather an implementation choice that's impossible for us to satisfy.

Alexander Monakov proposed an interesting suggestion why that is the case: the original GNU make implementation checks the file descriptor type when using the older jobserver variation based on anonymous pipes. That implementation relies simultaneously on the child process inheriting the file descriptor, and the environment variable specifying its number. In a pathological case, the file descriptor could be closed and replaced by some other open file; in which case checking if it's a pipe is a last resort protection against starting to read or write a random file. However, when passing an explicit path rather than file descriptor, such a check is not really necessary.

Acquiring tokens in a child process

Traditionally, jobserver integration is part of the job scheduler. The scheduler acquires tokens, starts new jobs and releases them once finished. However, there is at least one case where that's not the case: pytest-jobserver plugin hooks into child processes rather than the scheduler.

Such an implementation is much easier, since it seamlessly integrates with pytest-xdist, requiring no changes to the scheduler. However, it also has a major limitation: you can't dynamically adjust the job count. The xdist plugin starts the specified number of jobs (which effectively becomes the upper bound), and the jobserver plugin throttles the actual test execution based on the availability of job tokens.

From steve's point of view, the limitation of this approach is that every job token is associated with a different process. Features such as per-process job limits or round-robin token delivery can't work reliably, because steve can't really associate tokens across processes (at least for the time being).

By the way, another side effect of this design was that originally pytest-jobserserver did not obtain job tokens for test collection.

Problems with implicit slot

Another problem I've mentioned already is implicit slots. Per the documentation:

Second, every command make starts has one implicit job slot reserved for it before it starts. Any tool which wants to participate in the jobserver protocol should assume it can always run one job without having to contact the jobserver at all.

Job Slots (GNU make)

This design makes sense. The job starting GNU make (or any other jobserver client) should be covered by a job slot already, either implicit or using a job token. While starting nested jobs, the client uses little CPU time, and it makes no sense for it to block another job slot. Besides, if not for that, every nested make invocation would consume one more job token, and you'd soon run out of tokens.

However, it is easy to miss this fine point and actually try to acquire a token for every job started. As a result, you end up running one job too few. On top of that, it's not always actually easy to implement that. For example, pytest-jobserver implementation ended up special-casing xdist job "gw0", which is imperfect and could lead to locking: "gw0" uses the implicit slot even if it has nothing else to do, and any other job will have to acquire another token to finish its tasks.

On the other hand, nasm-rs implemented the implicit slot by writing an extra job token to the pipe. I mean, for named pipe it is sound: by adding one more token to the pool, you account for the parent process not needing one, the children get as many tokens as they need, and the total job count matches. However, for steve it meant we actually had to allow processes to temporarily hold −1 token. Doable but kinda ugly.

Writing back the wrong token

There's another curious point in GNU make documentation:

It's important that when you release the job slot, you write back the same character you read. Don't assume that all tokens are the same character; different characters may have different meanings to GNU make.

POSIX jobserver (GNU make)

Can you guess which implementation did not comply?

Yes, of course GNU make did not write back the same character. GCC also does not preserve the token. Admittedly, I suppose steve is the first jobserver to actually use characters with different meanings. I've used that to distinguish different jobs, and therefore be able to tell how long a particular job is running. It's not a critical feature, it can be helpful when debugging though.

Holding on to tokens throughout multiple jobs

What would you debug, actually? Well, for example you could notice that whenever Cargo is building something large, everything else stops. For example, you start five more emerge processes, and not one of them is able to start. You SIGUSR1 steve and confirm that Cargo is holding all the tokens for a few minutes now.

So you double-check the code, and file an issue: Cargo does not return tokens immediately after finishing a single job. But is that a bug? Arguably, the specification doesn't prohibit that. Apparently it works out better when building Firefox. For building multiple packages in parallel, it is much worse, though.

What's to come?

Jobservers aren't anything new. However, the widespread interest in jobserver support seems to have appeared recently. If I were to hazard a guess, that'd be due to a combination of large projects becoming more heterogeneous, and toolchains becoming more complex. It's perfectly normal to be building a large project comprising of C, C++ and Rust sources, and possibly including some bundled libraries that add both ninja and make to the mix. On top of that, LTO can overlay its own concurrency. The jobserver protocol is the obvious solution to throttling, and it stood the test of time. It's supported by ninja, by Cargo, by GCC… and the list continues to be growing.

The idea of using a shared jobserver isn't new either. Jobserver integration was proposed for Gentoo a long time ago. NixOS experimented with jobservers as well. However, steve is pretty new and by design inevitably different from GNU make. We've found many issues. And as testing continues and even more client implementations emerge, we're bound to find more.

As a footnote, many different people have been repeatedly pointing out the limitations of the protocol, as well as announcing that they're working on something new and better. I don't hold my hopes high, and honestly, I don't want to have to bother with another big transition. The GNU make protocol may be ugly, but it's well supported and there are reasonably good ways to make it work. Steve has proven that.

02 Sep 2026 5:40pm GMT

Bumping packages is one of the most common tasks of a Gentoo developer. It shouldn't then be surprising that it is the one most asking for some kind of automation, and that the pkgbump script would be one of the first scripts to become a part of the mgorny-dev-scripts package.

Today's pkgbump have come a long way from the trivial script of its first iteration. The most recent versions finally feature the feature I desired for a long time: version manipulation. This also made it possible for the script to become a complete version bumping tool rather than just a part of a larger workflow. In this post, I'd like to shortly tell the story behind the changes, and demonstrate the new options.

The first iteration

The initial version of pkgbump was quite trivial. It took two paths: source and destination. It copied the ebuild file, lowered the keywords, bumped the copyright date, updated the Manifest and run the pkgdiff (now pkgdiff-mg) tool to compare the unpacked archives. So a typical workflow would look like:

cdpkg svglib
pkgbump svglib-2.0.{1,2}.ebuild
vim svglib-2.0.2.ebuild  # if necessary
# test the package in another terminal
pkgcommit -sS . -m 'Bump to 2.0.2'

Obviously, there's some duplication here, and a potential for improvement. And over time, some improvements would happen. I've added support for removing obsolete Python implementations from PYTHON_COMPAT, support for PKGBUMPING variable that would be used to skip unpacking crates when diffing and integration with pkgcommit, so instead of repeating the version number, you'd do:

pkgcommit -sS . --bump

Bumping groups of packages

If pkgbump was sometimes cumbersome to use, that would be especially felt in groups of packages such as dev-python/botocore, dev-python/boto3 and app-admin/awscli. These three packages are released simulaneously, usually 5 times as week, often with slightly different version numbers. So you'd end up doing something like:

pkgbump botocore-1.24.{0,1}.ebuild
pkgbump boto3-1.21.{0,1}.ebuild
pkgbump awscli-1.22.{55,56}.ebuild

Thus, bump-boto was born. In its initial incarnation, it would take the old and new patch versions, and compute all the remaining numbers. So you'd just invoke:

bump-boto 0 1>

In this initial version, I'd have to update the script whenever upstream changed the minor version number. The next incarnation would actually grab the minor version number from the repository, but would still require changes whenever the alignment between patch numbers changed.

Eventually, I would replace the patch version arguments with an "increment". So you'd just specify:

bump-boto +1

And it would increment the patch version of all packages by one. Of course, whenever the minor version or alignment changed, I would still have to pkgbump them manually, but bump-boto would work just fine for the subsequent release, no modifications needed.

For a long time, this was a function unique to bump-boto, with a pretty dumb implementation. However, the period of frequent kernel updates, requiring me to keep retyping 7 version pairs, finally motivated me to make it more generic. Just imagine typing the equivalent of, twice a day:

bump-kernels 7.0.{12,13} 6.18.{35,36} 6.12.{93,94} 6.6.{142,143} 6.1.{175,176} 5.15.{209,210} 5.10.{258,259}

Generic increments in pkgbump

The latest versions of pkgbump still accept two positional arguments: source and destination. However, they do not have to be filenames anymore.

The destination can be a version number or an increment instead. So you could do either of:

pkgbump gentoo-kernel-7.0.12.ebuild 7.0.13
pkgbump gentoo-kernel-7.0.12.ebuild +1

What about the minor version changing? The increment can be followed by one or more version components, in which case the final components are set to the specified value, and the one preceding them is incremented. So you can do either of:

pkgbump gentoo-kernel-7.0.12.ebuild 7.1.0
pkgbump gentoo-kernel-7.0.12.ebuild +1.0

Or:

pkgbump gentoo-kernel-7.0.12.ebuild 7.1.2
pkgbump gentoo-kernel-7.0.12.ebuild +1.2

The source can be a pattern instead, in which case the script finds the ebuild with the highest version number matching it:

pkgbump 'gentoo-kernel-7.0.*.ebuild' +1
pkgbump 'gentoo-kernel-6.18.*.ebuild' +1

At this point, I could make the arguments optional. If source is omitted, '*.ebuild' is assumed. If destination is omitted, an increment of +1 is assumed. So the following are all valid:

pkgbump 'gentoo-kernel-6.18.*.ebuild'  # dest.: +1
pkgbump +2  # source: '*.ebuild'
pkgbump 1.2.3  # source: '*.ebuild'
pkgbump  # source: '*.ebuild', dest.: +1

More options for an integrated workflow

At the same time, I've added a few options to pkgbump to make it more suitable for an all-in-one tool. So effectively the process becomes, depending on options given:

1. Copy the ebuild, copybump, clean it.
2. Lower the keywords, unless -s / --stable is passed.
3. Run the editor to edit the ebuild, if -E / --edit-early is passed.
4. Update Manifest, unless -M / --no-manifest is passed.
5. Diff the working directories, unless -D / --no-diff is passed.
6. Run the editor to edit the ebuild, if -e / --edit is passed.
7. Commit the changes, if -c / --commit is passed.

So it would now be possible to reduce the initial workflow to a single call:

pkgbump -e -c

And that's it! It means "pick the latest ebuild, increment the final version component by one, manifest, diff, edit and commit."

Bumping kernels the easy way

This meant that I could finally make bumping kernels easier. The scripts are still work-in-progress, but now I can specify the branches instead of version pairs, to increment them all by one:

bump-kernels 7.0 6.18 6.12 6.6 6.1

Or I can just run the script without arguments to have it determine the available branches and bump them all:

bump-kernels

Snippets for the future

One of the features of pkgbump is that it attempts to do some minimal cleanup / modernization of ebuilds. For example, quite early I've added the cleanup of old Python implementations from PYTHON_COMPAT; it's something that's not urgent enough to justify the noise of mass updating, but worth doing if you're bumping the ebuild anyway.

Lately I wanted to add another similar function: updating the deprecated DISTUTILS_USE_PEP517 values. Again, this is nothing urgent; it will be required for EAPI 9 though. And similarly to PYTHON_COMPAT updates, it's quite specific to Python. It felt that pkgbump would end up with a lot of Python-specific logic for a generic tool.

But why restrict yourself to Python? I'm definitely open to adding other kinds of cleanup functions that people submit; except that keeping all that different logic inline feels kinda ugly. So the most recent version of pkgbump features cleanup snippets!

The Python-specific code is gone from the tool (only the generic copyright bumping and keyword lowering logic remains in the core), and now lives in /usr/lib/pkgbump.d/python. New scripts can also be dropped into that directory, either as part of mgorny-dev-scripts, or separately packaged. And there's /etc/pkgbump.d for user scripts, which can also be used to override system scripts. So if you don't want pkgbump to be doing its Python magic, you can drop a replacement /etc/pkgbump.d/python.

21 Jun 2026 3:29am GMT

I joined Gentoo over 15 years ago. I was a university student back then. I had lots of free time and energy. I was enthusiastic about novelties, and wanted to push them into Gentoo. Back then, bleeding edge was what I wanted out of it. Today, I have different priorities. I have less time to deal with breakage, and I want my Gentoo stable. I'm becoming somewhat wary of changes, and I find preserving what's great about Gentoo more important than adding new stuff. And what's really great about Gentoo is that it can accommodate both personas.

Gentoo has changed over these 15 years too. However, its core principles remained the same, and only recently I realized what they really are. The core value of Gentoo is respect. All the building from source, all the choice and flexibility, and all the community building power is because of this: Gentoo respects you. It doesn't try to waggle the dog, it just does what you tell it do. It may warn you that you're having a very bad idea and nobody will help you if you proceed, but in the end, you are free to pursue it.

However, respect goes beyond providing a working distribution for our users. It's in providing a reasonably vanilla development environment for software authors. It's in submitting patches upstream to ensure that everyone gets bug fixes. But most importantly, it lies in appreciating the human craft rather than taking the easy way out. And I believe that rejecting LLMs is important to keeping the Gentoo community whole and respected.

These days, I mostly handle Python packaging in Gentoo, build Distribution Kernels and a variety of odds and ends. I try to balance involvement in interesting high-level projects and the necessary ground work. I am employed at Quansight PBC where my work also primarily orients around Python packaging, but involving conda-forge and upstream work; it does not conflict with my Gentoo duties.

11 Jun 2026 11:41am GMT