04 Jun 2026

feedPlanet Debian

Reproducible Builds: Reproducible Builds in May 2026

Welcome to the May 2026 report from the Reproducible Builds project.

These reports outline what we've been up to over the past month, highlighting items of news from elsewhere in the increasingly-important area of software supply-chain security. As ever, if you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website.

In this month's report, we cover:

  1. Debian to ship reproducible packages in forky and beyond
  2. Holger Levsen on reproducing official Debian packages
  3. Reproducible Builds 2026 summit to be held in Gothenburg, Sweden
  4. Kettle: Attested Builds for Verifiable Software
  5. New rebuilderd version announced
  6. Reproducible open source messengers
  7. Distribution work
  8. Misc news
  9. Patches
  10. Documentation updates


Debian to ship reproducible packages in forky and beyond

In a huge change in Debian's reproducibility policy, the Debian Release Team announced that:

… we've decided it's time to say that Debian must ship reproducible packages. Since yesterday, we have enabled our migration software to block migration of new packages that can't be reproduced [on reproduce.debian.net] or existing packages in testing that regress in reproducibility.

That is to say, if newly-uploaded packages are not reproducible, they won't be considered candidates for inclusion in the next stable release of Debian codenamed forky. (Some exceptions may be granted.)

This news generated a number of articles and comments in various news outlets:


Holger Levsen on reproducing official Debian packages

Reproducible Builds developer Holger Levsen gave a talk at the 2026 Hamburg MiniDebconf this year on the topic of reproduce.debian.net - reproducing what is distributed from ftp.d.o.

Holger's talk announced that Debian intends to ship only reproducible packages in forky and beyond (see above), but also talked more broadly about reproducible builds, our testing framework and the Debian archive. That is to say, moving away from testing whether a package is reproducible in a theoretical sense (eg. whether we can build it twice in different environments and achieve the same result in our test system), and attempting to reproduce the same .deb files in the official Debian archive itself. This small-sounding distinction is actually essential, as this is the only means through which the reproducible builds technique can determine whether build systems are compromised are not.

A video (32m37s) of the talk is available, as are Holger's slides.


Reproducible Builds 2026 summit to be held in Gothenburg, Sweden

As initially announced in March 2026, we will be having our yearly Reproducible Builds summit 2026 in Gothenburg Sweden, from September 22 until 24, followed by two days of hacking!

Further information will be provided on our website and on the rb-general mailing list very soon.


Kettle: Attested Builds for Verifiable Software

André Arko and Amean Asad published a paper this month on Kettle, a build system that "produces cryptographically verifiable provenance for software built inside Trusted Execution Environments":

A Kettle build records the source commit, dependency set, toolchain, build environment and output artifact digests in a provenance document produced inside a measured confidential VM. The SHA-256 digest of that document is committed to the TEE platform's attestation report-data field, so the hardware-signed attestation report is itself the signature on the provenance, with the signing identity chaining to the TEE manufacturer's root of trust rather than to the build infrastructure operator. Because the CVM image is itself reproducible, its launch measurement is public and stable, which lets a build requester pre-attest the CVM before submitting any input and optionally deliver source over a TLS channel terminated inside it, so the build runs end-to-end confidentially without the host ever seeing source code in plaintext.

A PDF of the paper is available online.


New rebuilderd version announced

rebuilderd, our server designed for monitoring the official package repositories of Linux distributions and attempt to reproduce the observed results there; it powers, amongst other things, reproduce.debian.net.

A new version, 0.27.0, was released this month, with the following headline changes:

As kpcyrd's announcement mentions:

The new rebuilderd package is currently available in the extra-testing repository. Note the Arch Linux package is upgraded from v0.25.0 from v0.27.0; please be patient with the database migrations on first restart, and make yourself familiar with the breaking changes in v0.26.0 too.


Reproducible open source messengers

GitHub developer BarbossHack is maintaining an repository/page on GitHub to "track reproducibility status of open source messengers".


Distribution work

In Debian this month, the loong64 architecture was added to reproduce.debian.net. This is a 64-bit Reduced Instruction Set Computer (RISC) instruction set architecture developed by Loongson.

Vagrant Cascadian performed Non-Maintainer Uploads (NMUs) in Debian for several packages with outstanding patches over a year old. These included rocdbgapi, onevpl-intel-gpu, python-pytest-shell-utilities, python-mt-940 and pympress.

On tests.reproducible-builds.org, Vagrant Cascadian fixed the huge spike in build failures by adding passwd to the base tarballs, and re-enabled building gcc and binutils packages with PGO (Profile Guided Optimization) and LTO (Link Time Optimization) to avoid giving a false sense of reproducibility.

Inconsistencies on the reproducibility of the condor package were brought up on the Debian reproducible-builds mailing list. Following a hunch, Vagrant Cascadian eventually identified the issue was related to embedded kernel versions which was then fixed upstream and fixed in Debian as well.

Lastly, 40 reviews of Debian packages were added, 68 were updated and 75 were removed this month adding to our knowledge about identified issues. A number of issue types were updated, such as the addition of a new sphinx_reading_durations toolchain issue […], a golang_mango_generates_manpages_with_build_date issue […] and a random_offset_id_in_cython_linetrace […]. In addition, the timestamps_in_qhc issue was "refocused" to timestamps_in_qhc […].


In Fedora, Jelle van der Waa submitted a request for an official Fedora rebuilderd package which was reviewed by Neal Gompa.


Lastly, Bernhard M. Wiedemann posted another openSUSE monthly update for their reproducibility work there.


Misc news

On our mailing list this month:


Patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where applicable or possible. This month, we wrote a large number of such patches, including:


Documentation updates





Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

04 Jun 2026 7:12pm GMT

Jonathan Dowland: mount namespace for backup jobs (by hand)

It's been ten years since I configured mount on demand backups to reduce the risk of my backups being zapped by mistake. Way back then I wanted to go one step further and use dedicated mount namespaces for backup jobs, but systemd didn't provide the necessary support (and still doesn't, despite the promisingly-named JoinsNameSpaceOf= configuration option.)

I recently updated my setup to achieve this by hand. All backup jobs now have an extra pre-start instruction ExecStartPre=mkbackupns which runs a shell script to either set up a persistent mount namespace, or exit quietly if it already exists.

#!/bin/bash
set -euo pipefail

nsdir=/var/namespaces
nsfile=$nsdir/backup
nsfilex="$(echo $nsfile | sed 's#/#\\/#'g)"

private_propagation() {
    findmnt -o+PROPAGATION "$nsdir" | grep -q private
}
nsfs_is_mounted() {
    test "nsfs" = "$(awk "/$nsfilex/ { print \$3 }" /proc/mounts)"
}

if ! nsfs_is_mounted; then

    if ! private_propagation; then
        mkdir -p "$nsdir"
        mount --bind --make-private "$nsdir" "$nsdir"
    fi

    touch "$nsfile"
    unshare --mount="$nsfile" true

    nsenter --mount=/var/namespaces/backup mount /dev/phobos_backup/backup /backup
fi

I should note that I don't have the backup filesystem described in /etc/fstab to reduce the risk of it being mounted errantly in the main namespace.

The other change is to prefix an invocation of nsenter for every backup job command. E.g.:

ExecStart=/usr/bin/nsenter \
        --mount=/var/namespaces/backup \
        borgmatic -v 1 prune create

next steps

My backup scheme has lasted a decade with few tweaks (I moved it to Borg in 2020) which I am very grateful for. I want reliable, boring and robust.

Persistent mount namespaces are a lot less convoluted if you have a persistent process to associate them with. I didn't, but a subsequent improvement I am making is introducing one, so I will likely simplify the above accordingly.

04 Jun 2026 10:15am GMT

03 Jun 2026

feedPlanet Debian

Emmanuel Kasper: Running Linux i386 binary (steamcmd) via debootstrap foreign chroot

The Steam command line client, which I need to download the game data for the Doom3 BFG shooter, is only available as an Linux i386 binary. As my main home computer is an arm64 box, this could be an issue, but today we have no less than three different ways to run a Linux i386 binary on arm64: Fex, Box32/64 and the older qemu-user mode. According to the Box64 benchmarks, qemu-user is the slowest of the three. But since this is only to run a command line tool downloader, where network speed is the bottleneck, this doesn't matter a lot.

Running steamcmd outside of a chroot via qemu-user and dpkg multiarch support was failing me with the error i386-binfmt-P: Could not open '/lib/ld-linux.so.2': No such file or directory even after installing the i386 libc. So I went the way of qemu-user and a chroot environment, a bit more convoluted but I can run any i386 binaries there in the future.

Create a debian-i386 chroot environment via deboostrap:

$ sudo apt install qemu-user qemu-user-binfmt debootstrap
$ fakeroot debootstrap --foreign --arch=i386 debian-i386
$ sudo chroot debian-i386
# inside the chroot 
# /debootstrap/debootstrap --second-stage 
# exit

Add needed mounts to run binaries inside the chroot:

$ sudo mount --bind /dev/ debian-i386/dev/
$ sudo mount --bind /dev/pts debian-i386/dev/pts
$ sudo mount -t proc none  debian-i386/proc/

Install steamcmd in the chroot client:

$ sudo chroot debian-i386

# export LANG=C
# cat /etc/apt/sources.list
deb http://deb.debian.org/debian stable main contrib non-free
# apt update && apt install --yes steamcmd 
# useradd --create-home --shell /bin/bash steam
# su - steam
$ steamcmd 
... will download an updated version of the tool, and print a lot of tracing information

Steam> quit

From now on you can follow the Doom3 BFG instructions to download the game data.

Once you exit the chroot, the game data will be available at debian-i386/home/steam/

03 Jun 2026 8:50am GMT