05 Feb 2025

Planet Debian

Reproducible Builds: Reproducible Builds in January 2025

Welcome to the first report in 2025 from the Reproducible Builds project!

Our monthly reports outline what we've been up to over the past month and highlight items of news from elsewhere in the world of software supply-chain security when relevant. As usual, though, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website.

Table of contents:

reproduce.debian.net

The last few months saw the introduction of reproduce.debian.net. Announced at the recent Debian MiniDebConf in Toulouse, reproduce.debian.net is an instance of rebuilderd operated by the Reproducible Builds project. Powering that is rebuilderd, our server designed monitor the official package repositories of Linux distributions and attempt to reproduce the observed results there.

This month, however, we are pleased to announce that in addition to the existing amd64.reproduce.debian.net and i386.reproduce.debian.net architecture-specific pages, we now build for a three more architectures (for a total of five) - arm64 armhf and riscv64.

Two new academic papers

Giacomo Benedetti, Oreofe Solarin, Courtney Miller, Greg Tystahl, William Enck, Christian Kästner, Alexandros Kapravelos, Alessio Merlo and Luca Verderame published an interesting article recently. Titled An Empirical Study on Reproducible Packaging in Open-Source Ecosystem, the abstract outlines its optimistic findings:

[We] identified that with relatively straightforward infrastructure configuration and patching of build tools, we can achieve very high rates of reproducible builds in all studied ecosystems. We conclude that if the ecosystems adopt our suggestions, the build process of published packages can be independently confirmed for nearly all packages without individual developer actions, and doing so will prevent significant future software supply chain attacks.

The entire PDF is available online to view.

In addition, Julien Malka, Stefano Zacchiroli and Théo Zimmermann of Télécom Paris' in-house research laboratory, the Information Processing and Communications Laboratory (LTCI) published an article asking the question: Does Functional Package Management Enable Reproducible Builds at Scale?.

Answering strongly in the affirmative, the article's abstract reads as follows:

In this work, we perform the first large-scale study of bitwise reproducibility, in the context of the Nix functional package manager, rebuilding 709,816 packages from historical snapshots of the nixpkgs repository[. We] obtain very high bitwise reproducibility rates, between 69 and 91% with an upward trend, and even higher rebuildability rates, over 99%. We investigate unreproducibility causes, showing that about 15% of failures are due to embedded build dates. We release a novel dataset with all build statuses, logs, as well as full diffoscopes: recursive diffs of where unreproducible build artifacts differ.

As above, the entire PDF of the article is available to view online.

Distribution work

There as been the usual work in various distributions this month, such as:

Arch Linux developer kpcyrd has provided a status report for January 2025 related to Arch's progress towards full reproducibility. kpcyrd notes in particular progress towards to making a "minimal reproducible bootable system" - that is, an Arch installation containing only reproducible packages.

10+ reviews of Debian packages were added, 11 were updated and 10 were removed this month adding to our knowledge about identified issues. A number of issue types were updated also.
The FreeBSD Foundation announced that "a planned project to deliver zero-trust builds has begun in January 2025". Supported by the Sovereign Tech Agency, this project is centered on the various build processes, and that the "primary goal of this work is to enable the entire release process to run without requiring root access, and that build artifacts build reproducibly - that is, that a third party can build bit-for-bit identical artifacts." The full announcement can be found online, which includes an estimated schedule and other details.

Finally, for openSUSE, Bernhard M. Wiedemann published another report for that distribution.

On our mailing list…

On our mailing list this month:

Following-up to a substantial amount of previous work pertaining the Sphinx documentation generator, James Addison asked a question pertaining to the relationship between SOURCE_DATE_EPOCH environment variable and testing that generated a number of replies.
Adithya Balakumar of Toshiba asked a question about whether it is possible to make ext4 filesystem images reproducible. Adithya's issue is that even the smallest amount of post-processing of the filesystem results in the modification of the "Last mount" and "Last write" timestamps.
James Addison also investigated an interesting issue surrounding our disorderfs filesystem. In particular:

FUSE (Filesystem in USErspace) filesystems such as disorderfs do not delete files from the underlying filesystem when they are deleted from the overlay. This can cause seemingly straightforward tests - for example, cases that expect directory contents to be empty after deletion is requested for all files listed within them - to fail.

Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Bernhard M. Wiedemann:
- Komikku (nocheck)
- abseil-cpp (race)
- dunst (date)
- eclipse-egit (jar-mtime minor)
- exaile (race)
- gawk (bug)
- gimp3 (png date)
- intel (ASLR)
- ioquake3 (debugsource contains date and time)
- joker (sort)
- libchardet
- llama.cpp (random)
- llama.cpp (-march=native-related issue)
- nethack (race)
- netrek-client-cow (date)
- nvidia-modprobe (date)
- nvidia-persistenced (date)
- obs-build (toolchain bug, mis-parses changelog)
- perl-libconfigfile (race)
- pgvector (CPU)
- python-Django4 (FTBFS-2038)
- python-python-datamatrix (FTBFS)
- qore-ssh2-module (GIGO-bug)
- rpm (UID in cpio header from rpmbuild)
- zig (CPU-related issue)
Chris Lamb:
- #1092251 filed against kmetronome.
- #1092917 filed against rust-xh.
- #1093198 filed against parser.
- #1093199 filed against parser.
- #1093201 filed against rsync.
- #1094611 filed against wasistlos.
Egbert Eich:
- apptainer (randomness)
- spack (core-count and date)
Valentin Lefebvre:
- uki-tool (toolchain)
Marvin Friedrich:
- cargo-packaging/rusty_v8 (upstream toolchain bugfix)
James Addison:
- #1092870 filed against binutils.
Pol Dellaiera:
- PHP Ecosystem: composer/composer#12090 which was then gracefully fixed by Jordi Boggiano at composer/composer#12263.

diffoscope

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 285, 286 and 287 to Debian:

Security fixes:
- Validate the --css command-line argument to prevent a potential Cross-site scripting (XSS) attack. Thanks to Daniel Schmidt from SRLabs for the report. […]
- Prevent XML entity expansion attacks. Thanks to Florian Wilkens from SRLabs for the report.. […][…]
- Print a warning if we have disabled XML comparisons due to a potentially vulnerable version of pyexpat. […]
Bug fixes:
- Correctly identify changes to only the line-endings of files; don't mark them as Ordering differences only. […]
- When passing files on the command line, don't call specialize(…) before we've checked that the files are identical or not. […]
- Do not exit with a traceback if paths are inaccessible, either directly, via symbolic links or within a directory. […]
- Don't cause a traceback if cbfstool extraction failed.. […]
- Use the surrogateescape mechanism to avoid a UnicodeDecodeError and crash when any decoding zipinfo output that is not UTF-8 compliant. […]
Testsuite improvements:
- Don't mangle newlines when opening test fixtures; we want them untouched. […]
- Move to assert_diff in test_text.py. […]
Misc improvements:
- Drop unused subprocess imports. […][…]
- Drop an unused function in iso9600.py. […]
- Inline a call and check of Config().force_details; no need for an additional variable in this particular method. […]
- Remove an unnecessary return value from the Difference.check_for_ordering_differences method. […]
- Remove unused logging facility from a few comparators. […]
- Update copyright years. […][…]

In addition, fridtjof added support for the ASAR .tar-like archive format. […][…][…][…] and lastly, Vagrant Cascadian updated diffoscope in GNU Guix to version 285 […][…] and 286 […][…].

strip-nondeterminism is our sister tool to remove specific non-deterministic results from a completed build. This month version 1.14.1-1 was uploaded to Debian unstable by Chris Lamb, making the following the changes:

Clarify the --verbose and non --verbose output of bin/strip-nondeterminism so we don't imply we are normalizing files that we are not. […]
Bump Standards-Version to 4.7.0. […]

Website updates

There were a large number of improvements made to our website this month, including:

Arnout Engelen:
- Update the link to NixOS' reproducibility-related issue template on the NixOS-specific contribute page […] and remove an outdated link. […]
Holger Levsen:
- Check, deduplicate, update and generally cleanup a number of presentations linked on our Talks & Resources page. […][…][…][…]
James Addison:
- Add some file permissions hints and guidance on the Archive metadata page. […]
Michael R. Crusoe:
- Add an R example to the SOURCE_DATE_EPOCH documentation. […]
- Update the website's README to make the setup command copy & paste friendly. […]

Reproducibility testing framework

The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In January, a number of changes were made by Holger Levsen, including:

reproduce.debian.net-related:
- Add support for rebuilding the armhf architecture. […][…]
- Add support for rebuilding the arm64 architecture. […][…][…][…]
- Add support for rebuilding the riscv64 architecture. […][…]
- Move the i386 builder to the osuosl5 node. […][…][…][…]
- Don't run our rebuilders on a public port. […][…]
- Add database backups on all builders and add links. […][…]
- Rework and dramatically improve the statistics collection and generation. […][…][…][…][…][…]
- Add contact info to the main page […], thumbnails […] as well as the new, missing architectures. […]
- Move the amd64 worker to the osuosl4 and node. […]
- Run the underlying debrebuild script under nice. […]
- Try to use TMPDIR when calling debrebuild. […][…]
buildinfos.debian.net-related:
- Stop creating buildinfo-pool_${suite}_${arch}.list files. […]
- Temporarily disable automatic updates of pool links. […]
FreeBSD-related:
- Fix the sudoers to actually permit builds. […]
- Disable debug output for FreeBSD rebuilding jobs. […]
- Upgrade to FreeBSD 14.2 […] and document that bmake was installed on the underlying FreeBSD virtual machine image […].
Misc:
- Update the 'real' year to 2025. […]
- Don't try to install a Debian bookworm kernel from 'backports' on the infom08 node which is running Debian trixie. […]
- Don't warn about system updates for systems running Debian testing. […]
- Fix a typo in the ZOMBIES definition. […][…]

In addition:

Ed Maste modified the FreeBSD build system to the clean the object directory before commencing a build. […]
Gioele Barabucci updated the rebuilder stats to first add a category for network errors […] as well as to categorise failures without a diffoscope log […].
Jessica Clarke also made some FreeBSD-related changes, including:
- Ensuring we clean up the object directory for second build as well. […][…]
- Updating the sudoers for the relevant rm -rf command. […]
- Update the cleanup_tmpdirs method to to match other removals. […]
Jochen Sprickerhof:
- Fix logic for old files saved on buildinfos.debian.net. […]
- Rework and simplify the generation of statistics linked from reproduce.debian.net. […][…][…][…]
Roland Clobus:
- Update the reproducible_debstrap job to call Debian's debootstrap with the full path […] and to use eatmydata as well […][…].
- Make some changes to deduce the CPU load in the debian_live_build job. […]

Lastly, both Holger Levsen […] and Vagrant Cascadian […] performed some node maintenance.

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

IRC: #reproducible-builds on irc.oftc.net.
Mastodon: @reproducible_builds@fosstodon.org
Mailing list: rb-general@lists.reproducible-builds.org
Twitter: @ReproBuilds

05 Feb 2025 11:49am GMT

04 Feb 2025

Planet Debian

Dominique Dumont: Azure API throttling strikes back

In my last blog, I explained how we resolved a throttling issue involving Azure storage API. In the end, I mentioned that I was not sure of the root cause of the throttling issue.

Even though we no longer had any problem in dev and preprod cluster, we still faced throttling issue with prod. The main difference between these 2 environments is that we have about 80 PVs in prod versus 15 in the other environments. Given that we manage 1500 pods in prod, 80 PVs does not look like a lot.

To continue the investigation, I've modified k8s-scheduled-volume-snapshotter to limit the number of snaphots done in a single cron run (see add maxSnapshotCount parameter pull request).

In prod, we used the modified snapshotter to trigger snapshots one by one.

Even with all previous snapshots cleaned up, we could not trigger a single new snapshot without being throttled. I guess that, in the cron job, just checking the list of PV to snapshot was enough to exhaust our API quota.

Azure doc mention that a leaky bucket algorithm is used for throttling. A full bucket holds tokens for 250 API calls, and the bucket gets 25 new tokens per second. Looks like that not enough.

I was puzzled and out of ideas .

I looked for similar problems in AKS issues on GitHub where I found this comment that recommend using useDataPlaneAPI parameter in the CSI file driver. That was it!

I was flabbergasted by this parameter: why is CSI file driver able to use 2 APIs ? Why is one on them so limited ? And more importantly, why is the limited API the default one ?

Anyway, setting useDataPlaneAPI: "true" in our VolumeSnapshotClass manifest was the right solution. This indeed solved the throttling issue in our prod cluster.

But not the snaphot issue . Amongst the 80 PV, I still had 2 snaphots failing.

Fortunately, the error was mentioned in the description of the failed snapshots: we had too many (200) snapshots for these shared volumes.

What ?? All these snapshots were cleaned up last week.

I then tried to delete these snaphots through azure console. But the console failed to delete these snapshot due to API throttling. Looks like Azure console is not using the right API.

Anyway, I went back to the solution explained in my previous blog, I listed all snapshots with az command. I indeed has a lot of snaphots, a lot of them dated Jan 19 and 20. There was often a new bogus snaphot created every minute.

These were created during the first attempt at fixing the throttling issue. I guess that even though CSI file driver was throttled, a snaphot was still created in the storage account, but the CSI driver did not see it and retried a minute later. What a mess.