fidzu : debian

Five years or so ago, I had a look at trying to speed up dpkg's package installation; I concluded that it was probably possible to speed up, but that there was no appetite for this kind of large-scale changes. (You'd probably need to rewrite the transaction system to get rid of a lot of fsyncs, you'd ideally want to reduce the number of syscalls for unpack by io_uring and so on.)

This summer, I've been looking at something related on and off; it is possible to speed up the startup time? That's in a sense the opposite scenario; instead of installing lots of packages in a newly debootstrapped chroot (with very few packages), see how fast you can install one in a much more busy chroot (I just copied my laptop's dpkg dir, with ~6600 packages installed).

Before I show the numbers, I must stress that this is an investigation, not a fair benchmark, and you should not go shout at the dpkg maintainers that they need to get to "catch up". That said:

> sudo time dpkg --root=root -i hello_2.12.3-1_amd64.deb >/dev/null
Not building database; man-db/auto-update is not 'true'.
1.12user 0.49system 0:01.85elapsed 87%CPU (0avgtext+0avgdata 171880maxresident)k
0inputs+14648outputs (0major+72963minor)pagefaults 0swaps

> sudo time ./src/dpkg --root=root -i hello_2.12.3-1_amd64.deb > /dev/null
0.04user 0.01system 0:00.15elapsed 38%CPU (0avgtext+0avgdata 6520maxresident)k
0inputs+1080outputs (0major+2705minor)pagefaults 0swaps

How is it unfair? Well, for one, the code to run triggers is messed up so they're not run (but the trigger in question should be very fast). And there's one step at the end with detecting "disappearing packages" that doesn't run properly because it's a bit tricky in my model and I didn't want to deal with, well, difficult problems. But I think both are perfectly doable without really affecting the end time, it just requires engineering. There's a lot of work to be done, though; diving into the code makes me shudder at all the complexities that need to be in place to support all the corner cases of multiarch, for instance.

The code is extremely proof-of-concept, but it runs and can read (and write) metadata from SQLite instead of flat text files, it can resolve dependencies in the most basic fashion, it can keep track of installed files, it should be crash- and powerloss-proof. You know, the very very basic stuff, and without changing the model fundamentally (like e.g. Michael Stapelberg did with distri, fundamentally replacing packages with disk images and ending up in a very fast but rather different-looking system). So it was satisfying to see that it ends up around 10x even on my not-very-new laptop (plus a significant RAM reduction); I believe it should be possible to squeeze under 100 ms, but that would probably require also optimizing the unpacking itself, which I didn't look at this time.

Having a bunch of files being read into RAM and then processed freely was a design that made a lot of sense when dpkg was written (in 1995!) and Debian had ~250 binary packages in total (and you probably wouldn't install all of them). There was no reasonable database available for desktop systems; the closest thing you'd have was probably BerkeleyDB and that wasn't really it, so flat files and fsync made a lot of sense, and was easy to manipulate and persist. But now, SQLite is widely available and probably the most battle-tested code in history, a typical system has thousands of packages (you could easily install tens of thousands if you're doing heavy development), SSDs have replaced HDDs almost everywhere for system disks, and the environment has just changed a lot in general. So I hope that someone at some point will be crazy enough to pick this up and run with it, because it's a lot of work and I don't intend to. :-)

PS: I didn't look at apt; I think what I'd really love to see first and foremost is a package format change so that apt-listchanges can look at (or look for) NEWS.gz without having to unpack the entire package. Perhaps a control field saying "nothing new here"?

15 Jul 2026 7:15am GMT

I've just filed a Debian bug report about libproc-processtable-perl giving bad errors when SE Linux denies access to files under /proc [1].

Ran into unknown state (hex char: 0) at /tmp/test.pl line 8.

The errors are of the above form which Google didn't find before now so obviously isn't a common situation, below is my test program.

#!/usr/bin/perl
use strict;

use Proc::ProcessTable;

my $process_table = new Proc::ProcessTable('cache_ttys' => 0 );

foreach my $process ( @{$process_table->table} )
{
  print $process->fname . "\n";
}

Here is the relevant part of strace output:

newfstatat(AT_FDCWD, "/proc/2", 0x7fff19533c10, 0) = -1 EACCES (Permission denied)
openat(AT_FDCWD, "/proc/2/stat", O_RDONLY) = -1 EACCES (Permission denied)
access("/proc/2", F_OK)                 = 0
write(2, "Ran into unknown state (hex char: 0) at /tmp/test.pl line 8.\n", 61) = 61

Below is the apt sources.list line for my personal repository which has a version of the package with this fix. The gpg key is in the etbe-base package in that repository and the source is all there. To access it without apt use this web page [2].

deb [signed-by=/usr/share/keyrings/etbe.gpg arch=amd64 ] https://www.coker.com.au trixie misc

I've also done some work on the ps.monitor script in etbe-mon that uses this Perl package and made it better handle program names longer than 15 characters. That improvement apparently only works on Linux, Darwin, and Cygwin. People who want things to work better on BSD etc could patch libproc-processtable-perl accordingly.

• [1] https://tinyurl.com/2ckw4kh2
• [2] https://www.coker.com.au/dists/trixie/misc/binary-amd64/

15 Jul 2026 7:07am GMT

The twentieth release of the qlcal package arrivied at CRAN today, and has been built for r2u. This version synchronises with QuantLib 1.43 released today as well.

qlcal delivers the calendaring parts of QuantLib. It is provided (for the R package) as a set of included files, so the package is self-contained and does not depend on an external QuantLib library (which can be demanding to build). qlcal covers over seventy country / market calendars and can compute holiday lists, its complement (i.e. business day lists) and much more. Examples are in the README at the repository, the package page, and course at the CRAN package page.

This releases updates to several new calendars (see below), and extends the calendars for Israel to some added new conventions, updates a few helper functions, and turns on ccache for continuous integration builds.

The full details from NEWS.Rd follow.

Changes in version 0.1.2 (2026-07-14)

• Synchronized with QuantLib 1.43

• Calendar updates for India, Israel, and South Korea; small interface update for Israle

• New calendars for Croatia, Malta, Montenegro, North Macedonia, Serbia, Slovenia, Uzebekistan

• Updates to a number of QuantLib helper functions

• Continuous integration now uses ccache via a setup action

Courtesy of my CRANberries, there is a diffstat report for this release. See the project page and package documentation for more details, and more examples.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

15 Jul 2026 1:49am GMT