05 Jun 2026

feedPlanet Grep

Frederic Descamps: VillageSQL Features… déjà vu?

At events with the MySQL community, when we were enough "veterans," we had a little ritual: we played Cards Against the MySQL Community. A politically incorrect (but still lighthearted) game that followed the same rules as "Cards Against Humanity." Those were some really fun, unforgettable evenings! One of the most popular black cards, which always […]

05 Jun 2026 9:05am GMT

Frederic Descamps: MariaDB Hidden Gem: Create Aggregate Function

Have you ever written a query where the GROUP BY was easy, but the aggregate was the problem? You know how to group the rows.You know what result you want for each group.But none of the built-in aggregate functions really match your logic. So you end up with a long expression using SUM(), CASE, IF(), […]

05 Jun 2026 9:05am GMT

Dries Buytaert: Europe turns to Open Source for independence

A red chess pawn set in a niche carved into a white cube, on a blue background.

Today the European Commission released the European Technological Sovereignty Package, a big push to reduce Europe's dependence on foreign technology.

Earlier this year, the Commission ran a public consultation, and I contributed two articles to it: the Software Sovereignty Scale and a follow-up, The Sovereignty Prerequisite.

So when the package was published today, I skimmed it right away. I was pleasantly surprised to find one of my articles cited in a footnote on page 18!

I won't pretend to have fully digested it yet, but one part immediately caught my attention: a new Open Source Strategy for Europe (Section 4 of the PDF, starting on page 16).

The highlights are significant:

One of the best parts of the strategy is that it treats Open Source as infrastructure that needs sustained investment, not as free software that magically maintains itself. I'll admit, that made me happy.

It is an argument Open Source advocates have made for years, and one I made in Funding Open Source like public infrastructure. The Commission now seems to agree, pointing to the lack of sustained funding, uncertain maintenance, and procurement barriers that hold Open Source back.

Just as important, the strategy reframes why Open Source matters. The old argument for Open Source was mostly about saving money. Here, Open Source is treated as a path to Europe's technological independence: software that Europe can inspect, maintain, and control. In other words, software that gives Europe "freedom of action".

None of this came out of nowhere. The story starts with the 2024 Draghi report, the Commission's landmark diagnosis of why Europe fell behind the United States and China. The Commission spent the next year turning that diagnosis into policy, and today's strategy is one of the results.

You can see how far the thinking has moved just by counting. In Draghi's 412 pages, "Open Source" appears twice. In the new plan, it appears nearly 300 times, in roughly a tenth of the space. It really shows that Open Source has moved from the margins of Europe's competitiveness debate into the center of its sovereignty strategy.

Still, it is worth being clear about what kind of document this is. This is not a law. It does not require companies to use Open Source or rewrite procurement rules across Europe. But it still matters. It moves Open Source from principle to policy: part of Europe's sovereignty agenda, backed by real funding, and a step toward stronger procurement rules.

The strategy notes that "the EU currently spends EUR 264 billion a year mostly on US proprietary IT products and services". That is not the Commission's budget; it is what the broader European economy spends each year on American software.

Set against that number, €2 billion over seven years for Open Source is a start, but a very small one. Seven years of Europe's Open Source budget is roughly three days of its American software bill. Europe has started to treat Open Source as sovereignty infrastructure, but it is not yet funding it like sovereignty infrastructure.

The strategy also stops one word short. In procurement, it tells public bodies to choose Open Source "first", not that they must. But "first" is only a preference. It is the kind of thing you talk yourself out of when the demo is shiny and the deadline is close.

For the systems a society cannot afford to lose, Open Source should not be preferred. It should be required. Europe is not there yet, but this is an excellent step in that direction.

05 Jun 2026 9:05am GMT

feedPlanet Debian

Russell Coker: CPUs and Debian Package Building

Introduction

I have just bought a HP Z4 G4 with W-2125 CPU for $320 and I decided it was a good time to do some benchmarks on Debian package building to see which system I should use for that.

The W-2125 CPU scores only 9,954 on the passmark multithread test but scores 2,546 on single thread [1]. Passmark seems to have some limitations as the only DDR3 system that's important to me at the moment (the HP Z420 workstation my parents use which cost me $750 in 2021) with a E5-2620 CPU scoring 5,325 for multithread and 1,113 for single thread [2]. From the passmark results one would expect that the system is slightly more than twice as fast as the Z420 for operations that involve less than 4 CPU cores.

For the initial tests of the Z4 G4 I ran them with hyper-threading enabled as 4 cores isn't much by today's standards and also the machine in question is going to be less exposed to hostile data and contain less secret data than most of my systems so the security risks of hyper-threading are less of a concern.

I did some tests with a couple of tasks that are very important to me, building SE Linux policy packages (something I may do a dozen times in a day) and building Warzone 2100 (which I do less often but is the most intensive build process I regularly run). At the bottom of this post there are tables with the results from building these packages on my Z640 workstation with a E5-2696 v4 CPU [3], the Z420, and the new machine.

For the Warzone 2100 package I tested building on my Z840 dual CPU system [4]. I didn't test building the SE Linux policy on the Z840 this time because that package can't take advantage of even 22 cores. When I initially got the Z840 running it built the policy packages faster because the Z640 had an older CPU that was slower for single core operations than the CPUs in the Z840.

BTRFS Compression

For some time I have noticed significant differences in compile time on my workstation, a factor of more than 2. I did more tests and noticed that "top" showed something like the following, those kernel threads are all BTRFS related, except for "gfx" which is probably something graphical caused by running Chrome with about 300 tabs open.

2144316 root      20   0       0      0      0 I  26.6   0.0   0:36.76 kworker/u88:20-btrfs-endio-write                                                                                                                                                                             
2221470 root      20   0       0      0      0 I  23.7   0.0   0:01.85 kworker/u88:12-gfx                                                                                                                                                                                           
2221436 root      20   0       0      0      0 I  15.1   0.0   0:07.48 kworker/u88:8-btrfs-compressed-write                                                                                                                                                                         
2166191 root      20   0       0      0      0 I  12.8   0.0   0:15.80 kworker/u88:23-btrfs-compressed-write                                                                                                                                                                        
2126387 root      20   0       0      0      0 I  10.2   0.0   1:29.11 kworker/u88:4-events_unbound

I had been running BTRFS with the mount option "compress=zstd:15" which caused much of the performance problems when building. It was also a random performance issue which I think happened due to the BTRFS 30 second write-back sometimes taking more than 30 seconds during the build process which then caused a second write-back.

I did tests on ZSTD compression levels 5, 8, 10, and 15. 15 was never good and often really bad. 10 was not unbearable but consistently slower. 8 was sometimes as fast as 5 and sometimes quite a bit slower. I didn't test levels below 5 because I need to have some compression and it seemed that the benefits of reducing compression were dropping off below 8.

I found that the BTRFS compression delay is not counted in system time for the process. I think it's the fsync() system calls in the semodule and dpkg-deb programs that cause the delays related to BTRFS compression waiting for kernel threads.

BOINC

I have all my systems other than laptops running BOINC in the background so that CPU power is used for scientific research when I don't have any personal use for it [5]. I believe that it's immoral to waste CPU power when it could be used for research.

In the below table which has test results from building the package with and without BOINC, and with different ZSTD compression levels in BTRFS all the worst entries were from when BOINC was running apart from one where ZSTD level 15 compression was used. The really poor performance with ZSTD level 15 was an outlier, but it wasn't an uncommon outlier so I left it in.

Running BOINC in the background configured to use all CPU cores caused a significant increase in "user CPU time" (the time a CPU core spent actually running the program). My initial thought was that it's partly related to "turbo boost".

The Intel ARK page for the CPU in the Z420 shows that it's main clock speed is 2.0GHz with a 2.5GHz "turbo boost" [6]. The "turbo boost" is apparently largely based on temperature and apparently limited to one core, so if the other CPU cores are all being used then the CPU will probably be too hot to have the turbo boost and if it happens it might not happen for my compile processes.

The ARK page for the E5-2699 v4 (which is a similar CPU to the E5-2696 v4 that I'm using but is officially documented by Intel) [7] shows that it has a base clock speed of 2.2GHz and a turbo boost speed of 3.6 GHz. 322 vs 244 seconds of user CPU time means running 32% slower which can plausibly be explained by the lack of a 64% turbo boost with a bit of help from the 55MB L3 cache being thrashed.

Turbo boost would only be a noticeable issue for building packages like the SE Linux policy packages which doesn't take much advantage of multi-core CPUs. For a build process to average at best 362% CPU use there has to be large parts of the process that are limited to one or two cores which can potentially give a benefit from turbo-boost.

When building the Warzone 2100 packages most of the build time is running basis-universal which is a multi-threaded program to compress GPU texture data. This usually causes a load average of 300+ on the Z640 or 600+ on the Z840. But the build time is still increased by more than 50% on both the Z640 and the Z840 when BOINC is running in the background, which seems to be an indication that it's not related to turbo boost. I verified that BOINC is running at IDLE schedule priority with the following command:

# chrt -p $(pidof -s einstein_O4MD_2.01_x86_64-pc-linux-gnu)
pid 2974874's current scheduling policy: SCHED_IDLE
pid 2974874's current scheduling priority: 0

In theory this means that BOINC won't affect foreground processes.

Hyper Threading on the W-2125

The best claims I've seen about HT are 15% to 30% performance boost. The best I've actually seen in the past is about 18%. Seeing a 10% benefit for building Warzone 2100 is at the low end of the range I expected. 8 virtual cores is not many for a build process that causes a load average of 600+ when running on a system with 44 real cores.

I was surprised to see a 6% performance benefit in hyper-threading for building the SE Linux policy as I didn't think there was enough use of threading or multiple processes to allow that.

Many build scripts use a number of processes that match the number of apparent CPU cores. While "make -j 88" might give a theoretical performance benefit on a 44 core system it will also take a lot of RAM and any paging will outweigh the benefits of hyper-threading. On a system with only 4 real cores there's less potential for using too much RAM and as security isn't so important on that system I will leave it on.

Comparing the CPUs

The best results of the Z640 and Z4G4 are only 50% faster than the best results of the Z420.

The Z420 has a E5-2620 CPU which is far from the fastest CPU available for that system - the E5-2687W has 8 cores and rates 10,021/1,669 on passmark [8] which is far better than the 5,331/1,114 the E5-2620. The E5-2687W is the fastest CPU that HP lists as supported by the Z420 and it supports DDR3-1666 RAM as opposed to the DDR3-1333 that is the fastest that the E5-2620 supports. With suitable hardware upgrades the Z420 would probably only take about 20% longer to do builds of the SE Linux policy and other packages that can't take advantage of more than 8 CPU cores.

The Z4G4 system has 4 RAM channels which means that you should get some performance benefits from having 4 DIMMs, my system currently has 2 and I haven't yet managed to get more DDR4-2666 DIMMs. But I'd still expected a W-2125 CPU with 2*DDR4-2666 DIMMs outperform any E5-26xx CPU with 4*DDR4-DDR-2400 DIMMs for tasks that average less than 4 CPU cores.

In retrospect I would have been better off getting a HP Z820 (two socket server with DDR3 RAM) than the first DDR4 systems I got. It seems that for reasonable size builds a two socket system comes close to twice the speed of a single socket system. I did briefly own a HP ML350 two CPU system with DDR3 RAM but it was too noisy for my intended use as a deskside workstation so I sold it.

Things to Investigate

I plan to do more investigation on BTRFS compression, how to get the best compression without excessive delays and how to recognise when delays are happening. I have some SSDs that have sustained write speeds as low as 15MB/s (Crucial P1 series) so for those I could probably have very high compression levels without slowing the system down.

The fact that BIONC slows things down so much seems to be a bug. When processes are running with the IDLE scheduling class there shouldn't be such significant delays. Is it due to cache thrashing? How can I best get BOINC suitably throttled when I'm sitting at my workstation, I don't want BOINC connecting to the local X server (which it repeatedly tries to do). Do I need to tune my kernel for better handling of IDLE scheduling?

When I get more DIMMs in the Z4G4 I need to do more tests to see if it gives an overall performance boost.

Also the Z4G4 system has a BIOS option for "sub NUMA" which basically means treating the different RAM channels on a single CPU as NUMA zones, I enabled that option which does nothing presumably because I only have 2 DIMMs, the results when I have 4 DIMMs will be interesting. I will also do some NUMA tests on the Z840 to see what benefits it gives.

I have a selection of RAM speeds that will work in the Z4G4, if I have enough spare time I'll test what difference that makes for CPU bound tasks that matter to me.

For package building fsync() is not helpful, if the system crashes before it's done then I will just do the build again. For a build cluster it is probably a good feature and probably doesn't affect aggregate performance when multiple packages are built at the same time, but for the single user case probably not. I will investigate libeatmydata for package building [9].

Conclusion

The progress in CPUs seems to have slowed down a lot recently. The main benefits seem to be in more CPU cores and for newer sockets with more RAM channels.

The CPUs that do have improvements in single core performance are the i9 series (which mostly doesn't come with motherboards supporting ECC) and AMD CPUs (which is rare in enterprise class hardware). Maybe I should get a server with an i9 or AMD CPU for tasks that need a fast turn around with a small number of cores. That would probably outperform any CPU designed for large core counts for things like building the policy and setting up test VMs (which depends on package installation speed that is single core bottlenecked).

The W-21xx CPUs seem to offer little benefit over the E5-26xxv4 CPUs and not a lot of benefit over E5-26xx CPUs (with DDR3). Even the W-22xx CPUs look like they aren't going to offer a lot as they are only an incremental improvement over the W-21xx series. I had considered making the Z4G4 my main desktop workstation after the high end W CPUs become affordable, but it looks like that won't be worth it until such CPUs drop from the current ebay price of $900 to $100.

I think I'll keep waiting for a decent socket LGA3647 or DDR5 based server [10] for my next significant upgrade.

Tables

Building SE Linux Refpolicy

System BOINC Compression CPU Time Elapsed CPU%
Z640 no 8 248.82user 55.58system 1:23.88elapsed 362%CPU
Z4G4 no 5 245.15user 34.63system 1:24.93elapsed 329%CPU
Z640 no 5 244.75user 34.87system 1:25.98elapsed 325%CPU
Z4G4 no 10 245.21user 35.64system 1:29.63elapsed 313%CPU
Z640 no 8 248.71user 55.90system 1:33.01elapsed 327%CPU
Z640 no 10 250.90user 55.78system 1:42.12elapsed 300%CPU
Z640 yes 8 298.19user 69.30system 1:59.77elapsed 306%CPU
Z640 yes 10 300.58user 68.90system 2:01.53elapsed 304%CPU
Z420 no 5 359.01user 44.95system 2:07.33elapsed 317%CPU
Z640 yes 5 322.40user 71.82system 2:34.66elapsed 254%CPU
Z420 yes 5 372.03user 42.95system 2:42.15elapsed 255%CPU
Z640 yes 15 299.26user 67.18system 2:59.77elapsed 203%CPU
Z640 no 15 250.05user 54.60system 3:07.61elapsed 162%CPU

Building Warzone 2100

System BOINC Compression CPU Time Elapsed CPU%
Z840 no 10 6549.21user 89.46system 4:18.90elapsed 2564%CPU
Z840 no 5 6533.81user 90.50system 4:19.24elapsed 2555%CPU
Z640 no 5 7040.87user 183.12system 7:13.50elapsed 1666%CPU
Z840 yes 5 8039.52user 169.62system 8:02.86elapsed 1700%CPU
Z640 yes 5 7486.44user 205.03system 11:09.97elapsed 1148%CPU
Z4G4 no 5 7891.32user 74.45system 17:48.03elapsed 745%CPU
Z4G4 no 10 7942.10user 77.43system 17:58.72elapsed 743%CPU

Hyper-Threading

Build HT Compression CPU Time Elapsed CPU%
Warzone yes 5 7891.32user 74.45system 17:48.03elapsed 745%CPU
Warzone yes 10 7942.10user 77.43system 17:58.72elapsed 743%CPU
Warzone no 5 4492.45user 59.09system 19:59.01elapsed 379%CPU
Warzone no 10 4497.28user 59.46system 20:07.15elapsed 377%CPU
Refpolicy yes 5 245.15user 34.63system 1:24.93elapsed 329%CPU
Refpolicy yes 10 245.21user 35.64system 1:29.63elapsed 313%CPU
Refpolicy no 5 180.84user 29.74system 1:32.30elapsed 228%CPU
Refpolicy no 10 180.29user 30.07system 1:35.01elapsed 221%CPU

Related posts:

  1. HP z840 Many PCs with DDR4 RAM have started going cheap on...
  2. Firebuild After reading Bálint's blog post about Firebuild (a compile cache)...
  3. Matching Intel CPUs To run a SMP system with multiple CPUs you need...

05 Jun 2026 7:31am GMT

Birger Schacht: Status update, May 2026

Debian Related Work

DH Related Work

A big chunk of my DH related work went into designing & implementing a search app for the APIS framework. Our goal is to have a way of searching over various types of Django models. The app introduces a search model that indexes all registered models. We use a combination of PostgreSQLs full text search and Trigram Similarity to find the search results. Using a SearchVectorField and GinIndices for the trigram indexed fields we can reach a somewhat acceptable performance.

We released versions 0.63 and 0.64 of the APIS framework. The 0.63 release introduced the new entities app, which will soon hopefully replace the legacy apis_entities & apis_metainfo modules. Version 0.64 moved some logic from the legacy modules the entities module.

We made some progress in defining the endpoints for the PFP API.

05 Jun 2026 5:28am GMT

04 Jun 2026

feedPlanet Debian

Reproducible Builds: Reproducible Builds in May 2026

Welcome to the May 2026 report from the Reproducible Builds project.

These reports outline what we've been up to over the past month, highlighting items of news from elsewhere in the increasingly-important area of software supply-chain security. As ever, if you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website.

In this month's report, we cover:

  1. Debian to ship reproducible packages in forky and beyond
  2. Holger Levsen on reproducing official Debian packages
  3. Reproducible Builds 2026 summit to be held in Gothenburg, Sweden
  4. Kettle: Attested Builds for Verifiable Software
  5. New rebuilderd version announced
  6. Reproducible open source messengers
  7. Distribution work
  8. Misc news
  9. Patches
  10. Documentation updates


Debian to ship reproducible packages in forky and beyond

In a huge change in Debian's reproducibility policy, the Debian Release Team announced that:

… we've decided it's time to say that Debian must ship reproducible packages. Since yesterday, we have enabled our migration software to block migration of new packages that can't be reproduced [on reproduce.debian.net] or existing packages in testing that regress in reproducibility.

That is to say, if newly-uploaded packages are not reproducible, they won't be considered candidates for inclusion in the next stable release of Debian codenamed forky. (Some exceptions may be granted.)

This news generated a number of articles and comments in various news outlets:


Holger Levsen on reproducing official Debian packages

Reproducible Builds developer Holger Levsen gave a talk at the 2026 Hamburg MiniDebconf this year on the topic of reproduce.debian.net - reproducing what is distributed from ftp.d.o.

Holger's talk announced that Debian intends to ship only reproducible packages in forky and beyond (see above), but also talked more broadly about reproducible builds, our testing framework and the Debian archive. That is to say, moving away from testing whether a package is reproducible in a theoretical sense (eg. whether we can build it twice in different environments and achieve the same result in our test system), and attempting to reproduce the same .deb files in the official Debian archive itself. This small-sounding distinction is actually essential, as this is the only means through which the reproducible builds technique can determine whether build systems are compromised are not.

A video (32m37s) of the talk is available, as are Holger's slides.


Reproducible Builds 2026 summit to be held in Gothenburg, Sweden

As initially announced in March 2026, we will be having our yearly Reproducible Builds summit 2026 in Gothenburg Sweden, from September 22 until 24, followed by two days of hacking!

Further information will be provided on our website and on the rb-general mailing list very soon.


Kettle: Attested Builds for Verifiable Software

André Arko and Amean Asad published a paper this month on Kettle, a build system that "produces cryptographically verifiable provenance for software built inside Trusted Execution Environments":

A Kettle build records the source commit, dependency set, toolchain, build environment and output artifact digests in a provenance document produced inside a measured confidential VM. The SHA-256 digest of that document is committed to the TEE platform's attestation report-data field, so the hardware-signed attestation report is itself the signature on the provenance, with the signing identity chaining to the TEE manufacturer's root of trust rather than to the build infrastructure operator. Because the CVM image is itself reproducible, its launch measurement is public and stable, which lets a build requester pre-attest the CVM before submitting any input and optionally deliver source over a TLS channel terminated inside it, so the build runs end-to-end confidentially without the host ever seeing source code in plaintext.

A PDF of the paper is available online.


New rebuilderd version announced

rebuilderd, our server designed for monitoring the official package repositories of Linux distributions and attempt to reproduce the observed results there; it powers, amongst other things, reproduce.debian.net.

A new version, 0.27.0, was released this month, with the following headline changes:

As kpcyrd's announcement mentions:

The new rebuilderd package is currently available in the extra-testing repository. Note the Arch Linux package is upgraded from v0.25.0 from v0.27.0; please be patient with the database migrations on first restart, and make yourself familiar with the breaking changes in v0.26.0 too.


Reproducible open source messengers

GitHub developer BarbossHack is maintaining an repository/page on GitHub to "track reproducibility status of open source messengers".


Distribution work

In Debian this month, the loong64 architecture was added to reproduce.debian.net. This is a 64-bit Reduced Instruction Set Computer (RISC) instruction set architecture developed by Loongson.

Vagrant Cascadian performed Non-Maintainer Uploads (NMUs) in Debian for several packages with outstanding patches over a year old. These included rocdbgapi, onevpl-intel-gpu, python-pytest-shell-utilities, python-mt-940 and pympress.

On tests.reproducible-builds.org, Vagrant Cascadian fixed the huge spike in build failures by adding passwd to the base tarballs, and re-enabled building gcc and binutils packages with PGO (Profile Guided Optimization) and LTO (Link Time Optimization) to avoid giving a false sense of reproducibility.

Inconsistencies on the reproducibility of the condor package were brought up on the Debian reproducible-builds mailing list. Following a hunch, Vagrant Cascadian eventually identified the issue was related to embedded kernel versions which was then fixed upstream and fixed in Debian as well.

Lastly, 40 reviews of Debian packages were added, 68 were updated and 75 were removed this month adding to our knowledge about identified issues. A number of issue types were updated, such as the addition of a new sphinx_reading_durations toolchain issue […], a golang_mango_generates_manpages_with_build_date issue […] and a random_offset_id_in_cython_linetrace […]. In addition, the timestamps_in_qhc issue was "refocused" to timestamps_in_qhc […].


In Fedora, Jelle van der Waa submitted a request for an official Fedora rebuilderd package which was reviewed by Neal Gompa.


Lastly, Bernhard M. Wiedemann posted another openSUSE monthly update for their reproducibility work there.


Misc news

On our mailing list this month:


Patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where applicable or possible. This month, we wrote a large number of such patches, including:


Documentation updates





Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

04 Jun 2026 7:12pm GMT

01 Jun 2026

feedPlanet Lisp

Joe Marshall: Regression

Last year I wrote some Lisp related AI apps. There was a syntax highlighter that used the LLM to determine how to colorize and highlight syntax, and a prompt refiner that takes a wimpy LLM prompt and creates more elaborate prompt from them.

I took the apps down last week. They were `vibe coded' and therefore approximate and had bugs (but that's to be expected), but they had a security hole where you could hijack the LLM processing with your own prompt turning my app into an open relay using my API key. Last week I discovered that my AI spend on video creation was becoming serious. This is odd because I never create AI video. It turned out that my app was being hijacked by a proxy in Luxembourg and was generating videos on my dime.

So I shut down the apps. I knew they had the potential of being abused, and I was willing to tolerate a small amount of abuse, but it didn't occur to me that syntax highlighter could be hijacked to generate gigabytes of video at my expense. Future applications will be careful to obtain the API key from the user.

01 Jun 2026 7:00am GMT

31 May 2026

feedPlanet Lisp

Joe Marshall: CLRHack: Meta-object Protocol

Metaobject Protocol (MOP) Implementation in CLRHack

The Metaobject Protocol in CLRHack is a high-performance implementation of the Common Lisp Object System (CLOS) integrated into the .NET 8.0 Common Language Runtime (CLR). It provides a complete meta-compilation pipeline that bridges the gap between dynamic Lisp semantics and the static CIL (Common Intermediate Language) execution model.

Core Architecture

The MOP is implemented through three primary layers:

  1. The Metaobject Hierarchy (C#): A set of foundational classes in LispBase representing classes, methods, generic functions, and slot definitions.
  2. The Runtime Engine (MopRuntime): A centralized orchestrator that manages class finalization, method combination, dispatch caching, and instance allocation.
  3. The Compiler Bridge (Lisp): Transformations in ast.lisp that translate high-level CLOS forms (defclass, defmethod) into optimized runtime calls.

Instance Representation

Because the CLR type system is strictly single-inheritance and statically defined, CLRHack decouples Lisp-level inheritance from C# inheritance. All CLOS instances are represented by the StandardObjectInstance class, which contains:

The Dispatch Pipeline

Generic function invocation is the most complex part of the implementation. When a generic function is called:

  1. Cache Lookup: The DiscriminatingFunction first checks a thread-safe dispatchCache using an InvocationCacheKey (a stack-allocated struct) to find a previously computed effective method.
  2. Applicability & Precedence: If the cache misses, the runtime computes all applicable methods and sorts them based on specializer specificity and the Class Precedence List (CPL).
  3. Method Combination: The ComputeEffectiveMethod logic builds a nested execution chain following the Standard Method Combination rules:
    • :around methods are called first, with call-next-method progressing to the next around method or the main chain.
    • The main chain executes all :before methods, the primary method, and finally all :after methods in reverse order.
  4. Fast Invocation: The resulting effective method is compiled into a Func<object[], object> that uses direct delegate invocation to minimize overhead.

Challenges and Solutions

1. Thread-Safe Non-Local Exits (call-next-method)

Challenge: call-next-method and next-method-p require access to the current invocation's state (the remaining methods and original arguments). Passing this state through every function call would break compatibility with standard Lisp function signatures.

Solution: CLRHack utilizes [ThreadStatic] fields in MopRuntime to store the currentNextMethods and currentArguments. This ensures that even in highly concurrent environments (like a web server), each OS thread has its own isolated invocation context, allowing call-next-method to function correctly without state leakage.

2. Forward References and Lazy Finalization

Challenge: Lisp allows classes to refer to superclasses that haven't been defined yet. The runtime must handle these "zombie" classes without crashing the JIT compiler.

Solution: The system implements a ForwardReferencedClassMetaobject. When a class is defined, it is automatically finalized (computing its CPL and slot layout). If a superclass is missing, a forward reference is created. The EnsureFinalized protocol ensures that inheritance is resolved and slot locations are assigned the moment the class is first instantiated or used in dispatch.

3. Performance Overhead of the "MOP Bridge"

Challenge: A naive implementation of slot-value or generic dispatch using C# reflection or linear searches is orders of magnitude slower than native C# member access.

Solution: Three distinct optimizations were applied:

4. Bootstrapping the COMMON-LISP Package

Challenge: Core CLOS functions like make-instance must be available as symbols in the COMMON-LISP package before user code runs, but they rely on the MOP runtime being fully initialized.

Solution: A MopRuntime.Initialize() method is injected into the entry point (Main) of every generated assembly. This method interns the necessary symbols and binds them to GenericFunctionClosureAdapter objects, ensuring that the MOP is "alive" before the first line of Lisp code executes.

Vibe coding the MOP basically involved feeding chapters 4 and 5 of the Art of the Meta-Object Protocol into the LLM and telling it to make an implementation plan. It came up with a twenty-step plan to bootstrap CLOS. I then spent the rest of the day instructing an agent to take on each task of the twenty-step plan in sequential order. At the end of the day, I had a working MOP

This is the end of my series of posts on CLRHack.

31 May 2026 7:00am GMT

30 May 2026

feedPlanet Lisp

Joe Marshall: CLRHack: signal and error

Implementation of SIGNAL and ERROR in CLRHack

In CLRHack, the condition signaling system is implemented in the Lisp.HandlerControl class within the LispBase library. It leverages .NET's [ThreadStatic] storage to maintain a per-thread dynamic stack of active condition handlers.

SIGNAL Implementation

The Signal(object condition) method performs the following logic:

  1. Retrieval: It fetches the activeHandlers list for the current thread. This list is a chain of [LispBase]Lisp.Handler objects maintained by handler-bind.
  2. Iteration: It iterates linearly through the list from the most recently bound handler to the oldest.
  3. Type Matching: For each handler, it calls IsType(condition, handler.ConditionType).
    • If the condition is a symbol, it checks for symbol equality (supporting simple symbol-based conditions).
    • If the condition is a .NET object, it checks if the handler's type is assignable from the condition's runtime type (supporting interop with system exceptions).
    • It treats the symbols T or EXCEPTION as catch-all types.
  4. Handler Invocation: If a match is found:
    • Recursive Signal Protection: Before calling the handler function, the current handler list is temporarily shadowed. activeHandlers is set to cell.rest (the handlers bound outside the current one). This ensures that if the handler itself calls signal, it won't trigger itself recursively.
    • Execution: The handler's Closure is invoked with the condition object as its argument.
    • Restoration: A finally block ensures the original activeHandlers list is restored if the handler returns normally.

    ERROR Implementation

    The Error(object condition) method build upon Signal:

    1. Signaling Pass: It first invokes Signal(condition). If a handler performs a non-local exit (e.g., via handler-case), the Error method never returns.
    2. Debugger Entry: If Signal returns normally (meaning all handlers declined), Error calls EnterDebugger(condition).
    3. Interactive Debugging: The debugger:
      • Prints the condition and a list of available restarts (retrieved via RestartControl.GetActiveRestarts()).
      • Provides a prompt for the user to select a restart, launch the system-level debugger (Visual Studio/Rider), or abort.
      • If a restart is selected, it is invoked interactively (potentially gathering arguments from the user).
    4. Final Fallback: If the debugger is exited without invoking a restart, Error throws a C# Exception to ensure that execution does not continue on an invalid path.

    Notable Implementation Decisions and Edge Cases

signal and error complete the Common Lisp condition system implementation for CLRHack

30 May 2026 7:00am GMT

25 Apr 2026

feedFOSDEM 2026

All FOSDEM 2026 videos are online

All video recordings from FOSDEM 2026 that are worth publishing have been processed and released. Videos are linked from the individual schedule pages for the talks and the full schedule page. They are also available, organised by room, at video.fosdem.org/2026. While all released videos have been reviewed by a human, it remains possible that one or more issues fell through the cracks. If you notice any problem with a video you care about, please let us know as soon as possible so we can look into it before the video-processing infrastructure is shut down for this edition. To report any舰

25 Apr 2026 10:00pm GMT

29 Jan 2026

feedFOSDEM 2026

Join the FOSDEM Treasure Hunt!

Are you ready for another challenge? We're excited to host the second yearly edition of our treasure hunt at FOSDEM! Participants must solve five sequential challenges to uncover the final answer. Update: the treasure hunt has been successfully solved by multiple participants, and the main prizes have now been claimed. But the fun doesn't stop here. If you still manage to find the correct final answer and go to Infodesk K, you will receive a small consolation prize as a reward for your effort. If you're still looking for a challenge, the 2025 treasure hunt is still unsolved, so舰

29 Jan 2026 11:00pm GMT

26 Jan 2026

feedFOSDEM 2026

Call for volunteers

With FOSDEM just a few days away, it is time for us to enlist your help. Every year, an enthusiastic band of volunteers make FOSDEM happen and make it a fun and safe place for all our attendees. We could not do this without you. This year we again need as many hands as possible, especially for heralding during the conference, during the buildup (starting Friday at noon) and teardown (Sunday evening). No need to worry about missing lunch at the weekend, food will be provided. Would you like to be part of the team that makes FOSDEM tick?舰

26 Jan 2026 11:00pm GMT