Unfortunately, I managed to delete my WordPress database at a time when the most recent backup I had was from 11 years ago.

So… I will hopefully get some newer information uploaded again sometime.

But, most of my content is gone.

13 Sep 2024 3:46pm GMT

For the past two years Charles Zhang and I have been working on getting my game engine, Trial, running on the Nintendo Switch. The primary challenge in doing this is porting the underlying Common Lisp runtime to work on this platform. We knew going into this that it was going to be hard, but it has proven to be quite a bit more tricky than expected. I'd like to outline some of the challenges of the platform here for posterity, though please also understand that due to Nintendo's NDA I can't go into too much detail.

Current Status

I want to start off with where we are at, at the time of writing this article. We managed to port the runtime and compiler to the point where we can compile and execute arbitrary lisp code directly on the Switch. We can also interface with shared libraries, and I've ported a variety of operating system portability libraries that Trial needs to work on the Switch as well.

The above photo shows Trial's REPL example running on the Switch devkit. Trial is setting up the OpenGL context, managing input, allocating shaders, all that good stuff, to get the text shown on screen; the Switch does not offer a terminal of its own.

Unfortunately it also crashes shortly after as SBCL is trying to engage its garbage collector. The Switch has some unique constraints in that regard that we haven't managed to work around quite yet. We also can't output any audio yet, since the C callback mechanism is also broken. And of course, there's potentially a lot of other issues yet to rear their head, especially with regards to performance.

Whatever the case, we've gotten pretty far! This work hasn't been free, however. While I'm fine not paying myself a fair salary, I can't in good conscience have Charles invest so much of his valuable time into this for nothing. So I've been paying him on a monthly basis for all the work he's been doing on this port. Up until now that has cost me ~17'000 USD. As you may or may not know, I'm self-employed. All of my income stems from sales of Kandria and donations from generous supporters on Patreon, GitHub, and Ko-Fi. On a good month this totals about 1'200 USD. On a bad month this totals to about 600 USD. That would be hard to get by in a cheap country, and it's practically impossible in Zürich, Switzerland.

I manage to get by by living with my parents and being relatively frugal with my own personal expenses. Everything I actually earn and more goes back into hiring people like Charles to do cool stuff. Now, I'm ostensibly a game developer by trade, and I am working on a currently unannounced project. Games are very expensive to produce, and I do not have enough reserves to bankroll it anymore. As such, it has become very difficult to decide what to spend my limited resources on, and especially a project like this is much more likely to be axed given that I doubt Kandria sales on the Switch would even recoup the porting costs.

To get to the point: if you think this is a cool project and you would like to help us make the last few hurdles for it to be completed, please consider supporting me on Patreon, GitHub, or Ko-Fi. On Patreon you get news for every new library I release (usually at least one a month) and an exclusive monthly roundup of the current development progress of the unannounced game. Thanks!

An Overview

First, here's what's publicly known about the Switch's environment: user code runs on an ARM64 Cortex-A57 chip with four cores and 4 GB RAM, and on top of a proprietary microkernel operating system that was initially developed for the Nintendo 3Ds.

SBCL already has an ARM64 Linux port, so the code generation side is already solved. Kandria also easily fits into 4GB RAM, so there's no issues there either. The difficulties in the port reside entirely in interfacing with the surrounding proprietary operating system of the switch. The system has some constraints that usual PC operating systems do not have, which are especially problematic for something like Lisp as you'll see in the next section.

Fortunately for us, and this is the reason I even considered a port in the first place, the Switch is also the only console to support the OpenGL graphics library for rendering, which Trial is based upon. Porting Trial itself to another graphics library would be a gigantic effort that I don't intend on undertaking any time soon. The Xbox only supports DirectX, though supposedly there's an OpenGL -> DirectX layer that Microsoft developed, so that might be possible. The Playstation on the other hand apparently still sports a completely proprietary graphics API, so I don't even want to think about porting to that platform.

Anyway, in order to get started developing I had to first get access. I was lucky enough that Nintendo of Europe is fairly accommodating to indies and did grant my request. I then had to buy a devkit, which costs somewhere around 400 USD. The devkit and its SDK only run on Windows, which isn't surprising, but will also be a relevant headache later.

Before we can get on to the difficulties in building SBCL for the Switch, let's first take a look at how SBCL is normally built on a PC.

Building SBCL

SBCL is primarily written in Lisp itself. There is a small C runtime as well, which you use a usual C compiler to compile, but before it can do that, there's some things it needs to know about the operating system environment it compiles for. The runtime also doesn't have a compiler of its own, so it can't compile any Lisp code. In order to get the whole process kicked off, SBCL requires another Lisp implementation to bootstrap with, ideally another version of itself.

The build then proceeds in roughly five phases:

1. build-config
   This step just gathers whatever build configuration options you want for your target and spits them out into a readable format for the rest of the build process.

2. make-host-1

   Now we build the cross-compiler with the host Lisp compiler, and at the same time emit C header files describing Lisp object layouts in memory as C structs for the next step.

3. make-target-1

   Next we run the target C compiler to create the C runtime. As mentioned, this uses a standard C compiler, which can itself be a cross-compiler. The C runtime includes the garbage collector and other glue to the operating system environment. This step also produces some constants the target Lisp compiler and runtime needs to know about by using the C compiler to read out relevant operating system headers.

4. make-host-2

   With the target runtime built, we build the target Lisp system (compiler and the standard library) using the Lisp cross-compiler built by the Lisp host compiler in make-host-1. This step produces a "cold core" that the runtime can jump into, and can be done purely on the host machine. This cold core is not complete, and needs to be executed on the target machine with the target runtime to finish bootstrapping, notably to initialize the object system, which requires runtime compilation. This is done in

5. make-target-2

   The cold core produced in the last step is loaded into the target runtime, and finishes the bootstrapping procedure to compile and load the rest of the Lisp system. After the Lisp system is loaded into memory, the memory is dumped out into a "warm core", which can be loaded back into memory in a new process with the target runtime. From this point on, you can load new code and dump new images at will.

Notable here is the need to run Lisp code on the target machine itself. We can't cross-compile "purely" on the host, not in the least because user Lisp code cannot be compiled without also being run like batch-compiled C code can, and when it is run it assumes that it is in the target environment. So we really don't have much of a choice in the matter.

In order to deploy an application, we proceed similar to make-target-2: We compile in Lisp code incrementally and then when we have everything we need we dump out a core with the runtime attached to it. This results in a single binary with a data blob attached.

When the SBCL runtime starts up it looks for a core blob, maps it into memory, marks pages with code in them as executable, and then jumps to the entry function the user designated. This all is a problem for the Switch.

Building for the Switch

The Switch is not a PC environment. It doesn't have a shell, command line, or compiler suite on it to run the build as we usually do. Worse still, its operating system does not allow you to create executable pages, so even if we could run the compilation steps on there we couldn't incrementally compile anything on it like we usually do for Lisp code.

But all is not lost. Most of the code is not platform dependent and can simply be compiled for ARM64 as usual. All we need to do is make sure that anything that touches the surrounding environment in some way knows that we're actually trying to compile for the Switch, then we can use another ARM64 environment like Linux to create our implementation.

With that in mind, here's what our steps look like:

1. build-config
   We run this on some host system, using a special flag to indicate that we're building for the Switch. We also enable the fasteval contrib. We need fasteval to step in for any place where we would usually invoke the compiler at runtime, since we absolutely cannot do that on the Switch.

2. make-host-1

   This step doesn't change. We just get different headers that prep for the Switch platform.

3. make-target-1

   Now we use the C compiler the Nintendo SDK provides for us, which can cross-compile for the Switch. Unfortunately the OS is not POSIX compliant, so we had to create a custom runtime target in SBCL that stubs out and papers over the operating system environment differences that we care about, like dynamic linking, mapping pages, and so on.
   Here is where things get a bit weird. We are now moving on to compiling Lisp code, and we want to do so on a Linux host system. So we have to...

4. build-config (2)

   We now create a normal ARM64 Linux system with the same feature set as for the Switch. This involves the usual steps as before, though with a special flag to inform some parts of the Lisp process that we're going to ultimately target the Switch.

5. make-host-1 (2)

6. make-target-1 (2)

7. make-host-2

8. make-target-2

   With all of this done we now have a slightly special SBCL build for Linux ARM64. We can now move on to compiling user code.

9. For user code we now perform some tricks to make it think it's running on the Switch, rather than on Linux. In particular we modify *features* to include :nx (the Switch code name) and not :linux, :unix, or :posix. Once that is set up and ASDF has been neutered, we can compile our program (like Trial) "as usual" and at the end dump out a new core.

We've solved the problem of actually compiling the code, but we still need to figure out how to get the code started on the Switch, since it does not allow us to do the usual core-mapping strategy. As such, attaching the new core to the runtime we made for the Switch won't work.

To make this work, we make use of two relatively unknown features of SBCL: immobile-code, and elfination. Usually when SBCL compiles code at runtime, it sticks it into a page somewhere, and marks that page executable. The code itself however could become unneeded at some point, at which point we'd like to garbage collect it. We can then reclaim the space it took up, and to do so compact the rest of the code around it. The immobile-code feature allows SBCL to take up a different strategy, where code is put into special reserved code pages and remains there. This means it can't be garbage collected, but it instead can take advantage of more traditional operating system support. Typically executables have pre-marked sections that the operating system knows to contain code, so it can take care of the mapping when the program is started, rather than the program doing it on its own like SBCL usually does.

OK, so we can generate code and prevent it from being moved. But we still have a core at the end of our build that we now need to transform into the separate code and data sections needed for a typical executable. This is done with the elfination step.

The elfinator looks at a core and performs assembly rewriting to make the code position-independent (a requirement for Address Space Layout Randomisation), and then tears it out into two separate files, a pure code assembly file, and a pure data payload file.

We can now take those two files and link them together with the runtime that the C compiler produced and get a completed SBCL that runs like any other executable would. So here's the last steps of the build process:

10. Run the elfinator to generate the assembly files

11. Link the final binary

12. Run the Nintendo SDK's authoring tools to bundle metadata, shared libraries, assets, and the application binary into one final package

That's quite an involved build setup. Not to mention that we need at least an ARM64 Linux machine to run most of the build on, as well as either an AMD64 Windows machine (or an AMD64 Linux machine with Wine) to run the Nintendo SDK compiler and authoring tools.

I usually use an AMD64 Linux machine, so there's a total of three machines involved: The AMD64 "driver," the ARM64 build host, and a Windows VM to talk to the devkit with.

I wrote a special build system with all sorts of messed up caching and cross-machine synchronisation logic to automate all of this, which was quite a bit of work to get going, especially since the build should also be drivable from an MSYS2/Windows setup. Lots of fun with path mangling!

So now we have a full Lisp system, including user code, compiling for and being able to run on the Switch. Wow! I've skipped over a lot of the nitty-gritty dealing with getting the build properly aware of which target it's building for, making the elfinator and immobile-code working on ARM64, and porting all of the support libraries like pathname-utils, libmixed, cl-gamepad, etc. Again, most of the details we can't openly talk about due to the NDA. However, we have upstreamed what work we could, and all of the Lisp libraries don't have a private fork.

It's worth noting though that elfination wasn't initially designed to produce position independent executable Lisp code, which is usually full of absolute pointers. So we needed to do a lot of work in the SBCL compiler and runtime to support load time relocation of absolute pointers and make sure code objects (which usually contain code constants) no longer have absolute pointers, as the GC can't modify executable sections. Not even the OS loader is allowed to modify executable sections to relocate absolute pointer. We did this by relocating absolute pointers like code constants outside of the text space into a read-writable space close enough to rewrite constant references in code to load from this r/w space instead, which the loader and the moving GC can fixup pointers at.

Instead of interfacing directly with the Nintendo SDK, I've opted to create my own C libraries that have a custom interface the Lisp libraries interface with in order to access the operating system functionality it needs. That way I can at least publish the Lisp bits openly, and only keep the small C library private. Anyway, now that we can run stuff we're not done yet. Our system actually needs to keep running, too, and that brings us to

The Garbage Collector

Garbage collection is a huge topic in itself and there's a ton of different techniques to make it work efficiently. The standard GC for SBCL is called "gencgc", a Generational Garbage Collector. Generational meaning it keeps separate "generations" of objects and scans the generations in different frequencies, copying them over to another generation's location to compact the space. None of this is inherently an issue for the Switch, if it weren't for multithreading.

When multiple threads are involved, we can't just move objects around, as another thread could be accessing it at any time. The easiest way to resolve this conflict is to park all threads before engaging garbage collection. So the question becomes: when a thread wants to start garbage collection, how does it get the other threads to park?

On Unix systems a pretty handy trick is used: we can use the signalling mechanism to send a signal to the other threads, which then take that hint to park.

On the Switch we don't have any signal mechanism. In fact, we can't interrupt threads at all. So we instead need to somehow get each thread to figure out that it should park on its own. The typical strategy for this is called "safepoints".

Essentially we modify the compiler a little bit to inject some extra code that checks whether the thread should park or not. This strategy has some issues, namely:

• Adding a check isn't free. So we want to check as little as possible

• If we don't check frequently enough, we are going to stall all the other threads because GC can't begin until they're all parked

• If we have to inject a lot of instructions for a check, it is going to disrupt CPU cache lines and pipelining optimisations

The current safepoint system in SBCL was written for Windows, which similarly does not have inter-process signal handlers. However, unlike the Switch, it does still have signal handling for the current thread. So the current safepoint implementation was written with this strategy:

Each thread keeps a page around that a safepoint just writes a word to. When GC is engaged, those pages are marked as read-only, so that when the safepoint is hit and the other thread tries to write to the page, a segmentation fault is triggered and the thread can park. This is efficient, since we only need a single instruction to write into the page.

On the Switch we can't use this trick either, so we have to actually insert a more complex check, which can be tricky to get working as intended, as all parallel algorithms tend to be.

Since safepoints aren't necessary on any other platform than Windows, it also hasn't been tested anywhere else, so aside from modifying it for this new platform it's also just unstable. It is apparently quite a big mess in the code base and would ideally be redone from scratch, but hopefully we don't have to go quite that far.

I'd also like to give special mention to the issue that CLOS presents. Usually SBCL defers compilation of the "discriminating function" that is needed to dispatch to methods to the first call of the generic function. This is done because CLOS is highly dynamic and allows adding and removing methods pretty much at any time, and there's usually no good point in time that the system knows it is complete. Of course, on the Switch we can't invoke the compiler, so we can't really do this. For now our strategy has been to instead rely on the fast evaluator. We stub out the compile function to create a lambda that executes the code via the evaluator instead. This has the advantage of working with any user code that relies on compile as well, though it is obviously much slower for execution than it would be if we could actually compile.

This neatly brings us to

Future Work

The fasteval trick is mostly a fallback. Ideally I'd like to explore options to freeze as much of CLOS in place as possible right before the final image is dumped and compile as much as possible ahead of time. I'd also like to investigate the block compilation mode that Charles restored some years back more closely.

It's very possible that the Switch's underpowered processor will also force us to implement further optimisations, especially on the side of my engine and the code in Kandria itself. Up until now I've been able to get away with comparatively little optimisation, since even computers of ten years ago are more than fast enough to run what I need for the game. However, I'm not so sure that the Switch could match up to that even if it didn't also introduce additional constraints on performance with its lack of operating system support.

First, though, we need to get the garbage collector running fully. It runs enough to boot up and get into Trial's main loop, but as soon as it hits multi-generation compaction, it falls flat on its face.

Next we need to get callbacks from C working again. Apparently this is a part of the SBCL codebase that can only be described as "a mess," involving lots of hand-rolled assembly routines, which probably need some adjustments to work correctly with immobile-code and elfination. Callbacks fortunately are relatively rare, Trial only needs them for sound playback via libmixed.

There's also been some other issues that we've kept in the back of our heads but don't require our immediate attention, as well as some extra portability features I know I'll have to work on in Trial before its selftest suite fully passes on the Switch.

Conclusion

I'll be sure to add an addendum here should the state of the port significantly change in the future. Some people have also asked me if the work could be made public, or if I'd be willing to share it.

The answer to that is that while I would desperately like to share it all publicly, the NDA prevents us from doing so. We still upstream and publicise whatever we can, but some bits that tie directly into the Nintendo SDK cannot be shared with anyone that hasn't also signed the NDA. So, in the very remote possibility that someone other than me is crazy enough to want to publish a Common Lisp game on the Nintendo Switch, they can reach out to me and I'll happily give them access to our porting work once the NDA has been signed.

Naturally, I'll also keep people updated more closely on what's going on in the monthly updates for Patrons. With that all said, I once again plead with you to consider supporting me on Patreon, GitHub, or Ko-Fi. All the income from these will, for the foreseeable future, be going towards funding the SBCL port to the Switch as well as the current game project.

Thank you as always for reading, and I hope to share more exciting news with you soon!

13 Sep 2024 9:03am GMT

This post is somewhat prompted by a recent blog post by vindarel, about Common Lisp's various built-in equality predicates. It is aleo related to Marco Antoniotti's CDR 8, Generic Equality and Comparison for Common Lisp, implemented by Charles Zhang; Alex Gutev's GENERIC-CL; and Henry Baker's well-known 1992 paper on equality.

Let me start by summarizing those designs. CDR 8 proposes a generic equaity function equals, and a comparison function compare. These are both CLOS generic functions intended to be user-extended, though they also have some predefined methods. equals has several keyword parameters controlling its exact behavior. One of these is case-sensitive, which controls string comparison. Another is recursive, which controls its behavior on conses; if recursive is false (the default), conses are compared by eq, but if it's true, a tree comparison is done. compare is specified to return one of the symbols <, >, =, or /= to indicate the relative order of its arguments; it also has keyword parameters such as case-sensitive and recursive.

GENERIC-CL replaces many CL operations with CLOS generic functions, and also adds new ones. It touches many parts of the language other than equality and comparison, but I'll leave those aside for now. It has two generic equality functions: equalp, which, notwithstanding the name, is case-sensitive for characters and strings, and likep, which is case-insensitive. It also has comparison predicates lessp etc., along with a compare function (implemented using lessp) that can return :less, :equal, or :greater.

Henry's paper makes some interesting arguments about how a Common Lisp equality predicate should behave; he makes these concrete by defining a novel predicate egal. His most salient point, for my purposes, is that mutable objects, including vectors and conses, should always be compared with eq. I will argue below that FSet adheres to the spirit of this desideratum even though not to its letter.

FSet advertises itself as a "set-theoretic" collections library, and as such, requires a well-defined notion of equality. Also, since it is implemented using balanced binary trees, it requires an ordering function. FSet defines a generic function compare with these properties:

• It returns one of the symbols :less, :equal, :greater, or :unequal (:unequal is used in certain rare cases of values which are not equal but cannot be consistently ordered)
  
• It implements a strict weak ordering, with an additional constraint: along with incomparability (indicated by either :equal or :unequal) being transitive, equality is also transitive by itself
  
• It can compare any two Lisp objects; this is an element of FSet's design philosophy
  
• Being a generic function, it is of course user-extensible

FSet's equality predicate is equal?, which simply calls compare and checks that the result is :equal. Thus, the only step required to add a user-defined type to the FSet universe is to define a compare method for it. FSet provides a few utilities to help with this, which I'll go into below.

The cases in which compare returns :unequal to indicate unequal-but-incomparable arguments include:

• Numbers of equal value but different types; that is, = would return true on them, but eql would return false. Example: the integer 1 and the float 1.0.
• Distinct uninterned symbols (symbols whose symbol-package is nil) whose symbol-names are equal (by string=).
• Objects of a type for which no specific compare method has been defined, and which are distinct according to eql.
• If you create a package, rename it, then create a new package reusing the original name of the first package, the two packages compare :unequal. (FSet holds on to the original name, to protect itself from the effects of rename-package, which could otherwise be distastrous.) Also, two symbols with the same name, one from the old package and one from the new, also compare :unequal.
  
• Aggregates which are being compared component-wise, in the case where none of the component-wise comparisons returns :less or :greater, and at least one of them returns :unequal.
  

If compare's default method is called with objects of different classes, it returns a result based solely on the classes; the contents of the objects are not examined. Again, it is part of FSet's design philosophy to give you as much freedom as reasonably possible; this includes allowing you to have sets containing more than one kind of object.

(In general, FSet's built-in ordering has been chosen for performance, not for its likely usefulness to clients. For example, compare on two strings of different lengths orders the shorter one first, ignoring the contents, because this requires only an O(1) operation.)

Comparison with equal

FSet's equal? on built-in CL types behaves almost identically to CL's equal, with the one difference that on vectors (other than bit-vectors), equal just calls eq, but equal? compares the contents. (I just noticed that this is not true for multidimensional arrays, and have filed an FSet bug.) (On bit-vectors, they both compare the contents.)

Comparison with CDR 8

There are noticeable similarities between FSet and the CDR 8 proposal; the latter not only includes a comparison function, but even provides for it to return /=, corresponding to FSet's :unequal, to indicate unequal but incomparable arguments. But the idea that the behavior of equality and comparison could be modified via keyword parameters does not seem appropriate for FSet. I think it would make FSet quite a bit harder to use, for little gain. For example, FSet comparison on lists walks the lists, but CDR 8, by default, just calls eq on their heads; users would have to remember to pass :recursive t to get the behavior they probably expect. FSet collections would have to remember which options they were created with, and if you tried, say, to take the union of two sets which used different options, you'd get an error.

Years of programming experience - not only with FSet but also with Refine, the little-known proprietary language that inspired FSet - have left me with the clear impression that having a single global equality predicate is a great simplification and very rarely limiting, provided it was defined properly to begin with.

I also note that FSet has more predefined methods for its comparison function (and therefore for its equality predicate) than are proposed in CDR 8. In particular, CDR 8's default compare methods return /= in more cases (e.g. distinct symbols), which is not terribly useful, in my view; FSet tries to minimize its use of :unequal because its data structure code, in that case, has to fall back to using alists, which have much poorer time complexity than binary trees. (OTOH, Marco seems to have overlooked the other cases listed above that arguably should be treated as unequal but incomparable.)

Comparison with GENERIC-CL

Again, there are noticeable similarities between FSet's and GENERIC-CL's equality predicates and comparison functions. GENERIC-CL does have two different equality predicates, equalp and likep, but these have no parameters other than the objects to be compared; it does not follow the CDR 8 suggestion of specifying keyword parameters that alter their behavior. Its equalp is very similar to FSet's equal?, but not quite identical; one difference is that it returns true when called on the integer 1 and the float 1.0, where both fset:equal? and cl:equal return false.

That normally-minor discrepancy is related to a larger deficiency: GENERIC-CL's comparison operator has no defined return value corresponding to :unequal, to indicate unequal-but-incomparable arguments. That is, FSet and CDR 8 both recognize that comparison can't implement a total ordering over all possible pairs of objects, but GENERIC-CL overlooks this point.

There are other overlaps between FSet and GENERIC-CL, but I'll save an analysis of those for another time.

Comparison with EGAL

Henry is proposing an extension to Common Lisp, not an operator that can be written in portable CL. This shows up in two ways: first, some of his sample code implementing egal requires access to implementation internals; second, he proposes a distinction between mutable and immutable vectors and strings that does not exist in CL. The text also suggests adding an immutable cons type to CL, though the sample code doesn't mention this case.

I agree with Henry in principle: a mutable cons (or string, or vector) is a very different beast from an immutable one; as he puts it, "eq is correct for mutable cons cells and equal is correct for immutable cons cells". CL would have been a better language, in principle, had conses been immutable, and immutable strings and vectors been available (if perhaps not the default). But here I must invoke one of my favorite quips: "The difference between theory and practice is never great in theory, but in practice it can be very great indeed." The key design goal of CL, to unify the Lisp community by providing a language into which existing programs in various Lisp dialects could easily be ported, demanded that conses remain mutable by default. Adding immutable versions of these types was not, to my knowledge, a priority.

And as Henry himself points out, in the overwhelmingly most common usage pattern for these types, they are treated as immutable once fully constructed. For example, a common idiom is for a function to build a list in reverse order, then pass it through nreverse before returning it; at that point, it is fully constructed, and it won't be modified thereafter. Obviously, this is a generalization over real-world Lisp programs and won't always be true, but since Lisp encourages sharing of structure, I think Lisp programmers learn pretty early that they have to be very careful when mutating a list or string or vector that they can't easily prove they're holding the only pointer to (normally by virtue of having just created it). Given that this is pretty close to being true in practice, and that comparing these aggregates by their contents is usually what people want to do when they use them as members of collections, it would seem odd for FSet to distinguish them by identity.

Also, there's the simple fact that for these built-in types, CL provides no portable way to order or hash them by identity. Such functionality must exist internally for use by eq and eql hash tables, but the language does not expose any portable interface to it.

So in this case, both programming convenience and the hard constraints of implementability force a choice that is contrary to theoretical purity: FSet must compare these types by their contents. The catch, of course, is that one must be careful, once having used a list or string or vector as an element of an FSet collection, never to modify it, lest one break the collection's ordering invariant. But in practice, this rule doesn't seem at all onerous: if you found the object in the heap somewhere - as opposed to having just created it- don't mutate it.

When it comes to user-defined types, however, the situation is quite different. It is easy for the programmer, defining a class intended for mutation, to arrange for FSet to distinguish objects of the class by their identity rather than their contents. The recommended way to do this is to include a serial-number slot that is initialized, at object-creation time, to the next value from an integer sequence; then write a compare method that uses this slot. (I'll show some examples shortly.)

So if the design of your program involves some pieces of mutable state that are placed in collections, my strong recommendation is that such state should never be implemented as a bare list or string or vector, but should always be wrapped in an instance of a user-defined class. I believe this to be a good design principle in general, even when FSet is not involved, but it becomes imperative for programs using FSet.

Adding Support for User-Defined Classes

When adding FSet support for a user-defined class, the first question is whether instances of the class represent mutable objects or mathematical values. If it's a mathematical value, it should be treated as immutable once constructed. (Alas, CL provides no way to enforce immutability.) In that case, it should be compared by content. FSet provides a convenient macro compare-slots for this purpose. Here's an example:

This specifies that frobs shall be ordered first by position, then by color. compare-slots handles the details for you, including the complications that arise if one of the slot value comparisons returns :unequal.

For standard classes, best performance is obtained by supplying slot names as quoted symbols rather than function-quoted accessor names:

I am not sure whether to recommend the use of slot names for structure classes; the answer may depend on the implementation. At least on SBCL, you're probably better off using accessor functions for structs.

(Actually, the functions supplied don't have to be accessors; you could compare by some computed value instead, if you wanted. I haven't seen a use for this possibility in practice, though.)

Structure classes implementing mutable objects should do something like this:

For standard classes implementing mutable objects, FSet provides an especially convenient solution: just include identity-ordering-mixin as a superclass:

That's it!

More on FSet's Single Global Ordering

I sometimes get pushback, albeit mostly from people who haven't actually used FSet, about my design decision to have a single global ordering implemented by compare, rather than allowing collections to accept an ordering function when they are created. Let me defend this decision a little bit.

Because the ordering is extensible by defining new methods on compare, a programmer can always force a non-default ordering by defining a wrapper type. For example, if you have a map whose keys are strings and which you want to be maintained in lexicographic order, you can easily write a structure class to wrap the strings, and give that class a compare method that forces the strings to be compared lexicographically. (FSet even helps you out by providing a generic function compare-lexicographically, which you can just call.)

That said, I believe the need to write wrapper classes arises very rarely. It's needed only when there is a reason that a set or map needs to be continually maintained in the non-default order. If the non-default ordering is needed only occasionally - say, when the collection is being printed - it's usually easier to convert it to a list at that point (see FSet's generic function convert, about which I should write another blog post) and then just call sort or stable-sort on it.

And there is a wonderful simplicity to having the ordering be global. Ease of use is a very important design goal for FSet; collection-specific orderings would give the user another wrinkle to think about. I just don't see that the benefits, which seem to me very small, would outweigh the cost in cognitive load.

Perhaps the best way to put it is that FSet is primarily intended for application programming, not systems programming. The distinction is fuzzy, but broadly, if programmer productivity is more important to you than squeezing out the last few percent of performance, you're doing application programming, not systems programming. This is not necessarily a distinction about the kind of program being written - there certainly are applications that have performance-sensitive parts - but rather, about the amount of knowledge, experience, and mental effort required to write it. FSet is designed for general productivity, not necessarily for someone who needs maximal control to achieve maximal performance.

05 Sep 2024 6:58am GMT

My friend Rui was interviewed about Lisp and how we use it at SISCOG. The original interview is in Portuguese but you can read a translation via DeepL below:

SISCOG Engineering: get to know this cutting-edge Portuguese company

Find out how Lisp continues to drive innovation at SISCOG. Interview with Rui Patrocínio, Scheduling Team Leader at SISCOG

#1 How did you first get to know the Lisp programming language?

When I joined Técnico in 1999, the programming language taught in Introduction to Programming was Scheme, which is in the Lisp family. The curriculum closely followed the MIT 6.001 course (whose lectures from the 80s are on Youtube) and one of the best programming books ever for beginners and beyond: Structure and Interpretation of Computer Programs. The great advantage of learning to program with Scheme is that it is a very simple language, with a very simple syntax. All the effort goes into understanding the logic of what you're implementing, so there's no need to memorize the syntactical nuances of the language.

After that experience, Lisp reappeared as Common Lisp in the Artificial Intelligence course. Common Lisp is the current industrial version of the Lisp family of languages, which is what SISCOG uses on a daily basis. Common Lisp is a multi-paradigm language. Imperative, functional, object-oriented programming with very sophisticated meta-programming mechanisms available. As such, using the language as a whole implies some maturity and it's only natural that it should appear later in the curriculum of a computer engineering course.

I think it was the "power" of the Common Lisp language that made it particularly appealing to me and, despite the standard being from 1994, it is a very modern language. It should be noted that Guy Steele, one of the main figures behind the Common Lisp standard, was also heavily involved in the development of Scheme, C and Java, being hired by Sun at one point to improve Java. Here's a quote from him on a mailing list focused on the discussion of programming languages from the 1990s, talking about Java:

And you're right: we were not out to win over the Lisp programmers; we were after the C++ programmers. We managed to drag a lot of them about halfway to Lisp. Aren't you happy? -Guy Steele

#2 How SISCOG uses Lisp in its products

We use Common Lisp in the vast majority of our software. Both desktop applications and backend parts of web applications are implemented in Common Lisp. We also use C++ to develop specific modules, but Common Lisp still has the most lines of code in our repositories. It's a very expressive language that compiles to machine code reasonably efficiently without much effort on the part of the programmer. So it's an easy choice for most scenarios.

It remains to be said that our products have been in operation for more than 30 years in various national and international companies, such as the London and Lisbon Underground, or the railways of the Netherlands or Canada. They are decision support software for the optimized planning of these transport operators' resources, namely time and space, materialized in timetables, vehicles and personnel. These products help to plan and manage these operational resources as quickly and efficiently as possible, providing gains and savings in various areas, as well as, for example, greater satisfaction on the part of workers thanks to shifts and work schedules that better meet their preferences. And all with Lisp as a base!

#3 What are the main advantages of the Lisp syntax compared to other programming languages, such as C++?

The great advantage of the syntax is also the thing you'll find most strange at first: the brackets. The fact that everything uses a prefixed syntax also makes everything quite uniform. This combination of parentheses and prefixed syntax is called s-expressions in Lisp. S-expressions are code and data at the same time, and this is what allows you to have macros (functions that take code as an argument and return code as a value) that extend the language transparently, as if they belonged to the standard language.

#4 Can you explain the concept of "S-expressions" and how they contribute to the clarity of LISP code?

Basically, "S-expressions" are one of two things:

• atomic expressions (e.g. the number 2024 or the string "hello world" or the symbol +)
• a list, typically in the format (operator arg1 arg2 ... argn)

It's this uniformity, as I said earlier, that makes everything simpler. It also helps a lot to be able to generate code programmatically because it's all about manipulating lists, for which Common Lisp has a very reasonable API. This is where the power of Lisp macros comes from.

At first you find it strange, then you understand it. [nice try, DeepL. Rui wrote "Primeiro estranha-se, depois entranha-se." which is quoting a famous Coca-Cola slogan written by Fernando Pessoa in the 1940s. Richard Zenith, in his biography Pessoa: An Experimental Life, attempts to translate it as "On the first day you drink it slow. On the fifth day you can't say no." -Luís]

#5 How does Lisp allow rapid prototyping using untyped variables?

Static typing in large codebases is widely advocated today because it allows more automatic tools to check for problems in the code. Common Lisp has dynamic typing, but allows optional type declaration. It's quite common to only declare types when it's necessary to "squeeze" more performance out of a code segment. The compiler, via type inference, provides some information about problems encountered when there are enough type declarations to extract relevant information. This allows us to "fight the compiler" only in segments of code where it is very relevant and not be making premature optimizations to the whole code.

#6 How does Lisp facilitate metaprogramming?

There are two basic "tools" for metaprogramming in Common Lisp: the Meta Object Protocol and the macros mentioned earlier.

Meta Object Protocol is basically the mechanism behind the implementation of the Common Lisp object system (called CLOS, Common Lisp Object System). Although it doesn't belong to the standard, it is available in most Common Lisp implementations and is sometimes useful, particularly when implementing utilities to inspect the code or improve our development environment. For example, it's possible to implement something that shows us the relationships between classes, what methods exist, etc. This is not something you do on a day-to-day basis, but it can be done by you without depending on the guts of a specific IDE.

Another interesting mechanism is macros. When CLOS was introduced, it was basically a set of macros on top of the Lisp of the time (historically there were a few more iterations - CLOS wasn't the first object system developed). In other words, with macros it's possible to extend the language to introduce most of the mechanisms and paradigms that exist in other languages (such as object-oriented programming) in a way that feels natural to a Lisp programmer.

This type of mechanism allows SISCOG to implement undo and redo mechanisms in our software in a way that is practically transparent to the programmer. This is done by extending the class declaration. The programmer notes for which attributes of each class a history needs to be kept and the end user is able to undo their operations and see these data variations instantly (with the typical implementation using Command Pattern, multi-level undo is not instantaneous as it is necessary to re-execute operations, which may not be trivial; the tradeoff is, of course, the memory spent).

#7 What is your opinion of Lisp's learning curve compared to other programming languages?

Anyone with experience in a few different paradigm languages can quickly program in Common Lisp without major problems. Most of the concepts are familiar. Our experience at SISCOG, particularly with more recent hires who had no contact with Lisp in college, is that people adapt quickly and within a week are programming and making minor corrections.

Obviously, there are parts of the language that are less common and you need more maturity to use them effectively. These include the Meta-object protocol and the macros mentioned earlier.

In this case, the need for greater maturity comes simply from the fact that we are extending the language. Designing a new language isn't easy and there are people whose career it is to do this (like Guy Steele mentioned earlier). Of course, this is also rare and the most common thing is to introduce macros for small functionalities that make the programmer more productive with Domain Specific Languages.

#8 Why is Lisp a popular choice for specialized areas such as artificial intelligence and natural language processing?

It's essentially for historical reasons. Lisp was born in that context, there have been several "classic" artificial intelligence systems implemented in Lisp and so it has continued to be used. The fact that it's very easy to start a project quickly, it's easy to iterate and progressively improve what's been done, also helps in a research context where we're more concerned with testing ideas than making the software "bulletproof", which you can do progressively (treating errors, adding static typing, etc.) in Common Lisp.

#9 Can you give some examples of projects at SISCOG where Lisp has proved particularly advantageous?

SISCOG has relied on Lisp from the start. It's a very stable language, with good compilers and good performance, it's compact and has allowed us to go through the history of computer science across various platforms (Lisp machines, Unix, Windows) and maintain a code base with immense domain knowledge over the years. A company with software in production for over 30 years is not very common in the world and Lisp is clearly our little secret weapon.

#10 How does the Lisp development community compare with other programming language communities in terms of support and resources?

The Lisp community isn't very large, but what we lack in programmers, we make up for in enthusiasm. Paul Graham (of Y Combinators) was a great driving force behind the language in the late 1990s, early 2000s and Hacker News still talks about Common Lisp (and variants) on a regular basis. Many years ago there was a strong community on the old newsgroups, but that has largely moved on to Reddit and IRC. There are also a few annual conferences (e.g. European Lisp Symposium) where some of the most important members of the community usually gather. We don't have by far the largest number of libraries available and sometimes we have to implement things 'in house'. The community does, however, have enough to make high-quality software that is sold all over the world, as SISCOG has demonstrated.

#11 What are your predictions for the future of Lisp in the technology industry?

It seems to me that the future of a language depends a lot on fashions and the investments made in it. Google uses Lisp via the purchase of a company a few years ago (ITA Software, for flight search) and, as long as these large companies continue to invest, languages will thrive. At the moment, fashions have moved on to other platforms, but let's see what the future brings. Perhaps our little secret weapon will become less and less secret.

03 Sep 2024 9:52am GMT

Common Lisp has various equality functions: =, eq, eql, equal, equalp, string-equal, char-equal... but what are the differences?? We tell you everything, with examples.

As usual, this is best read on the Common Lisp Cookbook (a new page added on August, 2024). This is where it will get the updates.

In short:

• = is only for numbers and equal is the equality predicate that works on many things.
• you can't overload built-in operators such as = or equal for your own classes, unless you use a library.
• when you manipulate strings with functional built-ins (remove-if, find...) and you are surprised to get no results, you probably forgot the :test key argument: (find "foo" '("hello" "foo") :test #'equal).

Table of Contents

• = is for numbers (beware of NIL)
• eq is low-level. Think pointers, position in memory.
• eql is a better eq also for numbers of same types and characters.
• equal is also for strings (for objects whose printed representation is similar).
• equalp is case-insensitive for strings and for numerical value of numbers.
• Other comparison functions
  • null
  • eql is used by default by many CL built-ins
  • char-equal
  • strings and string-equal
  • Compare trees with tree-equal
  • Compare function table: to compare against (this), use (that) function
  • How to compare your own objects AKA built-in functions are not object-oriented
• Credits
• See also

= is for numbers (beware of NIL)

The = function compares the value of two or more numbers:

(= 2 2) ;; => T
(= 2 2.0 2 2) ;; => T
(= 2 4/2) ;; => T

(= 2 42) ;; => NIL

but = is only for numbers. In the below example we get an error with the interactive debugger. We show the error message, the condition type, and the backtrace, from SBCL.

(= 2 NIL)
;; => ERROR:
The value
  NIL
is not of type
  NUMBER
when binding SB-KERNEL::Y

   [Condition of type TYPE-ERROR]

Restarts:
  ...

Backtrace:
  0: (SB-KERNEL:TWO-ARG-= 2 NIL) [external]
  1: (SB-VM::GENERIC-=)
  2: (= 2 NIL)

Note how SB-KERNEL::Y refers to an internal variable of the compiler. No, you don't have a Y in your code.

As a consequence, if your equality check with numbers might contain NILs, you can use equalp, or encapsulate your variables with (or ... 0), or do prior checks with (null ...).

eq is low-level. Think pointers, position in memory.

(eq x y) is true if and only if x and y are the same identical object.

eq works for symbols and keywords.

Those are true:

(eq :a :a)
(eq 'a 'a)

If we compare an object with itself, it is eq:

(let ((x '(a . b)))
  (eq x x))
;; => T

eq does not work to compare numbers, lists, strings and other compound objects. It looks like it can, but it isn't specified to be true for all implementations.

As such, eq works for numbers on my implementation, but it might not on yours:

(eq 2 2) ;; => T or NIL, this is not specified

An implementation might allocate the exact same position in memory for the same number, but it might not. This isn't dictated by the standard.

Likewise, these might depend on the implementation:

(eq '(a . b) '(a . b)) ;; might be true or false.
(eq #\a #\a) ;; true or false

Comparing lists or strings are false:

(eq (list 'a) (list 'a)) ;; => NIL
(eq "a" "a") ;; => NIL

those strings (vectors of characters) are not equal by eq because the compiler might have created two different string objects in memory.

eql is a better eq also for numbers of same types and characters.

The eql predicate is true if its arguments are eq, or if they are numbers of the same type with the same value, or if they are character objects that represent the same character.

In terms of usefulness, we could say that eq < eql.

Now this number comparison is true:

(eql 3 3) ;; => T

but beware, this one isn't because 3 and 3.0 are not of the same type (integer and single float):

(eql 3 3.0) ;; => NIL

for complex numbers:

(eql #c(3 -4) #c(3 -4)) ;; is true.
(eql #c(3 -4.0) #c(3 -4)) ;; is false (because of -4.0 and -4)

Comparing two characters works:

(eql #\A #\A) ;; => T

And we still can't compare lists or cons cells:

(eql (cons 'a 'b) (cons 'a 'b)) ;; => NIL

equal is also for strings (for objects whose printed representation is similar).

The equal predicate is true if its arguments are structurally similar (isomorphic) objects. A rough rule of thumb is that two objects are equal if and only if their printed representations are the same.

Again, conceptually, we could say that eq < eql < equal.

We can still not compare numbers of different types:

(equal 3 3.0) ;; => NIL

but we can now compare lists and cons cells. Indeed, their printed representation is the same. No matter this time if they are different objects in memory.

(equal (cons 'a 'b) (cons 'a 'b)) ;; => T

(equal (list 'a) (list 'a)) ;; => T

We can compare strings!

(equal "Foo" "Foo") ;; => T

No matter if they are different objects in memory:

(equal "Foo" (copy-seq "Foo")) ;; => T

Case is important. Indeed, "FOO" doesn't print the same as "foo":

(equal "FOO" "foo") ;; => NIL

equalp is case-insensitive for strings and for numerical value of numbers.

Two objects are equalp if they are equal; if they are characters and satisfy char-equal, which ignores alphabetic case and certain other attributes of characters; if they are numbers and have the same numerical value, even if they are of different types; or if they have components that are all equalp.

Continuing with our ordering, we could say that eq < eql < equal < equalp.

We can compare two numbers, looking at their value, even if they have different types:

(equalp 3 3.0) ;; => T

Now look at our string comparison:

(equalp "FOO" "foo") ;; => T

equalp is case *in*sensitive for strings because a string is a sequence of characters, equalp compares all of its components and it uses char-equal for characters, which ignores the characters' case.

Other comparison functions

null

The function null returns true if its one argument is NIL.

eql is used by default by many CL built-ins

This is a common issue for newcomers who manipulate strings. Sometimes, you use a CL built-in function and you are puzzled why you get no result.

Look at this:

(find "foo" (list "test" "foo" "bar"))
;; NIL

we want to know if the string "foo" exists in the given list. We get NIL. What's happening?

This CL built-in function, as all that work for sequences, use eql for testing each elements. But (eql "foo" "foo") won't work for strings. We need to use another test function.

All of those functions accept a :test keyword parameter, that allows you to change the test function:

(find "foo" (list "test" "foo" "bar") :test #'equal)
;; => "foo"

We can also use equalp to ignore the string case:

(find "FOO" (list "test" "foo" "bar") :test #'equalp)
;; => "foo"

You will find more examples about those built-in functions in data-structures.

char-equal

We have a special operator to compare characters:

char-equal ignores alphabetic case and certain other attributes of characters

strings and string-equal

string-equal has a specific function signature to compare strings and substrings (you can specify the start and end boundaries for the comparison), but be aware that it uses char-equal, so the comparison is case-*in*sensitive. And it works with symbols.

(string-equal :foo "foo") ;; => T
(string-equal :foo "FOO") ;; => T

This is its docstring:

STRING-EQUAL

This is a function in package COMMON-LISP

Signature
(string1 string2 &key (start1 0) end1 (start2 0) end2)

Given two strings (string1 and string2), and optional integers start1,
start2, end1 and end2, compares characters in string1 to characters in
string2 (using char-equal).

See also our page strings.html.

Compare trees with tree-equal

Here you have it:

tree-equal returns T if X and Y are isomorphic trees with identical leaves

Compare function table: to compare against (this), use (that) function

To compare against...      Use...

Objects/Structs            EQ

NIL                        EQ (but the function NULL is more concise and probably cheaper)

T                          EQ (or just the value but then you don't care for the type)

Precise numbers            EQL

Floats                     =

Characters                 EQL or CHAR-EQUAL

Lists, Conses, Sequences   EQ (if you want the exact same object)
                           EQUAL (if you just care about elements)

Strings                    EQUAL (case-sensitive), EQUALP (case-insensitive)
                           STRING-EQUAL (if you throw symbols into the mix)

Trees (lists of lists)     TREE-EQUAL (with appropriate :TEST argument)

How to compare your own objects AKA built-in functions are not object-oriented

Use eq to check that two objects are identical, that they are the same object in memory

If you want to compare your own objects with a logic of your own (for example, two "person" objects will be considered equal if they have the same name and surname), you can't specialize a built-in function for this. Use your own person= or similar function, or use a library (see our links below).

While this can be seen as a limitation, not using generic functions has the advantage of being (much) faster.

As an example, let's consider the person class from the CLOS tutorial:

(defclass person ()
  ((name
    :initarg :name
    :accessor name)))

Let's create two person objects, they have the same name but are two different objects:

(defparameter *p1* (make-instance 'person :name "me"))
(defparameter *p2-same-name* (make-instance 'person :name "me"))

Use eq to compare two objects:

(eq *p1* *p1*) ;; => T
(eq *p1* *p2-same-name*) ;; => NIL

We use our own person= method to compare different objects and decide when they are equal:

(defmethod person= (p1 p2)
  (string= (name p1) (name p2)))

(person= *p1* *p2-same-name*)  ;; => T

If you really want to use = or equal, use a library, see below.

Credits

• CLtL2: Equality Predicates
• the compare table: Leslie P. Polzer on Stack-Overflow

See also

• equals - generic equality for Common Lisp.
• generic-cl - a generic function interface to CL built-ins.
  • we can use = or < on our own custom objects.

23 Aug 2024 10:53am GMT

Common Lisp's pathname system has many problems. Here is proposal to make the situation a little better in one respect. This is not a general fix: it's just trying to solve one problem.

The problem

The underlying problem is that on many platforms pathnames which 'look like' they contain wildcards are perfectly legal pathnames to the filesystem. So, on Unix & related systems [foo].* is a legal filename. On these platforms wildcard handling is generally implemented in a library, or often in multiple semi-compatible libraries¹.

CL then has two problems:

1. there is no portable way to construct pathnames which look wild but are not;
2. there is no portable way to parse a string which looks like a wild pathname but in fact should not be interpreted as one, for instance a string coming from some other application or library, or a filename stored in some file, such as an archive.

(1) happens because 19.2.2.3 says, in part

When examining wildcard components of a wildcard pathname, conforming programs must be prepared to encounter any of the following additional values in any component or any element of a list that is the directory component: […] A string containing implementation-dependent special wildcard characters. […]

That means that implementations are allowed to represent wildcard components of pathnames as strings, and that means that you can't portably construct a non-wildcard pathname.

(2) happens because there's no way to tell parse-namestring or pathname that the string you've handed to them is not wild, even though it looks like it is. That in turn means that to deal with this case you need to either write or find a pathname-parsing library which doesn't have this problem.

These problems arise in practice: for instance some programs create filenames which look like [foo].xml: SBCL at least parses strings like this as wild, as it is allowed to do. This then breaks programs which want to, for instance, process zip files, tar files or other archive formats.

A proposed solution

For (1) change 19.2.2.3 to say that wildcard components are never strings. Change the description of make-pathname to say that if the corresponding components to it are strings (or suitably-constrained lists for the directory component) then the pathname is not wild, except if the default provides a component which is wild.

For (2) add an extra argument to both parse-namestring and pathname named wild with a default of true. If given as nil this will force string parsing to construct a non-wild pathname. If that is not possible, such as when pathname is handed a pathname which is already wild, then an error will be signalled.

Notes

This is the smallest change I can think of which will solve the problem. Some implementations, SBCL for instance, already solve (1) in the suggested way. None, I think, solve (2).

For added value, it might be useful to specify that wildcard components can be given either as symbols or as lists whose first element is a symbol, and encourage implementations to return them as such if possible. So, for instance (:sequence "foo-" (:alternation "bar" "zap")) might represent a wild name which matches "foo-bar" and "foo-zap". I am not suggesting this particular notation however.

18 Aug 2024 5:00pm GMT

Like many programmers, I have programming hobbies. One of these is implementing new languages. My most recent language project, l1, was a Lisp dialect whose primary data types are symbols and arbitrarily-large integers.

I've been happy with l1, but it is interpreted; since I was actively working on it last (2022), I've been wondering about the best way to generate compiled standalone executable programs, written in l1 or any other language.

$ cat three.c
int x = 3;

int main() {
  return x;
}
$ clang three.c -o three
$ ./three; echo $?
3

$ clang -O3 -S three.c -o three.s
$ cat -n three.s
     1                .section        __TEXT,__text,regular,pure_instructions
     2                .build_version macos, 14, 0     sdk_version 14, 4
     3                .globl  _main                           ; -- Begin function main
     4                .p2align        2
     5        _main:                                  ; @main
     6                .cfi_startproc
     7        ; %bb.0:
     8        Lloh0:
     9                adrp    x8, _x@PAGE
    10        Lloh1:
    11                ldr     w0, [x8, _x@PAGEOFF]
    12                ret
    13                .loh AdrpLdr    Lloh0, Lloh1
    14                .cfi_endproc
    15                                                ; -- End function
    16                .section        __DATA,__data
    17                .globl  _x                              ; @x
    18                .p2align        2, 0x0
    19        _x:
    20                .long   3                               ; 0x3
    21
    22        .subsections_via_symbols

$ clang -S -emit-llvm three.c -o three.ll
$ cat -n three.ll
     1        ; ModuleID = 'three.c'
     2        source_filename = "three.c"
     3        target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
     4        target triple = "arm64-apple-macosx14.0.0"
     5
     6        @x = global i32 3, align 4
     7
     8        ; Function Attrs: noinline nounwind optnone ssp uwtable(sync)
     9        define i32 @main() #0 {
    10          %1 = alloca i32, align 4
    11          store i32 0, ptr %1, align 4
    12          %2 = load i32, ptr @x, align 4
    13          ret i32 %2
    14        }
    15
    16        attributes #0 = { noinline nounwind optnone ssp ;; .... very long list...
    17
    18        !llvm.module.flags = !{!0, !1, !2, !3, !4}
    19        !llvm.ident = !{!5}
    20
    21        !0 = !{i32 2, !"SDK Version", [2 x i32] [i32 14, i32 4]}
    22        !1 = !{i32 1, !"wchar_size", i32 4}
    23        !2 = !{i32 8, !"PIC Level", i32 2}
    24        !3 = !{i32 7, !"uwtable", i32 1}
    25        !4 = !{i32 7, !"frame-pointer", i32 1}
    26        !5 = !{!"Apple clang version 15.0.0 (clang-1500.3.9.4)"}

$ clang -O3 three.ll -o three
$ ./three; echo $?
3

$ cat 3.ll
target triple = "x86_64-apple-macosx14.0.0"

@x = global i32 3, align 4

define i32 @main() {
  %1 = load i32, ptr @x, align 4
  ret i32 %1
}
$ clang -O3 3.ll -o 3
$ ./3; echo $?
3

(def m1-target "arm64-apple-macosx14.0.0")

(defn target [t] (format "target triple = \"%s\"" t))

(defn arithm [op typ a b]
  (format "%s %s %s, %s"
          (name? op)
          (name? typ)
          (sigil a)
          (sigil b)))

(defn add [typ a b] (arithm :add typ a b))
(defn sub [typ a b] (arithm :sub typ a b))
(defn mul [typ a b] (arithm :mul typ a b))
(defn div [typ a b] (arithm :sdiv typ a b))

(comment
  (div :i32 :a :b)
  ;;=>
  "sdiv i32 %a, %b"

  (add :i8 :x 1)
  ;;=>
  "add i8 %x, 1")

(module
 (assign-global :i :i32 3)
 (def-fn :i32 :main []
   (assign :retval (load :i32 :i))
   (ret :i32 :retval)))

target triple = "arm64-apple-macosx14.0.0"

@i = global i32 3

define i32 @main() nounwind {
  %retval = load i32, i32* @i, align 4
  ret i32 %retval
}

(require '[babashka.process :as sh])

(defn sh
  "
  Use `bash` to run command(s) `s`, capturing both stdout/stderr
  as a concatenated string.  Throw an exception if the exit code
  is nonzero.
  "
  [s]
  (let [{:keys [out err]}
        (sh/shell {:out :string, :err :string}
                  (format "bash -c '%s'" s))]
    (str/join "\n" (remove empty? [out err]))))

(require '[babashka.fs :as fs])

(defn compile-to
  "
  Save IR `body` to a temporary file and compile it, writing the
  resulting binary to `progname` in the current working directory.
  "
  [progname body]
  (let [ll-file
        (str (fs/create-temp-file {:prefix "llbb-", :suffix ".ll"}))]
    (spit ll-file body)
    (sh (format "clang -O3 %s -o %s"
                ll-file
                progname))))

(compile-to "smallest-prog"
            (module
             (def-fn :i32 :main []
               (ret :i32 0))))

(sh "./smallest-prog; echo -n $?")
;;=>
"0"

;; Argument counting: return number of arguments as an exit code:
(compile-to "argcount"
            (module
             (def-fn :i32 :main [[:i32 :arg0]
                                 [:ptr :arg1_unused]]
               (assign :retptr (alloca :i32))
               (store :i32 :arg0 :ptr :retptr)
               (assign :retval (load :i32 :retptr))
               (ret :i32 :retval))))

(sh "./argcount; echo -n $?") ;;=> "1"
(sh "./argcount 1 2 3; echo -n $?") ;;=> "4"

(let [msg "Hello, World."
      n (inc (count msg))] ;; Includes string terminator
  (compile-to "hello"
              (module
               (external-fn :i32 :puts :i8*)
               (def-global-const-str :message msg)
               (def-fn :i32 :main []
                 (assign :as_ptr
                         (gep (fixedarray n :i8)
                              (star (fixedarray n :i8))
                              (sigil :message)
                              [:i64 0]
                              [:i64 0]))
                 (call :i32 :puts [:i8* :as_ptr])
                 (ret :i32 0)))))

(sh "./hello") ;;=> "Hello, World.\n"

(sh "ls -l hello")
;;=>
"-rwxr-xr-x  1 jacobsen  staff  33432 Aug 14 21:09 hello\n"

(def example "

2 2 +  \\ 4
5 *    \\ multiply by five to get 20
2 /    \\ divide by 2 -> 10
-1 +   \\ add -1 -> 9
8 -    \\ subtract 8 -> 1
.      \\ prints '1'

")

(defn strip-comments
  "
  Remove parts of lines beginning with backslash
  "
  [s]
  (str/replace s #"(?sm)^(.*?)\\.*?$" "$1"))

(defn tokenize
  "
  Split `s` on any kind of whitespace
  "
  [s]
  (remove empty? (str/split s #"\s+")))

(defrecord node
    [typ val] ;; A node has a type and a value
  Object
  (toString [this]
    (format "[%s %s]" (:typ this) (:val this))))

;; Allowed operations
(def opmap {"+" :add
            "-" :sub
            "/" :div
            "*" :mul
            "." :dot
            "drop" :drop})

(defn ast
  "
  Convert a list of tokens into an \"abstract syntax tree\",
  which in our Forth is just a list of type/value pairs.
  "
  [tokens]
  (for [t tokens
        :let [op (get opmap t)]]
    (cond
      ;; Integers (possibly negative)
      (re-matches #"^\-?\d+$" t)
      (node. :num (Integer. t))

      ;; Operations
      op (node. :op op)

      :else (node. :invalid :invalid))))

(->> example
     strip-comments
     tokenize
     ast
     (map str))

;;=>
("[:num 2]"
 "[:num 2]"
 "[:op :add]"
 "[:num 5]"
 "[:op :mul]"
 "[:num 2]"
 "[:op :div]"
 "[:num -1]"
 "[:op :add]"
 "[:num 8]"
 "[:op :sub]"
 "[:op :dot]")

(defn def-arithmetic-op [nam op-fn]
  (def-fn :void nam []
    (assign :sp (call :i32 :get_stack_cnt))
    (if-lt :i32 :sp 2
           (els)  ;; NOP - not enough on stack
           (els
            (assign :value2
                    (call :i32 :pop))
            (assign :value1
                    (call :i32 :pop))
            (assign :result (op-fn :i32 :value1 :value2))
            (call :void :push [:i32 :result])))
    (ret :void)))

;; ...
(def-arithmetic-op :mul mul)
(def-arithmetic-op :add add)
(def-arithmetic-op :sub sub)
(def-arithmetic-op :div div)
;; ...

$ cat example.fs

       \\ initial state          stack: []
3      \\ put 3 on stack.        stack: [3]
99     \\ put 99 on stack.       stack: [3, 99]
drop   \\ discard top item.      stack: [3]
drop   \\ discard top item.      stack: []
2 2    \\ put 2 on stack, twice: stack: [2, 2]
+      \\ 2 + 2 = 4.             stack: [4]
5 *    \\ multiply 4 * 5.        stack: [20]
2 /    \\ divide by 2.           stack: [10]
-1 +   \\ add -1                 stack: [9]
8 -    \\ subtract 8 -> 1        stack: [1]
.      \\ prints '1'             stack: [1]
drop   \\ removes 1.             stack: []

$ ./forth.bb example.fs
$ ./example
1

$ time ./example
1

real  0m0.007s
user  0m0.002s
sys   0m0.003s

$ ls -l ./example
-rwxr-xr-x  1 jacobsen  staff  8952 Aug 16 09:52 ./example

2 2 +
5 *
2 /
-1 +
8 -
.

$ cat example.lisp
(print (- (+ -1
             (/ (* 5
                   (+ 2 2))
                2))
          8))

(defn to-ssa [expr bindings]
  (if (not (coll? expr))
    expr
    (let [[op & args] expr
          result (gensym "r")
          args (doall
                (for [arg args]
                  (if-not (coll? arg)
                    arg
                    (to-ssa arg bindings))))]
      (swap! bindings conj (concat [result op] args))
      result)))

(defn convert-to-ssa [expr]
  (let [bindings (atom [])]
    (to-ssa expr bindings)
    @bindings))

(->> "example.lisp"
     slurp
     (edn/read-string)
     convert-to-ssa)
;;=>
[(r623 + 2 2)
 (r622 * 5 r623)
 (r621 / r622 2)
 (r620 + -1 r621)
 (r619 - r620 8)
 (r618 print r619)]

(def ops
  {'* #(mul :i32 %1 %2)
   '+ #(add :i32 %1 %2)
   '/ #(div :i32 %1 %2)
   '- #(sub :i32 %1 %2)
   'print
   #(call "i32 (i8*, ...)"
          :printf
          [:i8* :as_ptr]
          [:i32 (sigil %1)])})

(defn main [[path]]
  (when path
    (let [assignments (->> path
                           slurp
                           edn/read-string
                           convert-to-ssa)
          outfile (->> path
                       fs/file-name
                       fs/split-ext
                       first)
          ir (module
              (external-fn :i32 :printf :i8*, :...)
              (def-global-const-str :fmt_str "%d\n")
              (def-fn :i32 :main []
                (assign :as_ptr
                        (gep (fixedarray 4 :i8)
                             (star (fixedarray 4 :i8))
                             (sigil :fmt_str)
                             [:i64 0]
                             [:i64 0]))

                ;; Interpolate SSA instructions / operator invocations:
                (apply els
                       (for [[reg op & args] assignments
                             :let [op-fn (ops op)]]
                         (if-not op-fn
                           (throw (ex-info "bad operator" {:op op}))
                           (assign reg (apply op-fn args)))))
                (ret :i32 0)))]
      (compile-to outfile ir))))

(main *command-line-args*)

$ ./lisp.bb example.lisp
$ ./example
1

$ time ./example
1

real  0m0.006s
user  0m0.001s
sys   0m0.003s
$ ls -al ./example
-rwxr-xr-x  1 jacobsen  staff  33432 Aug 16 20:52 ./example

06 Aug 2024 12:00am GMT

In the heat of the summer (the coolest summer of the next ones), it is never a good thing to get an email from Xach telling you that "something does not compile on SBCL". In this case the issue was the usual, fascist STYLE-WARNING, that prevented a clean build on Quicklisp.

The fix was relatively easy, but it lead to a number of extra changes to properly test the fix itself.

Bottom line, a new version of HEΛP is available at helambdap.sf.net. Soon in Quicklisp as well.

Stay cool, hydrated and enjoy.

(cheers)

04 Aug 2024 11:01am GMT

Most uses of shadow and shadowing-import in Common Lisp packages point to design problems.

Let's assume you are designing a language which is going to be a variant CL: most of it will be just CL, but perhaps some things will be different. For example, let's imagine that you want if to have a mandatory else clause. You might start by designing your package like this:

(defpackage :my-language
  (:use :cl)
  (:shadow #:if)
  (:export #:if))

(in-package :my-language)

...

(defmacro if (test then else)
  `(cl:if ,test ,then ,else))

...

That all seems fine, right? Well, not so much. Consider for a minute people who want to use your language. They need to write something like this:

(defpackage :my-language-user-package
  (:use :cl :my-language)
  (:shadowing-import-from :my-language #:if))

(in-package :my-language-user-package)

...

'Oh well', you say, 'that's not so bad'. Well, now let's say you want to add a version of cond to your language which understands else and otherwise. So:

(defpackage :my-language
  (:use :cl)
  (:shadow #:if #:cond)
  (:export #:if #:cond #:else))

(in-package :my-language)

...

(defmacro if (test then else)
  `(cl:if test then else))

(defmacro cond (&body clauses)
  `(cl:cond
    ,@(mapcar (lambda (clause)
                (if (and (consp clause)
                         (member (first clause) '(else otherwise)))
                    `(t ,@(rest clause))
                  clause))
              clauses)))

...

And now every user of your language has to modify their package definitions:

(defpackage :my-language-user-package
  (:use :cl :my-language)
  (:shadowing-import-from :my-language #:if #:cond))

(in-package :my-language-user-package)

...

I'll say that again: every user of your language has to modify their package definitions every time you enhance it in a way which is not compatible with CL.

That … sucks. It's an absolutely terrible design. Wouldn't it be nice if it could be avoided?

It can. Rather than shadowing symbols, you can instead construct the packages you actually would like to exist. In the example above what you probably want people to be able to do is to say

(defpackage :my-language-user-package
  (:use :my-language))

and have that work, even when your language changes. So, you need the MY-LANGUAGE package to export most of the symbols from CL as well as a few of its own. You can do this by hand:

(defpackage :my-language
  (:use)
  (:export #:if #:cond #:else)
  (:import-from :cl
   cl:&allow-other-keys ...)
  (:export
   cl:&allow-other-keys ...))

Where the :import-from and the second :export clause specify all the symbols from CL except those which are replaced by ones defined by your language.

Note the empty :use clause: this avoids symbol clashes and therefore the need to shadow things.

You can then either define your language in this package or in an implementation package which uses it: the package has imported all of the external symbols from CL other than the ones it overrides, so it doesn't need to use the CL package at all.

The benefit of doing things this way is that it means that every user of this system doesn't have to care about the details of it and isn't forced to change their code because of implementation changes. That's worth it, even though writing the defpackage forms is laborious: you should do the work, not every user of your systm.

Of course, in real life you would not have to remember the names of all the symbols you are reexporting: you'd write a program to do it for you. You'd write, in fact, a macro.

Well other people have already done that for you, in particular I did this in 1998 when I decided that this idea was interesting. Other people have since done similar things I think and may have done so before me, but I will describe my version: conduit packages. In particular I'll mostly describe the functionality exported from the ORG.TFEB.CONDUIT-PACKAGES/DEFINE-PACKAGE package, which doesn't replace macros like defpackage and functions like export, but rather provides functionality under different names.

The basic notion is that packages can be conduits for one or more other packages: they serve to gather together and reexport subsets of the exported names from the packages for which they are conduits. define-package lets you define conduit packages easily, and define-conduit-package is even more specialised to the task.

Here is how you would define the package above

(define-package :my-language
  (:use)
  (:export #:if #:cond #:else)
  (:extends/excluding :cl
   #:if #:cond))

or with define-conduit-package:

(define-conduit-package :my-language
  (:export #:if #:cond #:else)
  (:extends/excluding :cl
   #:if #:cond))

Now you can quite happily define your language as before.

This works, of course, even if your package wants to extend other packages whose exports might change in a way that CL's are unlikely to do any time soon: the symbols to import & reexport are computed based on the state of the package system at the time the form is evaluated. In some cases - if the package you are extending is itself known to the system - the packages will be dynamically recomputed:

> (define-package :foo
    (:export #:one))
#<The FOO package, 0/16 internal, 1/16 external>

> (define-conduit-package :bar
    (:extends :foo))
#<The BAR package, 0/16 internal, 1/16 external>

> (do-external-symbols (s :bar (values)) (print (symbol-name s)))

"ONE"

> (define-package :foo
    (:export #:one #:foo))
Warning: Using DEFPACKAGE to modify #<The FOO package, 0/16 internal, 1/16 external>.
#<The FOO package, 0/16 internal, 2/16 external>

> (do-external-symbols (s :bar (values)) (print (symbol-name s)))

"FOO"
"ONE"

And thus was the abominable shadow cast into the outer darkness.

A remaining question is: are there good uses for shadowing? Well, conduit packages itself uses them in its implementation package, mostly because I was too lazy to write the code which would explicitly map over CL. And there must, I suppose, be other good uses, but it's very hard to think of them. The other common case, where you want to use two packages which export the same names, is dealt with by simply using a conduit of course.

I think it's worth remembering that when the CL package system was initially defined, people didn't really understand how such a thing should work. MACLISP didn't have a package system, Lisp Machine Lisp probably did (certainly Zetalisp did), but there was no great experience with what a package system should be like. Indeed the first CL version didn't have defpackage: instead you had to construct packages by hand, and there were all sorts of weirdnesses in the way the compiler handled make-package and other package functions (or you had to use eval-when all over the place).

Finally, when I wrote conduit packages I was still thinking that packages were big expensive objects, because in the late 1980s they were, and I hadn't yet realised that this was no longer true. In the late 1980s a big workstation on which you ran CL might have had 16MB of memory. Today laptops have perhaps a thousand times as much memory: data structures which ate a lot of precious memory in 1990 don't any more. So I think, today, it's appropriate to use packages in a fairly fine-grained way: having a few extra packages really is not hurting you very much.

So here is another way to define the little language above.

First, define a conduit for CL which exports just the symbols you want:

(define-conduit-package :my-language/cl
  (:extends/excluding :cl
   #:if #:cond))

Now define the implementation package for the language: this exports the new symbols:

(define-package :my-language/impl
  (:use :my-language/cl)
  (:export
   #:if #:cond #:else))

Now, finally, define the public package, which is a conduit for both MY-LANGUAGE/CL and MY-LANGUAGE/IMPL:

(define-conduit-package :my-language
  (:extends :my-language/cl :my-language/impl))

This is absurd overkill in this tiny example, but for real examples, where there might be several implementation packages, it lets you split things up in a nice way, while not burdening your users with lots of tiny packages.

01 Aug 2024 10:55am GMT

One good use of continuation passing style is to manage dynamic resources. The resource allocation function is written in continuation passing style and it takes a callback that it invokes once it has allocated and initialized the resource. When the callback exits, the resource is uninitialized and deallocated.

(defun call-with-resource (receiver)
  (let ((resource nil))
    (unwind-protect
        (progn
          (setq resource (allocate-resource))
          (funcall receiver resource))
      (when resource
        (deallocate-resource resource)))))

  ;; example usage:
  (call-with-resource
    (lambda (res)
      (do-something-with res)))

;;; In Lisp, we would provide a convenient WITH- macro
(defmacro with-resource ((resource) &body body)
  '(CALL-WITH-RESOURCE (LAMBDA (,resource) ,@body)))

  ;; example usage:
  (with-resource (res)
    (do-something-with res))

This pattern of usage separates and abstracts the resource usage from the resource management. Notice how the unwind-protect is hidden inside call-with-resource so that the user of the resource doesn't have to remember to deallocate the resource.

The with-resource macro is idiomatic to Lisp. You obviously can't provide such a macro in a language without macros, but you can still provide the call-with-resource function.

Continuation passing style for resource management can be used in other languages, but it often requires some hairier syntax. Because call-with-resource takes a callback argument, it is actually a higher-order function. The syntax for passing higher-order functions in many languages is quite cumbersome. The return value of the callback becomes the return value of call-with-resource, so the return type of the callback must be compatible with the return type of the function. (Hence the type of call-with-resource is actually parameterized on the return value of the callback.) Languages without sophisticated type inference may balk at this.

Another advantage of the functional call-with-resource pattern is that you can dynamically select the resource allocator. Here is an example. I want to resolve git hashes against a git repository. The git repository is large, so I don't want to clone it unless I have to. So I write two resource allocators: CallCloningGitRepository, which takes the URL of the repository to clone, and CallOpeningGitRepository which takes the pathname of an already cloned repository. The "cloning" allocator will clone the repository to a temporary directory and delete the repository when it is done. The "opening" allocator will open the repository and close it when it is done. The callback that will be invoked won't care which allocator was used.

Here is what this looks like in golang:

// Invoke receiver with a temporary directory, removing the directory when receiver returns.
func CallWithTemporaryDirectory(dir string, pattern string, receiver func(dir string) any) any {
        dir, err := os.MkdirTemp(dir, pattern)
        CheckErr(err)
        defer os.RemoveAll(dir)

        return receiver(dir)
}

// Invoke receiver with open git repository.
func CallOpeningGitRepository(repodir string, receiver func(string, *git.Repository) any) any {
        repo, err := git.PlainOpen(repodir)
        if err != nil {
                log.Fatal(err)
        }
        return receiver(repodir, repo)
}

// Invoke receiver with a cloned git repository, removing the repository when receiver returns.
func CallCloningGitRepository(dir string, pattern string, url string, receiver func(tmpdir string, repo *git.Repository) any) any {
        if url == "" {
                return nil
        }
        return CallWithTemporaryDirectory(
                dir,
                pattern,
                func(tempdir string) any {
                        log.Print("Cloning " + url + " into " + tempdir)
                        repo, err := git.PlainClone(tempdir, true, &git.CloneOptions{
                                Auth: &gitHttp.BasicAuth{
                                        Username: username,
                                        Password: password,
                                },
                                URL:      url,
                                Progress: os.Stdout,
                        })
                        CheckErr(err)
                        log.Print("Cloned.")
                        return receiver(tempdir, repo)
                })
}

You specify a repository either with a URL or a pathname. We select the appropriate resource allocator based on whether the specifier begins with "https".

func RepositoryGetter (specifier string) func (receiver func(_ string, repo *git.Repository) any) any {
        if strings.EqualFold(specifier[0:5], "https") {
                return GetRemoteGitRepository (specifier)
        } else {
                return GetLocalGitRepository (specifier)
        }
}

func GetRemoteGitRepository(url string) func(receiver func(_ string, repo *git.Repository) any) any {
        return func(receiver func(_ string, repo *git.Repository) any) any {
                return CallCloningGitRepository("", "git", url, receiver)
        }
}

func GetLocalGitRepository(repodir string) func(receiver func(_ string, repo *git.Repository) any) any {
        return func(receiver func(_ string, repo *git.Repository) any) any {
                return CallOpeningGitRepository(repodir, receiver)
        }
}

To open a repository, we call RepositoryGetter(specifier) to get a getRepository function. Then we invoke the computed getRepository function on a receiver callback that accepts the local pathname and the repository:

  getRepository := RepositoryGetter(specifier)

  return getRepository(
         func (_ string, repo *git.Repository) any {
                 // resolve git hashes against the repository
                 ....
                 return nil
         })

If given a URL, this code will clone the repo into a temporary directory and open the cloned repo. If given a pathname, it will just open the repo at the pathname. It runs the callback and does the necessary cleanup when the callback returns.

The biggest point of confusion in this code (at least to me) are the type specifiers of the functions that manipulate the resource allocators. Static types don't seem to mix well with continuation passing style.

31 Jul 2024 6:05pm GMT