25 May 2026

feedPlanet Lisp

Joe Marshall: CLRHack: Tail Recursion

Tail-Call Handling in CLRHack

I decided to make proper tail recursion a fundamental requirement in CLRHack. This prevents stack overflow errors during standard recursive patterns and ensures the runtime remains stable regardless of recursion depth. Technically, Common Lisp isn't required to be tail recursive, but I want mine to be.

1. Tail Position Identification

The compiler performs a structural analysis of the Abstract Syntax Tree (AST) to identify "tail positions." An expression is in a tail position if its value is the final result of the function, meaning no further work remains to be done in the current frame after the call returns. The generate-step2 walker propagates a tail-p flag through the following logic:

2. CIL Instruction Emission

To implement proper tail-call semantics, the compiler utilizes the native tail. prefix in the Common Intermediate Language (CIL). When a function call is detected in a tail position, the compiler applies the following mandatory transformation:

  1. The Prefix: It prepends the tail. opcode to the call or callvirt instruction.
  2. The Return: It immediately follows the call with a ret (return) instruction.

The tail. prefix instructs the .NET Just-In-Time (JIT) compiler to discard the current method's stack frame before jumping to the target function. This ensures that the call consumes zero additional stack space, turning the recursive call into a semantic jump.

3. Safety and Context Constraints

The implementation of tail-calls is subject to specific safety rules imposed by the Common Language Runtime (CLR) to maintain execution integrity:

Example CIL Output

Consider a recursive counter that must be able to run indefinitely:

  (defun count-down (n)
    (if (= n 0)
        "Done"
        (count-down (- n 1))))
  

The compiled CIL for the recursive branch is transformed to ensure stack neutrality:

      ; ... code to calculate (- n 1) ...
      tail.
      call object Program::'COUNT-DOWN'(object)
      ret
  

By strictly enforcing this pattern, CLRHack guarantees that recursive programs can execute with constant stack space, fulfilling my core requirement of tail recursion.

25 May 2026 7:00am GMT

24 May 2026

feedPlanet Lisp

Marco Antoniotti: Getting HEΛP, Finally!

As I wrote in my last blog entry, I went back hacking on HEΛP.

HEΛP is the Common Lisp code documentation tool I started writing many years ago.

Apart from a little necessary Javascript and CSS, HEΛP is a full Common Lisp program, geared towards producing static documentation sites for CL code. I finally got around to modernize it and it is now ready for testing.

Evolution from (X)HTML to HTML5

The original HEΛP release was producing only (X)HTML output, moreover based on FRAMESETs.

Alas, when the first HEΛP release was made, FRAMESETs were falling out of fashion, and they were eventually deprecated with the advent of HTML5. An "upgrade" to HTML5 became then a necessity.

After a very long process, I finally finished the HTML5 port, plus some bells and whistles. All in all, the implementation uses <div ... > sections plus CSS to lay out the display, as I understand it is the proper coding fashion nowadays. The port uses the W3.CSS styles, which facilitated a number of choices. The result is rather pleasing, as far as I am concerned.

Example: Producing the HEΛP documentation

The HEΛP documentation (a form of it) is produced with the following command (hlp is one of the package nicknames):

(hlp:document #P"./" ; Just a "top directory"...
              :documentation-title "HE&Lambda;P"
              :format :html5
              :exclude-directories
              (list "doc/"
                    "js/"
                    "css/"
                    ".git/"
                    "tests/"
                    "tmp/"
                    "tools/"
                    )

              :exclude-files ; I run this from LW.
              (list "impl-dependent/ccl.lisp"
                    "impl-dependent/sbcl.lisp"
                    "utilities/document-helambdap.lisp"
                    "utilities/lambda-list-parsing.lisp"
                    )
              :only-documented t
              :only-exported t
              )

After much printing, the resulting static web pages are deposited in docs/html5/, unless overridden. The system also relies on some defaults which are handled by CLAD library.

Viewing the result.
For the time being, you can find the main page of HEΛP here. Navigating the bar on the left will allow you to see different bits and pieces fo the documentation. You will notice that you have different views of the documentation: a system view and a package view. The system view gives you also a view of the files and folders (modules) it contains.

Documentation strings are mostly left alone.
Unike for Emacs Lisp, there is no real agreement in the CL community about how to format documentation strings (if there is, I do not agree with it by definition - obviously). HEΛP wants to be able to document code that does not adopt any documentation string convention, therefore it treats documentation strings pretty much as they are, only adding some text in the guise of the Hyperspec entries.

Screenshot

Here are a couple of screenshots. Apologies for the bad resolution.

The Main View

The HEΛP Main View

The DIctionary View

The HEΛP Dictionary View

Missing Pieces

There is one thing that is missing from HEΛP: the generation of proper crossreferencing. To do it correctly it will be necessary to somehow make some educated guesses about the content of documentation strings or agreeing on some markup to tag linkable items. Apart from that, at the time of this writing the doc strings are handled as enties in an has table, and that could be improved, as more indexes may be needed.

Of course, a major rewrite may also help, but time is a tyrant.

Any suggestion is welcome.

Try it!

Again, HEΛP is ready for you to try. You can clone the repository from Sourceforge.

... and remember: no Python or Ruby or Shell dependencies: pure CL (plus some Javascript, which is a functional language after all, whose first implementation was done in CL).

Enjoy!


'(cheers)

24 May 2026 8:15am GMT

Joe Marshall: CLRHack Lexical Variables

Lexical Closures in CLRHack

CLRHack implements lexical closures by transforming dynamic Lisp environments into static CIL class structures. Since the .NET Common Language Runtime (CLR) does not have a native concept of "nesting" functions within the lexical scope of another function's local variables, the compiler employs Lambda Lifting and Explicit Closure Conversion.

1. Lambda Lifting

Every lambda expression (including those generated by flet and labels) is extracted from its nesting site. The compiler generates a unique, standalone CIL class for each lambda. These classes inherit from the base [LispBase]Lisp.Closure class.

2. The Closure Class Structure

The generated class acts as a container for both the code (the lambda body) and the environment (the captured variables). It consists of:

3. Environment Capture

At the point in the code where the lambda is defined, the compiler emits a newobj instruction. It passes the current values of the required local variables into the closure's constructor. This "closes" over the variables, creating a persistent instance of the environment that lives on the heap.

  ; Lisp Source
  (let ((factor 10))
    (lambda (x) (* x factor)))

  ; Conceptual CIL Transformation
  .class private Lambda_1 extends [LispBase]Lisp.Closure {
      .field public object factor_captured

      .method public hidebysig specialname rtspecialname void .ctor(object f) {
          ldarg.0
          ldarg.1
          stfld object Lambda_1::factor_captured
          ret
      }

      .method public virtual object Invoke(object x) {
          ldarg.0
          ldfld object Lambda_1::factor_captured
          ldarg.1
          ; ... multiplication logic ...
      }
  }
  

4. Shared Mutability via ValueCells

Common Lisp requires that if an outer variable is mutated (via setq), all closures capturing that variable must see the change. To support this, CLRHack uses Indirection Cells:

5. Invocation

When a closure is invoked (e.g., via funcall), the Invoke method is called. Inside this method, the this pointer (ldarg.0 in CIL) provides the code with access to the captured environment fields. This allows the lifted function to behave as if it were still sitting inside its original lexical scope.

24 May 2026 7:00am GMT

23 May 2026

feedPlanet Lisp

Joe Marshall: CLRHack argument passing

Common Lisp Argument Passing in CLRHack

The CLRHack engine translates the dynamic, flexible argument-passing semantics of Common Lisp into the static, strongly-typed environment of the .NET Common Language Runtime (CLR). It achieves this through a combination of CIL method overloading, sentinel-based defaulting, and runtime list construction.

1. The Overloading Architecture

Since CIL methods have a fixed arity (number of arguments), but Common Lisp functions support variable arguments, the compiler generates multiple entry points for every defined function.

2. Parameter Type Implementation

Required Parameters

Required parameters are the simplest. They map directly to the leading object arguments in both the public overloads and the internal body method.

Optional Parameters (&optional)

Handling &optional involves a "Sentinel Pattern":

Rest Parameters (&rest)

Rest parameters are handled by runtime list construction:

Keyword Parameters (&key)

Keyword parameters are implemented as a transformation over the &rest mechanism:

  1. Trailing arguments are gathered into a "rest" list.
  2. The _Body method defines local variables for every keyword parameter, initialized to their Lisp default values.
  3. The Keyword Loop: At the start of the function, the engine emits a loop that scans the rest list in pairs. It uses Object.Equals to compare each key against the pre-interned keyword symbols (e.g., :TEST). When a match is found, the corresponding local variable is updated with the value following the key.

3. The Calling Mechanism

Direct Calls

When the compiler identifies a call to a known defun in the same assembly, it emits a direct call to the public overload that exactly matches the number of provided arguments. This provides near-native performance for fixed-arity calls.

Indirect Calls (Closures & Funcall)

All Lisp functions are instances of the Lisp.Closure class. This class provides virtual Invoke methods. When a closure is created, it captures its environment and provides overrides for these Invoke methods that jump into the appropriate Program static overloads.

4. Multiple Return Values

Note: Argument passing is only half the story; return values use a side-channel.

Because .NET methods return only one value, CLRHack uses a Thread-Static Side-Channel in the Lisp.Values class:

Example CIL Generation

; Lisp: (defun add-optional (x &optional (y 5)) (+ x y))

; Overload for 1 arg
.method public static object 'ADD-OPTIONAL'(object x) {
    ldarg 0
    ldsfld object [LispBase]Lisp.Undefined::Value
    tail. call object Program::'ADD-OPTIONAL_Body'(object, object)
    ret
}

; The Body Method
.method private static object 'ADD-OPTIONAL_Body'(object x, object y) {
    ; Defaulting logic for Y
    ldarg 1
    ldsfld object [LispBase]Lisp.Undefined::Value
    bne.un SKIP_DEFAULT
    ldc.i4 5
    box int32
    starg 1
SKIP_DEFAULT:
    ; ... rest of function ...
}
    

23 May 2026 11:57am GMT

18 May 2026

feedPlanet Lisp

Joe Marshall: CLRHack: FibBenchmark

The first thing to look at is the Fibonacci benchmark. The source code is here:

(in-package "CLRHACK")

(progn
  (defun fib (n)
    (if (

And it compiles to this IL code: (commentary after the code)
.assembly extern mscorlib {}
.assembly extern LispBase {}

.assembly 'FibBenchmark' {}
.module 'FibBenchmark.exe'

.class public auto ansi beforefieldinit Program
       extends [mscorlib]System.Object
{
    .field public static class [LispBase]Lisp.Symbol 'SYM_G545'

.method public static hidebysig specialname rtspecialname void '.cctor'() cil managed
{
    .maxstack 8
    ldsfld class [LispBase]Lisp.Package [LispBase]Lisp.Package::CommonLisp
    ldstr "T"
    callvirt instance class [LispBase]Lisp.Symbol [LispBase]Lisp.Package::'Intern'(string)
    stsfld class [LispBase]Lisp.Symbol Program::'SYM_G545'
    ret
}

.method public static hidebysig object 'FIB'(object) cil managed
{
    .maxstack 8
    .locals (object TEMP_B)
    ldarg 0
    ldc.i4 2
    box int32
    stloc TEMP_B
    unbox.any int32
    ldloc TEMP_B
    unbox.any int32
    clt
    brtrue TRUE543
    ldnull
    br END544
TRUE543:
    nop
    ldsfld class [LispBase]Lisp.Symbol Program::'SYM_G545'
END544:
    nop
    ldnull
    ceq
    brtrue ELSE546
    ldarg 0
    ret
ELSE546:
    nop
    ldarg 0
    ldc.i4 1
    box int32
    stloc TEMP_B
    unbox.any int32
    ldloc TEMP_B
    unbox.any int32
    sub
    box int32
    call object Program::'FIB'(object)
    ldarg 0
    ldc.i4 2
    box int32
    stloc TEMP_B
    unbox.any int32
    ldloc TEMP_B
    unbox.any int32
    sub
    box int32
    call object Program::'FIB'(object)
    stloc TEMP_B
    unbox.any int32
    ldloc TEMP_B
    unbox.any int32
    add
    box int32
    ret
BLOCK_END_FIB_532:
    nop
    ret
}

.method public static hidebysig object 'MAIN'() cil managed
{
    .maxstack 8
    ldstr "Fibonacci of 10:"
    call void [mscorlib]System.Console::'WriteLine'(object)
    ldnull
    pop
    ldc.i4 10
    box int32
    call object Program::'FIB'(object)
    call void [mscorlib]System.Console::'WriteLine'(object)
    ldnull
    ret
BLOCK_END_MAIN_536:
    nop
    ret
}

.method public static hidebysig void 'Main'() cil managed
{
    .entrypoint
    .maxstack 8
    call object Program::'MAIN'()
    pop
    ret
}

} // end of class Program


The first thing the fib program does is compare argument x to the literal number 2. The compiler pushes argument 0 on to the stack, and then the compiler pushes a integer 2 on to the stack and boxes it.
Next, the compiler has to perform the compare. In order to do this it must unbox both arguments. One argument is on top of the stack, so it is put into a local TEMP_B so we can get to the other argument. We unbox it. We then restore TEMP_B to the top of stack and unbox it. Finally we compare the two unboxed values for less than.
This pattern of unboxing a pair of elements from the top of stack by way of a temporary local is repeated several places in the compiled code as FIB rather inefficiently subtracts 1 or 2 from the argument and makes the recursive call.
This example shows that the compiler basically treats everything as a .NET object. It unboxes numbers at the last moment and boxes the results as soon as they are generated. It is not efficient code.

18 May 2026 6:39pm GMT

17 May 2026

feedPlanet Lisp

Joe Marshall: I Wrote a Compiler

I was bored so I wrote a compiler. I'm lazy so I vibe coded it. It compiles Lisp to .NET IL (the byte code that the .NET runtime executes). The IL is then JIT compiled to machine code and executed. You can use the dotnet runtime from Microsoft or the open source mono runtime as the runtime for the compiled code.

The basic idea of the compiler is to map lambda expressions to .NET classes. The lexical variables are stored as fields in the class. The body of the lambda is compiled to a method in the class. We use lambda lifting to flatten any nested lambdas. We use cell conversion to handle mutable variables and we simply copy the values of immutable variables into the lifted lambdas when they are closed over.

Although I `vibe coded` the compiler, I leveraged my experience with writing compilers to break down the problem into passes that were simple enough that `vibe coding` was possible. For instance, in order to implement lambda lifting, I first wrote a pass that determined the free variables of each lambda. That's a pretty simple operation that I could easily `vibe code`. In order to emit the correct IL, I first wrote a pass that segregated the variables into arguments, lexicals, and globals. Again, that's a simple operation that I could easily `vibe code`.

The trickiest part was the code generator. I had decided to implement tail recursion by using the `tail.` prefix in the IL. This is a hint to the JIT compiler that the call is a tail call and that it can optimize it by reusing the current stack frame. However, the JIT compiler is a bit picky about when it will actually perform the tail calls, and the other parts of the code generator kept moving the tail calls around so that they were no longer in tail position. I eventually had to add a pre-pass to the code generator that tracked the continuations in order to ensure that there was enough information later on to enforce tail position on the tail calls.

It... works? It compiles a number of the Gabriel Benchmarks, and some test programs that demonstrate lexical scoping, mutable variables, and tail recursion. It is most definitely a Lisp compiler, but if you look under the hood, well, be forewarned. It isn't pretty.

The compiler itself was vibe coded. The only restriction on the output code was that it had to implement what the input code specified. It did not have to conform to any particular notion of how to implement lisp features on the .NET runtime beyond the requirement that the output was correct. Choices that are typically made by a Lisp architect, such as how to deal with integers, the implementation of the standard library, etc., were all left up to the vibe coding process. I provided a couple of runtime libraries: a cell library for implementing mutable variables, and a List library for implementing singly linked lists. These were written in C#. The vibe coding process was allowed to modify the C# code in these libraries as well and it did so in a couple of places.

I started with one a simple benchmark and got it to compile and run. From there, I added more benchmarks and each time told the compiler to fix any errors that came up. I also added some test programs that were not part of the benchmarks in order to test specific features of the compiler. As I added more and more test programs, the `vibe coding process` added more and more features to the compiler. This ended up producing more and more complex compiler output code.

I'm going to devote a few blog posts to this compiler, so if it isn't up your alley, skip ahead a few posts.

17 May 2026 1:07pm GMT

05 May 2026

feedPlanet Lisp

ECL News: ECL 26.5.5 release

We are announcing a bugfix ECL release that addresses a few issues that has slipped through testing of the recent one.

Addressed issues:

This release is available for download in a form of a source code archive (we do not ship prebuilt binaries):

Happy Hacking,
The ECL Developers

05 May 2026 12:00pm GMT

Gábor Melis: DRef Leaves Home

Version 0.5 of DRef, the definition reifier, is now available. It has moved to its own repository, completing its separation from PAX, where it was originally developed.

This was a long time coming. Twelve years ago today, PAX was born. From the start, PAX used the concept of locatives to refer to definitions without first-class objects. For example, to generate documentation for the *MY-VAR* variable, one could use the VARIABLE locative as in (*MY-VAR* VARIABLE). PAX needed to be able to tell whether such a definition exists, as well as access its docstring and source location.

Over time, this mechanism evolved into a portable, extensible introspection library independent of PAX. I began separating the two projects two years ago and named the new library, though they continued to share a repository. I have now removed the remaining dependencies so that DRef can live on its own.

05 May 2026 12:00am GMT

01 May 2026

feedPlanet Lisp

Joe Marshall: Echoes of the Lisp Listener

The Lisp Machine Listener had an electric close parenthesis. When the user typed a close parenthesis, and this was the close parenthesis that finished the complete form at top level, the form would be sent to the REPL right away with no need to press enter. Here's how to get this behavior with SLY:

(defun my-sly-mrepl-electric-close-paren ()
  "Insert ')' and auto-send ONLY if we are closing a top-level Lisp form."
  (interactive)
  (let ((state (syntax-ppss)))
    (insert ")")
    ;; Safety checks:
    ;; 1. We were at depth 1 (so we are now at depth 0)
    ;; 2. We aren't in a string or comment
    ;; 3. The input actually starts with a paren (it's a form, not a sentence)
    (when (and (= (car state) 1)
               (not (nth 3 state))
               (not (nth 4 state))
               (string-match-p "^\\s-*(" 
                               (buffer-substring-no-properties (sly-mrepl--mark) (point))))
      (sly-mrepl-return))))

Another cool hack is to get the REPL to do double duty as a command line to the LLM chatbot. When you type RET in the REPL, it will check if the input is a complete lisp form. If so, it will send the form to the REPL as normal. If not, it will send the input to the chatbot. Here's how to do this:

(defun my-sly-mrepl-electric-return ()
  "Send to Lisp if it's a form/symbol, or wrap in (chat ...) if it's a sentence."
  (interactive)
  (let* ((beg (marker-position (sly-mrepl--mark)))
         (end (point-max))
         (input (buffer-substring-no-properties beg end))
         (trimmed (string-trim input)))
    (cond
     ;; If it's empty, just do a normal return
     ((string-blank-p trimmed)
      (sly-mrepl-return))
     
     ;; If it starts with a paren, quote, or hash, it's definitely a Lisp form
     ((string-match-p "^\\s-*[(#'\"]" trimmed)
      (sly-mrepl-return))
     
     ;; If it's a single word (no spaces), treat it as a symbol/form (e.g., *package*)
     ((not (string-match-p "\\s-" trimmed))
      (sly-mrepl-return))
     
     ;; Otherwise, it's a sentence. Wrap it and fire.
     (t
      (delete-region beg end)
      (insert (format "(chat %S)" trimmed))
      (sly-mrepl-return)))))

Install as follows:

;; Apply to SLY MREPL with a safety check for the mode map
(with-eval-after-load 'sly-mrepl
  (define-key sly-mrepl-mode-map (kbd "RET") 'my-sly-mrepl-electric-return)
  (define-key sly-mrepl-mode-map (kbd ")") 'my-sly-mrepl-electric-close-paren))

01 May 2026 5:29pm GMT

Tim Bradshaw: Making CLOS slot access less slow

Access to slots in CLOS instances is often very slow. It's probably not possible for it ever to be really fast, but the AMOP MOP does provide a way of making it, at least, less slow.

How slow is it?

Here are some benchmarks for accessing fields in objects of various kinds, using SBCL. All of these tests do something equivalent to

(defclass a ()
  ((i :initform 0 :type fixnum)))

(defclass a/no-fixnum ()
  ((i :initform 0)))

(defmethod svn ((a a) n)
  (declare (type fixnum n)
           (optimize speed (safety 0)))
  (dotimes (i n)
    (incf (the fixnum (slot-value a 'i)))))

(defmethod svn ((a a/no-fixnum) n)
  (declare (type fixnum n)
           (optimize speed (safety 0)))
  (dotimes (i n)
    (incf (the fixnum (slot-value a 'i)))))

They then call svn (or equivalent) with a large value of \(n\), do that a number of times \(m\) and then divide by \(2 \times n \times m\) to get an average time per access (incf accesses the slot twice).

For SBCL 2.6.3.178-a190d9710 on ARM64 Apple M1, seconds per access:

These numbers vary slightly, but this gives a good picture of what is going on. In particular you can see that slot-value within a method specialised on the class is more than 70 times slower than access for a structure slot, but if you can use standard-instance-access it is only about 6 times slower: standard-instance-access speeds things up by a factor of about 10, which changes CLOS slot access performance from laughably slow to merely pretty slow.

A macro

I've written a macro, called with-sia-slots which is like with-slots but uses standard-instance-access. It therefore has all the constraints imposed by that, but it is significantly faster than with-slots or slot-value. It has some overhead, as it has to dynamically compute the slot locations: this is better done outside any inner loop. This means that, for instance, you probably want to write code that looks like

(with-sia-slots (x) o
  (dotimes (i many)
    (setf x (... x ...))))

which will mean you only pay the overhead once.

The above tests don't use with-sia-slots, as I wrote them partly to see if something like this was worth writing. However on a current (at the time of writing) SBCL with-sia-slots is asymptotically about 10 times faster than with-slots as demonstrated by these tests.

Up to package names it should be portable to any CL with an AMOP-compatible MOP. It can be found in my implementation-specific hacks, linked from here.

01 May 2026 3:43pm GMT