fidzu : LISP

A year ago, you couldn't vibe code in Lisp. Even the SOTA models had trouble balancing parentheses, and they'd hallucinate packages and symbols that didn't exist. A year makes a big difference in this field, and the latest models are capable of vibe coding moderately sized programs in syntactically correct Lisp.

I have been experimenting with vibe coding in Common Lisp and I'm hooked. It is a blast. It is like having on hand a talented undergraduate who just took a Lisp course. If you give him small enough, focused tasks, he will churn out passable code. If you give him a good chunk of legacy code, he will churn out more code in the legacy style. The models are not good enough to do a full rewrite of a large codebase, but they are good enough to handle a small library with supervision.

I find myself accepting a large amount of code with just a glance-if it passes the Lisp reader, compiles, and the tests pass, I accept it. Unlike the code of a year ago, the generated code these days is far less buggy, and the models are pretty good at debugging their own code. I'll do spot checks on the code, but I don't bother reading it line by line unless I see something odd. If the model generates code in a style I don't like, I'll ask it to rewrite the code to be more to my liking.

But frankly, you don't need to read the code at all. If there is a good test suite, the model will generate code that passes tests. If the code is functionally correct, it doesn't matter if the code is pretty. In one way, it doesn't matter if the code is easy for a human to read and maintain because we ask the model to maintain it. We treat the code as a black box and we constrain it to pass the tests. (We accept machine code largely unread.)

Failure Modes

By far the most common failure mode is the model getting the number of closing parentheses wrong. The tail end of a block of code is usually a bunch of closing parentheses, and the model will be tokenizing them in groups of 2 or 3. But the likelihood of the "))" token isn't very much different from the likelihood of the ")))" token, so the model will sometimes grab the wrong one.

Depending on the model and the agent, when it tries to recover from the ensuing read error, it will re-compute the tokens in the output. It sometimes will thrash as it tries to balance parentheses, adding and removing them from various places in the code. (Sort of like a noob Lisp programmer.) Some models are more susceptible to this than others. I have found that the solution here is to pause the agent and manually fix the parentheses when the agent starts to thrash.

Vibe Coding Workflow

I've been using Copilot CLI and Gemini CLI to vibe code in Common Lisp. I start with a blank project directory and create an .asd file that loads the packages.lisp file and the main file for the project (which can start out as a "hello world"). Basically, make a minimal project that you can load with ASDF or Quicklisp.

The models can work at moderate levels of abstraction, but they do better if there is existing code supporting the abstraction level, and this suggests a `bottom-up` approach to the problem rather than a `stratified` design. But the models are actually quite capable of starting at a moderate level of abstraction right from the get-go.

So starting with a minimal project, I boot up the model and ask it to write the first things needed for the project-some data structures, some utilities, a few tests. The very simple stuff that is easy for the model to do ab initio. Then I ask the model to write a minimal main function that will implement the basic functionality of the project-a command loop, a server, what-have-you-with stubs for everything. Once a framework is in place, the models are easily able to extend it.

The agents will get into a loop of adding code, adding tests, and running all the tests. They will debug any test failures and only consider a task to be complete when all the tests pass.

The model does not write great code, and you will accumulate technical debt if you accept it as is. But the model can write code that works and passes the tests. It is a good idea to pause during development and simply ask the model to find the technical debt in the code, enumerate it, and rank it in order of importance. Then you ask the model to address each item in turn and the model will clean up the code. After a couple of iterations of cleanup, the code will look no worse than what I've seen in many professional codebases.

There are sort of two modes that you operate in: one is to modify the existing code (e.g. refactor) without disturbing the functionality; the other is to extend the functionality without disturbing the core operation. It is important to spend enough time refactoring and cleaning up. But the model is good at generating potential refactorings, and it is not good at knowing when to call it quits. It will happily churn away at your code making it `better' and doing more and more trivial refactorings. If you give the model one particular refactoring task and tell it to do just that one, it will do a good job.

Refactoring is satisfying in a certain way, but adding features gives you more instant gratification. The models are good at adding features and extending existing code, especially if the feature shares any similarity with existing code.

For more complex features and refactorings, tell the model that you want a 'plan' for the feature or refactoring. The model will come up with a multi-step plan, broken down into a series of tasks. The tasks in the plan are generally small enough to be handled by the model itself.

The models are good enough to maintain a codebase, so once you have a project up and running, the model will generally choose file names and a directory structure that is appropriate to put in the .asd file. If you get the model started with a test suite, it will extend the tests as it extends functionality, or you can ask it to add specific tests.

I have found that building a project by vibe coding it is an extremely rapid way to prototype. The model can churn out `obvious' code much faster than I can and it frees me up to think about the higher level design issues. I can build in a weekend what would have taken me a month before.

27 Jul 2026 7:39pm GMT

I wanted to run LLM models locally on my machine. I discovered that llama.cpp is how people run models locally, and that the popular LLM servers like Ollama and lmstudio and unsloth use llama.cpp under the hood.

llama.cpp is, of course, written in C++. I don't care for C++ and I prefer Common Lisp. With the appropriate declarations, Common Lisp code should be in the same performance ballpark as C++ code. So I decided to write a Common Lisp implementation of llama.cpp, which I call llambda.lisp.

It is available on GitHub.com/jrm-code-project/llambda If you care to contribute, it could use routing for architectures other than gemma, GPU support, NPU support, and other features.

12 Jul 2026 9:57pm GMT

Recently I stumbled on a funny kind of race in distributed systems. I believe even the classic texts don't cover that.

Say we have a system S sending commands to a receiver R over network. S maintains network outbox and inbox handled by separate threads. A command is expected to complete with certain result sent back before the deadline, or else S would assume a request timed out. Very basic so far.

However a certain class of commands (and only it) was timing out. They would fail regularly on some networks, sporadically on others and seemingly never on some. Perplexingly they were really simple commands, amounting to little else than sending back a reading of R's internal state. Increasing the command deadline had no apparent effect. Adding logging in places helped little and in fact often resulted in the issue disappearing.

Simplifying it quite a bit the S side looked something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

(defun handle-outbox (socket queue)
  (let ((command (pop queue)))
    (handler-case (send-command socket command))
      (network-error (c)
        (log-network-error c))
      (:no-error (c)
        (register-command-awaiting-response command))))

(defun handle-inbox (socket)
  (let* ((response (receive-incoming socket))
         (corresponding-command
          (command-awaiting-response response)))
    (when corresponding-command
      (process-reply-for-command command response))))

You see now what was happening: S was sending the command, R processing it, sending the reply and S handling the reply before the bookkeeping of outbox process would record the command. Then the response wouldn't match anything and be discarded as orphaned. Later the inbox would timeout the command (not shown).

The bookkeeping wasn't even that heavyweight: just storing some structures and perhaps couple expensive calls to set up a condition variable but that was enough.

Naturally any kind of latency (be it due to network load or extraneous syslog calls) would alleviate that. The fix was to make the code inelegant but correct by registering the command before the send attempt and un-registering in the event of failure.

tl;dr a network can be way too fast

06 Jul 2026 12:00am GMT

Lately I've been playing with writing a chatbot library in Common Lisp.

My previous gemini bindings were getting unweildy. I wanted to add the ability to run LLMs on my local machine but it turned out to be really kind of kludgy, so I decided to start from scratch with multiple back ends in mind.

I've got it to the point where in supports multiple back ends, so now I can prompt local LLMs from Lisp.

Recently I added the ability to recursively launch chatbots that can call each other. Since the chatbots do not share their contexts, this greatly reduces the context bloat of thet main chat because it can spawn off subtasks to a minion and not pollute the main context. This also allows you to create a federation of chatbots, each of which specializes in some topic and is overseen by a controlling chatbot that talks to the user.

Chatbots can be serialized and checkpointed, so if one is carrying out an agentic task and Lisp crashes, when we restart the agentic tasks are restarted as well and pick up where they left off.

IT turns out that recursive chats are a useful abstraction once you figure out how to use them. Basically any prompt you may issue may also want to be issued by an llm and this enables that to happen. It allows you to run subprocesses that would otherwise put junk in your context, for example reading the contents of a lange number of files. If you put that into a rocursive chatbot, it could slurp up the files into its context without adding tokens to the parent chat.

You can use a recursive chat as a `smart component'. The recursive chat can have a specialized system instruction and can preload its context with relevant information specific to it. It's context doesn't get diluted by the caller's context

28 Jun 2026 10:52pm GMT

I saw that there was some argument over how much slower slot access is than struct access, so I just decided to measure it naively. I made a two slot sruct and a CLOS version of a CONS cell with car and cdr slots and I ran LTAK using regular lists, `lists' made from CLOS conses, and `lists' made from structs. Here are the results:

D:\repositories\clos-benchmark>sbcl --script run-benchmarks.lisp
Benchmark: ltak over native cons cells, CLOS my-cons nodes, and my-cons-struct nodes
Inputs: x=15 y=9 z=4 repeats=35

Scenario                   min-ms     mean-ms      max-ms      ratio
--------------------------------------------------------------------
native standard               0.129      0.146      0.186
clos standard                 1.346      1.365      1.475       9.37x
struct standard               0.172      0.175      0.179       1.20x
native optimized              0.068      0.069      0.073
clos optimized                0.411      0.414      0.419       6.04x
struct optimized              0.068      0.069      0.073       1.01x

In this naive use case, structs are same as native cons cells, but CLOS objects are one ninth the speed of a struct or cons cell if you just use it unoptimized, and one sixth the speed if optimizations are turned on.

But the CLOS instance is more functional than the cons cell in mimics. For instance, I could add a slot to the class and all the instances would be lazily updated with the new slot. I can also subclass the CLOS class and the selector functions will continue to work. Finally, I can redefine the CLOS closs while I'm developing it and all the instances will be uppdated. THe machinery to keep all this running is costing us our factor of 9.

But this might be worth the cost if we are running on a network where the bulk of the time will be transmitting the answer down the pipe once it is computed. Taking a few extra milliseconds to compute the answer might be worth the convenience features of CLOS.

25 Jun 2026 4:11pm GMT

Back in the day, the US government had a program called SBIR (Small Business Innovation Research) that funded small businesses to do research and development. I recall sitting in our dorm in college, reading through a giant printed catalog of SBIR grants just to amuse ourselves by brainstorming solutions over bad pizza.

.

So, I got curious the other day: what does the SBIR landscape look like now?

I can tell you right now: do not even try to read an SBIR solicitation on your local machine. You are opening yourself up to a world of absolute, unmitigated pain.

You might think, what harm could there be in simply opening a file?

Well, in the modern compliance panopticon, any manipulation of digital information that comes from the govenment has the potential to spawn CUI (Controlled Unclassified Information). CUI is basically a digital pathogen; once you download that file, *anything whatsover* derived from it, including notes and metadata, instantly becomes CUI by association. The moment you read an SBIR on your computer, you've infected your system, rendering you subject to a nightmare of Byzantine federal regulations.

These days, the amount of beurocratic red tape surrounding CUI is insane. To even look at the file legally, you need a dedicated, air-gapped machine completely disconnected from the internet, conforming to a massive, expensive slew of NIST standards covering everything from hardware-level encryption to strict access controls. Alternatively you could contract with a cloud company that offers a pre-certified "CUI-compliant" environment.

And assuming you actually shell out the cash and jump through the hoops to set up this digital containment zone just to read a PDF, you must meticulously audit and account for every single action you take in its presence. Under current federal auditing logic, you are explicitly assumed to be attempting to defraud the government unless you can produce a mountain of paper proving otherwise. Want to bring in a partner to bounce ideas around? You can't just "know a guy." You have to navigate a labyrinth of federal subcontracting regulations.

I had intended on amusing myself by reading some SBIRs and daydreaming about solutions that might involve Lisp (an impossibility in the modern enterprise stack for entirely separate, depressing reasons). Instead, I quickly discovered I did not even own the physical hardware required to even read an SBIR without running afoul of federal regulations.

I wanted to read some clever and inspiring engineering proposals. I ended up reading a lot of very dry and boring compliance regulations.

18 Jun 2026 11:48am GMT

Last year I wrote some Lisp related AI apps. There was a syntax highlighter that used the LLM to determine how to colorize and highlight syntax, and a prompt refiner that takes a wimpy LLM prompt and creates more elaborate prompt from them.

I took the apps down last week. They were `vibe coded' and therefore approximate and had bugs (but that's to be expected), but they had a security hole where you could hijack the LLM processing with your own prompt turning my app into an open relay using my API key. Last week I discovered that my AI spend on video creation was becoming serious. This is odd because I never create AI video. It turned out that my app was being hijacked by a proxy in Luxembourg and was generating videos on my dime.

So I shut down the apps. I knew they had the potential of being abused, and I was willing to tolerate a small amount of abuse, but it didn't occur to me that syntax highlighter could be hijacked to generate gigabytes of video at my expense. Future applications will be careful to obtain the API key from the user.

01 Jun 2026 7:00am GMT

Metaobject Protocol (MOP) Implementation in CLRHack

The Metaobject Protocol in CLRHack is a high-performance implementation of the Common Lisp Object System (CLOS) integrated into the .NET 8.0 Common Language Runtime (CLR). It provides a complete meta-compilation pipeline that bridges the gap between dynamic Lisp semantics and the static CIL (Common Intermediate Language) execution model.

Core Architecture

The MOP is implemented through three primary layers:

1. The Metaobject Hierarchy (C#): A set of foundational classes in LispBase representing classes, methods, generic functions, and slot definitions.
2. The Runtime Engine (MopRuntime): A centralized orchestrator that manages class finalization, method combination, dispatch caching, and instance allocation.
3. The Compiler Bridge (Lisp): Transformations in ast.lisp that translate high-level CLOS forms (defclass, defmethod) into optimized runtime calls.

Instance Representation

Because the CLR type system is strictly single-inheritance and statically defined, CLRHack decouples Lisp-level inheritance from C# inheritance. All CLOS instances are represented by the StandardObjectInstance class, which contains:

• A reference to its ClassMetaobject.
• A private object[] storage array for instance slots, indexed by locations calculated during class finalization.

The Dispatch Pipeline

Generic function invocation is the most complex part of the implementation. When a generic function is called:

1. Cache Lookup: The DiscriminatingFunction first checks a thread-safe dispatchCache using an InvocationCacheKey (a stack-allocated struct) to find a previously computed effective method.
2. Applicability & Precedence: If the cache misses, the runtime computes all applicable methods and sorts them based on specializer specificity and the Class Precedence List (CPL).
3. Method Combination: The ComputeEffectiveMethod logic builds a nested execution chain following the Standard Method Combination rules:
   • :around methods are called first, with call-next-method progressing to the next around method or the main chain.
   • The main chain executes all :before methods, the primary method, and finally all :after methods in reverse order.
4. Fast Invocation: The resulting effective method is compiled into a Func<object[], object> that uses direct delegate invocation to minimize overhead.

Challenges and Solutions

1. Thread-Safe Non-Local Exits (call-next-method)

Challenge: call-next-method and next-method-p require access to the current invocation's state (the remaining methods and original arguments). Passing this state through every function call would break compatibility with standard Lisp function signatures.

Solution: CLRHack utilizes [ThreadStatic] fields in MopRuntime to store the currentNextMethods and currentArguments. This ensures that even in highly concurrent environments (like a web server), each OS thread has its own isolated invocation context, allowing call-next-method to function correctly without state leakage.

2. Forward References and Lazy Finalization

Challenge: Lisp allows classes to refer to superclasses that haven't been defined yet. The runtime must handle these "zombie" classes without crashing the JIT compiler.

Solution: The system implements a ForwardReferencedClassMetaobject. When a class is defined, it is automatically finalized (computing its CPL and slot layout). If a superclass is missing, a forward reference is created. The EnsureFinalized protocol ensures that inheritance is resolved and slot locations are assigned the moment the class is first instantiated or used in dispatch.

3. Performance Overhead of the "MOP Bridge"

Challenge: A naive implementation of slot-value or generic dispatch using C# reflection or linear searches is orders of magnitude slower than native C# member access.

Solution: Three distinct optimizations were applied:

• O(1) Slot Access: Each ClassMetaobject maintains a SlotDictionary. Slot names are mapped to physical array indices during finalization, allowing slot-value to perform a direct array access after a single dictionary lookup.
• Compiler Primitives: The compiler identifies SLOT-VALUE and MAKE-INSTANCE calls and emits direct CIL call instructions to optimized Lisp.MopRuntime methods, bypassing the general Funcall path.
• Zero-Allocation Cache Hits: By making InvocationCacheKey a readonly struct and avoiding the cloning of the argument array during cache probes, the hot-path for generic function dispatch generates zero garbage for the .NET Collector.

4. Bootstrapping the COMMON-LISP Package

Challenge: Core CLOS functions like make-instance must be available as symbols in the COMMON-LISP package before user code runs, but they rely on the MOP runtime being fully initialized.

Solution: A MopRuntime.Initialize() method is injected into the entry point (Main) of every generated assembly. This method interns the necessary symbols and binds them to GenericFunctionClosureAdapter objects, ensuring that the MOP is "alive" before the first line of Lisp code executes.

Vibe coding the MOP basically involved feeding chapters 4 and 5 of the Art of the Meta-Object Protocol into the LLM and telling it to make an implementation plan. It came up with a twenty-step plan to bootstrap CLOS. I then spent the rest of the day instructing an agent to take on each task of the twenty-step plan in sequential order. At the end of the day, I had a working MOP

This is the end of my series of posts on CLRHack.

31 May 2026 7:00am GMT

Implementation of SIGNAL and ERROR in CLRHack

In CLRHack, the condition signaling system is implemented in the Lisp.HandlerControl class within the LispBase library. It leverages .NET's [ThreadStatic] storage to maintain a per-thread dynamic stack of active condition handlers.

SIGNAL Implementation

The Signal(object condition) method performs the following logic:

1. Retrieval: It fetches the activeHandlers list for the current thread. This list is a chain of [LispBase]Lisp.Handler objects maintained by handler-bind.
2. Iteration: It iterates linearly through the list from the most recently bound handler to the oldest.
3. Type Matching: For each handler, it calls IsType(condition, handler.ConditionType).
   • If the condition is a symbol, it checks for symbol equality (supporting simple symbol-based conditions).
   • If the condition is a .NET object, it checks if the handler's type is assignable from the condition's runtime type (supporting interop with system exceptions).
   • It treats the symbols T or EXCEPTION as catch-all types.
4. Handler Invocation: If a match is found:
   • Recursive Signal Protection: Before calling the handler function, the current handler list is temporarily shadowed. activeHandlers is set to cell.rest (the handlers bound outside the current one). This ensures that if the handler itself calls signal, it won't trigger itself recursively.
   • Execution: The handler's Closure is invoked with the condition object as its argument.
   • Restoration: A finally block ensures the original activeHandlers list is restored if the handler returns normally.

   ERROR Implementation

   The Error(object condition) method build upon Signal:

   1. Signaling Pass: It first invokes Signal(condition). If a handler performs a non-local exit (e.g., via handler-case), the Error method never returns.
   2. Debugger Entry: If Signal returns normally (meaning all handlers declined), Error calls EnterDebugger(condition).
   3. Interactive Debugging: The debugger:
      • Prints the condition and a list of available restarts (retrieved via RestartControl.GetActiveRestarts()).
      • Provides a prompt for the user to select a restart, launch the system-level debugger (Visual Studio/Rider), or abort.
      • If a restart is selected, it is invoked interactively (potentially gathering arguments from the user).
   4. Final Fallback: If the debugger is exited without invoking a restart, Error throws a C# Exception to ensure that execution does not continue on an invalid path.

   Notable Implementation Decisions and Edge Cases

• Handler Shadowing: The decision to pop the handler list during invocation is critical for maintaining Common Lisp semantics. It prevents infinite loops and ensures that "outer" handlers can handle errors raised within "inner" handlers.
• Unified Exception Model: CLRHack attempts to unify Lisp conditions and .NET exceptions. IsType allows Lisp handlers to catch C# exceptions by their class name or Type object.
• Thread Isolation: By using [ThreadStatic] for activeHandlers, CLRHack ensures that condition signaling is thread-safe. One thread signaling an error will not interfere with the handler state of another thread.
• Debugger Capability: The SYSTEM-DEBUGGER option in EnterDebugger is a bridge to the underlying .NET environment, allowing developers to use professional IDE tools to inspect the state of the Lisp VM when an unhandled error occurs.

signal and error complete the Common Lisp condition system implementation for CLRHack

30 May 2026 7:00am GMT

In the CLRHack compiler, handler-bind is a primitive form used to register condition handlers in the dynamic environment. It operates by managing a thread-local list of active handler objects, ensuring that condition signaling follows the standard Common Lisp search and execution rules.

Handling of handler-bind

When the compiler processes a handler-bind form, it generates CIL code that performs the following steps:

1. Capture Previous State: It calls Lisp.HandlerControl::GetActiveHandlers() to retrieve the current list of active handlers and stores it in a frame-local variable.
2. Construct New List: For each binding, it evaluates the condition type and the handler function (which is typically a closure). It instantiates a new [LispBase]Lisp.Handler object and conses it onto the current handler list.
3. Install New State: It calls Lisp.HandlerControl::SetActiveHandlers(new_list) to update the dynamic environment for the current thread.
4. Protected Execution: The body of the handler-bind is wrapped in a CIL .try block.
5. Restoration: A finally block is emitted that calls SetActiveHandlers with the saved list. This ensures that handlers are properly uninstalled, regardless of whether the body completes normally, signals an error, or performs a non-local exit.

Lexical Non-Local Exits

Handlers in Common Lisp are executed in the dynamic environment of the signaller but have lexical access to the environment where they were defined. In CLRHack, if a handler function performs a non-local exit (such as a throw or return-from), the compiler utilizes its exception-based jump mechanism:

• If the exit is a throw, it uses the standard CatchThrowException mechanism.
• If the exit is a return-from to a block outside the handler closure, the compiler identifies this as a non-local exit during analyze-environment. It compiles the return-from into a throw of a BlockExitException, which is subsequently caught by the try/catch frame established by the target block.

Handler Search

The handler search is performed at runtime by the signal or error functions. These functions retrieve the active handlers list via HandlerControl.GetActiveHandlers() and iterate through them. For each handler, the runtime checks if the signaled condition is of the type (or a subtype of the type) the handler was registered for. If a match is found, the handler function is invoked. If the handler returns normally (declines), the search continues with the next applicable handler.

Dynamic Tags

The handler-bind implementation itself relies on the dynamic state of the thread-local activeHandlers list. However, when used in conjunction with handler-case, unique dynamic tags (typically fresh ListCell objects) are generated. These tags are used as the "target" for the throw performed by the handler, ensuring that the control flow returns exactly to the correct handler-case frame and doesn't conflict with other active handler or catch frames.

handler-case as an Extension of handler-bind

In CLRHack, handler-case is not a primitive but a macro that expands into a combination of block, catch, and handler-bind. It extends handler-bind by providing a mechanism to automatically exit the signaling context and execute a specific branch of code based on the condition caught.

The implementation details of the expansion are as follows:

• Exit Block: The entire form is wrapped in a block with a unique exit tag to allow the normal path to return immediately upon completion of the protected expression.
• Dynamic Setup: A unique dynamic tag is created for the catch frame. Local variables are established to store the captured condition and a unique ID identifying which clause was triggered.
• The Binding: A handler-bind is generated where each handler function is a closure that, when called:
  1. Saves the signaled condition into the local condition-var.
  2. Sets the id-var to a unique GENSYM representing that specific clause.
  3. Performs a throw to the dynamic tag.
• The Catch and Dispatch: A catch block surrounds the protected expression. If a handler performs the throw, the catch returns, and a cond statement (the dispatcher) checks the id-var. It then executes the body of the matching handler-case clause with the condition variable bound to the clause's parameter.

29 May 2026 7:00am GMT