03 Apr 2025
Planet Lisp
Joe Marshall: Blacksmithing and Lisp
One of my hobbies is blacksmithing (not ferrier work). Mild steel is an amazingly versatile material. It's very strong and hard at room temperature and it gets soft and easily workable when you heat it up. You use a hammer to move the metal once it is hot. You don't have to hit it very hard, just firmly. Hot metal is a very forgiving medium. If you make a mistake, simply heat the work up and try again. You rarely encounter mistakes you cannot recover from and you have to throw your work away.
A blacksmith uses tongs to manipulate work that would otherwise be too hot to handle. You don't want to drop a piece of hot steel, so you'd like your tongs to be a good fit on your work. Find some tongs that are an approximate fit and stick them in the fire to get good and hot. When they have softened, put the hot tongs around the cold workpiece and tap them into shape with your hammer. Voila! Custom tongs.
When I first saw this trick I was quite amused. It reminded me of Lisp programming - you can work on your problem, or you can customize the language to fit your problem better. And, yes, I'm the kind of nerd that sees a blacksmith trick and thinks "Lisp!"
Another computer-sciency question is one of bootstrapping. Where do tongs come from? How do you make tongs if you don't have them? This isn't too hard. A simple pair of tongs is basically two sticks with a rivet. You can shape half a pair of tongs (a single tong) by holding one end while you work the other. It will take a bit of time for the end in your hand becomes too hot to hold.
A good part of blacksmithing is creating ad hoc tools in pursuit of the end goal. You fairly often recur and create tools that help create tools. Metacircular blacksmithing.
The downside of working hot steel is that it is quite hot. You will get burned, but usually only mildly. Your reflexes take over pretty quick when you touch hot metal. Then you learn early on that if you drop something, you do not attempt to catch it.
03 Apr 2025 7:00am GMT
02 Apr 2025
Planet Lisp
Joe Marshall: Lisp at Work
Lisp is not a good fit for most of the projects I'm doing at work. Our company has few Lisp programmers, so there would be no one to help maintain any code I wrote, and no one has a Lisp environment set up on their machine. I'll sometimes do a quick prototype in Lisp, and I keep a REPL open on my machine, but I don't develop code I expect to share or deploy. Once I have a prototype or proof of concept, I'll write the real thing in a language that is more commonly used at work.
But sometimes, Lisp is such a good fit for a problem that it overrides the considerations of how well it fits into your company's ecosystem. Not often, but sometimes.
I had to migrate some builds from CircleCI 2 to CircleCI 4. We had tried (twice) and failed (twice) to do an in-place upgrade of the server, so we instead brought up CircleCI 4 in parallel and migrated the builds over. To migrate a build, we had to extract the build information from the old server and import it into the new server. The API to the old CircleCI server did not expose all the data we needed, and there was no API on the new server that let you import everything we needed.
But CircleCI is written in clojure. So you can connect to a running instance of CircleCI and open a REPL. The REPL has access to all the internal data structures. It is an easy matter to write some lisp code to extract the necessary data and print it to stdout. It is also an easy matter to write some lisp code to read some data from stdin and import it into the new server.
The migration code could be written in any language, but it had to talk to a clojure REPL, format data in such a way that the clojure REPL could read it, and parse the output from the clojure REPL, which was going to be a Lisp object. Any language with a Common Lisp reader and printer would do, but you sort of get these for free if you actually use Lisp. I knew that the migration code could be written in Lisp because I had already prototyped it.
So I wrote a migration function that would take the name of the project to be migrated. The I created a Dockerfile that would boot an instance of sbcl on a core file. I preloaded the migration code and dumped the core. I ran the Dockerfile and got an image that, when run, would migrate a single project read from the command line. I then created a server that a user could visit, enter the name of his project, and it would run the Docker image as a kubernetes job.
We migrated most of the projects this way. At one point, I wrote an additional script that would read the entire list of projects in the old server and simply print them to stdout. After we shut down the migration server, I'd get requests from people that didn't seem to understand what a deadline was. I could prime the migration script from the file and it would initialize a project on the new server with the old state that I dumped. Migration stragglers could often recover this way.
Using Lisp for the migration tool did not add risk to the company. Just using the tool did not require Lisp knowledge. Anyone that needed to understand the migration process in detail had to understand clojure anyway. The migration took place over a period of weeks, and the migration tool was shut down at the end of this period. Long term maintenance was not a concern. Users of the migration tool did not have to understand Lisp, or even know what language was being used. It was a black box kubernetes job. I got buy-in from management because the tool simply worked and solved a problem that had twice been been unsuccessfully attempted before.
02 Apr 2025 4:09pm GMT
01 Apr 2025
Planet Lisp
Joe Marshall: Vibe Coding, final word
I couldn't leave it alone. This AI was going to write some Lisp code if I had to force it. This isn't &lquo;vibing" anymore. We're going to be pecise, exact, and complete in our instructions, and we're going to check the results.
Again, I'm taking on a Minesweeper clone as the problem. All the code was to be written in a single file using a single package. The AI simply didn't understand the problem of forward references to symbols in other packages. Perhaps a game loop is beyond the ability of the AI. I wrote a basic game loop that initializes all the required libraries in correct order with unwind-protects to clean up in reverse order. I wrote a main function that creates a window and a renderer to draw on it, and a game loop that polls for events and handles keypresses and the quit event. This is a basic black window that has no behavior beyond the ability to quit. There should be no need for the AI to modify this code.
The AI used the GPT-4o model. Instructions were given in precise, imperative English. For example,
"Each cell on the board is in one of these states: hidden, flagging, flagged, unflagging, exposing, exposed Cells start out in hidden state. When a cell is hidden, it renders as a blank square. When a cell is hidden and the mouse is over the cell and the right button is down, the cell enteres the flagging state. When a cell is flagging and the mouse is over the cell and the right button is up, the cell enters the flagged mode. When a cell is flagged and the mouse is over the cell and the right button is down, the cell enters unflagging. When the cell is unflagging, the mouse is over the cell and and right button is up, the cell enters hidden. Cells that are flagging or flagged display as the flag texture. Cells that are hidden or unflagging display as the blank texture."
This is programming, not vibing. There is always room for misunderstanding, but I spelled out the details of part of the state transitions that I wanted the AI to implement. In particular, notice that when flagging a cell, there are hidden states beyond the flagged and unflagged states. These are necessary to make the effect of flagging and unflagging be edge triggered. I didn't trust the AI to know about this, so I spelled it out.
Sometimes I could write simple directions, such as:
"When rendering a cell, if it is under the mouse, highlight the cell with a red border."
Or:
"When the cell is in the exposing state, it should display as a blank, depressed cell."
But then again,
"When the mouse is over a cell, if the cell is in hidden state and left mouse button is down, the cell enters the exposing state. If a cell is in exposing state and the mouse is not over the cell, the cell reverts to hidden state. If a cell is in exposing state, and the mouse is over the cell, and the left mouse button is raised, the cell enters the exposed state."
The AI did a surprisingly good job at handling the parenthesis. There was one function that it kept adding a stray parenthesis to. I went back more than once to delete that extra parenthesis when the reader barfed.
At one point, the AI accidentally removed the abiity to place a flag on a cell. I asked it to restore that ability, but I didn't give instructions in as great detail as I did before. It added transitions to and from the flagged state, but it didn't add the intermediate flagging and unflagging states as I had explicitly directed it to before. As a result, flagging ended up being non-deterministic, with the flag being toggled on and off every 16 ms. I had to go back and add the intermediate states by hand again to restore the determinism.
After a few hours of this, I had a working Minesweeper clone. Let's look at the generated code.
;; -*- mode: lisp; package: "VIBE" -*- (in-package "VIBE") ;; Define constants for magic numbers (defparameter *tile-size* 32) ;; Size of each tile in pixels (defparameter *game-width* 25) ;; Number of columns (defparameter *game-height* 15) ;; Number of rows (defparameter *frame-delay* 16) ;; Delay between frames in milliseconds ;; Define cell states (defconstant +hidden+ 'hidden) (defconstant +flagging+ 'flagging) (defconstant +flagged+ 'flagged) (defconstant +unflagging+ 'unflagging) (defconstant +exposing+ 'exposing) (defconstant +exposed+ 'exposed) ;; Define game states (defconstant +playing+ 'playing) (defconstant +won+ 'won) (defconstant +lost+ 'lost)
It is a little odd to define the state constants as symbols. This adds a pointless extra level of symbolic indirection. I would have used keywords instead of symbols.
;; Global game state
(defparameter *game-state* +playing+)
;; The game board
(defparameter *board* nil)
;; The game board content (e.g., :mine or a number)
(defparameter *board-content* nil)
(defun initialize-board ()
"Initialize the game board with all cells in the hidden state and place mines."
;; Initialize the board state
(setf *board* (make-array (list *game-height* *game-width*)
:initial-element +hidden+))
;; Initialize the board content
(setf *board-content* (make-array (list *game-height* *game-width*)
:initial-element 0))
;; Place 50 mines randomly
(loop repeat 50
do (let ((row (random *game-height*))
(col (random *game-width*)))
;; Ensure no duplicate mines
(unless (eql (aref *board-content* row col) :mine)
(setf (aref *board-content* row col) :mine)
;; Update adjacent cells
(update-adjacent-cells row col)))))
(defun update-adjacent-cells (row col)
"Increment the mine count for all adjacent cells."
(loop for r from (max 0 (1- row)) to (min (1+ row) (1- *game-height*))
do (loop for c from (max 0 (1- col)) to (min (1+ col) (1- *game-width*))
do (unless (eql (aref *board-content* r c) :mine)
(incf (aref *board-content* r c))))))
You can tell that I didn't write this code. I hate loop.
The textures are loaded from PNG files just before beginning the game loop. The AI decided to maintain them as four separate values that are passed around to the functions that need them. This will get out of hand quickly. A table of resources would be better.
(defun render-frame (renderer blank-texture flag-texture mine-texture digit-textures)
(sdl2:set-render-draw-color renderer 0 0 0 0)
(sdl2:render-clear renderer)
;; Render game state
(loop for row from 0 below *game-height*
do (loop for col from 0 below *game-width*
do (render-tile renderer row col blank-texture flag-texture mine-texture digit-textures))))
(defun render-tile (renderer row col blank-texture flag-texture mine-texture digit-textures)
"Render a single tile based on its state and highlight it if under the mouse."
(let ((state (aref *board* row col))
(content (aref *board-content* row col)) ;; Content of the cell (e.g., :mine or a number)
(dest-rect (sdl2:make-rect (* col *tile-size*) (* row *tile-size*) *tile-size* *tile-size*)))
;; Render the tile based on its state
(cond
;; Hidden or unflagging state: render blank texture
((or (eql state +hidden+) (eql state +unflagging+))
(sdl2:render-copy renderer blank-texture :dest-rect dest-rect))
;; Flagging or flagged state: render flag texture
((or (eql state +flagging+) (eql state +flagged+))
(sdl2:render-copy renderer flag-texture :dest-rect dest-rect))
;; Exposing state: render blank texture with a depressed effect
((eql state +exposing+)
(sdl2:render-copy renderer blank-texture :dest-rect dest-rect)
(sdl2:set-render-draw-color renderer 50 50 50 100) ;; Semi-transparent dark overlay
(sdl2:render-fill-rect renderer dest-rect))
;; Exposed state: render mine or digit
((eql state +exposed+)
(cond
;; If the cell contains a mine, render the mine texture
((eql content :mine)
(sdl2:render-copy renderer mine-texture :dest-rect dest-rect))
;; If the cell contains a digit (positive number), render the corresponding digit texture
((and (integerp content) (> content 0))
(let ((digit-texture (gethash content digit-textures)))
(when digit-texture
(sdl2:render-copy renderer digit-texture :dest-rect dest-rect))))
;; Otherwise, render a default exposed tile (e.g., light gray)
(t
(sdl2:set-render-draw-color renderer 200 200 200 255) ;; Light gray for exposed
(sdl2:render-fill-rect renderer dest-rect)))))
;; Highlight the tile with a red border if it is under the mouse
(multiple-value-bind (mouse-x mouse-y _) (sdl2:mouse-state)
(declare (ignore _))
(let ((mouse-row (floor (/ mouse-y *tile-size*)))
(mouse-col (floor (/ mouse-x *tile-size*))))
(when (and (= row mouse-row) (= col mouse-col))
(sdl2:set-render-draw-color renderer 255 0 0 255) ;; Red color
(sdl2:render-draw-rect renderer dest-rect)
(sdl2:set-render-draw-color renderer 0 0 0 255)))))) ;; Reset color to black
Each time we go around the game loop, we update the state the cell under the mouse. The AI decided to pass the state input as spread arguments to the update function.
(defun update-cell-state (row col mouse-over left-button-down left-button-up right-button-down right-button-up)
"Update the state of a cell based on mouse input."
(let ((state (aref *board* row col)))
(cond
;; If the cell is hidden and the left mouse button is down, it enters the exposing state
((and (eql state +hidden+) mouse-over left-button-down)
(setf (aref *board* row col) +exposing+))
;; If the cell is exposing and the mouse is not over it, it reverts to hidden
((and (eql state +exposing+) (not mouse-over))
(setf (aref *board* row col) +hidden+))
;; If the cell is exposing, the mouse is over it, and the left mouse button is raised, it enters the exposed state
((and (eql state +exposing+) mouse-over left-button-up)
(setf (aref *board* row col) +exposed+)
;; If the cell contains a mine, the game is lost and all mines are exposed
(when (eql (aref *board-content* row col) :mine)
(setf *game-state* +lost+)
(expose-all-mines))
;; If the cell has zero neighboring mines, recursively expose neighbors
(when (and (integerp (aref *board-content* row col))
(= (aref *board-content* row col) 0))
(expose-neighbors row col)))
;; If the cell is hidden and the right mouse button is down, it enters the flagging state
((and (eql state +hidden+) mouse-over right-button-down)
(setf (aref *board* row col) +flagging+))
;; If the cell is flagging and the right mouse button is up, it enters the flagged state
((and (eql state +flagging+) mouse-over right-button-up)
(setf (aref *board* row col) +flagged+))
;; If the cell is flagged and the right mouse button is down, it removes the flag
((and (eql state +flagged+) mouse-over right-button-down)
(setf (aref *board* row col) +unflagging+))
((and (eql state +unflagging+) mouse-over right-button-up)
(setf (aref *board* row col) +hidden+)))))
(defun poll-mouse-and-update ()
"Poll the mouse position and button states, and update the board accordingly."
(when (eql *game-state* +playing+) ;; Only process mouse input if the game is playing
(multiple-value-bind (x y buttons) (sdl2:mouse-state)
(let ((row (floor (/ y *tile-size*)))
(col (floor (/ x *tile-size*)))
(left-button-down (logbitp 0 buttons)) ;; SDL_BUTTON_LEFT is bit 0
(right-button-down (logbitp 2 buttons))) ;; SDL_BUTTON_RIGHT is bit 2
(when (and (>= row 0) (< row *game-height*)
(>= col 0) (< col *game-width*))
;; Update the cell state based on mouse input
(update-cell-state row col
t ;; mouse-over is true for the current cell
left-button-down
(not left-button-down)
right-button-down
(not right-button-down)))))))
This illustrates that while the lights appear to be on, no one is at home. The mouse-over variable is always true, there is no need for it to exist at all. There is no need to pass both left-button-down and its complement. Same with right-button-down.
I did allow the AI to modify game-loop, but the modifications were subject to careful scrutiny to make sure that the game would continue to run. In particular, one time it wanted to add handlers for mouse events. I told it no, and that it could poll the mouse state as necessary instead.
(defun game-loop (window renderer blank-texture flag-texture mine-texture digit-textures game-over-texture)
"Main game loop."
(declare (ignore window))
;; Main game loop
(sdl2:with-event-loop (:method :poll)
(:idle ()
;; Clear the screen
(sdl2:set-render-draw-color renderer 0 0 0 255) ;; Black background
(sdl2:render-clear renderer)
;; Poll mouse and update game state
(poll-mouse-and-update)
;; Render the game frame
(render-frame renderer blank-texture flag-texture mine-texture digit-textures)
;; Render the "Game Over" overlay if the game is lost
(when (eql *game-state* +lost+)
(let ((screen-width (* *tile-size* *game-width*))
(screen-height (* *tile-size* *game-height*)))
;; Set blend mode and alpha for transparency
(sdl2:set-texture-blend-mode game-over-texture :blend)
(sdl2:set-texture-alpha-mod game-over-texture 192) ;; 75% transparency
;; Render the texture as a full-screen overlay
(let ((dest-rect (sdl2:make-rect 0 0 screen-width screen-height)))
(sdl2:render-copy renderer game-over-texture :dest-rect dest-rect))))
;; Present the rendered frame
(sdl2:render-present renderer)
;; Delay for the next frame
(sdl2:delay *frame-delay*))
(:keydown (:keysym keysym)
(cond
;; Reset the game when the 'o' key is pressed
((eql (sdl2:scancode keysym) :scancode-o)
(reset-game))
;; Quit the game when the 'x' key is pressed
((eql (sdl2:scancode keysym) :scancode-x)
(sdl2:push-quit-event))
;; Lose the game and expose all mines when the 'p' key is pressed
((eql (sdl2:scancode keysym) :scancode-p)
(progn
(setf *game-state* +lost+)
(expose-all-mines)))))
(:quit () t)))
Notice that in this game loop, we're not accounting for the time it takes to update the game state and render the frame. If this game really tried to animate anything, the animation would be jittery. A better game loop would track real time and refresh accordingly.
For a simple game such as this, it makes sense to load the all the bitmaps into memory at the get-go. For a more complicated game with many levels, you might not be able to fit them all in memory.
Passing the surfaces around as arguments is not going to work when you have a lot of them.
(defun initialize ()
"Initialize the game, load textures, and create the game board."
(initialize-board) ;; Initialize the game board
(let ((blank-surface nil)
(flag-surface nil)
(mine-surface nil)
(game-over-surface nil)
(digit-surfaces (make-hash-table)))
(unwind-protect
(progn
;; Load PNG surfaces
(setq blank-surface (sdl2-image:load-image
(asdf:system-relative-pathname "vibe" "textures/blank.png")))
(setq flag-surface (sdl2-image:load-image
(asdf:system-relative-pathname "vibe" "textures/flag.png")))
(setq mine-surface (sdl2-image:load-image
(asdf:system-relative-pathname "vibe" "textures/mine.png")))
;; Load digit textures (e.g., "1.png", "2.png", etc.)
(loop for i from 1 to 8
do (setf (gethash i digit-surfaces)
(sdl2-image:load-image
(asdf:system-relative-pathname "vibe" (format nil "textures/~a.png" i)))))
;; Create the "GAME OVER" surface
(setq game-over-surface (create-game-over-surface))
;; Create the window and renderer
(sdl2:with-window (window
:title "Vibe"
:x 0 :y 0
:w (* *tile-size* *game-width*)
:h (* *tile-size* *game-height*)
:flags '(:shown))
(sdl2:with-renderer (renderer window :index -1 :flags '(:accelerated))
(let ((blank-texture (sdl2:create-texture-from-surface renderer blank-surface))
(flag-texture (sdl2:create-texture-from-surface renderer flag-surface))
(mine-texture (sdl2:create-texture-from-surface renderer mine-surface))
(digit-textures (make-hash-table))
(game-over-texture (sdl2:create-texture-from-surface renderer game-over-surface)))
;; Convert digit surfaces to textures
(maphash (lambda (key surface)
(setf (gethash key digit-textures)
(sdl2:create-texture-from-surface renderer surface)))
digit-surfaces)
(unwind-protect
(game-loop window renderer blank-texture flag-texture mine-texture digit-textures game-over-texture)
;; Cleanup textures
(sdl2:destroy-texture blank-texture)
(sdl2:destroy-texture flag-texture)
(sdl2:destroy-texture mine-texture)
(sdl2:destroy-texture game-over-texture)
(maphash (lambda (_key texture)
(declare (ignore _key))
(sdl2:destroy-texture texture))
digit-textures)))))))
;; Cleanup surfaces
(when flag-surface (sdl2:free-surface flag-surface))
(when blank-surface (sdl2:free-surface blank-surface))
(when mine-surface (sdl2:free-surface mine-surface))
(when game-over-surface (sdl2:free-surface game-over-surface))
(maphash (lambda (_key surface)
(declare (ignore _key))
(sdl2:free-surface surface))
digit-surfaces)))
In Minesweeper, if you click on a cell with no neighboring mines, all the neighboring cells are exposed. This will open up larger areas of the board. The AI did a good job of implementing this, but I was careful to specify that only the hidden cells should be exposed. Otherwise, the recursion would not bottom out because every cell is a neighbor of its neighbors.
(defun expose-neighbors (row col)
"Recursively expose all hidden neighbors of a cell with zero neighboring mines."
(loop for r from (max 0 (1- row)) to (min (1+ row) (1- *game-height*))
do (loop for c from (max 0 (1- col)) to (min (1+ col) (1- *game-width*))
do (when (and (eql (aref *board* r c) +hidden+)) ;; Only expose hidden cells
(setf (aref *board* r c) +exposed+)
;; If the neighbor also has zero mines, recursively expose its neighbors
(when (and (integerp (aref *board-content* r c))
(= (aref *board-content* r c) 0))
(expose-neighbors r c))))))
We need a way to get the game back to the initial state.
(defun reset-game () "Reset the game by reinitializing the board and setting the game state to playing." (initialize-board) (setf *game-state* +playing+))
The AI writes buggy code. Here is an example. It is trying figure out if the player has won the game. You can state the winning condition in couple of different ways.
- All the cells that are not mines are exposed.
- All the cells that are mines are flagged, all flagged cells contain mines.
This does't quite achieve either of these.
(defun check-win-condition ()
"Check if the player has won the game."
(let ((won t)) ;; Assume the player has won until proven otherwise
(loop for row from 0 below *game-height*
do (loop for col from 0 below *game-width*
do (let ((state (aref *board* row col))
(content (aref *board-content* row col)))
(when (and (not (eql state +exposed+)) ;; Cell is not exposed
(not (or (eql state +flagged+) ;; Cell is not flagged
(eql content :mine)))) ;; Cell does not contain a mine
(setf won nil)))))
(when won
(setf *game-state* +won+))))
create-game-over-surface prepares a surface with the words "Game Over" writ large.
(defun create-game-over-surface ()
"Create a surface for the 'GAME OVER' splash screen using SDL2-TTF."
(let ((font nil)
(text-surface nil))
(unwind-protect
(progn
;; Load the font (adjust the path and size as needed)
(setq font (sdl2-ttf:open-font (asdf:system-relative-pathname "vibe" "fonts/arial.ttf") 72))
;; Render the text "GAME OVER" in red
(setq text-surface (sdl2-ttf:render-text-solid font "GAME OVER" 255 0 0 255)))
;; Cleanup
(when font (sdl2-ttf:close-font font)))
text-surface))
The main function initializes the SDL2 library and its auxiliar libraries along with unwind-protects to uninitialize when we leave the game. The AI was not permitted to change this code.
(defun main ()
(sdl2:with-init (:video)
(unwind-protect
(progn
(sdl2-image:init '(:png))
(unwind-protect
(progn
(sdl2-ttf:init)
(initialize))
(sdl2-ttf:quit)))
(sdl2-image:quit))))
If you step on a mine, it exposes the other mines.
(defun expose-all-mines ()
"Expose all mines on the board."
(loop for row from 0 below *game-height*
do (loop for col from 0 below *game-width*
do (when (eql (aref *board-content* row col) :mine)
(setf (aref *board* row col) +exposed+)))))
Conclusion
This wasn't "vibe coding". This was plain old coding, but filtered through an English language parser. It added an extra level of complexity. Not only did I have to think about what should be coded, I had to think about how to phrase it such that the AI would generate what I had in mind and not disturb the other code.
Whenever I tried to let go and "vibe", the AI would generate some unworkable mess. Programming is a craft that requires training and discipline. No dumb pattern matcher (or sophisticated one) is going to replace it.
In languages other that Common Lisp, you might get further. Consider Java. It takes a page and half of boilerplate to specify the simplest first-class object. An AI can easily generate pages and pages of boilerplate and appear to be quite productive. But you've missed the point if you think that it is better to generate boilerplate automatically than to use abstractions to avoid it and a language that doesn't need it.
01 Apr 2025 7:00am GMT