15 May 2025
Planet Lisp
Gábor Melis: PAX PDF Output
Thanks to Paul A. Patience, PAX now has PDF support. See pax-manual-v0.4.1.pdf and dref-manual-v0.4.1.pdf. The PDF is very similar to the HTML, even down to the locative types (e.g [function]
) being linked to the sources on GitHub, but cross-linking between PDFs doesn't work reliably on most viewers, so that's disabled. Also, for reading PDFs so heavy on internal linking to be enjoyable, one needs a viewer that supports going back within the PDF (not the case with Chrome at the moment). Here is a blurry screenshot to entice:
There is a bit of a Christmas tree effect due to syntax highlighting and the colouring of the links. Blue links are internal to the PDF, maroon links are external. I might want to change that to make it look more like the HTML, but I have not found a way on LaTeX to underline text without breaking automatic hyphenation.
15 May 2025 12:00am GMT
13 May 2025
Planet Lisp
Joe Marshall: Purchasing White Elephants
As a software engineer, I'm constantly trying to persuade management to avoid doing stupid things. Management is of the opinion that because they are paying the engineers anyway, the software is essentially free. In my experience, bespoke software is one of the most expensive things you can waste money on. You're usually better off setting your money on fire than writing custom software.
But managers get ideas in their heads and it falls upon us engineers to puncture them. I wish I were less ethical. I'd just take the money and spend it as long as it kept flowing. But I wouldn't be able to live with myself. I have to at least try to persuade them to avoid the most egregious boondoggles. If they still insist on doing the project, well, so be it.
I'm absolutely delighted to find that these LLMs are very good at making plausible sounding proposals for software projects. I was asked about a project recently and I just fed the parameters into the LLM and asked it for an outline of the project, estimated headcount, time, and cost. It suggested we could do it in 6 months with 15 engineers at a cost of $3M. (I think it was more than a bit optimistic, frankly, but it was a good start.) It provided a phased breakdown of the project and the burn rate. Management was curious about how long it would take 1 engineer and the LLM suggested 3-6 years.
Management was suitably horrified.
I've been trying to persuade them that the status quo has been satisfying our needs, costs nothing, needs no engineers, and is ready today, but they didn't want to hear it. But now they are starting to see the light.
13 May 2025 8:05pm GMT
12 May 2025
Planet Lisp
Marco Antoniotti: Getting into a rabbit's hole and - maybe - getting out: Emacs Make Compile (EMC)
In the past years I fell form one rabbit's hole into another. For the first time in a rather long time, I feel I am getting out of one.
First of all let me tell you what I produced. I built the "Emacs Make Compile", or "Emacs Master of Ceremonies", package EMC. Soon it will be available in melpa. The EMC package is a wrapper around compile.el
that allows you to semi-transparently invoke make
or cmake
from Emacs. Either on UN*X, MacOS or Windows.
Once you have loaded the emc
library in the usual ways, you can just issue the Emacs command emc:run
(yes: Common Lisp naming conventions). The command is just the most general one available in EMC; other ones are the more specialized emc:make
and emc:cmake
. Emacs will then ask you for the necessary bits and pieces to ensure that you can run, say, make
. The README file included in the distribution explains what is available in more details.
Where did it all begin?
Why not stick with compile.el
? Because it does not have "out-of-the-box" decent defaults under Windows. At least, that was my original excuse.
I fell into this rabbit's hole coming from another one of course.
Some time ago, I started fiddling around with Emacs Dynamic Modules. I wanted to compile them directly from Emacs in order to "simplify" their deployment. Therefore, I set out to write a make
function that would hide the compile
setup.
Alas, I found out that, because of the necessary setup, invoking the Microsoft Visual Studio toolchain is not easy before you can get to cl
and nmake
. That was not all that difficult as a problem to solve, but then I made the mistake of learning to cmake
. You know; to ensure that the building process was "more portable". The basic machinery for make
and nmake
worked to also get cmake
up and running. But then I made another mistake: I started to want to get EMC to be usable in the "Emacs" way: at a minimum getting interactive commands working. That got me deeper and deeper in the rabbit's hole.
At the bottom of the hole (yep: I got there!)
I found out many things on my way to the bottom. That is, I learned many things about the Emacs Lisp ecosystem and wasted a lot of time in the process. I never was a fast learner. All in all, I think I can now say two things.
- Making a command, i.e., an
interactive
function is not trivial, especially if your function has many arguments. Bottom line: your Emacs commands should have *few* arguments. I should have known better. - The Emacs
widget
library is woefully underdocumented (which, of course, brings up the question: why did you want to use it?)
In any case, what I was able to concot is that hitting M-x emc:make
does what you expect, assuming you have a Makefile
in the directory; if not you will be asked for a "makefile", say stuff.mk
to be used as in
make -f stuff.mk
- or
nmake /F stuff.mk
Issuing C-u M-x emc:make
will ask you for the "makefile", the "source directory", the "build directory", "macros", and "targets".
In what other ways could I have wasted some time? By coming up with a widget-based UI! (See my previous post about DeepSeek and the widget library). The result can be invoked by using the command emc:emc
, which pops up the window below.
Getting out of the rabbit hole by popping the stack
I kind of consider EMC finished. I am pleased by the result; it was fun to solve all the problems I encountered, although the code is not exaclty nice or nicely organized. Is EMC useful? Probabiy not so much, but I have the luxury of wasting hacking time. I just hope somebody will like it: please try it out and report bugs and suggestions (the minor mode and associated menu need work for sure, as well as emc:emc
).
Having said so, I can now go back to play with Emacs Dynamic Modules, which is where I was coming from. After being satisfied with that, I will be able to climb back up a bit more from the rabbit's hole; that is, I will be able to go back to the magiciel
library (which kind of works already). You may ask why I am writing magiciel
, but you will have to reach down several levels in the rabbit's hole.
In any case, I finished one thing. It's progress.
'(cheers)
12 May 2025 1:41pm GMT
Paolo Amoroso: Changing text style for DandeGUI window output
Printing rich text to windows is one of the planned features of DandeGUI, the GUI library for Medley Interlisp I'm developing in Common Lisp. I finally got around to this and implemented the GUI:WITH-TEXT-STYLE
macro which controls the attributes of text printed to a window, such as the font family and face.
GUI:WITH-TEXT-STYLE
establishes a context in which text printed to the stream associated with a TEdit window is rendered in the style specified by the arguments. The call to GUI:WITH-TEXT-STYLE
here extends the square root table example by printing the heading in a 12-point bold sans serif font:
(gui:with-output-to-window (stream :title "Table of square roots")
(gui:with-text-style (stream :family :sans :size 12 :face :bold)
(format stream "~&Number~40TSquare Root~2%"))
(loop
for n from 1 to 30
do (format stream "~&~4D~40T~8,4F~%" n (sqrt n))))
The code produces this window in which the styled column headings stand out:
The :FAMILY
, :SIZE
, and :FACE
arguments determine the corresponding text attributes. :FAMILY
may be a generic family such as :SERIF
for an unspecified serif font; :SANS
for a sans serif font; :FIX
for a fixed width font; or a keyword denoting a specific family like :TIMESROMAN
.
At the heart of GUI:WITH-TEXT-STYLE
is a pair of calls to the Interlisp function PRINTOUT
that wrap the macro body, the first for setting the font of the stream to the specified style and the other for restoring the default:
(DEFMACRO WITH-TEXT-STYLE ((STREAM &KEY FAMILY SIZE FACE)
&BODY BODY)
(ONCE-ONLY (STREAM)
`(UNWIND-PROTECT
(PROGN (IL:PRINTOUT ,STREAM IL:.FONT (TEXT-STYLE-TO-FD ,FAMILY ,SIZE ,FACE))
,@BODY)
(IL:PRINTOUT ,STREAM IL:.FONT *DEFAULT-FONT*))))
PRINTOUT
is an Interlisp function for formatted output similar to Common Lisp's FORMAT
but with additional font control via the .FONT
directive. The symbols of PRINTOUT
, i.e. its directives and arguments, are in the Interlisp package.
In turn GUI:WITH-TEXT-STYLE
calls GUI::TEXT-STYLE-TO-FD
, an internal DandeGUI function which passes to .FONT
a font descriptor matching the required text attributes. GUI::TEXT-STYLE-TO-FD
calls IL:FONTCOPY
to build a descriptor that merges the specified attributes with any unspecified ones copied from the default font.
The font descriptor is an Interlisp data structure that represents a font on the Medley environment.
#DandeGUI #CommonLisp #Interlisp #Lisp
Discuss... Email | Reply @amoroso@oldbytes.space
12 May 2025 9:35am GMT
02 May 2025
Planet Lisp
Gábor Melis: Adaptive Hashing
At the 2024 ELS, I gave a talk on adaptive hashing, which focusses on making general purpose hash tables faster and more robust at the same time.
Theory vs Practice
Hash table theory most concerns itself with the asymptotic worst-case cost with a hash function chosen randomly from a family of hash functions. Although these results are very relevant in practice,
-
those pesky constant factors, that the big-O cost ignores, do matter, and
-
we don't pick hash functions randomly but fix the hash function for the lifetime of the hash table.
There are Perfect Hashing algorithms, that choose an optimal hash function for a given set of keys. The drawback is that they either require the set of keys to be fixed or they are too slow to be used as general purpose hash tables.
Still, the idea that we can do better by adapting the hash function to the actual keys is key. Can we do that online, that is, while the hash table is being used? Potential performance gains come from improving the constant factors mentioned above by
-
having fewer collisions, and
-
being more cache-friendly.
The first image above plots the regret (the expected number of comparisons of per lookup minus the minimum achievable) and the measured run-time of PUT operations vs the number of keys in the hash table with a particular key distribution. Green is Murmur (a robust hash function), Blue is SBCL's expedient EQ
hash. The wiggling of the graphs is due to the resizing of the hash table as keys are added to it.
Note how SBCL's regret starts out much lower and becomes much higher than that of Murmur, but if anything, its advantage in run time (second image) grows.
Implementation
The general idea is sound, but turning it into real performance gains is hard due to the cost of choosing a hash function and switching to it. First, we have to make some assumption about the distribution of keys. In fact, default hash functions in language runtimes often make such assumptions to make the common cases faster, usually at the cost of weakened worst-case guarantees.
The rest of this post is about how SBCL's built-in hash tables, which had been reasonably fast, were modified. The core switching mechanism looks at
-
the length of the collision chain on PUT operations,
-
the collision count on rehash (when the hash table is grown), and
-
the size of the hash table.
Adapting EQ
hash tables
-
Init to to constant hash function. This a fancy way of saying that we do linear search in a vector internally. This is an
EQ
hash table, so key comparison is as single assembly instruction. -
When the hash table is grown to more than 32 keys and it must be rehashed anyway, we switch to a hash function that does a single right shift with the number of bits to shift determined from the longest common run of low-bits in the keys.
-
If too many collisions, we switch to the previous default SBCL
EQ
-hash function that has been tuned for a long time. -
If too many collisions, we switch to Murmur, a general purpose hash. We could also go all the way to cryptographic hashes.
In step 2, the hash function with the single shift fits the memory allocator's behaviour nicely: it is a perfect hash for keys forming arithmetic sequences, which is often approximately true for objects of the same type allocated in a loop.
In this figure, the red line is the adaptive hash.
Adapting EQUAL
hash tables
For composite keys, running the hash function is the main cost. Adaptive hashing does the following.
-
For string keys, hash only the first and last 2 characters.
-
For list keys, only hash the first 4 elements.
-
If too many collisions, double the limit.
So, SBCL hash tables have been adaptive for almost a year now, gaining some speed in common cases, and robustness in others.
The full paper is here.
02 May 2025 12:00am GMT
01 May 2025
Planet Lisp
Joe Marshall: It Still Sucks
Don't get me wrong. I"m not saying that the alternatives are any better or even any different.
Unix has been around more than forty years and it is still susceptible to I/O deadlock when you try to run a subprocess and stream input to it and output from it. The processes run just fine for a while, then they hang indefinitely waiting for input and output from some buffer to synchronize.
I'm trying to store data in a database. There aren't any good database bindings I could find, so I wrote a small program that reads a record from stdin and writes it to the database. I launch this program from Common Lisp and write records to the input of the program. It works for about twenty records and then hangs. I've tried to be careful to flush and drain all streams from both ends, to no avail.
I have a workaround: start the program, write one record, and quit the program. This doesn't hang and reliably writes a record to the database, but it isn't fast and it is constantly initializing and creating a database connection and tearing it back down for each record.
You'd think that subprocesses communicating via stream of characters would be simple.
01 May 2025 12:29pm GMT
30 Apr 2025
Planet Lisp
Neil Munro: Ningle Tutorial 6: Database Connections
Contents
- Part 1 (Hello World)
- Part 2 (Basic Templates)
- Part 3 (Introduction to middleware and Static File management)
- Part 4 (Forms)
- Part 5 (Environmental Variables)
- Part 6 (Database Connections)
Introduction
Welcome back, in this tutorial we will begin looking at how to work with SQL databases, specifically SQLite3, MySQL, and PostgreSQL. We will be using the mito ORM to create user models and save them to the database using the form
we created previously. Mito
itself is a basic ORM and includes several mixins to provide additional functionality, we will use one called mito-auth to provide password hashing and salting.
It is important to know that mito
is based on top of a SQL library known as SXQL, we will occasionally use SXQL
to write queries with mito
, while it's not always required to use SXQL, there are times where it will make life easier. To achieve this, I elected to :use
SXQL in my package definition.
(defpackage ningle-tutorial-project
(:use :cl :sxql)
Part of working with databases using an ORM is creating the initial database/tables and managing changes over time, called migrations
, mito
appears to have a migrations system, although I was unable to get it working, but I developed a means by which to apply migrations, so perhaps in a future tutorial the subject can be revisited. As such, in addition to seeing how to connect to the respective SQL databases, we will write implementation specific migration functions.
We will follow the example of setting up a secure user registration system across all three SQL implementations. One thing to bear in mind is that it is beyond the scope of this tutorial to instruct how to setup MySQL or PostgreSQL, I would recommend learning how to set them up using docker. All that having been said, let's have a look at the different databases and how to connect to them!
Please bear in mind that when working with SQLite remember to add .db
to your .gitignore
as you most certainly don't want to accidentally commit a database into git! SQLite, being a file based database (unlike MySQL and PostgreSQL) will create a file that represents your database so this step only applies to SQLite.
Installing Packages
To begin with we will need to ensure we have installed and added the packages we need to our project asd file, there are three that we will be installing:
As normal you will need to add them in the :depends-on
section. Please note however that there is an issue in mito-auth
that prevents it from working in MySQL, I have submitted a fix but it has not been merged yet, so for now you can use my branch, if you do, please ensure you check it out via git into your quicklisp/local-projects
directory.
:depends-on (:clack
:cl-dotenv
:djula
:cl-forms
:cl-forms.djula
:cl-forms.ningle
:ingle
:mito
:mito-auth
:ningle)
Mito is a package for managing models/tables in our application, mito-auth is a mixin
that enables models to have a secure password field, not all models will need this, but our user model will! ingle
is a small library that includes some very useful utilities, one of which is a redirect
function which will be very useful indeed!
Now that that is done, we must set up the middleware, you might remember from Part 3 that middleware is placed in the lack.builder:builder
function call in our start
function.
SQL Middleware
Mito provides middleware to establish and manage database connections for SQLite3
, MySQL
, and PostgreSQL
, when you build your solution you will need to pick a database implementation, for production systems I suggest PostgreSQL
, but if you are just starting out, you can use SQLite
.
SQLite3
(defun start (&key (server :woo) (address "127.0.0.1") (port 8000))
(djula:add-template-directory (asdf:system-relative-pathname :ningle-tutorial-project "src/templates/"))
(djula:set-static-url "/public/")
(clack:clackup
(lack.builder:builder
:session
`(:mito
(:sqlite3
:database-name ,(uiop:getenv "SQLITE_DB_NAME")))
(:static
:root (asdf:system-relative-pathname :ningle-tutorial-project "src/static/")
:path "/public/")
*app*)
:server server
:address address
:port port))
MySQL
(defun start (&key (server :woo) (address "127.0.0.1") (port 8000))
(djula:add-template-directory (asdf:system-relative-pathname :ningle-tutorial-project "src/templates/"))
(djula:set-static-url "/public/")
(clack:clackup
(lack.builder:builder
:session
`(:mito
(:mysql
:database-name ,(uiop:native-namestring (uiop:parse-unix-namestring (uiop:getenv "MYSQL_DB_NAME")))
:username ,(uiop:getenv "MYSQL_USER")
:password ,(uiop:getenv "MYSQL_PASSWORD")
:host ,(uiop:getenv "MYSQL_ADDRESS")
:port ,(parse-integer (uiop:getenv "MYSQL_PORT"))))
(:static
:root (asdf:system-relative-pathname :ningle-tutorial-project "src/static/")
:path "/public/")
*app*)
:server server
:address address
:port port))
PostgreSQL
(defun start (&key (server :woo) (address "127.0.0.1") (port 8000))
(djula:add-template-directory (asdf:system-relative-pathname :ningle-tutorial-project "src/templates/"))
(djula:set-static-url "/public/")
(clack:clackup
(lack.builder:builder
:session
`(:mito
(:postgres
:database-name ,(uiop:native-namestring (uiop:parse-unix-namestring (uiop:getenv "POSTGRES_DB_NAME")))
:username ,(uiop:getenv "POSTGRES_USER")
:password ,(uiop:getenv "POSTGRES_PASSWORD")
:host ,(uiop:getenv "POSTGRES_ADDRESS")
:port ,(parse-integer (uiop:getenv "POSTGRES_PORT"))))
(:static
:root (asdf:system-relative-pathname :ningle-tutorial-project "src/static/")
:path "/public/")
*app*)
:server server
:address address
:port port))
Testing The Connection
Before we go further with building models and migration functions, we should test that the connections work and the most basic of SQL statements. We will be working on our register controller, so that seems like as good a place as any to place a simple check.
(setf (ningle:route *app* "/register" :method '(:GET :POST))
(lambda (params)
(let ((query (mito:retrieve-by-sql "SELECT 2 + 3 AS result")))
(format t "Test: ~A~%" query))
(let ((form (cl-forms:find-form 'register)))
...
Here, in the controller we have added a small (temporary) check to ensure that the database connections are set up correctly, when you run the application and perform a GET
request on this route, you should see the output printed in the console for:
Test: ((RESULT 5))
It might look a little odd, but rest assured that this is proof that everything is right and the connection works! We will be removing this later as it serves just as a small check. So with that done, we can begin to look into writing our first model, our user model.
Creating Models
Models are a way to represent both a generic object, and any specific object of that type in a relational database system. For example you might have a Book model, that represents a book table, however a book is just a way to classify something any doesn't tell you anything specific about any individual book. So here we will create a User model, that refers to all users, but each instance of a User will contain the specific information about any given user.
We will create a new file called models.lisp
:
(defpackage ningle-tutorial-project/models
(:use :cl :mito)
(:export #:user))
(in-package ningle-tutorial-project/models)
(deftable user (mito-auth:has-secure-password)
((email :col-type (:varchar 255) :initarg :email :accessor email)
(username :col-type (:varchar 255) :initarg :username :accessor username))
(:unique-keys email username))
Now, mito
provides a deftable
macro
that hides some of the complexities, there is a way to use a regular class and change the metaclass
, but it's much less typing and makes the code look nicer to use the deftable
syntax. It is important to note however that we use the mixin
from mito-auth
called has-secure-password
. Obviously this mixin wouldn't be needed in all of our models, but because we are creating a user that will log into our system, we need to ensure we are handling passwords securely.
Writing Migrations
Now that we have this we need to write the migration code I mentioned earlier, databases (and their models) change over time as application requirements change, as columns get added, removed, changed, etc it can be tricky to get right and you certainly would prefer to have these done automatically, a stray SQL query in the wrong connected database can do incredible damage (trust me, I know!), so migrations allow us to track these changes and have the database system manage them for us.
This code will set up connections to the implementation we want to use and delegate migrations to mito, so pick your implementation and place it in migrations.lisp
.
SQLite3
(defpackage ningle-tutorial-project/migrations
(:use :cl :mito)
(:export #:migrate))
(in-package :ningle-tutorial-project/migrations)
(defun migrate ()
"Explicitly apply migrations when called."
(dotenv:load-env (asdf:system-relative-pathname :ningle-tutorial-project ".env"))
(format t "Applying migrations...~%")
(mito:connect-toplevel
:sqlite3
:database-name (uiop:native-namestring (uiop:parse-unix-namestring (uiop:getenv "SQLITE_DB_NAME"))))
(mito:ensure-table-exists 'ningle-tutorial-project/models:user)
(mito:migrate-table 'ningle-tutorial-project/models:user)
(mito:disconnect-toplevel)
(format t "Migrations complete.~%"))
MySql
(defpackage ningle-tutorial-project/migrations
(:use :cl :mito)
(:export #:migrate))
(in-package :ningle-tutorial-project/migrations)
(defun migrate ()
"Explicitly apply migrations when called."
(dotenv:load-env (asdf:system-relative-pathname :ningle-tutorial-project ".env"))
(format t "Applying migrations...~%")
(mito:connect-toplevel
:mysql
:database-name (uiop:native-namestring (uiop:parse-unix-namestring (uiop:getenv "MYSQL_DB_NAME")))
:username (uiop:getenv "MYSQL_USER")
:password (uiop:getenv "MYSQL_PASSWORD")
:host (uiop:getenv "MYSQL_ADDRESS")
:port (parse-integer (uiop:getenv "MYSQL_PORT")))
(mito:ensure-table-exists 'ningle-tutorial-project/models:user)
(mito:migrate-table 'ningle-tutorial-project/models:user)
(mito:disconnect-toplevel)
(format t "Migrations complete.~%"))
PostgreSQL
(defpackage ningle-tutorial-project/migrations
(:use :cl :mito)
(:export #:migrate))
(in-package :ningle-tutorial-project/migrations)
(defun migrate ()
"Explicitly apply migrations when called."
(dotenv:load-env (asdf:system-relative-pathname :ningle-tutorial-project ".env"))
(format t "Applying migrations...~%")
(mito:connect-toplevel
:postgres
:database-name (uiop:getenv "POSTGRES_DB_NAME")
:host (uiop:getenv "POSTGRES_ADDRESS")
:port (parse-integer (uiop:getenv "POSTGRES_PORT"))
:username (uiop:getenv "POSTGRES_USER")
:password (uiop:getenv "POSTGRES_PASSWORD"))
(mito:ensure-table-exists 'ningle-tutorial-project/models:user)
(mito:migrate-table 'ningle-tutorial-project/models:user)
(mito:disconnect-toplevel)
(format t "Migrations complete.~%"))
It will be necessary to add these two files into the :components
section of your project asd file.
:components ((:module "src"
:components
((:file "models")
(:file "migrations")
(:file "forms")
(:file "main"))))
Just remember if you are using MySQL or PostgreSQL, you will need to ensure that the database you want to connect to exists (in our case ntp), and that your connecting user has the correct permissions!
Running Migrations
Now that everything is set up, we will need to perform our initial migrations:
(ningle-tutorial-project/migrations:migrate)
If this has worked, you will see a lot of output SQL statements, it's quite verbose, however this only means that it is working and we can move onto actually creating and saving models.
Removing Connection Check
Now that we have migrations and models working we should remember to remove this verification code that we wrote earlier.
(let ((query (mito:retrieve-by-sql "SELECT 2 + 3 AS result")))
(format t "Test: ~A~%" query))
Registering & Querying Users
What we are going to do now is use the user register form and connect it to our database, because we are registering users we will have to do some checks to ensure since we stated that usernames and email addresses are unique, we might want to raise an error.
(when valid
(cl-forms:with-form-field-values (email username password password-verify) form
(when (mito:select-dao 'ningle-tutorial-project/models:user
(where (:or (:= :username username)
(:= :email email))))
(error "Either username or email is already registered"))
We can see from this snippet here that mito uses the SXQL Domain Specific Language for expressing SQL statements. Using the select-dao
we can query the user table and apply where
clauses using a more Lispy like syntax to check to see if an account with the username or email already exists. Such DSLs are common when interacting with SQL inside another programming language, but it's good to know that from what we learned earlier that it can handle arbitrary SQL strings or this more Lispy syntax, so you can use pure SQL syntax, if necessary.
While having this check isn't necessary, it does make the error handling somewhat nicer, as well as exploring parts of the mito api. We will also add a check to raise an error if the passwords submitted in the form do not match each other.
(when (string/= password password-verify)
(error "Passwords do not match"))
If both of these pass (and you can test different permutations of course), we can continue to using mito to create our first user object!
(mito:create-dao 'ningle-tutorial-project/models:user
:email email
:username username
:password password)
The final thing to add is that we should redirect to another route, which we can do with the ingle:redirect
function.
(ingle:redirect "/people")
You will notice that we are redirecting to a route that doesn't (yet) exist, we will write the controller below after we have finished this controller, the multiple-value-bind
section of which, when completed, looks like this:
(multiple-value-bind (valid errors)
(cl-forms:validate-form form)
(when errors
(format t "Errors: ~A~%" errors))
(when valid
(cl-forms:with-form-field-values (email username password password-verify) form
(when (mito:select-dao 'ningle-tutorial-project/models:user
(where (:or (:= :username username)
(:= :email email))))
(error "Either username or email is already registered"))
(when (string/= password password-verify)
(error "Passwords do not match"))
(mito:create-dao 'ningle-tutorial-project/models:user
:email email
:username username
:password password)
(ingle:redirect "/people"))))
Getting All Users
We will look at two final examples of using mito before we finish this tutoral, as mentioned earlier we will write a new /people
controller, which will list all the users registered in the system, and we will create a /people/:person
controller to show the details of an individual user.
Starting with the /people
controller, we create a controller like we have seen before, we then use a let
binding to store the result of (mito:retrieve-dao 'ningle-tutoral-project/model:user)
, this is how we would get all rows from a table represented by the class 'ningle-tutorial-project/models:user
. We then pass the results into a template.
(setf (ningle:route *app* "/people")
(lambda (params)
(let ((users (mito:retrieve-dao 'ningle-tutorial-project/models:user)))
(djula:render-template* "people.html" nil :title "People" :users users))))
The html for this is written as such:
{% extends "base.html" %}
{% block content %}
<div class="container">
<div class="row" >
<div class="col-12">
{% for user in users %}
<div class="row mb-4">
<div class="col">
<div class="card">
<div class="card-body">
<h5 class="card-title"><a href="/people/{{ user.username }}">{{ user.username }}</a></h5>
<p class="card-text"><a href="/people/{{ user.email }}">{{ user.email }}</a></p>
<p class="text-muted small"></p>
</div>
</div>
</div>
</div>
{% endfor %}
{% if not users %}
<div class="row">
<div class="col text-center">
<p class="text-muted">No users to display.</p>
</div>
</div>
{% endif %}
</div>
</div>
</div>
{% endblock %}
Getting A Single User
In our individual person view, we see how a route may have variable data, our :person
component of the URL string, this will be either a username or email, it doesn't really matter which as we can have a SQL query that will find a record that will match the :person
string with either the username or email. We also take advantage of another ingle
function, the get-param
, which will get the value out of :person
. We use a let*
binding to store the user
derived from :person
and the result of mito:select-dao
(using the person
), we then pass the user
object into a template.
As we saw before this query was used to check for the existence of a username or email address in our register
controller.
(setf (ningle:route *app* "/people/:person")
(lambda (params)
(let* ((person (ingle:get-param :person params))
(user (first (mito:select-dao
'ningle-tutorial-project/models:user
(where (:or (:= :username person)
(:= :email person)))))))
(djula:render-template* "person.html" nil :title "Person" :user user))))
And here is the template for an individual user.
{% extends "base.html" %}
{% block content %}
<div class="container">
<div class="row">
<div class="col-12">
<div class="row mb-4">
<div class="col">
{% if not user %}
<h1>No users</h1>
{% else %}
<div class="card">
<div class="card-body">
<h5 class="card-title">{{ user.username }}</h5>
<p class="card-text">{{ user.email }}</p>
<p class="text-muted small"></p>
</div>
</div>
{% endif %}
</div>
</div>
</div>
</div>
</div>
{% endblock %}
Conclusion
This was a rather large chapter and we covered a lot, looking at the different means by which to connect to a SQL database, defining models, running migrations and executing queries, of course we are just getting started but this is a massive step forward and our application is beginning to take shape. I certainly hope you have enjoyed it and found it useful!
To recap, after working your way though this tutorial you should be able to:
- Explain what a model is
- Explain what a migration is
- Write code to connect to a SQL database
- Implement a model
- Implement a migration
- Execute a migration
- Write controllers that write information to a database via a model
- Write controllers that read information from a database via a model
Github
- The link for the SQLite version of this tutorial code is available here.
- The link for the MySQL version of this tutorial code is available here.
- The link for the PostgreSQL version of this tutorial code is available here.
Resources
30 Apr 2025 9:30pm GMT
28 Apr 2025
Planet Lisp
Paolo Amoroso: Adding window clearing and message printing to DandeGUI
I continued working on DandeGUI, a GUI library for Medley Interlisp I'm writing in Common Lisp. I added two new short public functions, GUI:CLEAR-WINDOW
and GUI:PRINT-MESSAGE
, and fixed a bug in some internal code.
GUI:CLEAR-WINDOW
deletes the text of the window associated with the Interlisp TEXTSTREAM
passed as the argument:
(DEFUN CLEAR-WINDOW (STREAM)
"Delete all the text of the window associated with STREAM. Returns STREAM"
(WITH-WRITE-ENABLED (STR STREAM)
(IL:TEDIT.DELETE STR 1 (IL:TEDIT.NCHARS STR)))
STREAM)
It's little more than a call to the TEdit API function IL:TEDIT.DELETE
for deleting text in the editor buffer, wrapped in the internal macro GUI::WITH-WRITE-ENABLED
that establishes a context for write access to a window.
I also wrote GUI:PRINT-MESSAGE
. This function prints a message to the prompt area of the window associated with the TEXTSTREAM
passed as an argument, optionally clearing the area prior to printing. The prompt area is a one-line Interlisp prompt window attached above the title bar of the TEdit window where the editor displays errors and status messages.
(DEFUN PRINT-MESSAGE (STREAM MESSAGE &OPTIONAL DONT-CLEAR-P)
"Print MESSAGE to the prompt area of the window associated with STREAM. If DONT-CLEAR-P is non NIL the area will be cleared first. Returns STREAM."
(IL:TEDIT.PROMPTPRINT STREAM MESSAGE (NOT DONT-CLEAR-P))
STREAM)
GUI:PRINT-MESSAGE
just passes the appropriate arguments to the TEdit API function IL:TEDIT.PROMPTPRINT
which does the actual printing.
The documentation of both functions is in the API reference on the project repo.
Testing DandeGUI revealed that sometimes text wasn't appended to the end but inserted at the beginning of windows. To address the issue I changed GUI::WITH-WRITE-ENABLED
to ensure the file pointer of the stream is set to the end of the file (i.e -1
) prior to passing control to output functions. The fix was to add a call to the Interlisp function IL:SETFILEPTR
:
(IL:SETFILEPTR ,STREAM -1)
#DandeGUI #CommonLisp #Interlisp #Lisp
Discuss... Email | Reply @amoroso@oldbytes.space
28 Apr 2025 10:52am GMT
27 Apr 2025
Planet Lisp
Joe Marshall: Senior Programmers Have Nothing to Fear From AI
I have, through experimentation, discovered that vibe coding in Common Lisp is not effective. But other kinds of AI coding are highly effective and have been saving me hours of work. AI is not going to replace senior programmers, but it will take over many of the tasks that junior programmers do. I'm not worried about my job, but were I a junior programmer, I'd be concerned.
Part of my job as a senior programmer is to identify tasks that are suitable for junior programmers. These tasks have certain properties:
- They are well-defined.
- They are repetitive, making them suitable for development of a program to carry them out.
- They are not too difficult, so that a junior programmer can complete them with a little help.
- They have a clear acceptance criteria, so that the junior programmer can tell when they are done.
- They have a clear abstraction boundary so that integrating the code after the junior programmer is done is not too difficult.
But because junior programmers are human, we have to consider these factors as well:
- The task must not be too simple or too boring.
- The task must be written in a popular programming language. Junior programmers don't have the inclination to learn new programming languages.
- The task must not be time critical because junior programmers are slow.
- The task should not be core critical to the larger project. Junior programmers write crappy code, and you don't want crappy code at the heart of your project.
Oftentimes, I find some tasks that fits many of these criteria, but then I find that I can do it myself better and faster than a junior programer could.
AI coding can handle many of the tasks that I would otherwise assign to a junior programmer. It works best when the task is well defined, not too difficult, and written in a popular language. It doesn't care if the task is boring and repetitive. AI coding is much faster than a junior programmer, and it writes code that tends to follow standard conventions. If you can specify good abstraction barriers, the AI can do a good job of coding to them. While AI coding is not perfect, neither are junior programmers. In either case, a senior programmer needs to carefully review the code.
AI coding is not going to replace senior programmers. The AI will not generate code without a prompt, and the better the prompt, the better the generated code. Senior programmers can take a large program and break it down into smaller tasks. They can create definitions of the smaller tasks and define the acceptance criteria, the API, and the abstractions to be used. They can carefully and precisely craft the prompts that generate the code. Senior programmers are needed to drive the AI.
Which leads to the question of where senior programmers will come from if junior programmers are no longer needed. I don't have a good answer for this.
27 Apr 2025 2:33pm GMT
26 Apr 2025
Planet Lisp
Nicolas Martyanoff: Working with Common Lisp pathnames
Common Lisp pathnames, used to represent file paths, have the reputation of being hard to work with. This article aims to change this unfair reputation while highlighting the occasional quirks along the way.
Filenames and file paths
The distinction between filename and file paths is not always obvious. On POSIX systems, the filename is the name of the file, while a file path represents its absolute or relative location in the file system. Which also means that all filenames are file paths, but not the other way around.
Common Lisp uses the term filename for objects which are either pathnames or namestrings, both being representation of file paths. We will try to avoid confusion by using the terms filenames, pathnames and namestrings when referring to Common Lisp concepts and we will talk about file paths when referring to the language-agnostic file system concept.
Pathnames
Pathnames are an implementation-independent representation of file paths made of six components:
- an host identifying either the file system or a logical host;
- a device identifying the logical of physical device containing the file;
- a directory representing an absolute or relative list of directory names;
- a name;
- a type, a value nowadays known as file extension;
- a version, because yes file systems used to support file versioning.
While this representation might seem way too complicated -it originates from a time where the file system ecosystem was much richer- it still is suitable for modern file systems.
The make-pathname
function is used to create pathnames and lets you specificy all components. For example the following call yields a pathname representing the file path represented on POSIX systems by /var/run/example.pid
:
(make-pathname :directory '(:absolute "var" "run") :name "example" :type "pid")
Common Lisp functions manipulating file paths of course accept pathnames, letting you keep the same convenient structured representation everywhere, only converting from/to a native representation at the boundaries of your program.
Special characters
What happens when you construct a pathname with components containing separation characters, e.g. a directory name containing /
on a POSIX system or a type containing .
? According to Common Lisp 19.2.2.1.1, the behaviour is implementation-defined; but if the implementation accepts these component values it must handle quoting correctly.
For example:
- CLISP rejects separator characters in component values, signaling a
SIMPLE-ERROR
condition. - CCL accepts them and quotes them when converting the pathname to a namestring. So
(namestring (make-pathname :name "foo/bar" :type "a.b"))
yields"foo\\/bar.a\\.b"
on Linux. - SBCL accepts and quotes them but does not quote
.
in type components, yielding"foo\\/bar.a.b"
for the example above. - ECL accepts them but fails to quote them when converting the pathname to a namestring.
One could wonder about which implementation, CCL or SBCL, is correct regarding the quoting of the .
character in type strings on POSIX platforms. While everyone understands that /
is special in file and directory names, .
is debatable because POSIX does not mention the type extension in its definitions: foo.txt
is the name of the file, not a combination of a name and a type. As such, I would argue that quoting and not quoting are both correct. And as you will realize then reading about namestrings further in this article, it is irrelevant since namestrings are not POSIX paths.
Note that whether ECL violates the standard or not is unclear since there is no character quoting for POSIX paths. In other words, there is no such thing as a directory named a/b
, because it could not be referenced in a way different from a directory named b
in a directory named a
. This behaviour derives directly from POSIX systems treating paths as strings and not as structured objects.
Invalid characters
The Common Lisp standard mentions special characters but is silent on the subject of invalid characters. For example POSIX forbids null bytes in filenames. But since it is not a separation character, implementations are free to deal with it as they see fit.
When testing implementations with a pathname containing a null byte using (make-pathname :name (string (code-char 0)))
, CCL, SBCL and ECL accept it while CLISP signals an error mentioning an illegal argument.
I am not convinced by CLISP's behaviour since null bytes are only invalid in POSIX paths, not in Common Lisp filenames, meaning that the error should occur when the pathname is internally converted to a format usable by the operating system.
Pathname component case
A rarely mentioned property of pathnames is the support for case conversion. MAKE-PATHNAME
and function returning pathname components (e.g. PATHNAME-TYPE
) support a :CASE
argument, either :COMMON
or :LOCAL
indicating how how to handle character case in strings.
With :LOCAL
-which is the default value-, these functions assume that component strings are already represented following the conventions of the underlying operating system. It also dictates that if the host only supports one character case, strings must be returned converted to this case.
With :COMMON
, these functions will use the default (customary) case of the host if the string is provided all uppercase, and the opposite case if the string is provided all lowercase. Mixed case strings are not transformed.
These behaviours are not intuitive and made much more sense at a time where some file systems only supported one specific case. You should probably stay away from component case handling unless you really know what you are doing.
On a personal note, as someone running Linux and FreeBSD, I am curious about the behaviour of various implementations on Windows and MacOS since both NTFS and APFS are case insensitive.
Unspecific components
While all components can be null, some of them can be :UNSPECIFIC
(which ones is implementation-defined). The only real use case for :UNSPECIFIC
is to affect the behaviour of MERGE-PATHNAMES
: if a component is null, the function uses the value of the component in the pathname passed as the :DEFAULTS
argument; if a component is :UNSPECIFIC
, the function uses the same value in the resulting pathname.
For example:
(merge-pathnames (make-pathname :name "foo")
(make-pathname :type "txt"))
yields the "foo.txt"
namestring, but
(merge-pathnames (make-pathname :name "foo" :type :unspecific)
(make-pathname :type "txt"))
yields "foo"
.
Unfortunately the inability to rely on its support for specific component types (since it is implementation-defined) makes it interesting more than useful.
Namestrings
Namestrings are another represention for file paths. While pathnames are structured objects, namestrings are just strings. The most important aspect of namestrings is that unless they are logical namestrings (something we will cover later), the way they represent paths is implementation-defined (c.f. Common Lisp 19.1.1 Namestrings as Filenames). In other words the namestring for the file foo
of type txt
in directory data
could be data/foo.txt
. Or data\foo.txt
. Or data|foo#txt
. Or any other non-sensical representation. Fortunately implementations tend to act rationally and use a representation as similar as possible to the one of their host operating system.
One should always remember that even though namestrings look and feel like paths, they are still a representation of a Common Lisp pathname, meaning that they may or may not map to a valid native path. The most obvious example would be a pathname whose name is the null byte, created with (make-pathname :name (string (code-char 0)))
, whose namestring is a one character string that has no valid native representation on modern operating systems.
Pathnames can be converted to namestrings using the NAMESTRING
function, while namestrings can be parsed into pathnames with PARSE-NAMESTRING
. The #P
reader macro uses PARSE-NAMESTRING
to read a pathname. As such, #P"/tmp/foo.txt"
is identical to #.(parse-namestring '"/tmp/foo.txt")
.
Note that most functions dealing with files will accept a pathname designator, i.e. either a pathname, a namestring or a stream.
Native namestrings
An unfortunately missing feature from Common Lisp is the ability to parse native namestrings, i.e. paths that use the representation of the underlying operating system.
To understand why it is a problem, let us take *.txt
, a perfectly valid filename at least on any POSIX platform. In Common Lisp, you can construct a pathname representing this filename with (make-pathname :name "*" :type "txt")
. No problem whatsoever. However the "*.txt"
namestring actually represents a pathname whose name component is :WILD
. There is no namestring that will return this pathname when passed to PARSE-NAMESTRING
.
As a result, when processing filenames coming from the external world (a command line argument, a list of paths in a document, etc.), you have no way to handle those that contain characters used by Common Lisp for wild components.
There is no standard way of solving this issue. Some implementations provide functions to parse native namestrings, e.g. SBCL with SB-EXT:PARSE-NATIVE-NAMESTRING
or CCL with CCL:NATIVE-TO-PATHNAME
. If you use ASDF, you can also use UIOP:PARSE-NATIVE-NAMESTRING
.
Wildcards
Up to now pathnames may have looked like a slightly unusual representation for paths. But we are just getting started.
Pathname can be wild, meaning that they contain one or more wild components. Wild components can match any value. All components can be made wild with the special value :WILD
. Directory elements also support :WILD-INFERIORS
which matches one or more directory levels.
As such
(make-pathname :directory '(:absolute "tmp" :wild) :name "foo" :type :wild)
is equivalent to the /tmp/*/foo.*
POSIX glob pattern, while
(make-pathname :directory '(:absolute "tmp" :wild-inferiors "data" :wild)
:name :wild :type :wild)
is equivalent to /tmp/**/data/*/*.*
.
Wild pathnames only really make sense for the DIRECTORY
function which returns files matching a specific pathname.
Logical pathnames
We currently have talked about pathnames representing either paths to physical files or pattern of filenames. Logical pathnames go further and let you work with files in a location-independent way.
Logical pathnames are based on logical hosts, set as pathname host components. Logical pathnames can be passed around and manipulated as any other pathnames; when used to access files, they are translated to a physical pathname, i.e. a pathname referring to an actual file in the file system.
SBCL uses logical pathnames for source file locations. While SBCL is shipped with its source files, their actual location on disk depends on how the software was installed. Instead of manually merging pathnames with a base directory value everywhere, SBCL uses the SYS
logical host to map all pathnames whose directory starts with SRC
to the actual location on disk. For example on my machine:
(translate-logical-pathname "SYS:SRC;ASSEMBLY;MASTER.LISP")
yields #P"/usr/share/sbcl-source/src/assembly/master.lisp"
.
Another example would be CCL which maps pathnames with the HOME
logical host to subpaths of the home directory of the user.
Note that logical hosts are global to the Common Lisp environment. While SYS
is reserved for the implementation, all other hosts are free to use by anyone. To avoid collisions, it is a good idea to name hosts after their program or library.
Logical namestrings
Logical namestrings are implementation-independent, meaning that you can safely use them in your programs without wondering about how they will be interpreted. Their syntax, detailed in section 19.3.1 of the Common Lisp standard, is quite different from usual POSIX paths. The host is separated from the rest of the path by a colon character, and directory names are separated by semicolon characters.
For example "SOURCE:SERVER;LISTENER.LISP"
is the logical namestring equivalent of the /server/listener.lisp
POSIX path for the SOURCE
logical host.
The astute reader will notice the use of uppercase characters in logical namestrings. It happens that the different parts of a logical namestring are defined as using uppercase characters, but that the implementation translates lowercase letters to uppercase letters when parsing the namestrings (c.f. Common Lisp 19.3.1.1.7). We use the canonical uppercase representation for clarity.
Translation
Translation is controlled by a table that maps logical hosts to a list of pattern (wild pathnames or namestrings) and their associated wild physical pathnames.
One can obtain the list of translations for a logical host with LOGICAL-PATHNAME-TRANSLATIONS
and update it with (SETF LOGICAL-PATHNAME-TRANSLATIONS)
. Each translation is a list where the first element is a logical pathname or namestring (usually a wild pathname) and the second element is a pathname or namestring to translate into.
The translation process looks for the first entry that satisfies PATHNAME-MATCH-P
, which is guaranteed to behave in a way consistent with DIRECTORY
. When there is match, the translation processes replaces corresponding patterns for each components.
And of course if translation results in a logical pathname, it will be recursively translated until a physical pathname is obtained.
A simple example would be the use of a logical host referring to a temporary directory. This lets a program manipulates temporary pathnames without having to know their actual physical location, the translation process being controlled in a single location.
(setf (logical-pathname-translations "tmp")
(list (list (make-pathname :host "tmp"
:directory '(:absolute :wild-inferiors)
:name :wild :type :wild)
(make-pathname :directory '(:absolute "var" "tmp" :wild-inferiors)
:name :wild :type :wild))))
or if we were to use namestrings:
(setf (logical-pathname-translations "tmp")
'(("TMP:**;*.*.*" "/var/tmp/**/*.*")))
Translating pathnames or namestrings using the TMP
logical host yields the expected results. For example (translate-logical-pathname "TMP:CACHE;DATA.TXT")
yields #P"/var/tmp/cache/data.txt"
.
Caveats
While logical pathnames are an elegant abstraction, they are plagued by multiple issues that make them hard to use correctly and in a portable way.
Logical namestring components can only contain letters, digits and hyphens (or the *
and **
sequences for wild namestrings). This limitation probably comes from a need to be compatible with all existing file systems, but it can be a showstopper if one needs to refer to files whose naming scheme is not controlled by the program.
Namestring parsing is confusing: calling PARSE-NAMESTRING
on an invalid namestring (because it contains invalid characters or because the host is not a known logical host) will not fail. Instead the string will be parsed as a physical namestring, introducing silent bugs. The LOGICAL-PATHNAME
can be used to validate logical pathnames and namestrings.
The way translation converts between both pathname patterns is unclear. It is not specified by the Common Lisp standard. Debugging patterns can quickly become very frustrating, especially with implementations unable to produce quality error diagnostics.
Finally, the behaviour of logical pathnames with other functions is rarely obvious, leading to frustrating debugging sessions.
They nevertheless are a unique and helpful feature for very specific use cases.
Recipes
Resolving a path
Files are accessible through multiple paths. For example, on POSIX systems, foo/bar/baz.txt
, foo/bar/../bar/baz.txt
refer to the same file. If your operating system and file system support symbolic links, you can refer to the same physical file from multiple links, themselves being files.
It is sometimes useful to obtain the canonical path of a file. On POSIX systems, the realpath
function serves this purpose. In Common Lisp, this canonical path is called truename, and the TRUENAME
function returns it.
Transforming paths
The :DEFAULTS
option of MAKE-PATHNAME
is useful to construct a pathname that is a variation of another pathname. When a component passed to MAKE-PATHNAME
is null, the value is taken from the pathname passed with :DEFAULTS
.
For example to create the pathname of a file in the same directory as another pathname:
(make-pathname :name "bar"
:defaults (make-pathname :directory '(:absolute "tmp") :name "foo"))
Or to create a wild pathname matching the same file names but with any extension:
(make-pathname :type :wild
:defaults (make-pathname :name "foo" :type "txt"))
Or to obtain a pathname for the directory of a file:
(make-pathname :name nil
:defaults (make-pathname :directory '(:relative "a" "b" "c")
:name "foo"))
Joining two paths
Joining (or concatenating) two paths can be done with MERGE-PATHNAMES
. In general calling (MERGE-PATHNAMES PATH1 PATH2)
returns a new pathname whose components are taken either from PATH1
when they are not null, or from PATH2
when they are. As a special case, if the directory component of PATH1
is relative, the directory component of the result pathname is the concatenation of the directory components of both paths.
In other words
(merge-pathnames (make-pathname :directory '(:relative "x" "y"))
(make-pathname :directory '(:absolute "a" "b" "c")))
yields "/a/b/c/x/y/"
but
(merge-pathnames (make-pathname :directory '(:absolute "x" "y"))
(make-pathname :directory '(:absolute "a" "b" "c")))
yields "/x/y/"
.
Finding files
The DIRECTORY
function returns files matching a pathname, wild or not.
If the pathname is not wild, DIRECTORY
returns a list of one or zero element depending on whether a file exists at this location or not.
If the pathname is wild, DIRECTORY
behaves similarly to POSIX globs. Due to the way pathnames are structured, with the name and type being two different components, a common error is to specify a wild name without a type. In this case, DIRECTORY
will not return any file with an extension (since their pathname has a non-null type). To match all files with any extension, set both the name and the type to :WILD
.
Another interesting possibility is to only match directories. Directories are represented by pathnames with a non-null directory component and a null name component. Therefore to find all directories in /tmp
(top-level only):
(directory (make-pathname :directory '(:absolute "tmp" :wild)))
Note that DIRECTORY
returns truenames, i.e. pathnames representing the canonical location of the files. An unexpected consequence is that the function will resolve symlinks. Since the Common Lisp standard explicitely allows extra optional arguments, some implementations have a way to disable symlink resolving, e.g. SBCL with :RESOLVE-SYMLINKS
or CCL with :FOLLOW-LINKS
.
Resolving tildes in paths
It is commonly believed that tilde characters in paths is a universal feature. It is not. Tilde prefixes are defined in POSIX in the context of the shell (cf. POSIX 2017 2.6.1 Tilde Expansion) and are only supported in very specific locations.
To obtain the path of a file relative to the home directory of the current user, use the USER-HOMEDIR-PATHNAME
function.
For example:
(merge-pathnames (make-pathname :directory '(:relative ".emacs.d")
:name "init" :type "el")
(user-homedir-pathname))
26 Apr 2025 6:00pm GMT