21 Aug 2025
Planet Lisp
TurtleWare: Using Common Lisp from inside the Browser
Table of Contents
- Scripting a website with Common Lisp
- JS-FFI - low level interface
- LIME/SLUG - interacting from Emacs
- Injecting CL runtime in arbitrary websites
- Current Caveats
- Funding
Web Embeddable Common Lisp is a project that brings Common Lisp and the Web Browser environments together. In this post I'll outline the current progress of the project and provide some technical details, including current caveats and future plans.
It is important to note that this is not a release and none of the described APIs and functionalities is considered to be stable. Things are still changing and I'm not accepting bug reports for the time being.
The source code of the project is available: https://fossil.turtleware.eu/wecl/.
Scripting a website with Common Lisp
The easiest way to use Common Lisp on a website is to include WECL and insert script tags with a type "text/common-lisp". When the attribute src is present, then first the runtime loads the script from that url, and then it executes the node body. For example create and run this HTML document from localhost:
<!doctype html>
<html>
<head>
<title>Web Embeddable Common Lisp</title>
<link rel="stylesheet" href="https://turtleware.eu/static/misc/wecl-20250821/easy.css" />
<script type="text/javascript" src="https://turtleware.eu/static/misc/wecl-20250821/boot.js"></script>
<script type="text/javascript" src="https://turtleware.eu/static/misc/wecl-20250821/wecl.js"></script>
</head>
<body>
<script type="text/common-lisp" src="https://turtleware.eu/static/misc/wecl-20250821/easy.lisp" id='easy-script'>
(defvar *div* (make-element "div" :id "my-ticker"))
(append-child [body] *div*)
(dotimes (v 4)
(push-counter v))
(loop for tic from 6 above 0
do (replace-children *div* (make-paragraph "~a" tic))
(js-sleep 1000)
finally (replace-children *div* (make-paragraph "BOOM!")))
(show-script-text "easy-script")
</script>
</body>
</html>
We may use Common Lisp that can call to JavaScript, and register callbacks to be called on specified events. The source code of the script can be found here:
- https://turtleware.eu/static/misc/wecl-20250821/easy.html
- https://turtleware.eu/static/misc/wecl-20250821/easy.lisp
Because the runtime is included as a script, the browser will usually cache the ~10MB WebAssembly module.
JS-FFI - low level interface
The initial foreign function interface has numerous macros defining wrappers that may be used from Common Lisp or passed to JavaScript.
Summary of currently available operators:
- define-js-variable: an inlined expression, like
document
- define-js-object: an object referenced from the object store
- define-js-function: a function
- define-js-method: a method of the argument, like
document.foobar()
- define-js-getter: a slot reader of the argument
- define-js-setter: a slot writer of the first argument
- define-js-accessor: combines define-js-getter and define-js-setter
- define-js-script: template for JavaScript expressions
- define-js-callback: Common Lisp function reference callable from JavaScript
- lambda-js-callback: anonymous Common Lisp function reference (for closures)
Summary of argument types:
type name | lisp side | js side |
---|---|---|
:object | Common Lisp object | Common Lisp object reference |
:js-ref | JavaScript object reference | JavaScript object |
:fixnum | fixnum (coercible) | fixnum (coercible) |
:symbol | symbol | symbol (name inlined) |
:string | string (coercible) | string (coercible) |
:null | nil | null |
All operators, except for LAMBDA-JS-CALLBACK
have a similar lambda list:
(DEFINE-JS NAME-AND-OPTIONS [ARGUMENTS [,@BODY]])
The first argument is a list (name &key js-expr type)
that is common to all defining operators:
- name: Common Lisp symbol denoting the object
- js-expr: a string denoting the JavaScript expression, i.e "innerText"
- type: a type of the object returned by executing the expression
For example:
(define-js-variable ([document] :js-expr "document" :type :symbol))
;; document
(define-js-object ([body] :js-expr "document.body" :type :js-ref))
;; wecl_ensure_object(document.body) /* -> id */
;; wecl_search_object(id) /* -> node */
The difference between a variable and an object in JS-FFI is that variable expression is executed each time when the object is used (the expression is inlined), while the object expression is executed only once and the result is stored in the object store.
The second argument is a list of pairs (name type)
. Names will be used in the lambda list of the operator callable from Common Lisp, while types will be used to coerce arguments to the type expected by JavaScript.
(define-js-function (parse-float :js-expr "parseFloat" :type :js-ref)
((value :string)))
;; parseFloat(value)
(define-js-method (add-event-listener :js-expr "addEventListener" :type :null)
((self :js-ref)
(name :string)
(fun :js-ref)))
;; self.addEventListener(name, fun)
(define-js-getter (get-inner-text :js-expr "innerText" :type :string)
((self :js-ref)))
;; self.innerText
(define-js-setter (set-inner-text :js-expr "innerText" :type :string)
((self :js-ref)
(new :string)))
;; self.innerText = new
(define-js-accessor (inner-text :js-expr "innerText" :type :string)
((self :js-ref)
(new :string)))
;; self.innerText
;; self.innerText = new
(define-js-script (document :js-expr "~a.forEach(~a)" :type :js-ref)
((nodes :js-ref)
(callb :object)))
;; nodes.forEach(callb)
The third argument is specific to callbacks, where we define Common Lisp body of the callback. Argument types are used to coerce values from JavaScript to Common Lisp.
(define-js-callback (print-node :type :object)
((elt :js-ref)
(nth :fixnum)
(seq :js-ref))
(format t "Node ~2d: ~a~%" nth elt))
(let ((start 0))
(add-event-listener *my-elt* "click"
(lambda-js-callback :null ((event :js-ref)) ;closure!
(incf start)
(setf (inner-text *my-elt*)
(format nil "Hello World! ~a" start)))
Note that callbacks are a bit different, because define-js-callback
does not accept js-expr
option and lambda-js-callback
has unique lambda list. It is important for callbacks to have an exact arity as they are called with, because JS-FFI does not implement variable number of arguments yet.
Callbacks can be referred by name with an operator (js-callback name)
.
LIME/SLUG - interacting from Emacs
While working on FFI I've decided to write an adapter for SLIME/SWANK that will allow interacting with WECL from Emacs. The principle is simple: we connect with a websocket to Emacs that is listening on the specified port (i.e on localhost). This adapter uses the library emacs-websocket
written by Andrew Hyatt.
It allows for compiling individual forms with C-c C-c
, but file compilation does not work (because files reside on a different "host"). REPL interaction works as expected, as well as SLDB. The connection may occasionally be unstable, and until Common Lisp call returns, the whole page is blocked. Notably waiting for new requests is not a blocking operation from the JavaScript perspective, because it is an asynchronous operation.
You may find my changes to SLIME here: https://github.com/dkochmanski/slime/, and it is proposed upstream here: https://github.com/slime/slime/pull/879. Before these changes are merged, we'll patch SLIME:
;;; Patches for SLIME 2.31 (to be removed after the patch is merged).
;;; It is assumed that SLIME is already loaded into Emacs.
(defun slime-net-send (sexp proc)
"Send a SEXP to Lisp over the socket PROC.
This is the lowest level of communication. The sexp will be READ and
EVAL'd by Lisp."
(let* ((payload (encode-coding-string
(concat (slime-prin1-to-string sexp) "\n")
'utf-8-unix))
(string (concat (slime-net-encode-length (length payload))
payload))
(websocket (process-get proc :websocket)))
(slime-log-event sexp)
(if websocket
(websocket-send-text websocket string)
(process-send-string proc string))))
(defun slime-use-sigint-for-interrupt (&optional connection)
(let ((c (or connection (slime-connection))))
(cl-ecase (slime-communication-style c)
((:fd-handler nil) t)
((:spawn :sigio :async) nil))))
Now we can load the LIME adapter opens a websocket server. The source code may be downloaded from https://turtleware.eu/static/misc/wecl-20250821/lime.el:
;;; lime.el --- Lisp Interaction Mode for Emacs -*-lexical-binding:t-*-
;;;
;;; This program extends SLIME with an ability to listen for lisp connections.
;;; The flow is reversed - normally SLIME is a client and SWANK is a server.
(require 'websocket)
(defvar *lime-server* nil
"The LIME server.")
(cl-defun lime-zipit (obj &optional (start 0) (end 72))
(let* ((msg (if (stringp obj)
obj
(slime-prin1-to-string obj)))
(len (length msg)))
(substring msg (min start len) (min end len))))
(cl-defun lime-message (&rest args)
(with-current-buffer (process-buffer *lime-server*)
(goto-char (point-max))
(dolist (arg args)
(insert (lime-zipit arg)))
(insert "\n")
(goto-char (point-max))))
(cl-defun lime-client-process (client)
(websocket-conn client))
(cl-defun lime-process-client (process)
(process-get process :websocket))
;;; c.f slime-net-connect
(cl-defun lime-add-client (client)
(lime-message "LIME connecting a new client")
(let* ((process (websocket-conn client))
(buffer (generate-new-buffer "*lime-connection*")))
(set-process-buffer process buffer)
(push process slime-net-processes)
(slime-setup-connection process)
client))
;;; When SLIME kills the process, then it invokes LIME-DISCONNECT hook.
;;; When SWANK kills the process, then it invokes LIME-DEL-CLIENT hook.
(cl-defun lime-del-client (client)
(when-let ((process (lime-client-process client)))
(lime-message "LIME client disconnected")
(slime-net-sentinel process "closed by peer")))
(cl-defun lime-disconnect (process)
(when-let ((client (lime-process-client process)))
(lime-message "LIME disconnecting client")
(websocket-close client)))
(cl-defun lime-on-error (client fun error)
(ignore client fun)
(lime-message "LIME error: " (slime-prin1-to-string error)))
;;; Client sends the result over a websocket. Handling responses is implemented
;;; by SLIME-NET-FILTER. As we can see, the flow is reversed in our case.
(cl-defun lime-handle-message (client frame)
(let ((process (lime-client-process client))
(data (websocket-frame-text frame)))
(lime-message "LIME-RECV: " data)
(slime-net-filter process data)))
(cl-defun lime-net-listen (host port &rest parameters)
(when *lime-server*
(error "LIME server has already started"))
(setq *lime-server*
(apply 'websocket-server port
:host host
:on-open (function lime-add-client)
:on-close (function lime-del-client)
:on-error (function lime-on-error)
:on-message (function lime-handle-message)
parameters))
(unless (memq 'lime-disconnect slime-net-process-close-hooks)
(push 'lime-disconnect slime-net-process-close-hooks))
(let ((buf (get-buffer-create "*lime-server*")))
(set-process-buffer *lime-server* buf)
(lime-message "Welcome " *lime-server* "!")
t))
(cl-defun lime-stop ()
(when *lime-server*
(websocket-server-close *lime-server*)
(setq *lime-server* nil)))
After loading this file into Emacs invoke (lime-net-listen "localhost" 8889)
. Now our Emacs listens for new connections from SLUG (the lisp-side part adapting SWANK, already bundled with WECL). There are two SLUG backends in a repository:
- WANK: for web browser environment
- FRIG: for Common Lisp runtime (uses
websocket-driver-client
)
Now you can open a page listed here and connect to SLIME:
<!doctype html>
<html>
<head>
<title>Web Embeddable Common Lisp</title>
<link rel="stylesheet" href="easy.css" />
<script type="text/javascript" src="https://turtleware.eu/static/misc/wecl-20250821/boot.js"></script>
<script type="text/javascript" src="https://turtleware.eu/static/misc/wecl-20250821/wecl.js"></script>
<script type="text/common-lisp" src="https://turtleware.eu/static/misc/wecl-20250821/slug.lisp"></script>
<script type="text/common-lisp" src="https://turtleware.eu/static/misc/wecl-20250821/wank.lisp"></script>
<script type="text/common-lisp" src="https://turtleware.eu/static/misc/wecl-20250821/easy.lisp">
(defvar *connect-button* (make-element "button" :text "Connect"))
(define-js-callback (connect-to-slug :type :null) ((event :js-ref))
(wank-connect "localhost" 8889)
(setf (inner-text *connect-button*) "Crash!"))
(add-event-listener *connect-button* "click" (js-callback connect-to-slug))
(append-child [body] *connect-button*)
</script>
</head>
<body>
</body>
</html>
This example shows an important limitation - Emscripten does not allow for multiple asynchronous contexts in the same thread. That means that if Lisp call doesn't return (i.e because it waits for input in a loop), then we can't execute other Common Lisp statements from elsewhere because the application will crash.
Injecting CL runtime in arbitrary websites
Here's another example. It is more a cool gimmick than anything else, but let's try it. Open a console on this very website (on firefox C-S-i) and execute:
function inject_js(url) {
var head = document.getElementsByTagName('head')[0];
var script = document.createElement('script');
head.appendChild(script);
script.type = 'text/javascript';
return new Promise((resolve) => {
script.onload = resolve;
script.src = url;
});
}
function inject_cl() {
wecl_eval('(wecl/impl::js-load-slug "https://turtleware.eu/static/misc/wecl-20250821")');
}
inject_js('https://turtleware.eu/static/misc/wecl-20250821/boot.js')
.then(() => {
wecl_init_hooks.push(inject_cl);
inject_js('https://turtleware.eu/static/misc/wecl-20250821/wecl.js');
});
With this, assuming that you've kept your LIME server open, you'll have a REPL onto uncooperative website. Now we can fool around with queries and changes:
(define-js-accessor (title :js-expr "title" :type :string)
((self :js-ref)
(title :string)))
(define-js-accessor (background :js-expr "body.style.backgroundColor" :type :string)
((self :js-ref)
(background :string)))
(setf (title [document]) "Write in Lisp!")
(setf (background [document]) "#aaffaa")
Current Caveats
The first thing to address is the lack of threading primitives. Native threads can be implemented with web workers, but then our GC wouldn't know how to stop the world to clean up. Another option is to use cooperative threads, but that also won't work, because Emscripten doesn't support independent asynchronous contexts, nor ECL is ready for that yet.
I plan to address both issues simultaneously in the second stage of the project when I port the runtime to WASI. We'll be able to use browser's GC, so running in multiple web workers should not be a problem anymore. Unwinding and rewinding the stack will require tinkering with ASYNCIFY and I have somewhat working green threads implementation in place, so I will finish it and upstream in ECL.
Currently I'm focusing mostly on having things working, so JS and CL interop is brittle and often relies on evaluating expressions, trampolining and coercing. That impacts the performance in a significant way. Moreover all loaded scripts are compiled with a one-pass compiler, so the result bytecode is not optimized.
There is no support for loading cross-compiled files onto the runtime, not to mention that it is not possible to precompile systems with ASDF definitions.
JS-FFI requires more work to allow for defining functions with variable number of arguments and with optional arguments. There is no dynamic coercion of JavaScript exceptions to Common Lisp conditions, but it is planned.
Funding
This project is funded through NGI0 Commons Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet program. Learn more at the NLnet project page.
21 Aug 2025 12:00am GMT
19 Aug 2025
Planet Lisp
Scott L. Burson: FSet 1.5.0 gets custom orderings!
The ordering of the "setlike" collections - sets, maps, and bags - in FSet has always been determined by the generic function fset:compare. This approach is often very convenient, as it allows you to define the ordering of a new type simply by adding a method on compare; there is no need to supply the ordering explicitly every time you create a new collection.
However, as people have complained from time to time, it is also a bit limiting. Say you want to make something like a telephone directory (anyone remember telephone directories?) which maps string keys to values, and you would like it maintained in lexicographic order of the keys. To do this with FSet, you have heretofore had to define a wrapper class, and then a compare method on that class, something like:
Then you would have to wrap your keys in lexi-strings before adding them to your map. That seems a little wasteful of both time and space.
A second problem with always using fset:compare is that you have to pay the cost of the generic function dispatch several times every time the collection gets searched for an element, as in contains? on a set or lookup on a map. (The number of such calls is roughly the base-2 logarithm of the size of the collection.) One micro-benchmark I ran showed this cost to be around 20% of the access time, which is not insignificant.
So, in response to popular demand, I have added custom orderings to FSet: you can supply your own comparison functions when creating collections, and FSet will call those instead of compare. Use of this feature is completely optional; existing code is not affected. But if you want to do it, now you can!
I refer you to the PR description for the details.
There is one aspect of this change that might surprise you. When given objects of different classes, fset:compare doesn't compare the contents of the objects; it just compares their class names and returns :less or :greater accordingly. So, for instance, a list cannot be equal? to a vector or seq, even if they have the same elements in the same order. This rule now also covers cases where the objects are collections of the same kind (sets, bags, or maps) but with different orderings. So just as a wb-set and a ch-set can never be :equal, so two wb-sets with different orderings can never be :equal; compare will just look at the comparison function names to impose an artificial ordering.
I'm not suggesting this is an ideal situation, but I don't see a way around it. Since comparing two wb-sets of the same ordering relies on that ordering, a combined relation on wb-sets of different orderings would in general fail to be transitive; you would get situations where a < b and b < c, but c < a.
19 Aug 2025 8:59am GMT
16 Aug 2025
Planet Lisp
Joe Marshall: Dinosaurs
What did the dinosaurs think in their twilight years as their numbers dwindled and small scurrying mammals began to challenge their dominance? Did they reminisce of the glory days when Tyrannosaurus Rex ruled the land and Pteranodon soared through the air? Probably not. They were, after all, just dumb animals.
Our company has decided to buy in to Cursor as an AI coding tool. Cursor is one of many AI coding tools that have recently been brought to market, and it is a fine tool. It is based on a fork of VSCode and has AI coding capabilities built in to it. One of the more useful ones (and one that is available in many other AI tools) is AI code completion. This anticipates what you are going to type and tries to complete it for you. It gets it right maybe 10-20% of the time if you are lucky, and not far wrong maybe 80% of the time. You can get into a flow where you reflexively keep or discard its suggestions or accept the near misses and then correct them. This turns out to be faster than typing everything yourself, once you get used to it. It isn't for everyone, but it works for me.
Our company has been using GitHub Copilot for several months now. There is an Emacs package that allows you to use the Copilot code completion in Emacs, and I have been using it for these past few months. In addition to code completion, it will complete sentences and paragraphs in text mode and html mode. I generally reject its suggestions because it doesn't phrase things the way I prefer, but I really like seeing the suggestions as I type. It offers an alternative train of thought that I can mull over. If the suggestions wildly diverge from what I am thinking, it is usually because I didn't lay the groundwork for my train of thought, so I can go back and rework my text to make it clearer. It seems to make my prose more focused.
But now comes Cursor, and it has one big problem. It is a closed proprietary tool with no API or SDK. It won't talk to Emacs. So do I abandon Emacs and jump on the Cursor bandwagon, or do I stick with Emacs and miss out on the latest AI coding tools? Is there really a question? I've been using Emacs since before my manager was born, and I am not about to give it up now. My company will continue with a few GitHub Copilot licenses for those that have a compelling reason to not switch to Cursor, and I think Emacs compatibility is pretty compelling.
But no one uses Emacs and Lisp anymore but us dinosaurs. They all have shiny new toys like Cursor and Golang. I live for the schadenfreude of watching the gen Z kids rediscover and attempt to solve the same problems that were solved fifty years ago. The same bugs, but the tools are now clumsier.
16 Aug 2025 3:04pm GMT