07 Sep 2017

feedPlanet Lisp

Nicolas Hafner: Portacle Release - Confession 75

I've written about Portacle on a previous occasion, where I talked mostly about the issues I've faced. This time, however, I'm excited to announce that Portacle has finally reached version 1.0. This means that there are no obvious remaining issues that I am aware of. Everything should Just Work™.

In case you're confused about what Portacle even is, it stands for the Portable Common Lisp development Environment. It's a combination of Emacs, SBCL, Quicklisp, GIT, and a variety of other, smaller components that together bring you a fully-fledged IDE that runs on the three major operating systems in use today. It is installable with a simple extraction and fully contained in its own directory. It can thus be loaded onto a USB stick for use on the go as well.

Portacle is primarily intended to target both complete newcomers, for which the installation procedure of a full Emacs setup otherwise involves a lot of confusing and complicated steps, and advanced users that simply need a quick way to set up a running environment on a machine. Portacle is especially convenient to test your libraries on different systems.

I have personally tested Portacle to run properly on the following platforms:

Note that currently the following platform versions are supported:

You can download the current release here. If your system falls within these constraints and Portacle won't run properly for you, please do file an issue so that I can see what else needs fixing.

If you otherwise have suggestions regarding documentation extension, adding features, or smoothing out rough edges, please file an issue as well, or hop onto the #shirakumo IRC channel on Freenode to chat directly with me. I'd be happy to hear your thoughts.

07 Sep 2017 5:06pm GMT

Zach Beane: September of Sly

I like the idea of sly: like slime, but cooler. Less conservative with changes, less concerned about backwards-compatibility, more features, cleaner implementation, etc. But I don't know that much about it in detail, and I've never tried it - until now.

I'm going to use sly exclusively for the month of September. As I bump into differences from slime, I'm taking notes and will share them here. I hope to give people an idea about what it's like to switch and help them decide if it's worthwhile for them, and figure out if I'll be switching back on October 1.

So here are a few quick notes from getting started:

Stay tuned for more!

07 Sep 2017 12:06pm GMT

05 Sep 2017

feedPlanet Lisp

TurtleWare: Tutorial: Working with FiveAM

What is FiveAM?

FiveAM is a simple-yet-mature test framework. It makes test suites for your project easy to implement, maintain, organize and run.


While it can't be said that there are no learning materials provided for FiveAM, it feels like they are lacking in both clarity and detail. Beginners are in need of gentle, friendly guidance. Experienced Lisp hackers are able to make do without it, but even they probably spend a little extra time tinkering, experimenting and skimming source code to "get" the framework. This shouldn't be necessary.

This tutorial assumes familiarity with Common Lisp and a basic understanding of ASDF system definitions.

Our building blocks

We will start with a bit of theorizing. Be not afraid, however - there won't be too much of it.

The essential terms you will need to be familiar with are:


A check is, essentially, a single assertion - a line of code that makes sure something that should be true is indeed true. FiveAM tries to make assertions as simple as possible. The form of a basic check definition looks like this:

(is test &rest reason-args)

In this case,test is the assertion we want to make. A function (or special operator) application with any number of arguments can be used as the assertion. If it returns a true value, the assertion succeeds; if it returns NIL, it fails.

If the test parameter matches any of the 4 "templates" below, FiveAM will try to reason a little about what is what and attempt to print the explanations of failures in a more readable way. Arguably.

(predicate value)
(predicate expected value)
(not (predicate value))
(not (predicate expected value))

The logic FiveAM follows when reasoning is thus:

The first expression checks whether value satisfies predicate.

In the second one, the predicate is usually some form of equality test. The assertion makes sure the value we got (by calling some function we're testing) matches the expected value according to the predicate.

The last two tests are the same things, only negated.

In practice, these declarations look like this:

(is (listp (list 1 2)))   ; is (list 1 2) a list?
(is (= 5 (+ 2 3)))        ; is (+ 2 3) equal 5?

Simple, right? If we were implementing standard Lisp functions, we could use the above to test whether list generates a list as it should, and whether + sums properly. Or, well, at least we'd ascertain that for the above cases.

And if we wanted to negate:

(is (not (listp (list 1 2))))  ; is (list 1 2) not a list?
(is (not (= 5 (+ 2 3))))       ; is (+ 2 3) not equal 5?

As you may have noticed, we haven't used the optional reason-args argument. It's used to specify what's printed as the reason for a failed check. Sometimes FiveAM's reasoning just isn't good enough. We will get back to it when we start hacking away.


We know how to write checks, but there's not much we can actually do with just this knowledge. The is syntax is only available in the context of a test definition.

A test, as defined by FiveAM, is simply a collection of checks. Each such collection has a name so that we can easily run it later. Defining one is easy:

(test test-+
  "Test the + function"     ;optional description
  (is (= 0 (+ 0 0)))
  (is (= 4 (+ 2 2)))
  (is (= 1/2 (+ 1/4 1/4))))

We're sticking to the basics for now, but you should know there are some additional keyword parameters you can pass in order to declare dependencies, explicitly specify the parent suite, specify the fixture, change the time of compilation and/or collect profiling information.

A fixture is something that ensures a test is run in a specific context. Sometimes it's necessary to reproduce results consistently. For example, if you had a pathfinding algorithm, you'd probably have to load some sort of a map before you could test it. Apparently, using FiveAM's fixture functionality isn't recommended by the current maintainer. Perhaps it's best to just set up macros for those.

As for profiling information, this functionality doesn't seem to actually be implemented yet. Instead, Metering is a good option if needed.

You'll most likely end up defining a single test for a single function, but nothing stops you from slicing the pie up differently. Maybe a particularly complex function requires a lot of checks that are best divided into categories? Maybe a set of simple, related functions can be covered by a single test for simplicity? Your common sense is the best advisor here.


The final piece of the puzzle. These are not obligatory, but very useful. Suites are containers for tests, good if you need more hierarchy - which, honestly, you will. Speaking of hierarchy: suites can parent other suites, so you can have plenty of that.

The way suites are defined and used is roughly analogous to packages.

(def-suite tutorial-suite
    :description "A poor man's suite"
    :in some-parent-suite)

(in-suite tutorial-suite)

The first form defines a test suite called tutorial-suite. The in keyword is used to set the parent suite.

Just like in-package sets the *package* special variable, in-suite sets the *suite* one. Test definitions pick up on it when provided. Thanks to that, any test definitions after (in-suite tutorial-suite) will be included in tutorial-suite. Other suite definitions, however, won't be automagically contained in the suite pointed to by *suite*. For that reason, you always need to explicitly set the in keyword when defining a child suite.

And that's actually all there is to suites.

The story so far

Time for a quick summary - from the top, our tests are organized like this:

A practical example

Now that all that is clear, let's try doing something with it. Imagine you are building an RPG game according to some existing pen-and-paper system. One day, it will surely rival the likes of AAA+ titles out there.

...for now, though, you only have the character generation facility down. Oh well, got to start somewhere. According to the specification of the system you're using, the stats of a character are generated randomly, but prior to the generation, a player can choose two stats they wish to "favor". Unfavored stats are decided on with a roll of two 8-sided dice, while favored ones - a roll of three 8-sided dice. You've defined a little utility function for rolling an arbitrary number of dice with an arbitrary number of sides.

You've written this basic functionality, wrapped it up in a package, defined an ASDF system, checked that everything compiles without warnings... So far, so good. But now you want to go the extra mile to make sure this is going to be a well-built piece of software. You want to integrate tests.

If you'd like to follow all the outlined steps and integrate FiveAM with me, just clone the master branch of the quasirpg repository.

git clone https://github.com/uint/quasirpg.git

Ideally, if you have quicklisp, do that in ~/quicklisp/local-projects/. Otherwise, clone the repository to either ~/common-lisp/ or ~/.local/share/common-lisp/source/.

If you wish, you can also look through the commit history of the test branch to see exactly how I've done all the work detailed in the following sections. It might come in useful if you get stuck.

If you want to see the code in action, try these:

CL-USER> (ql:quickload 'quasirpg)
CL-USER> (in-package #:quasirpg)
QUASIRPG> (roll-dice 3 6) ; throw three 6-sided dice
QUASIRPG> (make-character)
QUASIRPG> (make-character "Bob" '("str" "int"))

In case you don't have quicklisp, you can use this to load the system:

CL-USER> (asdf:load-system 'quasirpg)

Keep in mind that without quicklisp, you will also have to download FiveAM by hand. In the same directory you cloned quasirpg to, try:

git clone https://github.com/sionescu/fiveam.git


Tests shouldn't be a part of your software's main system. Why would they be? People who simply want to download your application and use it don't need them. Neither do they need to pull FiveAM as a dependency. So let's define a new system for tests. We could create a separate .asd file, but I like to have just one .asd file around. In this case, any additional systems defined after the main quasirpg one should be named quasirpg/some-name. So we append this to our quasirpg.asd:

(asdf:defsystem #:quasirpg/tests
  :depends-on (:quasirpg :fiveam)
  :components ((:module "tests"
            :serial t
            :components ((:file "package")
                         (:file "main")))))

We also create the new files tests/package.lisp and tests/main.lisp to make true to the above declaration. As you might guess, we're planning to define a separate package for tests. This isn't as important as separate systems, but it's always good to keep namespaces separate. Nice and tidy.

;;;; tests/package.lisp

(defpackage #:quasirpg-tests
  (:use #:cl #:fiveam)
  (:export #:run!

And finally the star of the show:

;;;; tests/main.lisp

(in-package #:quasirpg-tests)

(def-suite all-tests
    :description "The master suite of all quasiRPG tests.")

(in-suite all-tests)

(defun test-quasi ()
  (run! 'all-tests))

(test dummy-tests
  "Just a placeholder."
  (is (listp (list 1 2)))
  (is (= 5 (+ 2 3))))

Defining a simple, argument-less test runner for the whole system (test-quasi here) isn't strictly necessary, but it's going to spare us some potential headaches with ASDF.

We define a meaningless test just so we can check whether the whole setup works. If you've done everything correctly, you should be able to load the test system in your REPL

CL-USER> (ql:quickload 'quasirpg/tests)

and run the test runner

CL-USER> (quasirpg-tests:test-quasi)

Running test suite ALL-TESTS
 Running test DUMMY-TESTS ..
 Did 2 checks.
    Pass: 2 (100%)
    Skip: 0 ( 0%)
    Fail: 0 ( 0%)


So far, so good!

ASDF integration

Integrating the tests with ASDF is a good idea. That way we get hooked up to the standard, abstracted way of triggering system tests. First, we add this somewhere to our quasirpg/tests system definition.

:perform (test-op (o s)
            (uiop:symbol-call :fiveam :run! 'quasirpg-tests:all-tests))

From now on, we can run all-tests with:

CL-USER> (asdf:test-system 'quasirpg/tests)

Next, we tell ASDF that when someone wants to test quasirpg, they really want to run the quasirpg/tests test-op. Somewhere in the quasirpg system definition:

:in-order-to ((test-op (test-op "quasirpg/tests")))

Now all we need to do to test our game is:

CL-USER> (asdf:test-system 'quasirpg)

Adding real tests

Most of the character generation system's math is within the dice-rolling function - it's probably a good idea to tackle that one. The only problem is it's not a very predictable one. We can, however, still do some useful things.

(defun test-a-lot-of-dice ()
  (every #'identity (loop for i from 1 to 100
                       collecting (let ((result (quasirpg::roll-dice 2 10)))
                                    (and (>= result 2)
                                         (<= result 20))))))

(test dice-tests
  :description "Test the `roll-dice` function."
  (is (= 1 (quasirpg::roll-dice 1 1)))
  (is (= 3 (quasirpg::roll-dice 3 1)))
  (is-true (test-a-lot-of-dice)))

The first two checks simply provide arguments for which the function should always spew out the same values - we're throwing one-sided dice. Just... try not to think too hard about it.

The function test-a-lot-of-dice returns true only if every one of 100 throws of two 10-sided dice is within the expected bounds, that is 2-20. All we have to do is check whether that function returns true. We can just write (is (test-a-lot-of-dice)), but I recommend using is-true instead, since the way it prints failures is more readable in cases like this.

In all honesty, test-a-lot-of-dice could be improved in terms of optimization (for example by making it a macro that wraps the 100 checks in an and) or functionality (the parameters passed to roll-dice could be random). But this version is simple and sufficient for this tutorial.

Now let's see this thing in action.

Running test suite ALL-TESTS
 Running test DICE-TESTS fff
 Did 3 checks.
    Pass: 0 ( 0%)
    Skip: 0 ( 0%)
    Fail: 3 (100%)

And there we go. We've just detected a bug that would never be caught by the compiler. A look at the first fail gives us a hint:


 evaluated to 


 which is not 




A look at the function in question should be enough to see the problem.

  (let ((result (loop for i from 1 to n summing (random sides))))

What (random sides) does is generate a number from 0 to (sides - 1). That's not what we want.

  (let ((result (loop for i from 1 to n summing (1+ (random sides)))))

And now we re-run the tests:

Running test suite ALL-TESTS
 Running test DICE-TESTS ...
 Did 3 checks.
    Pass: 3 (100%)
    Skip: 0 ( 0%)
    Fail: 0 ( 0%)

The true power of tests, however, is that if we now ever decide to modify our dice-throwing facility, any bugs we introduce by accident will most likely be caught by the tests already in place. And so we'll avoid nasty, hard-to-debug consequences further down the line. All that without having to test things by hand each time we make changes.

Handling invalid parameters

What happens when someone passes a non-positive integer to roll-dice? Or a fractional one? We should probably control that behavior. And we should probably test to make sure when the unexpected happens, it's handled as expected.

Let's say our specification tells us that when any of the arguments is fractional, it should just be rounded down. So we append two additional tests to dice-tests:

  (is (= 3 (quasirpg::roll-dice 3.8 1)))
  (is (= 3 (quasirpg::roll-dice 3 1.9)))

The first one actually passes. It just so happens that loop is responsible for looping N times over the random number generation for each die. loop rounds down a fractional number if it's passed one.

The second test requires our attention. It fails. The problem is that random is passed a fractional argument, and it thinks it's meant to give a fractional number in response. Simple fix and we're back on track:

  (let ((result (loop for i from 1 to n summing (1+ (floor (random sides))))))

Now for something more interesting. Let's say our specification tells us that if any argument is not a positive number, we should get a SIMPLE-TYPE-ERROR. It's time to introduce yet another kind of check.

(signals condition &body body)

Not a lot to explain here. BODY is expected to cause CONDITION to be signaled. Our check only succeeds if it does. We can use this:

  (signals simple-type-error (quasirpg::roll-dice 3 -1))
  (signals simple-type-error (quasirpg::roll-dice 3 0))
  (signals simple-type-error (quasirpg::roll-dice -1 2))
  (signals simple-type-error (quasirpg::roll-dice 0 2))
  (signals simple-type-error (quasirpg::roll-dice -1 1))

Again, some of the work is already done for us. random will signal a SIMPLE-TYPE-ERROR in response to a non-positive arg. What's left to do is to handle the number of throws, so we add the appropriate code to the beginning of roll-dice:

(if (< n 1)
      (error 'simple-type-error
         :expected-type '(integer 1)
         :datum n
         :format-control "~@<Attempted to throw dice ~a times.~:>"
         :format-arguments (list n)))

And voila. Once more, all checks pass.

Random number generators

So far, we've used specific numbers. We can do better, though. We can run a large amount of checks based on random data. This is where the fiveam:for-all check comes in that runs tests 100 times, randomizing specified variables each time.

(for-all bindings &body; body)

bindings is a list of forms of this type:

(variable generator)

generator is a function (or function-bound symbol) that returns random data. variable is the variable binding that stores the results from generator.

body can contain other kinds of checks.

For example, let's try replacing (is-true (test-a-lot-of-dice)) with something more comprehensive.

(for-all ((n (gen-integer :min 1 :max 10))
          (sides (gen-integer :min 1 :max 10)))
  "Test whether calls with random positive integers give results within expected bounds."
  (let ((min n)
        (max (* n sides))
        (result (quasirpg::roll-dice n sides)))
    (is (<= min result))
    (is (>= max result))))

(gen-integer :min 1 :max 10) is a function provided by FiveAM that returns a random integer generator with the specified bounds. We keep the numbers small here so that the tests don't take forever trying to throw a lot of dice, and so that there's a reasonable chance of edge cases getting tested.

We can also replace the rounding checks. Since FiveAM doesn't provide a suitable generator, we have to write our own. It's not difficult, though, thanks to CL's ease of creating higher-order functions:

(defun gen-long-float (&key (max (1+ most-positive-long-float))
                            (min (1- most-negative-long-float)))
  (lambda () (+ min (random (1+ (- max min))))))

With that definition in place, we can write the new checks:

  (for-all ((valid-float (gen-long-float :min 1 :max 100)))
    "Test whether floats are rounded down."
    (is (= (floor valid-float) (quasirpg::roll-dice valid-float 1)))
    (is (>= (floor valid-float) (quasirpg::roll-dice 1 valid-float))))

Finally, we can replace our condition checking too:

  (for-all ((invalid-int (gen-integer :max 0))
            (invalid-int2 (gen-integer :max 0))
            (valid-int (gen-integer :min 1)))
    "Test whether non-positive numbers signal SIMPLE-TYPE-ERROR."
    (signals simple-type-error (quasirpg::roll-dice valid-int invalid-int))
    (signals simple-type-error (quasirpg::roll-dice invalid-int valid-int))
    (signals simple-type-error (quasirpg::roll-dice invalid-int invalid-int2))))

If you run these tests, you'll notice only a few checks in the results. That's because FiveAM treats each for-all declaration as a single check, regardless of the contents or the hundreds of tests that actually get run.


When the tests we've written failed, the output we got was mostly descriptive enough. That's not always the case. It's hard to expect the testing framework to know what sort of information is meaningful to us, or what the concept behind the functions we write is.

So let's say when we make-character, we want the name to be automatically capitalized. We care about punctuation and won't allow our players to get sloppy with it. Pshaw.

We add a new test:

(test make-character-tests
  :description "Test the `make-character` function."
  (let ((name (quasirpg::name (quasirpg::make-character "tom" '("str" "dex")))))
    (is (string= "Tom" name))))

Obviously, it fails.

 Failure Details:

 evaluated to 


 which is not 





We can understand it, but put yourself in the position of someone who isn't all that familiar with the make-character function. Imagine that person just got the above output while testing the entire game. They're probably really scratching their head trying to piece this together. Let's make life easy for them. Attempt number 2:

(test make-character-tests
  :description "Test the `make-character` function."
  (let ((name (quasirpg::name (quasirpg::make-character "tom" '("str" "dex")))))
    (is (string= "Tom" name)
    "MAKE-CHARACTER should capitalize the name \"tom\", but we got: ~s" name)))

We use the &rest reason-args parameter of the is check. You can use format directives and pass it arguments, just like in a format call. Now the test result is much easier to interpret:

 Failure Details:
      MAKE-CHARACTER should capitalize the name "tom", but we got: "tom".


Let's imagine what happens when the project grows. For one thing, we'll probably write many more tests, until having all of them in one file looks rather messy.

We'll also probably eventually end up reorganizing the code. roll-dice might eventually end up a part of a collection of utilities for generating randomized results, while make-character could get moved to chargen.lisp. It would be good if the hierarchy of our tests reflected those changes and let us test only random-utils.lisp or chargen.lisp if we want to.

So above all of our dice-testing code we tuck this in:

(def-suite random-utils-tests
    :description "Test the random utilities."
    :in all-tests)

(in-suite random-utils-tests)

Now all-tests contains random-utils-tests, which in turn contains dice-tests.

Let's do the same for character generation:

(def-suite character-generation-tests
    :description "Test the random utilities."
    :in all-tests)

(in-suite character-generation-tests)

(test make-character-tests
  :description "Test the `make-character` function."
  (let ((name (quasirpg::name (quasirpg::make-character "tom" '("str" "dex")))))
    (is (string= "Tom" name)
    "MAKE-CHARACTER should capitalize the name \"tom\", but we got: ~s" name)))

You can check that running (asdf:test-system 'quasirpg) still runs all of our tests, since it launches the parent suite all-tests. But we can also do (fiveam:run! 'quasirpg-tests::make-character-tests).

The next logical step is moving the test suites to separate files. If you wish to see how I've done it, just look at this commit or at the end result in the test branch.

What else is there?

A few different kinds of checks and a way to customize the way test results and statistics are presented.

So far, we've always used run! to run all the tests, which is really a wrapper for (explain! (run 'some-test)). You can, therefore, replace the explain! function with your own.

How can you learn about those things? The best I can do is point you to the FiveAM documentation and possibly source code.

Happy hacking!

05 Sep 2017 12:00am GMT

03 Sep 2017

feedPlanet Lisp

Eugene Zaikonnikov: Also ALSA

After having some issues with microphone input handling in portaudio I took a shortcut and sketched Also ALSA: an interface to Advanced Linux Sound Architecture library. As the name suggests, it's not the first CL wrapping of it. It is however small, reasonably self-contained and can handle both input and output.

LGPL to comply with alsa-lib.

03 Sep 2017 3:00pm GMT

30 Aug 2017

feedPlanet Lisp

Quicklisp news: August 2017 Quicklisp dist update now available

New projects:

Updated projects: 3bgl-shader, 3d-matrices, 3d-vectors, acclimation, alexandria, architecture.builder-protocol, array-utils, asd-generator, calispel, cepl, cepl.glop, cepl.sdl2, checkl, chirp, cl-ana, cl-fond, cl-forms, cl-gamepad, cl-gpio, cl-groupby, cl-k8055, cl-marshal, cl-mixed, cl-monitors, cl-mpg123, cl-ntp-client, cl-online-learning, cl-out123, cl-pixman, cl-random-forest, cl-soloud, cl-spidev, cl-tidy, clip, clml, closer-mop, clss, clx, coleslaw, colleen, croatoan, crypto-shortcuts, deeds, deferred, deploy, dexador, dirt, dissect, do-urlencode, documentation-utils, drakma, easy-routes, eco, esrap, fast-io, fiasco, flac-parser, flare, flow, fn, for, form-fiddle, fs-utils, fset, fxml, gamebox-dgen, gamebox-grids, glkit, glsl-spec, glsl-toolkit, halftone, harmony, hu.dwim.graphviz, hu.dwim.perec, hu.dwim.reiterate, humbler, ieee-floats, json-streams, lambda-fiddle, lass, legit, liblmdb, lichat-protocol, lichat-serverlib, lichat-tcp-client, lichat-tcp-server, lichat-ws-server, lionchat, lquery, magicffi, maiden, mcclim, mito, modularize, modularize-hooks, modularize-interfaces, nineveh, north, opticl, overlord, papyrus, parachute, parse-float, parseq, pathname-utils, piping, plump, plump-bundle, plump-sexp, plump-tex, png-read, qt-libs, qtools, qtools-ui, quux-hunchentoot, racer, random-state, ratify, redirect-stream, sanitized-params, scalpl, serapeum, simple-inferiors, simple-tasks, skitter, slime, softdrink, south, spinneret, staple, stumpwm, sxql, trivial-arguments, trivial-benchmark, trivial-indent, trivial-main-thread, trivial-mimes, trivial-thumbnail, ubiquitous, unit-formula, varjo, verbose, websocket-driver, woo, workout-timer, xsubseq.

To get this update, use (ql:update-dist "quicklisp").


30 Aug 2017 9:25pm GMT

28 Aug 2017

feedPlanet Lisp

Wimpie Nortje: Getting started with Common Lisp

Are you trying to get started with Common Lisp but find it difficult to make progress? People keep on saying that there are plenty free resources available online. Maybe you have even found most of them ... and yet the questions remain ... What is the Lisp syntax? Are there any libraries and how do I find them? Can I compile my program into a binary or are users forced to install a compiler?

The list goes on: What is the meaning of those asterisks around the *VARIABLE* names? Which implementation should I use? Where do I go for help when I get stuck?

How do I connect my IDE to the debugger because really ... who in their right mind want learn Emacs just to use Common Lisp?

There are plenty learning resources on the web. For free. In the wonderful world of Common Lisp you always have multiple choices to achieve your goal. Starting with the choice of which implementation you should use.

In some aspects this post is similar to many of the other "Getting Started" articles and in some aspects it is different. This is my take on which initial choices to make and why. The choices are neither better nor worse than the alternatives but importantly, they are not binding. At any point in the future you can switch away from any of the choices I made without severe impact on your progress. The important thing is to accept some set of initial choices and to get started. You will only discover through experience whether you agree or disagree with the choices.

Common Lisp is a vast ocean of possibilities, stretching infinitely and with no horizon... And here I am pretending to understand the MOP while trying not to end up in r/badcode, like a child playing in the surf...

Comment in the Crane library's source code

The steps

Before you can start working seriously you need a functional development environment. The Portacle project can give you a jump start here. It packages all the recommended tools to provide a ready-made Lisp development environment. If you choose this option configure Portacle then start with step 4 below. Installation instructions are on the Portacle page.

I prefer to install software natively. If you choose this option follow the steps from the beginning.

  1. Set up a Lisp implementation.
  2. Set up the library installer.
  3. Set up the development environment.
  4. Locate reference documentation.
  5. Pick a project.
  6. Deploy your program.

Set up an implementation

There are a number of Common Lisp implementations available, each with its own unique features. If you don't know why you need to use a specific implementation, use CCL (Clozure Common Lisp) or SBCL (Steel Bank Common Lisp). They are easy to set up because they don't have any external dependencies. Since Common Lisp is a standardised language you can trivially switch implementations later if you really need to.

Many people prefer SBCL. If your OS has an installation package for it, use that. Otherwise you can find the installation instructions on the website.

I prefer to use CCL because it has faster compile times than SBCL.

Install CCL

Download the version for your OS. Use the SVN method because that makes it easier to upgrade later on.

$ svn co http://svn.clozure.com/publicsvn/openmcl/release/1.11/linuxx86/ccl

Copy the file ccl/scripts/ccl64 to a directory that is in your path. On Linux a good place is /usr/local/bin. Rename it to "ccl"

$ cp ccl/scripts/ccl64 /usr/local/bin/ccl

Edit the script in /usr/local/bin. Change CCL_DEFAULT_DIRECTORY=/usr/local/src/ccl to point to the place where you installed CCL

Run "Hello world"

Start CCL

$ ccl

In the REPL, execute

? (format t "Hello world!")

The command should print

Hello world!

Exit the REPL with

? (quit)

Set up the library installer

The easiest way to obtain and install libraries is to use Quicklisp.

The following steps set up Quicklisp in your Lisp implementation:

  1. Download the setup script. In Linux it would be

    $ wget https://beta.quicklisp.org/quicklisp.lisp

  2. Install Quicklisp in CCL:

    Start CCL

    $ ccl

    In the REPL, execute

    (load "quicklisp.lisp")
  3. Test the installation

    ? (ql:system-apropos "alexandria")

    The command should print information about the Alexandria library. Something like this:

    #<SYSTEM alexandria / alexandria-20170630-git / quicklisp 2017-07-25>
    #<SYSTEM alexandria-tests / alexandria-20170630-git / quicklisp 2017-07-25>

Set up a development environment

Picking a text editor

One of Common Lisp's biggest productivity advantages is the REPL (Read-Evaluate-Print Loop). It looks and functions a lot like the operating system's command line interface which executes commands as they are entered. Variables and defined functions remain available throughout the REPL session. New functions can be defined one at time.

In contrast to languages like C where your program must be compilable before you can test any change, in the REPL you can compile only the function you want to test without having a complete application.

To understand the implication, suppose you have function A which is called by many other functions throughout the application and you want to change A's argument list. In C you'd have to update all the functions that call A before you can compile the program to actually test the change. In Common Lisp you can make the change, update one function and test it. Only when you are satisfied with the results you have to update the other calling functions.

The described workflow is completely possible with any random text editor and loading the source files directly in the REPL. After making and testing changes in the REPL only the final modifications need to be transferred to the source files. While using Lisp in this way is possible, there is a disconnect between the text editor and the Lisp implementation which can negate the advantages brought by the REPL.

When the text editor is able to control the REPL it allows you to make exploratory changes directly in the source files then compile and test only the modifications. At the end of this exploratory cycle the source files are already up-to-date and all that remain is to update the rest of the calling functions.

The tool that enables control over the REPL is called SLIME1. It is the most advanced and mature tool for this purpose and it is specifically developed for Emacs. SLIME has been adapted for use in other editors2 but they have fewer users and contributors and thus always lag behind the latest SLIME development.

When setting out to learn Common Lisp the path of least resistance is to use Emacs, even with all its foreign concepts and non-standard shortcut keys. Once one appreciates the power and productivity because of SLIME it is easier to look past Emacs' aged appearance and arcane user interface.

Install SLIME

In the CCL REPL, run the following:

? (ql:quickload "quicklisp-slime-helper")

Install Emacs

Installing Emacs should be trivial. Most Linux distributions provide installation packages in their repositories and the Emacs web page provides installation instructions for Windows and macOS.

When Emacs is working, edit or create the file ~/.emacs. To locate the file on Windows, use the File | Visit New File menu item and type ~/.emacs as the filename.

Place the following code in .emacs.

;; Set up CCL
(setq inferior-lisp-program "ccl")

;;Load Quicklisp slime-helper
(load (expand-file-name "~/quicklisp/slime-helper.el"))

Exit and restart Emacs.

Using Emacs

Download the Essential Emacs key bindings cheatsheet .

Emacs' key bindings (i.e. shortcut keys) differ significantly from the standard shortcut keys used in most other editors. If you don't know some basics and your menu bar happens to become hidden you will be helpless. This is a bare minimum introduction to using Emacs so that you will be able to edit Lisp files and run them with SLIME. The EmacsWiki key reference page has a much more thorough list.

Emacs shortcuts are a combination of a modifier key with a normal key. The modifiers are:

Key combinations are written as C-x which means press and hold Control and then press x.

C-x C-c Exit Emacs
C-x C-f Open or create file
C-x C-s Save file
C-x k Close file
M-w Copy selected text
C-w Cut selected text
C-y Paste text
C-/ Undo
C-x h Select all
C-x b Switch between buffers
M-x slime Start SLIME
C-c C-z Switch to SLIME from a Lisp buffer
C-c C-c Compile the function at the cursor
C-c M-q Indent the function at the cursor
C-up/down Move through history in SLIME buffer

If you really want to do copy, cut and paste with the same keys used in other applications then you can search for Emacs CUA mode.

Run "Hello world"


M-x slime

The first time you run SLIME there may be some delay while it is being compiled. When it is ready you should see the following in Emacs:

; SLIME 2.19

In the REPL, execute

CL-USER> (format t "Hello world!")

You should see

Hello world!

printed on the screen.

Programming and debugging

With the power of the SLIME / Emacs combination and Common Lisp's "vast ocean of possibilities" there are many ways to use the tools. Each programmer will eventually develop his or her own distinct technique.

Without spending too much time learning Emacs or SLIME early on one can become quite productive using the following approach:

Big changes:

  1. Edit a file
  2. Save the file - C-x C-s
  3. Jump to the SLIME REPL - C-c C-z
  4. Load the file - (load "app.lisp")
  5. Run the code - (some-function)

Small changes

  1. Edit a file
  2. Save the file - C-x C-s
  3. Compile the function - C-c C-c
  4. Jump to the SLIME REPL - C-c C-z
  5. Run the code - (some-function)

Locate reference documentation

When you get stuck Stack Overflow can be useful but often you get answers faster if you know about other sources of information. I posted previously on the topic of finding answers. Here is another list which is more relevant if you are only getting started.

Learn the language
Practical Common Lisp
Naming conventions
Find libraries and their documentation
Common Lisp Language reference
Help with Emacs
Use SLIME more effectively
SLIME manual
More introductory material
Articulate Lisp
When all these are not enough
Reddit r/lisp sidebar

Pick a project

To learn a new language you must write a program and for that you need a project. I have posted some ideas for beginner projects before but I think it is easier to stay motivated when you work on your own idea.

With the project in hand you now have everything to start programming and learning. At the very beginning it is easier to write all the code in one file. Loading the program is straightforward and it removes a lot of extra variables introduced when system definition files and multiple source files come into play. They are essential for bigger programs but they cause an unnecessary mental burden when you are still finding your feet.

The moment a single file becomes too convoluted to continue, stop programming and create a project skeleton with Quickproject. It creates a minimal project with all the necessary files to make it loadable with ASDF and Quicklisp.

The steps to create a project skeleton are:

(ql:quickload :quickproject)
(quickproject:make-project "path/to/project/")

Deploy your program

Common Lisp programs can be compiled into standalone executables or they can run from source code.

Buildapp is my tool of choice for creating executables. I posted before on how use it.

Server-side software are usually run from source code. Depending on how your application startup code looks like it can be as simple as

ccl -l app.lisp


  1. SLIME - The Superior Lisp Interaction Mode for Emacs.

  2. See the "Tools" section in the Reddit sidebar.

28 Aug 2017 12:00am GMT

10 Aug 2017

feedPlanet Lisp

Quicklisp news: July 2017 Quicklisp download stats

Here are the raw download stats for the top 100 projects in Quicklisp for July:

11470  alexandria
8732 babel
8521 closer-mop
7779 split-sequence
7534 trivial-features
7197 cffi
7170 iterate
7095 cl-ppcre
7061 bordeaux-threads
6863 trivial-gray-streams
6526 anaphora
6062 flexi-streams
5589 cl+ssl
5588 trivial-garbage
5327 trivial-backtrace
5122 let-plus
5071 nibbles
4811 cl-fad
4648 usocket
4353 puri
4124 cl-base64
4119 drakma
4107 local-time
4006 named-readtables
3949 chunga
3661 chipz
3201 ironclad
3164 esrap
3058 cl-unicode
3043 cl-interpol
3010 cl-yacc
2807 more-conditions
2789 md5
2526 utilities.print-items
2523 fiveam
2511 asdf-flv
2472 log4cl
2250 slime
2198 parse-number
2178 trivial-types
2154 trivial-indent
2152 cl-annot
2122 trivial-utf-8
2113 cl-syntax
1969 array-utils
1914 cl-json
1913 gettext
1894 symbol-munger
1882 plump
1875 arnesi
1826 collectors
1825 cl-slice
1805 access
1794 djula
1767 cl-locale
1766 cl-parser-combinators
1742 cl-utilities
1732 metabang-bind
1695 lift
1668 cl-containers
1666 asdf-system-connections
1664 optima
1662 metatilities-base
1633 quri
1631 hunchentoot
1599 simple-date-time
1567 lparallel
1566 fast-io
1562 uuid
1531 cl-clon
1461 bt-semaphore
1438 trivial-mimes
1437 closure-common
1421 cxml
1409 static-vectors
1406 mcclim
1327 clack
1322 cl-vectors
1281 ieee-floats
1220 salza2
1197 fast-http
1165 clx
1160 fare-utils
1116 fare-quasiquote
1114 lack
1105 architecture.hooks
1087 prove
1087 cl-colors
1057 uffi
1040 cl-ansi-text
997 inferior-shell
997 fare-mop
991 postmodern
979 rfc2388
978 proc-parse
961 quicklisp-slime-helper
942 pythonic-string-reader
940 xsubseq
940 plexippus-xpath
934 cl-jpeg

10 Aug 2017 7:40pm GMT

29 Jul 2017

feedPlanet Lisp

Timofei Shatrov: Your personal DIY image search

Hi everyone, it's been a while! I bet you forgot this blog even existed. I happen to be a big supporter of quality over quantity, so while my work on parsing Japanese counters earlier this year was pretty interesting, I already wrote way too many articles about Ichiran/ichi.moe so I decided to keep it to myself. Recently I've been working on a little side-project and now that it finally works, I think it deserves a full-fledged blog post.

For a bit of a nostalgia trip, let's go back to the early 00s. Remember when TinEye first appeared? It was amazing. For the first time you could easily find where that one image you once saved from some random phpBB forum is really from. It didn't matter if your image was resized, or slightly edited from the original, it still worked. That shit was magic, my friends. Of course these days nobody is impressed by this stuff. Google Image Search indexes pretty much anything that exists on the Internet and even uses neural networks to identify content of an image.

Back to the present day. I discovered I have an image hoarding problem. Over the years of using the Intertubes, I have accumulated a massive number of images on my hard drive. When I see an image I like my first thought is "do I have this one saved already?" because how could I possibly remember? At this point I need my own personal Google Image Search. And (spoiler alert) now I have one.

First of all, I needed an actual image matching technology. These days the cloud is all the rage, so I definitely wanted to have this thing running in the cloud (as opposed to my local PC) so that I could search my images from anywhere in the world. After a cursory search, my eyes fell on a thing called Pavlov Match which runs from a Docker container, so should be pretty easy to install. I installed docker and docker-compose on my VPS, and then git-cloned Match and ran make dev according to instructions. This will actually run an Elasticsearch instance on the same VPS, and apparently the damn thing eats memory for breakfast, at least with the default settings. I'm using a cheap 2GB RAM Linode, so the memory is actually a very finite resource here, as I will find out later. The default settings will also completely expose your match installation AND elasticsearch to the world. But don't worry, I figured this out so that you don't have to. Let's edit docker-compose.yml from match repository as follows:

version: '2'
    image: pavlov/match:latest
    command: ["/wait-for-it.sh", "-t", "60", "elasticsearch:9200", "--", "gunicorn", "-b", "", "-w", "4", "--preload", "server:app"]
    - elasticsearch
    image: elasticsearch
      - "ES_JAVA_OPTS=-Xms256m -Xmx256m"
      - bootstrap.mlockall=true
    - "9200"

This will make match server only available on local network within the VPS on port 8888, and elasticsearch only available to these two docker containers. It will also restrict elasticsearch RAM consumption to 512mb and --preload flag reduces the amount of memory gunicorn workers consume.

To make match server available from outside I recommend proxying it through nginx or some other proper web server. You can also add authentication/IP whitelist in nginx because the match server has no authentication features whatsoever, so anyone will be able to search/add/delete the data on it.

That was the backend part. No programming required here! But this is a Lisp blog, so the next step is writing a Lisp client that can communicate with this server. The first step is reading the match API documentation. You might notice it's a bit… idiosyncratic. I guess REST is out of fashion these days. Anyway, I started implementing a client using the trusty drakma, but I quickly hit a limitation: match expects all parameters to be sent encoded as form data, but drakma can only encode POST parameters as form data and not, say, DELETE parameters. Not to be foiled by a badly designed API, I tried dexador, and while dex:delete does not encode parameters as form data, dex:request is flexible enough to do so. Each response (a JSON string) is parsed using jsown.

(defun parse-request (&rest args)
  (when *auth*
    (setf args `(,@args :basic-auth ,*auth*)))
  (multiple-value-bind (content return-code)
      (handler-bind ((dex:http-request-failed #'dex:ignore-and-continue))
        (apply 'dex:request args))
      ((<= 400 return-code 499)
        ("status" "fail")
        ("error" content)
        ("code" return-code)))
      (t (let ((obj (jsown:parse content)))
           (jsown:extend-js obj ("code" return-code)))))))

(defun add-local (file &key path (metadata "{}"))
  "Add local image to Match server"
   (api-url "/add")
   :method :post
   :content `(("image" . ,(pathname file))
              ("filepath" . ,(or path file))
              ("metadata" . ,metadata))))

With this basic client in place, I can add and delete individual images, but it would be incredibly cumbersome to manage thousands of images with it. I had to write some code that would scan specified directories for images, track any changes and then add/update/delete information from Match server as needed. I already wrote something like this before, so this was pretty easy. Of course SBCL's "sb-posix:stat doesn't work on Unicode filenames" bug has reared its head again, but I already knew the workaround. This time I completely relied on UIOP for recursively walking directories (uiop:subdirectories and uiop:directory-files are your friends). Each image file is represented as CLOS object and saved into a hash-table which is serialized to a file using CL-STORE. The object has a status attribute which can be :new, :update, :delete, :ok and so on. Based on status, an action needs to be performed, such as uploading an image to Match server (for :new and :update).

Now, I could just send a bunch of requests one after another, but that would be a waste. Remember, we have 4 gunicorn workers running on our server! This clearly calls for a thread pool. I thought PCALL would be perfect for this, but nope. It uses sb-thread:interrupt-thread which is incredibly unsafe and the result is that you basically can't safely make http requests from thread workers. Debugging this took way too much time. In the end, I implemented a thread pool based on lparallel promises which is kind of an overkill for such a simple use case, but at least it worked.

  (setf *cache* (update-cache))
  (let ((lparallel:*kernel* (lparallel:make-kernel threads)))
         (loop for value in (alexandria:hash-table-values *cache*)
            collect (worker value) into futures
            finally (map nil 'lparallel:force futures))
  (save-cache *cache*))

Note that you must be very careful when doing things that affect global state inside the threads. For example :delete action removes a key from the hash table *cache*. This is not guaranteed to be an atomic operation, so it's necessary to grab a global lock when doing it.

(defvar *cache-lock* (bordeaux-threads:make-lock "match-cache-lock"))
   (bordeaux-threads:with-lock-held (*cache-lock*)
      (remhash key *cache*))

Printing messages to REPL from inside threads also requires a separate lock and (force-output), otherwise it will look like a complete mess!

(defun format-msg (str &rest args)
  (bordeaux-threads:with-lock-held (*msg-lock*)
    (apply 'format t str args)

Now that the required functionality is implemented, it's time to test upload a bunch of stuff… and get back a bunch of errors. It took some sleuthing to discover that gunicorn workers of my Match server are routinely getting killed by "OOM killer". Basically, the server runs out of memory and the system in desperation kills a process that it doesn't like. Remember, I only have 2Gb of memory there!

I figured out that it's images with very large dimensions that are the most problematic in terms of memory usage. If I were to resize these images to some reasonable size, the matching should still work pretty well. In order to execute this plan, I thought I'd use some Lisp to ImageMagick interface. There's in fact a pure Lisp solution called OptiCL but would it really handle any image? Remind me to test that later! Anyway, back to ImageMagick. Neither lisp-magick nor lisp-magick-wand would work with the most recent ImageMagick version (seems its API has changed a bit). However the last one I tried cl-graphicsmagick, which uses a fork of ImageMagick called GraphicsMagick, has unexpectedly worked (at least on my Windows laptop. Note that you need to install Microsoft Visual C Redistributable 2008 otherwise the library wouldn't load with CFFI) so I went with that.

Using very useful temporary files functionality of UIOP (uiop:with-temporary-file), I resize each oversized image to reasonable dimensions and save into a temporary file, which is then uploaded to Match server. I also send the file's original and resized dimensions as metadata. Thankfully this completely eradicated the memory issue. There's a minor problem where GraphicsMagick cannot do Unicode pathnames on Windows, so I copy the original image into a temporary file with ASCII-only name in that case.

(defun resize-image (input-path output-path
                     &key (max-width *max-dimension*) (max-height *max-dimension*)
                       (filter :%QuadraticFilter) (blur 1))
  (gm::with-magick-wand (wand)
    (handler-case (gm::%MagickReadImage wand input-path)
      ;; graphicsmagick cannot read Unicode filenames on Windows so attempt to load a copy
      (gm::magick-error ()
        (uiop:with-temporary-file (:pathname tmp :prefix "gm" :type (pathname-type input-path))
          (uiop:copy-file input-path tmp)
          (setf wand (gm::%NewMagickWand))
          (gm::%MagickReadImage wand (namestring tmp)))))
    (let ((w (gm::%MagickGetImageWidth wand))
          (h (gm::%MagickGetImageHeight wand))
          (res nil))
      (multiple-value-bind (fw fh) (gm::fit-width-height w h max-width max-height)
        (unless (and (= w fw) (= h fh))
          (gm::%MagickResizeImage wand fw fh filter blur)
          (gm::%MagickWriteImage wand output-path)
          (setf res output-path))
        (values res w h fw fh)))))

Later I tested this code on an Ubuntu machine with GraphicsMagick installed from Apt repository and SBCL crashed into ldb debugger mode straight away… Welp. The helpful folks of #lisp told me the problem is with signal handlers established by GraphicsMagick library, somehow they confuse SBCL. Based on that advice, eventually I succeeded making this work. Uninstall apt Graphicsmagick and grab the sources. Find the file called magick.c and replace the line

InitializeMagickSignalHandlers(); /* Signal handlers */


// InitializeMagickSignalHandlers(); /* Signal handlers */

(commenting it out). Then do configure --enable-shared (see readme for possible options), make and sudo make install. This will make it work when called from SBCL on Linux.

Anyways, the full code of MATCH-CLIENT can be found at my Github. It's not installable from quicklisp for obvious reasons, in fact it's a complete pain to install as you might've already guessed, but if you wanna try it, you're welcome. The main two commands are update and match. The first is called to upload all images in your *root-dirs* to the server and then to update them if anything changes. match is used to match any image on the Internet (passed as URL string) or a local pathname (passed as pathname object) compared to the server. It returns a list of jsown objects (basically alists) that contain score (up to 100 for exact match), path (with "local tag" which can be different per device) and metadata containing original and resized dimensions.

((:OBJ ("score" . 96.00956)
  ("filepath" . "[HOME] d:/foo/bar/baz.jpg")
  ("metadata" :OBJ ("rw" . 1218) ("rh" . 2048) ("w" . 3413) ("h" . 5736))))

Anyway, this was a fun (although often frustrating) thing to build and ended up being quite useful! Thanks for reading and see you next time.

29 Jul 2017 6:47pm GMT

McCLIM: Progress report #9

Dear Community,

McCLIM code is getting better on a weekly basis depending on developer time. We are happy to see the project moving forward.

Some highlights for this iteration:

Moreover many bug fixes have been proposed and merged into the codebase.

All McCLIM bounties (both active and already solved) may be found here. Default bounty expiration date is 6 months after posting it (a bounty may be reissued after that time period).

To answer recurring requests for native Windows and OSX support, we have posted bountes for finishing the Windows backend and fixing the OSX backend. Moreover, to improve portability a bounty for closer-mop support has been posted.

Bounties solved this iteration:

Active bounties ($1700):

Our current financial status is $800 for bounties and $267 recurring monthly contributions from the supporters (thanks!).

Suggestions as to which other issues should have a bounty on them are appreciated and welcome. Please note that Bountysource has a functionality "Suggest an Issue" which may be found on the bounties page. If you would like to work on an issue that is not covered by the existing bounties, feel free to suggest a new bounty.

If you have any questions, doubts or suggestions - please contact me either by email (daniel@turtleware.eu) or on IRC (my nick is jackdaniel).

Sincerely yours,
Daniel Kochmański

29 Jul 2017 1:00am GMT

25 Jul 2017

feedPlanet Lisp

Quicklisp news: June 2017 Quicklisp download stats

Here are the raw download stats for the top 100 projects in Quicklisp for June:

 9081  alexandria
7797 closer-mop
7437 split-sequence
6863 cl-ppcre
6790 babel
6498 trivial-features
6303 iterate
6222 bordeaux-threads
6173 anaphora
6099 trivial-gray-streams
5522 trivial-garbage
5367 cffi
5056 flexi-streams
4911 nibbles
4729 let-plus
4702 usocket
4592 puri
4582 cl-base64
4286 trivial-backtrace
4181 chipz
4145 cl+ssl
4021 cl-fad
3959 chunga
3381 drakma
3292 named-readtables
3281 ironclad
3221 more-conditions
3153 esrap
3144 local-time
2928 utilities.print-items
2587 parse-number
2439 cl-yacc
2149 metabang-bind
2142 cl-unicode
2131 cl-interpol
2101 trivial-utf-8
2084 md5
2083 fiveam
2056 asdf-flv
1930 optima
1918 lparallel
1897 log4cl
1879 slime
1869 lift
1854 trivial-indent
1822 closure-common
1808 cxml
1795 array-utils
1746 plump
1743 uuid
1612 bt-semaphore
1561 trivial-types
1541 simple-date-time
1513 cl-clon
1472 cl-json
1429 cl-utilities
1392 architecture.hooks
1390 quri
1342 cl-containers
1340 metatilities-base
1330 cl-annot
1319 cl-syntax
1317 asdf-system-connections
1291 ieee-floats
1253 plexippus-xpath
1113 salza2
1079 trivial-mimes
1070 postmodern
1067 arnesi
1052 cl-slice
1050 fare-utils
1047 fast-io
1040 static-vectors
1027 fare-quasiquote
1015 symbol-munger
1009 djula
1007 collectors
1003 access
996 gettext
982 cl-parser-combinators
980 cl-locale
925 hunchentoot
904 cl-sqlite
896 inferior-shell
894 fare-mop
887 prove
885 rfc2388
868 cl-log
865 command-line-arguments
859 trivia
858 lisp-namespace
851 cl-colors
824 py-configparser
821 cl-markdown
821 cl-ansi-text
821 asdf-finalizers
820 dynamic-classes
819 cl-mssql
818 garbage-pools
805 cl-abnf

25 Jul 2017 7:13pm GMT