17 May 2016

feedPlanet Lisp

Vsevolod Dyomkin: Improving Lisp UX One Form at a Time

At the recent ELS, I presented a lightning talk about RUTILS and how I see it as a way of "modernizing" CL, i.e. updating the basic language elements to be simpler, clearer and more generic. Thus improving the everyday user experience and answering the complaints of outsiders about "historical cruft" in the Lisp standard. Indeed, Lisp has a lot of unrecognizable names (like mapcar and svref) or just unnecessary long ones (multiple-value-bind or defparameter), and out-of-the-box it lacks a lot of things that many current programmers are used to: unified generic accessors, generators, literal syntax for defining hash-tables or dynamic vectors etc. This may not be a problem for the people working with the language on a regular basis (or if it is they probably have a personal solution for that already), but it impedes communication with the outside world. I'd paid extra attention to that recently as I was preparing code examples for the experimental course on algorithms, which I teach now using Lisp instead of pseudocode (actually, modulo the naming/generics issue, Lisp is a great fit for that).

Unfortunately, the lightning talk format is too short for a good presentation of this topic, so here's a more elaborate post, in which I want to show a few examples from the RUTILS library of using Lisp's built-in capabilities to introduce clear, uniform, and generic syntactic abstractions that may be used alongside the standard Lisp operators, as well as replace them in the cases when we want to get more concise and understandable code.

What's cool about this problem is that, in Lisp, besides a common way to extend the language with functions and methods (and even macros/templates, which find they way into more and more languages), there are several other approaches to the problem that allow to tackle issues that can't be covered by functions and even macros. Those include, for instance, reader macros and aliasing. Aliasing is, actually, a rather simple idea (and can be, probably, implemented in other dynamic languages): duplicating functionality of existing functions or macros with a new name. The idea for such operator came from Paul Graham's "On Lisp" and it may be implemented in the following way (see a full implementation here):

(defmacro abbr (short long &optional; lambda-list)
((macro-function ',long)
(setf (macro-function ',short) (macro-function ',long)))
((fboundp ',long)
(setf (fdefinition ',short) (fdefinition ',long))
,(when lambda-list
`(define-setf-expander ,short ,lambda-list
(values ,@(multiple-value-bind
(dummies vals store store-form access-form)
(cons long (remove-if (lambda (sym)
(member sym '(&optional; &key;)))
(let ((expansion-vals (mapcar (lambda (x) `(quote ,x))
(list dummies
(setf (second expansion-vals)
(cons 'list vals))
(t (error "Can't abbreviate ~a" ',long)))
(setf (documentation ',short 'function) (documentation ',long 'function))

As you may have noticed, it is also capable of duplicating a setf-expander for a given function if the lambda-list is provided. Using abbr we can define a lot of shorthands or alternative names, and it is heavily used in RUTILS to provide more than 50 alternative names; we'll see some of them in this post. What this example shows is the malleability of Lisp, which allows approaching its own improvement from different angles depending on the problem at hand and the tradeoffs you're willing to make.

Introducing generic element access

One of the examples of historic baggage in CL is a substantial variety of different methods to access elements of collections, hash-tables, structures, and objects with no generic function unifying them. Not to say that other languages have a totally uniform accessor mechanism. Usually, there will be two or three general-purpose ways to organize it: dot notation for object field access, something square-braketish for array and other collections access, and some generic operator like get for all the other cases. And occasionally (e.g. in Python or C++) there are hooks to plug into the built-in operators. Still, it's a much smaller number than in Lisp, and what's more important, it's sufficiently distinct and non-surprising.

In Lisp, actually, nothing prevents us from doing even better - both better than the current state and than other languages - i.e. from having a fully uniform and extensible solution. At first approximation, it's just a matter of defining a generic function that will work on different container types and utilize all the existing optimized accessor functions in its methods. This interface will be extensible for any container object. In RUTILSX (a part of RUTILS where any experiments are allowed) this function is called generic-elt:

(defgeneric generic-elt (obj key &rest; keys)
(:method :around (obj key &rest; keys)
(reduce #'generic-elt keys :initial-value (call-next-method obj key))))

One important aspect you can see in this definition is the presence of an :around method that allows to chain multiple accesses in one call and dispatch each one to an appropriate basic method via call-next-method. Thus, we may write something like (generic-elt obj 'children 0 :key) to access, for instance, an element indexed by :key in a hash-table that is the first element of a sequence that is the contents of the slot children of some object obj.

The only problem with this function is its long name. Unfortunately, most of good short element access names, like elt and nth are already taken in the Common Lisp standard, while for RUTILS I've adopted a religious principle to retain full backward compatibility and don't alter anything from the standard. This is a critical point: not redefining CL, but building on top of it and extending it!

Moreover, element access has two features: it's a very common operation and it's also not a usual function that does some computation, so ideally it should have a short but prominent look in the code. The perfect solution occurred to me at one point: introduce an alias ? for it. Lisp allows to name operations with any characters, and a question mark, in my opinion, matches very well the inner intent of this operation: query a container-like object using a certain key. With it, our previous example becomes very succinct and cool: (? obj 'children 0 :key).

Additionally to element reading, there's also element write access. This operation in Lisp, like in most other languages, has a unified entry point called setf. There's a special interface to provide specific "methods" for it based on the accessor function. Yet, what to do when an access function is polymorphic? Well, provide polymorphic setter companion. (defsetf generic-elt generic-setf). Like generic-elt, generic-setf defers work to already defined specific setters:

(defmethod generic-setf ((obj list) key &rest; keys-and-val)
(setf (nth key obj) (atomize keys-and-val)))

And it also supports key chaining, so you can write: (setf (? obj 'children 0 :key) new-value).

Having this unified access functionality is nice and cool, but some people may still linger for the familiar dot object slot access syntax. We can't blame them: habits are a basis of good UX. Unfortunately, this is contrary to the Lisp way... But Lisp is a pro-choice and future-proof language: if you want something badly, even something not in the usual ways, almost always you can, actually, find a clean and supported means of implementing it. And this case is not an exception. If you can tolerate an small addition - a @-prefix to the object reference (that's also an extra prominent indicator of something unusual going on) - when accessing its slots you can define a reader macro that will expand forms @obj.slot into our (? obj 'slot) or a standard (slot-value obj 'slot). With it, we can write something like (? tokens @dep.govr.id), which is much more succinct and, arguably, readable than (elt tokens (slot-value (slot-value dep 'govr) 'id)).

Still, one issue remains unsolved in this approach: the preferred Lisp slot-access method is not via slot-value, but with an accessor method that is exported. And one of the reasons for it is that slot-names, which are usually short and can clash, are kept private to the package where they are defined. It means that in most cases @obj.slot will not work across packages. (Unlike the OO-languages in which every class is its own namespace, in Lisp, this function is not "complected" within the OO-system, and packages are a namespacing method, while objects serve for encapsulation and inheritance.)

There are two ways to tackle this problem. As I said, Lisp is future-proof: being thoroughly dynamic and extensible, CLOS defines a method that is called when there's a problem accessing an object's slot - slot-missing. Once again, we can define an :around method that will be a little smarter (?) and try to look up slot-name not only in the current package, but also in the class' original package.

(defmethod slot-missing :around
(class instance slot-name (operation (eql 'slot-value)) &optional; new-value)
(declare (ignore new-value))
(let ((class-package (symbol-package (class-name (class-of instance)))))
(if (eql class-package (symbol-package slot-name)) ;; to avoid infinite looping
(if-it (find-symbol (string-upcase slot-name) class-package)
(slot-value instance it)

This is a rather radical way and comes at a cost: two additional virtual function calls (of the slot-missing method itself and an additional slot-value one). But in most of the cases it may be worth paying it for convenience's sake, especially, since you can always optimize a particular call-site by changing the code to the most direct (slot-value obj 'package::slot) variant. By the way, using slot accessor method is also costlier than just slot-value, so we are compensating here somewhat. Anyway, it's cool to have all the options on the table: beautiful slow and ugly fast method that our backward-compatibility approach allows us. As usual, you can't have a cake and eat it too...

Though, sometimes, you can. :) If you think more of this it becomes apparent that slot-value could be implemented this way from the start: look up the slot name in the class'es original package. As classes or structs are defined together with their slots it is very rare if not almost impossible to see slot-names not available in the package where their class is defined (you have to explicitly use a private name from another package when defining a class to do such a trick). So, slot-value should always look for slot names in the class'es package first. We can define a "smart" slot-value variant that will do just that, and with our nice generic-elt frontend it can easily integrated without breaking backward-compatibility.

(defun smart-slot-value (object slot-name)
(slot-value object
(or (find-symbol (string-upcase slot-name)
(symbol-package (class-name (class-of instance))))

Unifying variable binding with with

Almost everything in functional variable definition and binding was pioneered by Lisp at some point, including the concept of destructuring. Yet, the CL standard, once again, lacks unification in this area. There are at least 4 major constructs: let and let*, destructuring-bind and multiple-value-bind, and also a few specialized ones like with-slots or ppcre:register-groups-bind. One more thing to mention is that parallel assignment behavior of plain let can be implemented with destructuring-bind and multiple-value-bind. Overall, it just screams for uniting in a single construct, and already there have been a few attempts to do that (like metabang-bind). In RUTILS, I present a novel implementation of generic bind that has two distinct features: a more plausible name - with - and a simple method-based extension mechanism. The implementation is very simple: the binding construct selection is performed at compile-time based on the structure of the clause and, optionally, presence of special symbols in it:

(defmacro with ((&rest; bindings) &body; body)
(let ((rez body))
(dolist (binding (reverse bindings))
(:= rez `((,@(call #'expand-binding binding rez)))))
(first rez)))

A very short number of methods covering the basic cases are defined:

(defun expand-binding (binding form)
(append (apply #'bind-dispatch binding)

(defgeneric bind-dispatch (arg1 arg2 &rest; args)
(:method ((arg1 symbol) arg2 &rest; args)
(if args
`(multiple-value-bind (,arg1 ,arg2 ,@(butlast args)) ,(last1 args))
`(let ((,arg1 ,arg2)))))
(:method ((arg1 list) (arg2 (eql '?)) &rest; args)
`(let (,@(mapcar (lambda (var-key)
`(,(first (mklist var-key))
(? ,(first args) ,(last1 (mklist var-key)))))
(:method ((arg1 list) (arg2 (eql '@)) &rest; args)
(with-gensyms (obj)
`(let* ((,obj ,(first args))
,@(mapcar (lambda (var-slot)
`(,(first (mklist var-slot))
(smart-slot-value ,obj ',(last1 (mklist var-slot)))))
(:method ((arg1 list) arg2 &rest; args)
`(destructuring-bind ,arg1 ,arg2)))

In a sense, it's a classic example of combining generic-functions and macros to create a clean and extensible UI. Another great benefit of using with is reduced code nesting that can become quite deep with the standard operators. Here's one of the examples from my codebase:

(with (((stack buffer ctx) @ parser)
(fs (extract-fs parser interm))
(((toks :tokens) (cache :cache)) ? ctx))

And here's how it would have looked in plain CL:

(with-slots (stack buffer ctx) parser
(let ((fs (extract-fs parser interm)))
(toks (gethash :tokens ctx))
(cache (gethash :cache ctx)))

Implementing simple generators on top of signals

One of my friends and a Lisp enthusiast, Valery Zamarayev, who's also a long-time Python user, once complained that the only thing that he misses in CL from Python is generators. This feature is popular in many dynamic languages, such as Ruby or Perl, and even Java 8 has introduced something similar. Sure, there are multiple ways to implement lazy evaluation in Lisp with many libraries for that, like SERIES, pygen or CLAZY. We don't have to wait for another version of the spec (especially, since it's not coming 8-)

In RUTILS I have discovered, I believe, a novel and a very clean way to implement generators - on top of the signal system. The signal or condition facility is, by the way, one of the most underappreciated assets of Common Lisp that often comes to rescue in seemingly dead ends of control flow implementation. And Kent Pitman's description of it is one of my favorite reads in Computer Science. Anyway, here's all you need to implement Python-style generators in Lisp:

(define-condition generated ()
((item :initarg :item :reader generated-item)))

(defun yield (item)
(restart-case (signal 'generated :item item)
(resume () item)))

(defmacro doing ((item generator-form &optional; result) &body; body)
(with-gensyms (e)
`(block nil
(handler-bind ((generated (lambda (,e)
(let ((,item (generated-item ,e)))
(invoke-restart (find-restart 'resume))))))

The doing macro works just like dolist, but iterating the generator form instead of an existing sequence. As you can see from this example, restarts are like generators in disguise. Or, to be more correct, they are a more general way to handle such functionality, and it takes just a thin layer of syntactic sugar to adapt them to a particular usage style.

And a few mischiefs

We have seen three different approaches to extending CL in order to accommodate new popular syntactic constructs and approaches. Lastly, I wanted to tread a little in the "danger zone" that may be considered unconventional or plain bad-style by many lispers - modifying syntax at the reader level. One thing that Clojure (following other dynamic languages before it), I believe, has proven is the importance of shorthand literal notation for popular operations. CL standard has predated this understanding: although it has specific print representations for various important objects, and even a special syntax for static arrays. Yet, the language is really future-proof in this respect, because it provides a way to hook into the reader mechanism by modifying the readtables. It was further smoothed and packaged by the popular NAMED-READTABLES library, which allows to treat readtables similar to packages. In RUTILS I have defined several extended readtables that implement a few shortcuts that are used literally in every second function or macro I define in my code. These include:

Overall, I have experimented a lot with naming - it was sort of my obsession in this work to find short and obvious names for new things, many of which substitute the existing functionality, under the constraints of not altering what's already in the standard. For this sake, I've ventured into non-character symbols and even the keyword package - a major offence, I reckon... And here are a few of the findings I wanted to share (besides ? and with mentioned previously):

The only thing I failed to find a proper renaming for so far is mapcar. It is another one of those emblematic operations that should be familiar to everyone, yet -car creates confusion. For now, I resist the temptation to rename map into map-into and make map smarter by using the first sequence's type for the result expression. However, there's no plausible alternative variant I was able to find even among the zoo of other language's naming of this concept. Any thoughts?

PS. Those were a few prominent examples, but RUTILS, in fact, has much more to offer. A lot of stuff was borrowed from other utility projects, as well as implemented from scratch: anaphoric operators, the famous iter - a replacement for loop, Clojure-style threading macros, a new semantic pair data type to replace cons-cells, lots of utilities to work with the standard data structures (sequences, vectors, hash-tables, strings) making them truly first-class, iteration with explicit indices etc etc. With all that in the toolbox, there's now no ground to claim that Lisp is in any aspect inferior in terms of day-to-day UX compared to some other language, be it Haskell, Ruby or Clojure. Surely, I'm not talking about the semantic differences here.

17 May 2016 12:02pm GMT

Wimpie Nortje: Do you really want to use conditional compilation?

Edit (2016-05-18): Use featurep to improve readability in run time check. Thanks to @ogamita for the pointer.

Edit (2016-05-21): Fix spelling mistakes. featurep not featuresp. Thanks to @ngnghm for pointing that out.

How do I conditionally include code?

When code must change behaviour based on build time settings people often reach for the conditional reader macros (#+ and #-).

These macros have two properties one must be aware of.

  1. The compiler sees different code based on the condition.
  2. The macros are only evaluated at compile time.

Another thing to be aware of is that ASDF only recompiles files when their modification timestamps have changed. This is an optimization to decrease compilation time.

There are two situations where conditional code inclusion are most often used. The first is for writing code which is portable between different computing environments and the second is for setting behaviour options at build time.

In the first scenario the same source file must work on different platforms (e.g. 32 bit and 64 bit) and different compilers. The variance in target environments makes it a necessity to present different code to the compiler based on the environment. Once the file is compiled there is no reason to recompile it until it is moved to a new environment. For this scenario the two properties above (and ASDF's partial compilation) is exactly what is needed and the correct solution is the #+ and #- macros.

The second scenario happens when the application's behaviour can be modified by setting appropriate variables at build time. This technique if often used to switch a code base between development and production modes.

If conditional macros are used to perform this task it is easy to end up with files which were compiled under different conditions. This will almost certainly result in transient bugs, i.e. bugs which disappear when the complete project is recompiled.

Since a complete recompile avoids transient bugs the next logical step is to do a complete recompile every time strange behaviour is encountered. Doing such a recompile negates much of the benefit of ASDF's partial compilation.

Another issue is that compiling different code based on the environment means that you can never test the complete code base in a single environment. One problem with this is that there is always uncertainty about the source of a bug which is present in only one of the environments. Another problem is that one can get into a situation where buggy code is only ever present in an environment with no debugging facilities1.

In summary, using conditional macros to implement build time settings have the following problems2:

These problems can be avoided while keeping the build time settings by using run time checks instead of compile time checks. The code examples below illustrate both the compile time and run time methods.

Conditional behaviour using reader macros

This method causes trouble.

(pushnew :app-release *FEATURES*)


Conditional behaviour using run time checks

One possible solution to rid your code of conditional macros.

(pushnew :app-release *FEATURES*)

(if (uiop:featurep :app-release) 

Conditional reader macros or run time detection?

Though I have not seen much discussion about this topic, I have seen a few projects which implement build time settings as I suggest in this post.

Method Use case
#+ and #- The same functionality is implemented by different pieces of code for different environments.
Run time detection. Different behaviours are selected based on build time options.

  1. An example is to move between SLIME and Buildapp with debugging disabled.

  2. Also see Buildapp fails when using uncompiled libraries for another problem caused by conditional macros.

17 May 2016 12:00am GMT

11 May 2016

feedPlanet Lisp

Nicolas Hafner: 9th European Lisp Symposium - Confession 62

As I'm writing this I'm still in Krakow. Sitting next to me is Till, who joined me for ELS this year. It was a blast, but I'm also really exhausted and my throat is still hurting a bit from talking all the time the past three days. Our flight back to Zürich is in about an hour from now and I have a test to study for on the coming Thursday; I actually would've really liked to stay a bit longer, especially considering there were a few people I would've loved to talk to a bit more. Alas, you can't always get what you want.

But before I go through the entire thing by backtracking, let's instead reverse time all the way back to Sunday. Our flight to Krakow was scheduled for 17:00, so we had ample time to lounge around at home and try to relax a bit before the inevitable stress that is airport security and flying in general. Till had also packed way too much stuff, so we unloaded a bunch to make the carrying lighter. In hindsight I'm really glad we did that, as it turned out that we had to walk around quite a bit in Krakow.

At around two we then set off for the airport, where we had a quick lunch and noticed that Till had left his boarding pass in a book he had packed but we then left at home. Fortunately enough -after a bit of trouble with the Swiss website- we managed to download a copy of it to his tablet, so that all turned out fine. I suppose you really can't go for any kind of journey without at least some kind of oversight that gives you a hefty scare.

The plane we flew in was a small jet, but it was still pretty packed. I assume it was mostly Polish people returning home after a quick holiday break. On the flight my stomach got upset a bit, but otherwise it went by just fine. Once we finally arrived in Krakow we got a bit confused about the airport layout, as it was under pretty heavy construction. After some wandering about we managed to find the proper bus stop and get our tickets. I also exchanged way too much money for zloty, most of which is still in my wallet now. I didn't get much of an opportunity to waste it.

The bus ride to the hotel took around three quarters of an hour, so we got to have a good look around the outskirts of the city and its landscapes. The hotel itself was located in an area that looked rather worn down, the streets were not up to par and long stretches of the sidewalk were opened up for construction. After check-in and a short look-see at our room we decided to head on down to the bar and wait for someone to show up. Most of the conference people had already arrived in Krakow before us and were having a jolly time at the pre-conference registration party from what I overheard.

About an hour later we were joined by Christian Schafmeister and Joram Schrijver and the discussions immediately fired up. We talked a lot about Clasp and its near future- Christian was pretty worried about what he could show for his talk. He had the impression that people wouldn't be impressed by a new Common Lisp alone. I can't say I agree with that viewpoint, Clasp brings a lot of new stuff to the table that makes it a great addition to the list of implementations. The C++ interop alone is already noteworthy enough, but there's lots of smaller features that could prove very useful for larger projects. The biggest problem with Clasp remains however; there's just not enough people working on it to move it along quicker. Even with Christian's incredible speed and dedication, there's only so much he can do on his own. I've been trying to push Clasp into a situation where it is more accessible to other people for quite a while now, but especially with recent changes there's a lot left to be done for that- something that was reflected again throughout the discussions we had during the conference.

Later we were joined by a group of other lispers that were just returning from their previous party to start a new one at the bar. Things got rather lively and all sorts of topics got brought up. At around midnight I had to excuse myself however, as I wanted to be at least somewhat fresh on the coming morning. The hotel room was alright, at least it didn't smell terribly and was otherwise nicely roomy. Unfortunately the heating was also turned up enough that I couldn't fall asleep for about an hour. Opening the window cooled things down sufficiently and we finally managed to get some good rest in.

Finding the conference building in the morning was a bit tricky, but we managed to discover a kiosk along the way to get some snacks and drinks in. The conference provided for plenty of that on its own, but I was still glad to have a nice bottle of ice tea in my bag at all times. We got to the conference hall on time for the registration, but the actual conference organisation was oddly delayed. Nevertheless, discussion between the few people that had already showed up sparked almost immediately, so it didn't feel like we had to wait at all.

It was great to see Robert Strandh again as well, although I barely got to talk to him this year, much to my dismay. I'm hoping to remedy that next year. I also met Masatoshi Sano again, but I only got to talk to him on the second day. I met a few other people that I already knew from the previous ELS and had the pleasure of talking to them, but I unfortunately am terrible at remembering names, so I can't list them all here. My apologies.

Moving on to the talks. The first was about Lexical Closures and Complexity, by Francis Sergeraert. At points it was unfortunately -despite the rather heavy focus on Math at ETH- a bit over my head or moving too quickly, so I had a bit of trouble following along what exactly was happening. From what I could gather he uses closures to model potentially infinite or very large problem spaces and then perform various computations and mappings on those, thus still being able to compute real-value results without wasting enormous amounts of resources to try and model it all.

Next up was the language design section of the talks, the first of which focused on automated refactoring tools to aid students in finding style problems in their Racket code. It was fairly interesting to see a brief introduction to the tools used to both analyse and restructure source forms automatically to determine more succinct and idiomatic ways of achieving the same semantic result. It was also rather depressing to see some real-world snippets of code they had gathered from actual students. I can't say I'm surprised that this kind of absolutely horrendous code gets written by people being introduced to a language or programming in general, but one can't help but wonder if these 20-level nested ifs might stem from something other than the writer being new to it.

Following this was the demonstration of a library that extended the CL type system for a way to type-check sequences, allowing you to express things like plists as a type that you otherwise could not. It appears to me that this system might prove useful for succinct pattern checking, but unfortunately because these type definitions don't actually really communicate with the compiler in any way it is rather useless for inference or other potential optimisations that could be done if the compiler had actual knowledge of what kind of structure is being described. The code presented also used (declare (type ..)), which is the wrong way to go about something like this. Declarations are intended as promises from the programmer to the compiler. That all sane compilers also insert checks for the type on standard optimisation levels is not something that should be relied upon. check-type on the other hand would be perfectly suitable for this. However, at that point you might as well drop the type charade altogether and just have something like a check-pattern function that performs the test. Still, the talk presented an interesting view into how the actual sequence type descriptors are compiled into efficient finite state machines. The mechanism behind the type set merging was very intriguing.

Afterwards we heard a talk from Robert about his implementation of editor buffers, presenting an efficient and extensible way to handle text editing. I had read his paper on it before so I was already familiar with the ideas behind it, but it was a nice refresher to hear about it again. I'll make sure to see about hooking in his system when I inevitably get to the point of writing a source editor for QUI. It was also pretty surprising to hear about a topic like this since usually when using editors, one doesn't think about the potential efficiency problems presented by them- it all just works so well most of the time already. The biggest grievance for me in Emacs at the moment isn't necessarily the editing of text itself, even if that slows down to a crawl sometimes when I'm cruising about with a couple hundred cursors at the same time, no, the problem is with dynamic line wrapping. Emacs goes down to a complete crawl if you have any kind of buffer with long lines without line breaks. This however I assume has more to do with the displaying of the buffer than the internal textual manipulation algorithm. Maybe Robert has ideas for a good way to solve that problem as well.

During Lunch I had the great pleasure of meeting and talking to Chris Bagley of CEPL fame. I'll really have to look into that myself to see which parts of it can be incorporated into Trial. I'll definitely incorporate the Varjo part so that we can use sexprs for GLSL code, but there might be lots of other little gems in there that can be repurposed.

Following Lunch was the session on DSLs, starting off with a system to describe statistical tests in lisp. For this talk too I felt like I wasn't familiar enough with the areas it touched upon to really be able to appreciate what was being done. Apparently the system was used to do some hefty number-crunching. Besides, it's always great to see new discoveries in language evolution that allow a convenient description of a problem without sacrificing computational power for it.

The following talk stepped right into that line as well, presenting a high-performance image processing language called CMera (I believe). It does some really nifty stuff like automatically unrolling loops to avoid having to do edge testing in your tight loops all the time, expanding a single loop over an image into nine different parts that are all optimised for their individual parts. When comparing the code written against an implementation of the same algorithm in hand-written C++, the difference is absolutely astounding. Not only does this reduction in code size make things much more readable and maintainable, it also allows them to move much faster and prototype things quickly, all without having to sacrifice any computation speed. If anything they can gain speed by letting the compiler use higher-level information about the code to transform it into specialised and large but performant code.

Finally the last session of the day focused on demonstrations, firing right off with CL-MPI, a library to use the MPI system from within lisp while taking care of all the usual C idiosyncrasies, thus presenting a very handy API to run highly parallel and distributed code. Really interesting to me was their system to synchronise memory between the individual machines automatically. While the system seems pretty neat I couldn't help but wonder whether this might a bit too easily lead to either assuming synchronisation when none is present, or introducing a bottleneck in the system when it has to synchronise too often. Either way, I'm glad I don't have worry about highly distributed systems myself- measly threading alone is enough of a headache for me.

After this we had a really nice showing of an interactive computer vision system in Racket using the Kinect. That sounded like a very fun way to introduce students to Racket and computer vision in general. The few demos he showed also seemed very promising. Given last year's talk on computer vision, I might really have to take the time to look into this stuff more closely some time.

The last talk for the day focused on the problem of lexical variables in CL when debugging. Since lexical variables are often compiled away completely, it's tough to see their values during debugging. SBCL often does a pretty good job at retaining this information when compiling with (debug 3) in my experience, but there's certainly times when it doesn't, and I can see the value in implementing a system that can ensure that this gets preserved in every case, especially one that's portable across implementations. Apparently there's some really nasty code walking necessary to get the job done, and there's apparently still no code walker around that actually works as expected on all major implementations, which is a bit of a downer.

As usual closing off the day was a session of lightning talks. This year mine was a form of continuation on my last year's talk about Qtools. I talked very briefly about Qtools-UI, the effort to replace Qt parts that aren't extensible enough and thus provide a more convenient base for people to work with. I'm not sure if I managed to convince anyone to contribute to it, but hopefully it'll at least linger around in some people's heads so that they might remember it if they ever come across the need to write a GUI.

The rest of the lightning talks I'm afraid to say I can't quite remember. My memory is rather shoddy at the moment and the only reason I remember all the other talks is because I looked up their titles on the website. So, my apologies for skipping out on this, but I think the article is already plenty long as it is so going into detail on all these would only make it all the longer.

The first day was concluded by a Chris, Christian, Joram, Till, and I going back to the hotel for a brief chat at the bar, followed by a quest in search for pizza. We looked up a bunch of places near the hotel and went on our way. The first we encountered was too full and the other was near a campus that was filled with students drinking booze and doing BBQ; we deemed it a bit too lively for us. The third one was inside a student dorm building, but had enough space for us to spend some hours talking and eating. The pizza tasted very differently from what I'm used to. It wasn't bad, but also not really my kind of thing.

Some more talking and a good night's rest later it was already Tuesday. Time flies when you're having a blast. We got up a tad later this time around and walked through a convenience store. I was relieved to see that just like all the stores I'm used to the layout is as confusing as possible so that you have to waste lots of time walking by everything except what you're looking for.

Since we were close on time and there was a group photo to shoot we didn't get any time to talk before the first talk. It started off with a presentation of the Julia language by Stefan Karpinski. Julia seems like a really nice replacement for Matlab and I'd very much welcome it if it got more ground that way. However, some of the points that were presented here didn't really seem to make much sense to me. One thing that was emphasised as distinguishing Julia is that number types and arithmetic aren't in the specification, but rather defined in Julia code itself. This sounds like a neat little thing to do for curiosity's sake, but I just can't see the benefit of it. Not to mention that now instead of reading some pages of a spec you have to read some pages of code with possibly arcane interconnections and optimisations going on. Whether this makes anything more clear is really dubious to me. I'm guessing that this part was mostly mentioned at all to at least bring something new to the table since it would otherwise be pretty hard to impress lispers. Another thing I was confused about is that he seemed to hint at the possibility of writing functions that get the information about the inferred type of the compiler and can use that to generate different code, which is something that I've missed in CL in places where macro functions could be further optimised with that kind of information, but the example he showed didn't seem to use that in any way or even get it at all, so I'm not sure if I didn't catch that part or what exactly is going on with that.

A quick break later we got to the implementations part of the talks. Robert presented his modern implementation of the loop macro, which uses a system of combinatory parsing and full CLOS to allow it to be extensible. I'd love to have a portable way to extend loop as iterate really doesn't appeal to me much at all and there's currently nothing else that is extensible for custom sequences and clauses and the like. I'm not sure if his implementation will be adopted, but I would definitely welcome it.

The next two talks, which were about source translation in Racket and STM in Clojure, I'm sad to say I can't really talk about because I was distracted by a bug I had discovered momentarily and couldn't help myself but try to fix. I got absorbed all too easily, so I didn't catch much of it.

During the lunch break I got to talk to Masatoshi Sano for a good while, we mostly discussed the prospect of using Roswell for my Portacle project and talked about some of the difficulties or ways to deal with what I'll just call "The Windows Situation". I later talked to Joram a bit about potentially getting him involved in the Colleen3 or Trial projects, for which I'd heartily welcome some contributors or even just discussion partners. I'm very excited about the prospect of working with him on that.

And then came the big one. Christian's talk was, just like last year, pretty comparable to a bomb dropping. Some suspect that he's not of this world. Ignoring the question of his conception, hearing him in his element talking about Chemistry is always a treat. He showed off a nice demo of what CANDO is capable of and it really looks like a nicely lispy way of performing chemistry modelling. This is said from what I can tell with my practically nonexistent knowledge of chemistry, so I can't really claim to have a grasp on what you can actually do with it. Given that he's been at this for such a long time though, I'm convinced he knows what he needs to do in order to create things like what he presented to us- a perfect water filtering membrane. I'm still glad to have gotten involved with Clasp, it has given me lots of really great talking and thinking opportunities. It's exciting to listen with or discuss the in-depth details of what's going on inside Clasp. Now though Christian needs to get his chemistry stuff off the ground so that he can get enough funding to continue Clasp. Unfortunately grants have been hard to come by for him and that's a looming pressure that has been haunting the project many times before. Hopefully he'll be able to prove the worth of Clasp and Cando in the near future. I wish him all the luck.

Next up we had a presentation about the question of how different implementations of a depth of field effect perform on different hardware. This was mostly a concerning example as to how much code still needs to be tuned to the hardware it's being run on today. Maybe the effect is actually even more so now, since lots of hardware that lies in the same category is still very differing on what it is adept at. Thankfully I am mostly staying clear of such optimisation lunacy.

Some more coffee passed by and we were ready for the last session of talks for the conference. The debut was made by James Anderson, presenting his research into how source files are connected with each other and what kind of dependencies exist between them. He wrote a system that analysed the entire Quicklisp ecosystem's files for symbol references between things and then crunched all that data down into interesting graphs using his own database technology. The graphs for larger systems like qtools-ui look like a complete jumble as expected. He also mentioned the difficulties of trying to extract this kind of relationship information since he could only inspect code by reading it in. This is particularly a problem for methods, since they are likely to be defined from lots of different source files and potentially packages, but without at least type inference or even runtime information you can't really know where the dependency goes. Initially his idea for doing this seemed to be that he doesn't want to have to write the dependency information into his system definition files and the system should be able to infer it automatically. I'm not so sure that this is is a good idea, or that it is in fact such a problem. It seems like a rather minor inconvenience to me, but then again I've never written systems on a very large scale.

Closing it all off we had a presentation of Bazel and how it can be used to build Lisp. The most promising feature of it all being that you are able to use it to statically link libraries into an SBCL binary. I'm not convinced that Bazel is the tool to use for building unless you have a gigantic project or ecosystem surrounding it already however. It seems ridiculously heavy-weight and paying the price for it and its different way of configuration and operation does not seem worth the benefits unless you really need static linking and cannot do with shared libraries. Still, it gave me some things to think about for the eventual effort of writing my own build system, whenever that will happen.

Then as before we had some more lightning talks to round it all off. I didn't do a second one this time around, mostly because I didn't really know what to talk about for Trial. It did not seem finished enough to present yet. Maybe next year.

Finally the conference was rounded off by some goodbye messages from the conference organisation and the announcement that the next ELS might be happening in Brussels. We then had two hours left before the banquet. Michal Herda guided us into the inner parts of the city where we got to see some nice architecture. Along the way Chris Bagley and I chatted about the problems in writing game engines and games in general and some other assorted topics.

Once we arrived at the banquet I was pretty beat. The place we stood at looked oddly high-brow. The tables were set with all the usual you get in fancy restaurants- multiple wine glasses, forks and knives. It seemed in an odd conflict with the rest of the getup of the conference attendees. We all looked far too casual for this kind of thing. Due to my pickiness I couldn't eat much of the food that was being served either. It certainly looked fancy, but I don't think the taste was in accordance to that. From what I've heard from others or noticed in their expressions it was nothing exceptional. No matter for me either way though, since I came all this way not to eat, but to finally be with people that understood me and vice versa. And I got ample opportunity to do exactly that. During the dinner I mostly talked with Joram about Colleen3 and Markless.

Soon enough it hit 22:00 and we had to leave. Due to the long way back to the hotel and other delays in saying goodbye to everyone, we only arrived around two hours later, at which point I just slammed myself into bed after making sure that I got the boarding pass for next morning.

And so today I woke up at six, just to make sure that we had plenty of time for potential mistakes. Three quarters of an hour later we were on the bus to the airport. Half an hour later we had already passed through security and were waiting at the gate, at which point I started writing this. The flight after was rather annoying. It was pretty packed and there were lots of screaming children on board, the bane of any flight passenger. I have no idea why there were so many families on board, let alone on a Wednesday morning, let alone from Krakow to Zürich. Despite hellish screams of tortured souls haunting us along the way we made it back safely.

Now it's already 16:00 and aside from getting home and continuing to write this I only got the time to cook some nice lunch- the kitchen remains to be cleaned.

I suppose I should try to form some sort of a conclusion here to end this overly long article on a good note. If it wasn't already apparent from my descriptions, I had a grand time at the conference and I'm really glad that I could attend again this year. A huge thanks to everyone that was willing to talk to me and especially to all the people that got involved to make it all happen. Hopefully it'll happen again next year; I'm definitely looking forward to it.

For now though I have to get back to work hacking lisp. There's so much left to be done.


11 May 2016 2:58pm GMT

10 May 2016

feedPlanet Lisp

Vsevolod Dyomkin: European Lisp Symposium 2016

The last two days, I'm at the ELS2016. So far, it's being a great experience - I've actually forgotten the joy of being in one room with several dozens of Lisp enthusiasts. The peculiarity of this particular event is that it's somewhere in the middle between a scientific conference, like ACL, that I had a chance to attend in the recent years thanks to my work at Grammarly, and a tech gathering: it employs the same peer reviewed approach and a scientific presentation style you will find at the research conferences, but most of the topics are very applied and engineering-related.

Anyway, the program was really entertaining with several deep and insightful presentations (here are the proceedings). The highlights for me were the talks on the heterogenous sequences type-checker implementation based on the Lisp declare facility (that I'm growing more and more fond) by Jim Newton and a presentation of an image-processing DSL that's an excellent example of the Lisp state-of-the-art approach in DSL design by Kai Selgrad. Other things like a description of the editor buffers protocol, local variables preservation technic were also quite insightful. And other good stuff is coming...

It's also great to hear new people bringing fresh ideas alongside old-timers sharing their wisdom and perspective - one of the things I appreciate in the Common Lisp community.

Near the end, I'm going to present a lightning talk about RUTILS and how I view it as a vehicle for evolving the Common Lisp user experience.

Sugaring Lisp for the 21st Century from Vsevolod Dyomkin

10 May 2016 11:19am GMT

07 May 2016

feedPlanet Lisp

Christophe Rhodes: not going to els2016

I'm not going to the European Lisp Symposium this year.

It's a shame, because this is the first one I've missed; even in the height of the confusion of having two jobs, I managed to make it to Hamburg and Zadar. But organizing ELS2015 took a lot out of me, and it feels like it's been relentless ever since; while it would be lovely to spend two days in Krakow to recharge my batteries and just listen to the good stuff that is going on, I can't quite spare the time or manage the complexity.

Some of the recent complexity: following one of those "two jobs" link might give a slightly surprising result. Yes, Teclo Networks AG was acquired by Sandvine, Inc. This involved some fairly intricate and delicate negotiations, sucking up time and energy; some minor residual issues aside, I think things are done and dusted, and it's as positive an outcome for all as could be expected.

There have also been a number of sadder outcomes recently; others have written about David MacKay's recent death; I had the privilege to be in his lecture course while he was writing Information Theory, Inference, and Learning Algorithms, and I can trace the influence of both the course material and the lecturing style on my thought and practice. I (along with many others) admire his book about energy and humanity; it is beautifully clear, and starts from the facts and argues from those. "Please don't get me wrong: I'm not trying to be pro-nuclear. I'm just pro-arithmetic." - a rallying cry for advocates of rationality. I will also remember David cheefully agreeing to play the viola for the Jesus College Music Society when some preposterous number of independent viola parts were needed (my fallible memory says "Brandenburg 3"). David's last interview is available to view; iPlayer-enabled listeners can hear Alan Blackwell's (computer scientist and double-bassist) tribute on BBC Radio 4's Last Word.

So with regret, I'm not travelling to Krakow this year; I will do my best to make the 10th European Lisp Symposium (how could I miss a nice round-numbered edition?), and in the meantime I'll raise a glass of Croatian Maraschino, courtesy of my time in Zadar, to the success of ELS 2016.

07 May 2016 8:30pm GMT

04 May 2016

feedPlanet Lisp

drmeister: Linking LLVM bitcode files for a dynamic language

Clasp Common Lisp is a dynamic language in which every top-level form needs to be evaluated in the top level environment. Clasp compiles Common Lisp code to bitcode files and then links them together into a shared library or an executable. When the library or the executable are loaded, each top-level form needs to be evaluated. This is until Clasp gains the ability to save a running environment to a file, which it doesn't have yet. Even then, the ability to play back the top-level forms will be needed to create the environment to write to a file.

So the compiled bitcode files need to keep track of the top-level forms so that they can be played back at startup. Clasp does this by defining a "main" function with internal linkage (called "run-all") for each llvm::Module. "run-all" evaluates every top-level form that was compiled into the llvm:Module. The tricky part is how does this main function get exposed to the outside world so that it can be called if a single bitcode file is loaded into clasp or linked together into a library or executable and invoked with other "run-all" functions from other modules.

Clasp creates a global variable in each module called "global-run-all-array" that stores an array of initially one function pointer that points to the module's "run-all" function. The "global-run-all-array" global variable is defined with "appending" linkage. What this does is when bitcode files get linked together by the system linker, the "global-run-all-array" will have all of the "run-all" functions appended together and put back into the "global-run-all-array" global variable.

Then there is the problem to determine the number of entries in the "global-run-all-array". Clasp solves that by ensuring that the last module linked in a list of modules has a two element "global-run-all-array" where the second element is NULL and by adding a second global variable called "global-epilogue".

When a bitcode file or a shared library is loaded into Clasp, it checks for the "global-epilogue" symbol, if it finds it then it knows that the "global-run-all-array" contains a NULL terminated array of function pointers to call. If "global-epilogue" is not present, then it knows that "global-run-all-array" contains a single function pointer.

Clasp then invokes each of the "global-run-all-array" functions one after the other and each one of them invokes the compiled functions for the top-level forms for each of the bitcode files.

This only takes a few seconds when starting up Clasp.

Note: I haven't been actively blogging - because I'm very, very actively programming. If you want to say hi, I'm on IRC, freenode.org #clasp almost every day.

04 May 2016 6:20pm GMT

03 May 2016

feedPlanet Lisp

CL Test Grid: quicklisp 2016-04-21

The difference between this and previous months:

Grouped by lisp implementation first and then by library:

Grouped by library first and then by lisp impl:

(Both reports show the same data, just arranged differently)

As usually, some new libraries start to fail on old lisp implementations because they need newer ASDF.

3d-vectors redefines constant to a value not eql to the previous value. clinch starts to fail on several lisps. hyperluminal-mem refers undefined variable MOST-POSITIVE-B+SIZE. rutils crashes CCL. And some other failures. There are improvements too of course.

If you're interested in some particular failure or need help investigating something, just ask in the comments or on the mailing list.

03 May 2016 1:44am GMT

Wimpie Nortje: Buildapp fails when using uncompiled libraries.

Note: I use CCL 64-bit on Linux. I have not checked this on anything else.

I discovered that Buildapp fails to build a project when the FASLs1 for some 'standard' libraries are not available. 'Standard' meaning well known, mature libraries like Alexandria.

When I started using Common Lisp I tended to use the reader macros #+ and #- to conditionally compile for development or production. Since ASDF uses file modification times instead of code dependencies to determine what to recompile it would often happen that the project's various files were compiled using different compilation conditions. This caused many mysterious bugs.

The solution to this environment mismatch is to force a recompile of the whole project. I use a two-phase process of generating a manifest and then building the application binary. The recompile is forced by deleting2 the project FASLs before each phase.

At one point I tracked an elusive bug to a library which uses conditional reader macros for conditions which differ between my development and production environments. I extended my FASL deletion to include all the Quicklisp libraries for both building phases. This caused Buildapp to fail.

By using (ql:quickload) in its verbose mode during the build process I saw that CCL emitted 'compilation failure' warnings for some libraries while generating the manifest. Many mature and often used libraries caused such warnings. Compilation completed successfully in spite of the warnings and these libraries have been used like that for years so it does not seem to be a serious problem.

During the binary creation phase Buildapp exited at the first occurrence of a 'compiler failure' warning with an error. It seems that Buildapp escalated all warnings to errors which caused it to fail.

When your own code triggers this behaviour it is useful because it helps you ship better software. However, when external libraries trigger the failure it is extremely annoying because it blocks your development effort.

The solution for making a complete build in a consistent environment is to do a full clean before generating the manifest and project-only clean before building the binary. This enables Buildapp to load the libraries while still compiling the complete set in a known environment but it requires that the environment conditions for the libraries remain constant during the manifest generation and building phases3.

  1. 'FASL' is short for 'FASt Loading'. It is a binary file containing compiled code. ASDF prefers to load code from a FASL rather than a source file when it determines that the source has not changed since being compiled.

  2. ASDF documentation provides three options for forcing a recompile: (1) The (clear-system) API call, (2) touching the system's .asd file, and (3) deleting the project FASLs.

  3. This can be a tricky requirement because some libraries use #+quicklisp which definitely changes from manifest to building.

03 May 2016 12:00am GMT

28 Apr 2016

feedPlanet Lisp

Zach Beane: New version of ZS3 supports AWS4 auth

I just published ZS3 1.2.8. It's available on my website and will be in the next Quicklisp dist update in May. The main difference is support for the latest AWS authentication system. This makes ZS3 work properly with the latest AWS regions like Frankfurt and Seoul.

If you have any trouble using it, please let me know!

(This work was for a paying customer; if you are interested in specific updates and features in ZS3 or any of my other software, get in touch.)

28 Apr 2016 4:42pm GMT

22 Apr 2016

feedPlanet Lisp

Daniel Kochmański: Creating a project homepage with the SCLP


In this short tutorial I'll describe how to bootstrap easily a project website. In fact that's what I did today with the Embeddable Common-Lisp website in order to provide the RSS feed and make putting there the news easier.

Additionally I'm showing here, how to create a standalone executable for coleslaw with clon after providing quicklisp independant bundle of systems.

Quick start

First clone the repository:

$ cd /home/p/ecl
$ git clone https://gitlab.common-lisp.net/dkochmanski/sclp.git website
$ cd website

Now you should adjust the appropriate files. Edit .coleslawrc (file is self-explanatory), static pages and posts.

Each file with the extension *.page is a static page. pages/main.page is an example template with a static page - don't forget to link it in the .coleslawrc's sitenav section. Exact URL of the page is declared in the file's header.

Files named *.post represent blog/news posts which appear in the RSS feed. They are indexed and accessible from the root URL. Supported file formats are markdown, html and cl-who (if enabled).

When you're done, you could just load coleslaw with your favorite CL implementation, using Quicklisp load coleslaw and call the function main on the website directory:

(ql:quickload 'coleslaw)
(coleslaw:main "/home/p/ecl/website/")

We will take more ambitious road - we'll create a standalone executable with a proper command line arguments built from a clean bundle produced by Zach Bane's Quicklisp. CLI arguments will be handled by Clon - the Command-Line Options Nuker, an excellent deployment solution created by Didier Verna.

Creating the bundle

Bundle is a self-containing tree of systems packed with their dependencies. It doesn't require internet access or Quicklisp and is a preferred solution for the application deployment.

Some dependencies aren't correctly detected - Quicklisp can't possibly know, that our plugin will depend on the cl-who system, and it can't detect cl-unicode's requirement during the build phase - flexi-streams (this is probably a bug). We have to mention these systems explicitly.

Clon is added to enable the clonification (keep reading).

(ql:bundle-systems '(coleslaw flexi-streams
                     cl-who cl-fad
                   :to #P"/tmp/clw")

Clonifying the application

(in-package :cl-user)
(require "asdf")

(load "bundle")
(asdf:load-system :net.didierverna.clon)
(asdf:load-system :coleslaw)
(asdf:load-system :cl-fad)

(use-package :net.didierverna.clon)
(defsynopsis (:postfix "DIR*")
  (text :contents "Application builds websites from provided directories.")
  (flag :short-name "h" :long-name "help"
        :description "Print this help and exit."))

(defun main ()
  "Entry point for our standalone application."
  (when (getopt :short-name "h")
  (print (remainder))
  (handler-case (mapcar
                 #'(lambda (p)
                      (cl-fad:pathname-as-directory p)))
    (error (c) (format t "Generating website failed:~%~A" c)))

(dump "coleslaw" main)

You may generate the executable with sbcl and ccl (ecl has some problems with the coleslaw dependency - esrap, I'm working on it). I have used ccl, because it doesn't "derp" on the symbol exit and produces slighly smaller executable than sbcl.

Issue the following in the bundle directory (/tmp/clw):

ccl -n -l clonify.lisp

This command should create native executable named coleslaw in the same directory. On my host ccl produces binary with the approximate size 50M.

Executable usage

This is a very simple executable definition. You may extend it with new arguments, more elaborate help messages, even colors.

To generate a websites with sources in directories /tmp/a and /tmp/b you call it as follows:

./coleslaw /tmp/a /tmp/b

That's all. Deployment destination is set in the .coleslawrc file in each website directory.

Adding GIT hooks

You may configure a post-receive hook for your GIT repository, so your website will be automatically regenerated on each commit. Let's assume, that you have put the coleslaw standalone executable in place accessible with the PATH environment variable. Enter your bare git repository and create the file hooks/post-receive:

cd website.git

cat > hooks/post-receive <<EOF
########## CONFIGURATION VALUES ##########



if cd `dirname "$0"`/..; then
    cd $OLDPWD || exit 1
    exit 1

git clone $GIT_REPO $TMP_GIT_CLONE || exit 1

while read oldrev newrev refname; do
    if [ $refname = "refs/heads/master" ]; then
        echo -e "\n  Master updated. Running coleslaw...\n"
        coleslaw $TMP_GIT_CLONE


That's all. Now, when you push to the master branch your website will be regenerated. By default .gitignore file lists directory static/files as ignored to avoid keeping binary files in the repository. If you copy something to the static directory you will have to run coleslaw by hand.


Coleslaw is a very nice project simplifying managing project website with easy bootstrapping the site without any need to maintain working lisp process on the server (this is static content which may be served with nginx or apache) and allowing easy blogging (write a post in markdown and push to the repository).

Sample Common-Lisp Project is a pre-configured website definition with a theme inspired by the common-lisp.net projects themes with some nice features, like RSS feed and blog engine (thanks to coleslaw).

We have described the process of creating a simple website, creating a standalone executable (which may be shared by various users) and chaining it with git hooks.


22 Apr 2016 12:00am GMT