fidzu : LISP

I have recently looked at Julia, a new programming language developed at MIT that promises to be a dynamic programming language that is suitable for scientific computing with a high-performance implementation. It is an interesting project that heavily borrows from Common Lisp, Dylan, and Scheme, and you can rightfully argue that Julia itself is actually a Lisp dialect. While I wasn't very impressed with some other recent Lisp dialects (Arc, Clojure), Julia puts a couple of very interesting features on the table. Below is a discussion of the more salient aspects of Julia, as seen from a Common Lispers perspective. It is based on the documentation of the prerelease version 0.3 of Julia.

Julia is closest to Dylan in many regards. It uses a somewhat mainstream syntax rather than s-expressions. Unlike Dylan, you can nevertheless write 'full' macros, since macro definitions are implemented in Julia, not some template language, and backquote/quasiquote is integrated with the Julia syntax. Julia is a Lisp-1 (like Scheme or Dylan) rather than a Lisp-2 (like Common Lisp or ISLISP), which makes it necessary to add macro hygiene features. Fortunately, this does not mean you have to deal with the rather painful syntax-case construct of some Scheme dialects, but you can still use far simpler backquote/quasiquote constructions, just with macro hygiene taken care of by default. Julia also allows you to selectively break hygiene. Although I usually strongly prefer the simplicity of Common Lisp's non-hygienic macro system, the fact that Julia is a Lisp-1 turns macro hygiene into a real problem, so I guess this is a reasonable design.

Julia provides object-oriented programming from the ground up, similar to Dylan. It centers on generic functions rather than classes, where methods are defined outside classes and allow for multiple dispatch, just like in Dylan and Common Lisp. Like Dylan, and unlike Common Lisp, it does not distinguish between functions and generic functions: All functions can have methods, you do not have to make up your mind whether you want methods or plain functions. Unlike in Common Lisp, there are no method combinations, no before/after/around methods, and call-next-method is not directly supported, but has to be done manually. This is probably to simplify method dispatch, maybe to have some performance advantages, though I find it hard to imagine that adding method combinations would make things substantially worse.

You still need a class hierarchy to drive method dispatch. Unlike in Common Lisp, there is no multiple inheritance, only single inheritance. In fact, there is actually no real inheritance, because in Julia, only leaf classes of the class hierarchy are allowed to define slots/fields. All superclasses are required to be "abstract," without any slot definitions. Also, Julia classes cannot be redefined at runtime, so in fact Julia classes are much closer to Common Lisp's structured types rather than classes.

Julia's execution model is based on dynamic compilation. As a user, you don't have to compile your code at all, source code is just compiled on the fly (similar as in Clozure Common Lisp). Julia inlines functions on the fly, including generic functions, and can de-optimize when function definitions change at runtime. This is more flexible than in Common Lisp, where inlined functions can get out of sync with their potentially changed definitions. Also, while the Common Lisp specification does not say anything with regard to being able to inline generic functions or not, there are aspects in the CLOS MOP specification that prevent generic functions from being inlined, at least for user-defined extensions of generic functions. Julia definitely seems more "modern" here.

In Julia, there is no distinction between variable binding and variable assignment. If you assign to a variable that has not been used before in the same lexical environment, it is silently introduced. In Common Lisp/Scheme/Dylan, there is a distinction between 'let forms that introduce variable bindings, and assignments (setq/setf/set!) that perform assignments. I'm highly skeptical of Julia's design here, because this potentially leads to bugs that are hard to find: A simple typo in your source code just may go unnoticed.

In Julia, all variables are lexically scoped (except for some seemingly quirky scoping semantics for global variables, see below). There are no special / dynamically scoped variables in Julia, which is a major omission in my book. Some academics don't like special scoping, but their presence in Common Lisp is incredibly useful in practice, especially but not only for multi-threading!

Julia's default representation for integers is either 32-bit or 64-bit integers, depending on the target architecture, which silently wrap around. Julia also supports "BigInts" that can be arbitrarily large, but you have to ask for them explicitly. In Common Lisp, integers are by default arbitrarily large, which I think is an advantage. Due to type tagging in Common Lisp implementations, even "big" integers are typically allocated as immediate values rather than on the heap when they fall into the "fixnum" range. I didn't find anything in the Julia documentation that discusses this aspect of "BigInt."

In Julia, all mathematical operations are generic functions and can be extended by user-defined methods. This is a strong advantage for Julia. In Common Lisp, mathematical operations are "plain" functions which cannot be extended. Due to some aspects in the design of Common Lisp's generic functions, it's hard to inline (or open-code) them, which is why for performance reasons, it's better to express mathematical (and other such performance-critical functions) as "plain" functions. Apart from that, the support for number types seems to be on par between Julia and Common Lisp (complex types, rational numbers, floating point numbers, etc.)

In Julia, strings are unicode strings by default. My knowledge about unicode support in Common Lisp implementations is limited, so I cannot really make any comparisons here. One interesting aspect in Julia's string support is that there can be user-defined macros to parse them and construct other syntactic entities out of them. This feels somewhat similar to read macros in Common Lisp, although with a slightly different scope.

Julia's support for functions is similar to Common Lisp: They can be first class and anonymous (lambda expressions). There are varargs (&rest), optional and keyword arguments. In Common Lisp, optional and keyword arguments cannot be dispatched on in methods. In Julia, optional arguments can be dispatched on, but not keywords. (This is a pity, dispatch on keyword arguments would be very interesting, and is something I wanted to add as part of Closer to MOP for a long time!)

Julia's support for control flow is much more limited than in Common Lisp. There are equivalents for progn, cond/if, for and while loops. Unlike Common Lisp, there is no support for a full loop facility, or even for a simple goto construct. Common Lisp clearly wins here. Julia's support for exception handling is also limited: No handler-bind, no restarts, unlike in Common Lisp, which are also really useful features.

Julia's type system has some interesting differences to Common Lisp: There is a distinction between mutable and immutable classes. Immutable classes disallow any side effects on their fields. This seems to be primarily directed at enabling stack allocation as an optimization. In Common Lisp, you would use dynamic extent declarations when allocating structs (or other data types) to achieve similar performance improvements. I'm not sure why it would matter that the fields need to be read-only for such an optimization, but if this covers most cases, maybe this is good enough.

Julia allows for returning and receiving multiple values from function calls, similar to Common Lisp. This is a feature I like a lot in Common Lisp, so I'm happy it's also in Julia. In Julia, this is achieved by an explicit tuple type representation which doesn't exist in this form in Common Lisp. In Lisp, you could also return/receive lists instead of multiple values, which would correspond to this kind of tuple type, but lists add an additional performance overhead, which multiple values and, presumably, tuples in Julia don't have.

Julia supports parametric types. I don't see at the moment why this is relevant. You could achieve similar functionality also with some macrology. Maybe I'm missing something here.

There is a whole section on constructors (make-instance / make-struct) in the Julia manual. This makes me suspicious, this should be an easier topic.

Julia has a module system. It supports export and automatic import of explicitly exported identifiers. You can also still access non-exported identifiers with additional syntax. This is good, because module designers may not always perfectly anticipate what users actually need to access. Common Lisp's package system supports a similar distinction between external and internal definitions that can be accessed in different ways. I slightly prefer Common Lisp's ability to use the package name as a prefix even for explicitly exported definitions. There is no feature in Julia to rename imported identifiers, which is something where Common Lisp's support for explicit package name prefixes can come in very handy. I would like to see something like Oberon's support for renaming identifiers on import in some Lisp dialect someday, because I believe that is the most complete solution for dealing with potential name conflicts.

In terms of meta-programming, apart from macros, Julia also supports (top-level) eval, like Common Lisp. Julia's support for "reflection" is much weaker than Common Lisp's CLOS MOP: You can only inspect types at runtime, but you cannot modify them (and language designers should stop calling something "reflection" that is clearly just "introspection").

Both Julia and Common Lisp support multi-dimensional arrays. Common Lisp's arrays are in row-major order, starting at index 0 in every dimension. Julia's arrays are column-major order, starting at index 1 in every dimension. Julia's support for multi-dimensional arrays is a library feature, whereas Common Lisp's support is built into the language. Julia supports both dense and sparse matrices, where Common Lisp supports only dense matrices out of the box.

There are a lot of libraries that ship with Julia targeted at scientific computing.

Julia supports parallel programming with a model built on top of message passing: If you want to run parallel algorithms, you essentially start several instances of Julia that communicate with each other. The model does not support shared memory, and there is no multi-threading within a Julia instance (although there seem to be discussions among the Julia designers to add this in the future). The model is built on top of MPI as an implementation backend. However, the actual programming model supports single-sided communication: You can ask a function to be executed in some other Julia worker process, and can later synchronize with it to fetch results. On top of that, there are some high-level constructs provided as library features, such parallel maps and loops. Julia's message passing model ensures that within a Julia instance, only one task is executed at a time, so there is yet no need to provide low-level synchronization mechanisms, such as locks or atomic operations. The lack of shared-memory parallelism is problematic because many parallel algorithms that are very easy to express with shared memory become quite complicated in a distributed memory setting. On the other hand, Julia's model easily supports true distributed programming: You can configure Julia to run several instances across a cluster, and use them in a quite straightforward way: substantially easier than what you have to do with, say, MPI, and much closer with regard to ease of use to modern PGAS languages like Chapel or X10.

The ANSI specification for Common Lisp does not mention anything about multi-threading or parallel programming at all, but many Common Lisp implementations add support for shared-memory parallelism. I will not go into details here, but let me just briefly state that, for example, the LispWorks implementation of Common Lisp provides excellent support for symmetric multiprocessing that is at least on par with what you can find in most other language implementations in terms of parallel programming support. However, unfortunately, support for true distributed memory models seems almost non-existent in Common Lisp, apart for some basic support for MPI in a library that was maintained only for a very short period of time a couple of years ago. Julia looks like a good source of inspiration for adding such features to Common Lisp.

However, one aspect of Julia's message passing approach seems problematic, as far as I can tell: You can pass closures between different instances of Julia, but it's not clear how free variables in a lambda expression are bound. It seems that lexical variables are bound and serialized to another process, but global variables are not serialized and need to be present in any presence that may execute the closure. Experiments with adding side effects to free variables in lambda expressions that are passed to other processes seem to suggest that the semantics of this combination of features are unpredictable. At least, I have not been able to figure out what happens when - sometimes variables seem to be updated at the sender's side, sometimes not - and I didn't find a discussion of this topic in the Julia manual.

Anyway, as you can tell, there are a lot of interesting features in Julia, and it's definitely worth a try.

20 Jul 2014 2:49pm GMT

Emacs Live is a frakkin' epic compilation of customizations and packages, streamlined for the ultimate hacking experience. As I mentioned in my last post, Adventures in Clojure, it's under development by the same team as the Clojure bridge to SuperCollider, Overtone. The one downside? It was designed for hacking Clojure, so it doesn't include SLIME and Common Lisp support out of the box, and of course, it completely replaces your ~/.emacs.d/ directory and ~/.emacs config file, so you lose your existing SLIME setup (and all your other customizations) when you install Emacs Live. Don't panic, the Emacs Live installer is smart enough to move your existing ~/.emacs.d/ folder and ~/.emacs config file to a safe place.

Emacs Live does, however, offer a pretty neat interface for creating Live Packs, boxed collections of Emacs packages and customizations, that can all be loaded together as a single package via ~/.emacs-live.el, stored outside the managed ~/emacs.d/ directory so that they can be maintained across updates. This made it only slightly less trivial than normal to get SLIME set up and running in Emacs Live.

To get the full Emacs Live experience for Common Lisp, however, you also need another package, AC-SLIME. It provides the auto-completion and pop-up documentation for Common Lisp, both in file buffers and the REPL.

I have packaged both together in an Emacs Live pack, which you can get at: https://github.com/thephoeron/slime-pack. Installation is a cinch. After installing Emacs Live, just clone SLIME-PACK into your ~/.live-packs/ directory, and add the following line to your ~/.emacs-live.el config file:

(live-append-packs '(~/.live-packs/slime-pack))

The default inferior-lisp-program for SLIME-PACK is SBCL. You can change this, as normal, by setq'ing inferior-lisp-program to your chosen Lisp implementation after the code above.

Once that's all done and saved, either restart Emacs or compile and load your modified config file. You can then simply M-x slime as normal, and enjoy all the extra Emacs Live awesomeness for Common Lisp!

17 Jul 2014 1:58pm GMT

New projects:

• bit-smasher - Common Lisp library for handling bit vectors, bit vector arithmetic, and type conversions. - MIT
• cl-cli - Command line parser - WTFPL
• cl-voxelize - Cl-voxelize is a library to convert polygon models into voxel models for particle-based physics simulation. - LLGPL
• equals - Generic Equality and Comparison for Common Lisp - MIT
• hash-set - An implementation of the hash-set data structure. - Unlicense
• mpc - Monadic Parser Combinators for Common Lisp. MPC tries to be simple and practical while being powerful, well documented and fairly performant. A friendly fork of Drew Crampsies _Smug_ library. - GNU Affero General Public License
• pp-toml - TOML parser - LLGPL
• printv - printv: a batteries-included tracing and debug-logging macro - Apache 2.0
• purl - Parse and print URLs as described in RFC 1738. - GNU AGPL
• software-evolution - programmatic modification and evaluation of extant software - GPL V3
• south - Simple OaUTH library for oAuth1.0 - Artistic

Updated projects: 3bmd, access, aws-sign4, btrie, caveman, chirp, cl-ana, cl-async, cl-autowrap, cl-charms, cl-colors, cl-conspack, cl-coroutine, cl-ftp, cl-fuse-meta-fs, cl-html5-parser, cl-ltsv, cl-mustache, cl-plplot, cl-ply, cl-project, cl-qrencode, cl-rdfxml, cl-rethinkdb, cl-sdl2, cl-xul, clack, closer-mop, clsql-helper, coleslaw, colleen, com.informatimago, conium, contextl, crane, datafly, drakma-async, esrap, function-cache, gbbopen, glyphs, hctsmsl, ieee-floats, lisp-interface-library, lisp-unit2, marching-cubes, mime4cl, ningle, packet, paiprolog, plump, protobuf, racer, readable, repl-utilities, rutils, sexml, shelly, slime, spinneret, talcl, trivial-ldap, vgplot, weblocks, weblocks-utils, wookie.

SBCL 1.2.1 changed some internals that SLIME 2.7 relied on. This update includes SLIME 2.8, which works fine with SBCL 1.2.1.

To get the dist update, use (ql:update-dist "quicklisp").

Enjoy!

14 Jul 2014 2:26pm GMT

I'm going to the International Lisp Conference in 2014 and so should you. It's great to meet Lispers in person and swap stories. Montreal is also a great city.

The deadline for early registration is tomorrow, Monday, July 14th. If you join the ALU for $25, early registration is only $300. (If you don't join the ALU, early registration is $400.) After tomorrow the registration fee goes up signifcantly!

Go register today! See you in Montreal next month!

13 Jul 2014 11:10pm GMT

This morning I got access to the new Cloud9 IDE beta-and I have to say… WOW. It's slicker, it's faster, it's more stable, auto-complete recognizes Lisp definition forms from your open workspace files such as defun and defmacro, and most importantly, it only takes seconds to get your workspace set up with RLWRAP, SBCL and Quicklisp.

The new Cloud9 IDE is running on an Ubuntu backend workspace. Cloud9 has had terminal access to your project workspace for quite some time now, but I've found the terminal experience to be significantly smoother in the new beta. It stays connected now, no longer timing-out on you when switching tabs or stepping away from the computer for a minute. Users can also use sudo for root access, and as a result install any debian package from apt (amongst many other things, of course). Emacs 24 is already installed by default. I suspect that SSH tunneling to a remote SWANK server from the Cloud9 workspace is also now possible.

The Collaboration tools seem to be more streamlined. Workspace Members, Notifications, and Group Chat all appear together in one panel. I expect, with all the other improvements in the beta, that collaborative editing of your workspace files is likewise improved.

There's a new Outline panel that lists the symbol-names of your top-level definition forms for the current active view-yes, even for Lisp. You can select a symbol-name and jump right to its definition in the file. Also, this functionality appears to be integrated with auto-complete, allowing you to jump to a definition of a function, macro, or variable from the auto-complete list as you type.

An interesting set of features I have not yet tried, is the custom Run/Build configurations. These features appear to allow you to write custom Run/Build scripts for arbitrary programming languages, so you should now be able to integrate Lisp into the IDE better, and write/test/debug/deploy your Lisp applications for the most part automatically.

One step closer to my hopes stated in my previous post on Cloud9 IDE from January, the Cloud9 IDE beta includes a JavaScript REPL. Combined with all the other helpful tools that they've included to support Lisp cloud development, it seems reasonable to suppose that a full REPL plugin is in the works.

I've barely scratched the surface here-there are so many new features to try out, I'll probably be discovering new things every day for the next week. And to think, this is just the Beta! If you do your part and show your support for Cloud9 as a Lisp Hacker, I'm quite certain that the next full version of Cloud9 IDE will include everything we need to Hack on Lisp seamlessly in the Cloud.

If you want to get beta access to the new Cloud9 IDE yourself, all you have to do is follow the instructions on their blog post. If your Cloud9 username is different from your Twitter handle, you may need to provide that to them as well to get Beta access.

As always, Happy Hacking!

08 Jul 2014 3:21pm GMT

A collection of Lisp Usenet gems, including articles from Kent Pitman, Erik Naggum, Chris Riesbeck, Pascal Costanza, and Will Hartung.

Things I Want in Common Lisp by Robert Smith.

Video demo of cl-notebook by lnaimathi. By the same author: [T]he primary endeavor of every programmer is twofold: To understand, and to be understood, which demonstrates generating code from a visual diagram.

Looking to start a band? Stumped on a name? See this twitter thread for inspiration.

simple-search is a "smaller alternative to Montezuma" that "allows you to index documents you have stored in-memory and query them in various ways." Looks good. From Andrew Lyon.

It is not hard to read Lisp code, by Jisang Yoo.

mathkit is "a purely math-related utility kit, providing functions which can be useful for games, 3D, and GL in general" by Ryan Pavlik.

08 Jul 2014 1:18pm GMT

(Updated) Program and registration details of next month's International Lisp Conference can now be found at http://international-lisp-conference.org/. Note the changed dates; note that the deadline for early registration is July 14.

02 Jul 2014 1:37pm GMT

I played with Docker some more. It's still in beta so, unsurprisingly, I ran into some problems. It's cool, none the less.

I made a repository for running OpenMCL, aka ccl, inside a container. I set this up so the Lisp process expects to be managed using slime/swank. So it exports port where swank listens for clients to connect. When you run it you export that port, i.e. "-p 1234:4005″ in the example below.

Docker shines at making it easy to try things like this. Fire it up: "docker run -name=my_ccl -i -d -p 1234:4005 bhyde/crate-of-ccl". Docker will spontaneously fetch the everything you need. Then you M-x slime-connect to :1234 and you are all set. Well, almost, the hard part is .

I have run this in two ways, on my Mac, and on DigitalOcean. On the Mac you need to have a virtual machine running linux that will hold your containers - the usual way to do that is the boot2docker package. On Digital Ocean you can either run a Linux droplet and then installed Docker, or you can use the application which bundles that for you.

I ran into lots of challenges getting access to the exported port. In the end I settled on using good old ssh LocalForward statements in my ~/.ssh/config to bring the exported port back to my workstation. Something like "LocalForward 91234 172.17.42.1:1234″ where that IP address that of an interface (docker0 for example) on the machine where the container is running. Lots of other things look like they will work, but didn't.

Docker consists of a client and a server (i.e. daemon). Both are implemented in the same executable. The client chats with the server using HTTP (approximately). This usually happens over a Unix socket. But you can ask the daemon to listen on a TCP port, and if you LocalForward that back to your workstation you can manage everything from there. This is nice since you can avoid cluttering you container hosting machine with source files. I have bash functions like this one "dfc () { docker -H tcp://localhost:2376 $@ ; }" which provides a for chatting with the docker daemon on my Digital Ocean machine.

OpenMCL/ccl doesn't really like to be run as a server. People work around by running it under something like screen (or tmux, detachtty, etc.). Docker bundles this functionality, that's what the -i switch (for interactive) requests in that docker run command. Having done that you can then uses "docker log my_ccl" or "docker attach my_ccl" to dump the output or open a connection to Lisp process' REPL. You exit a docker attach session using control-C. That can be difficult if you are inside of an Emacs comint session, in which case M-x comint-kill-subjob is sometimes helpful.

For reasons beyond my keen doing "echo '(print :hi)' | docker attach my_ccl" get's slightly different results depending on Digital Ocean v.s. boot2docker. Still you can use that to do assorted simple things. UIOP is included in the image along with Quicklisp, so you can do uiop:runprogram calls … for example to apt-get etc.

Of course if you really want to do apt-get, install a bundle of Lisp code, etc. you ought to create a new container built on this one. That kind of layering is another place where Docker shines.

So far I haven't puzzled out how to run one liners. Something like: "docker run -rm bhyde/crate-of-ccl ccl -e '(print :hi)'" doesn't work out as I'd expect. It appears that argument pass thru, arg. quoting, and that the plumbing of standard IO et. al. is full of personality which I haven't comprehended. Or maybe there are bugs.

That's frustrating - I undermines my desire to do sterile testing.

26 Jun 2014 3:17pm GMT

Location: Marbella, Spain
No. of positions available: 1

Position immediately available for an experienced software professional. You will work with an international team of developers skilled in Common Lisp, PL/SQL, Java and Python.

The ideal candidate will have excellent skills as a software engineer, with a strong computer science background and professional experience delivering quality software. You must be fluent in modern software development practices, including multi-threading, distributed systems, and cloud computing. If you are not already an expert in Common Lisp, you aspire to become one. Innovative problem solving and engaging human interaction drive you. With high degree of independence, you will design and implement maintainable software in Common Lisp based on loose and changing specifications.

Familiarity with SQL including query optimization and PL/SQL is very much a plus. Comfort in a growing, fast-paced environment with a premium on problem solving is required. Must be adaptable and willing to learn new technologies. You work successfully in a small team environment, with a willingness to teach and to learn. Lead reviews of your code and participate in the reviews of others.

The ability to communicate effectively in English, both in writing and verbally is a must. Knowledge of Spanish is not a business requirement. European Union legal working status is strongly preferred.

Email CV and a Cover Letter to employment@ravenpack.com with subject "Lisp Developer".

24 Jun 2014 5:25pm GMT

BIT-SMASHER is a lean, straightforward and admittedly naive Common Lisp library for handling the oft-overlooked bit-vector type, bit-vector arithmetic, and type conversion between bit-vectors, octet-vectors, hexadecimal strings, and non-negative integers, extending the related functionality in the Common Lisp standard. While of little use to the average Lisp project, it was designed for those cases where working with bit-vectors is either necessary, or would be ideal if it were not for the lack of the functions this library provides.

You can get BIT-SMASHER now at: https://github.com/thephoeron/bit-smasher - or wait for it to come out in the next Quicklisp release.

The most obvious use-case for BIT-SMASHER is when you need to convert universally between bit-vectors, octet-vectors, hexadecimal strings, and non-negative integers. The library provides manual conversion functions for all twelve cases, plus type-checking, type-casting style convenience functions:

• hex<-, octets<-, int<-, and bits<-
• hex->bits, hex->octets, and hex->int
• octets->hex, octets->bits, and octets->int
• int->hex, int->bits, and int->octets
• bits->hex, bits->octets, and bits->int

For example:

; universal type-casting style functions
(bits<- "F0") => #*11110000
(bits<- 240) => #*11110000
(int<- #*11110000) => 240

; manual conversions without type-checking
(hex->bits "F0") => #*11110000
(int->bits 10) => #*00001010
(octets->bits (int->octets 244)) => #*11110100

The lack of bit-vector arithmetic in the Common Lisp standard was my main motivation to write this library. However, an obvious limitation of the type to values in the set of non-negative integers forced an equal limitation on return results from the bit-vector arithmetic functions. That is to say, when an arithmetic function would normally return a negative integer, float, or fraction as one of its values, it returns the absolute ceiling value (as a bit-vector) instead. For example:

(bit+ #*0100 #*0010) => #*00000110 ; +6, as expected

(bit- #*0000 #*0010) => #*00000010 ; returns +2, not -2

(bit/ #*1010 #*0010 #*0010) => #*00000010 #*00000001 ; returns +2, remainder +1, not 0.5

The library also contains measurement and predicate utility functions, hopefully serving all your bit-vector needs excluded from the standard.

If you encounter any bugs, or have a feature you would like to see, let me know in the comments or create an issue on GitHub.

24 Jun 2014 4:12pm GMT