15 Jun 2016

feedPlanet Lisp

ECL News: ECL Quarterly Volume IV

Table of Contents

1 Preface

Hello,

I've managed to assemble the fourth volume of the ECL Quarterly. As always a bit off schedule but I hope you'll find it interesting.

This issue will revovle around ECL news, some current undertakings and plans. Additionally we'll talk about Common Lisp implementations in general and the portability layers. I believe it is important to keep things portable. Why? Keep reading!

Lately we're working with David O'Toole on making support for ECL on Android better. He wants to distribute his games on this platform and was kind enough to write an article for ECL Quarterly. Thanks to his work we've discovered various rough edges and bugs in ECL and gained some invaluable insight into the cross compilation problems of Common Lisp applications.

As the final remark - I've found some time to establish a proper RSS subscription feed for ECL and ECL Quarterly. I hope that this issue will finally land on the Planet Lisp - a well known Lisp-related blog posts aggregator maintained by Zach Beane.

I want to thank for the valuable feedback and proofreading to many people, especially Antoni Grzymała, Javier Olaechea, Michał Posta, Ilya Khaprov and David O'Toole.

Have a nice lecture,

-
Daniel Kochmański ;; aka jackdaniel | TurtleWare
Poznań, Poland
June 2016

2 ECL's "what's going on"

I've added a milestone with a deadline for the ECL 16.1.3 release with the bugs I want to fix. You may find it here. I'm very happy to receive a lot of positive feedback, merge requests and awesome bug reports. Thank you for that! :-)

Backporting CLOS changes from CLASP was successful but we won't incorporate them in the main branch. The recently resurrected cl-bench has shown that these changes impact performance and consing negatively (check benchmarks). If you are curious about the changes, you may checkout the branch feature-improve-clos in the repository.

I'm slowly working on the new documentation. This is very mundane task which I'm not sure I'll be able to finish. Rewriting DocBook to TexInfo and filling the missing parts is hard. I'm considering giving up and improving the DocBook instead.

In the near future I plan to make a crowdfunding campaign to improve support for cross-compilation, Android and Java interoperability in order to boost development. More details will be probably covered in the next Quarterly issue.

3 Porting Lisp Games to Android with Embeddable Common Lisp, Part 1

3.1 Introduction

Recently I ported my Common Lisp game engine "Xelf" to the Android operating system using Embeddable Common Lisp.

Some work remains to be done before I can do a proper beta test release, but ECL Quarterly provides a good opportunity to pause and share the results thus far. This is the first part of a two-part article. The focus of Part 2 will be on performance optimization, testing, and user interface concerns.

Special thanks to Daniel Kochmański, 3-B, oGMo, and the rest of the Lisp Games crew for their inspiration and assistance.

3.1.1 About the software

Xelf is a simple 2-D game engine written in Common Lisp. It is the basis of all the games I have released since 2008, and can currently be used with SBCL to deliver optimized standalone game executables for GNU/Linux, MS Windows, and Mac OSX.

I've also published a Git repository with all the work-in-progress scripts, patches, and libraries needed to compile Xelf for Android with Embeddable Common Lisp, OpenGL, and SDL.

Please note that this is a pre-alpha release and is mainly intended for Common Lisp developers looking to get a head start in building an Android game. Use with caution.

Xelf is not required; you can substitute your own Lisp libraries and applications and just use the repo as a springboard.

I would like to add support for CL-SDL2 as well, both as a prelude to porting Xelf to SDL 2.0, and as a way to help the majority who use SDL 2.0 for current projects.

3.2 Problems

3.2.1 Choosing an implementation

As I use only Free Software for my projects, I did not consider any proprietary Lisps.

Steel Bank Common Lisp now runs on Android, but SBCL as a whole cannot yet be loaded as a dynamic shared library. This is a show-stopper because Android requires the entry point of a native application to be in a shared library specially embedded in the app.

Xelf works very well with Clozure Common Lisp, but CCL's Android support is not fully functional at present. So I've been quite happy to discover Embeddable Common Lisp. Its technique of translating Common Lisp into plain C has made integration with the Android NDK toolchain relatively simple.

3.2.2 Cross-compilation

For performance reasons the Lisp stack (meaning LISPBUILDER-SDL, CL-OPENGL, CFFI, Xelf, the game, and all their dependencies) must be compiled to native ARM machine code and loaded as shared libraries.

There is a complication in this task as regards ECL. The latter produces native code by translating Common Lisp into plain C, and then invoking the C compiler. But the C compiler toolchain is not typically present on Android, and building one that is properly configured for this task has proved difficult so far.

Therefore we must cross-compile the entire Lisp stack. ECL's Android build procedure already cross-compiles the Lisp contained in ECL, but there were additional difficulties in compiling Lisp libraries which I'll cover below in the "Solutions" section.

3.2.3 Legacy code

Xelf has improved a lot over time and gained new features, but is now outdated in some respects. When I first wrote Xelf in the 2006-2007 period SDL 1.2 was current and OpenGL Immediate mode had not yet been officially deprecated. This hasn't been a terrible problem in practical terms, given that both are still widely supported on PC platforms. But porting to Android would mean I could not procrastinate any longer on updating Xelf's SDL and OpenGL support.

3.3 Solutions

3.3.1 CommanderGenius to the rescue

Help arrived for my SDL woes in the form of Sergii Pylypenko's "CommanderGenius", a fancy port of SDL 1.2/2.0 to Android. I can utilize the existing LISPBUILDER-SDL bindings for SDL, SDL-MIXER, SDL-TTF, SDL-IMAGE, and SDL-GFX. Not only that, there are extra features such as gamepad support, floating virtual joysticks, access to touchscreen gesture data and Android system events, support for the Android TV standard, and much more.

CommanderGenius is actually designed from the start to rebuild existing SDL 1.2 / 2.0 / OpenGL projects as Android applications, and includes dozens of examples to work with. So in mid-May this year I set about splicing together Daniel Kochmański's ECL-ANDROID Java wrapper and startup code (which together load ECL as a shared object from within the app) into the CommanderGenius SDL application code and build procedures.

The result is a fullscreen SDL/OpenGL application with Embeddable Common Lisp, optionally running Swank. There's even a configurable splash screen!

3.3.2 Do a little dance with ASDF

ECL can compile an entire system into one FASL file, but I ran into a snag with the ASDF-based build procedure. The typical way is to compile each Lisp file and then load the resulting compiled file. But on the cross-compiler,

(load (compile-file "myfile.lisp"))

fails because the output of COMPILE-FILE is a binary for the wrong architecture. Likewise, alien shared libraries cannot be loaded during Lisp compilation, which broke CL-OPENGL and LISPBUILDER-SDL.

My temporary solution was to redefine the function ASDF:PERFORM-LISP-LOAD-FASL in my build script. My modified version does something like this instead:

(compile-file "myfile.lisp")
(load "myfile.lisp")

I then invoke ECL's system builder, which spits out a big binary FASB file containing the whole system. But thanks to the LOAD statements, each Lisp file has had access to the macros and other definitions that preceded it in compilation.

I'm sure this is really wrong, but it works, and the resulting FASBs load very quickly. (App startup time went from over 30 seconds when loading byte-compiled FASCs, to about 3.5 seconds.)

In the end, it was simple to deal with CL-OPENGL and LISPBUILDER-SDL wanting to open shared libraries during compilation. I used Grep to find and then comment out calls to CFFI:USE-FOREIGN-LIBRARY, leaving the DEFINE-FOREIGN-LIBRARY definitions intact. This allows cross-compilation to proceed normally.

Then on Android, after the FASBs are loaded I invoke USE-FOREIGN-LIBRARY on each of the required definitions.

So tricking ASDF works. But aside from being a hack, it's not enough for some of the things I'd like to do. The INLINED-GENERIC-FUNCTION technique looks like a highly promising way to increase performance, but my cross-compilation trick led in this case to invalid FASB's with embedded FASC bytecodes. Indeed, to work with ECL in this situation would require actually loading the ARM-architecture compiled INLINED-GENERIC-FUNCTION binary before compiling systems that use inlining-which as mentioned above cannot be done during cross-compilation.

I'm exploring other potential solutions, such as installing a GNU/Linux container on my Android development unit in order to give ECL access to a native C compiler toolchain (see below). I may even attempt to write a custom cross-compilation procedure using Clang and LLVM. But this is less urgent for now, because tweaking ASDF is sufficient to produce a working application.

3.3.3 Use OpenGL ESv1 with CL-OPENGL

Luckily the the path of least resistance could prevail here. OpenGL ES version 1 is widely supported on Android devices, and is easier to port to from Immediate mode than is GLESv2. CL-OPENGL supports it right out of the box. (I'd like to thank 3-B and oGMo for their help in bridging the gap with my own code.)

Some tasks remain to be done here but most of Xelf's drawing functions are now working, including TrueType fonts and vertex coloring.

I've also written some code to partially emulate vertex coloring as a way of increasing render performance, and this will be covered in the forthcoming Part 2 of this article.

3.3.4 ProTip: Use the byte-compiler

One issue has gone unmentioned. How do I interactively redefine functions and set variables in order to develop the running game via SLIME/Swank, if everything must be cross-compiled on an X86 system?

The answer is that ECL's built-in bytecode compiler is used in these cases, and the bytecoded definitions replace the originals. I can freely use COMPILE-FILE, LOAD, and even ASDF:LOAD-SYSTEM during "live" development; under normal circumstances the only real difference is execution speed of the resulting code. The final game app will ship without Swank, of course, and with a fully native Lisp stack.

Now you have a new problem, which is how to edit the Lisp files on your Android device so that Swank can compile and load them.

3.3.5 ProTip: Use Emacs TRAMP with ADB

To make this useful you need a rooted android device.

(add-to-list 'tramp-default-user-alist '("adb" nil "root"))
(find-file "/adb::/")

This can integrate with Emacs' "bookmarks" and "desktop" features for even more convenience.

3.3.6 ProTip: Use Emacs to inspect your APK package

They're just zip files. Missing libraries or assets? Check the APK by opening it as a file in GNU Emacs.

3.3.7 ProTip: Use a GNU/Linux container for SSH and native Emacs with X11!

You can actually install a GNU/Linux "container" with Debian, Ubuntu, or some other distribution on your Android development system in order run the Secure Shell daemon and many other applications. I use it to run a graphical Emacs on the Android box, with Emacs' X11 connection forwarded through SSH so that its windows open on my desktop GNU/Linux PC's X server-right alongside my native Emacs. I use different color themes to avoid mixing them up.

This gives me full access to everything on both systems from a single mouse/keyboard/monitor, and I can cut and paste text freely between applications.

Setting up such a container is beyond the scope of this article, but I highly recommend it. It was pretty easy on a rooted device, and works very well.

3.4 Conclusion

In less than a month we went from "let's do it" to "wow, it works!" What more can you ask for?

This concludes Part 1 of my article on building Lisp games for Android with Embeddable Common Lisp. To read my running commentary and see news and test results as they are posted, you can visit the project README:

https://gitlab.com/dto/ecl-android-games-src/blob/master/README.org

More details and all scripts and configurations can be found in that repository.

Thanks for reading,

-
David O'Toole (dto@xelf.me)
11 June, 2016

4 Common Lisp implementations

Some time ago I've created with the help of many kind people (most notably from Rainer Joswig and Fare Rideau) a graph presenting Common Lisp implementations and the relations between them. This version is improved over drafts presented on twitter and linkedin. If you find any errors, please contact me.

all-hierarchy.png

It is worth noting that LispWorks and VAX share the code with Spice Lisp which later evolved into Common Lisp implementation CMUCL. Striped lines lead to CMUCL, because I didn't want to add pre-CL implementations.

There is also suspicion that Lucid shares code with Spice Lisp and/or VAX, but I couldn't confirm that, so I'm leaving it as is.

"JavaScript Lisp Implementations" classifies some lisps as CL, but I've added only Acheron and parenscript to the list, because rest is just CL-ish, not even being a subset.

Resources I've found on the internet: CMU FAQ, ALU list, CLiki overview, Wikipedia article, JavaScript Lisp Implementations.

5 Building various implementations

I've built various lisps to perform some benchmarks and to have some material for comparison. Ultimately I've decided to polish it a little and publish. I had some problems with Clasp and Mezzano so I've decided to not include them and leave building these as an exercise for the reader ;-). Also, if you feel adventurous, you may try to build Poplog, which has Common Lisp as one of the supported languages.

If you want to read about various implementations, please consult Daniel's Weinreb Common Lisp Implementations: A Survey (material from 2010, definitely worth reading).

First we create a directory for the lisp implementations (we'll build as an ordinary user) and download the sources. Each implementation has a list of building prerequisites, but it may be not comprehensive.

export LISPS_DIR=${HOME}/lisps
mkdir -p ${LISPS_DIR}/{src,bin}
pushd ${LISPS_DIR}/src

# Obtain sources
svn co http://abcl.org/svn/trunk/abcl/ abcl
# git clone git@github.com:drmeister/clasp.git
svn co http://svn.clozure.com/publicsvn/openmcl/trunk/linuxx86/ccl ccl
hg clone http://hg.code.sf.net/p/clisp/clisp clisp
git clone git@common-lisp.net:cmucl/cmucl.git cmucl
git clone https://gitlab.com/embeddable-common-lisp/ecl.git ecl
git clone git://git.sv.gnu.org/gcl.git gcl
git clone git@github.com:davazp/jscl.git jscl
# git clone https://github.com/froggey/Mezzano.git
git clone https://gitlab.common-lisp.net/mkcl/mkcl.git mkcl
git clone git://git.code.sf.net/p/sbcl/sbcl sbcl
git clone git@github.com:wadehennessey/wcl.git wcl
git clone https://github.com/gnooth/xcl.git

5.0.1 ABCL (Armed Bear Common Lisp)

Requires
jdk, ant
pushd abcl
ant
cp abcl ${LISPS_DIR}/bin/abcl-dev
popd

5.0.2 CCL (Clozure Common Lisp)

Requires
gcc, m4, gnumake
pushd ccl
echo '(ccl:rebuild-ccl :full t)' | ./lx86cl64 -n -Q -b

# installation script is inspired by the AUR's PKGBUILD
mkdir -p ${LISPS_DIR}/ccl-dev
cp -a compiler contrib level-* lib* lisp-kernel objc-bridge \
   tools x86-headers64 xdump lx86cl64* examples doc \
   ${LISPS_DIR}/ccl-dev

find ${LISPS_DIR}/ccl-dev -type d -name .svn -exec rm -rf '{}' +
find ${LISPS_DIR}/ccl-dev -name '*.o' -exec rm -f '{}' +
find ${LISPS_DIR}/ccl-dev -name '*.*fsl' -exec rm -f '{}' +

cat <<EOF > ${LISPS_DIR}/bin/ccl-dev
#!/bin/sh
exec ${LISPS_DIR}/ccl-dev/lx86cl64 "\$@"
EOF
chmod +x ${LISPS_DIR}/bin/ccl-dev
popd

5.0.3 CLISP

Requires
gcc, make
Notes
don't build with ASDF (it's old and broken)
pushd clisp
./configure --prefix=${LISPS_DIR}/clisp-dev/ \
            --with-threads=POSIX_THREADS \
            build/
cd build
make && make install
ln -s ${LISPS_DIR}/clisp-dev/bin/clisp ${LISPS_DIR}/bin/clisp-dev
popd

5.0.4 CMUCL (CMU Common Lisp)

Requires
cmucl binary, gcc, make, openmotif
Notes
it needs another CMUCL to bootstrap (release 21a)
pushd cmucl
mkdir -p prebuilt
pushd prebuilt
wget https://common-lisp.net/project/cmucl/downloads/release/21a/cmucl-21a-x86-linux.tar.bz2 \
     https://common-lisp.net/project/cmucl/downloads/release/21a/cmucl-21a-x86-linux.extra.tar.bz2
mkdir ${LISPS_DIR}/cmucl-21a
tar -xf cmucl-21a-x86-linux.tar.bz2 -C ${LISPS_DIR}/cmucl-21a/
tar -xf cmucl-21a-x86-linux.extra.tar.bz2 -C ${LISPS_DIR}/cmucl-21a/
cat <<EOF > ${LISPS_DIR}/bin/cmucl-21a
#!/bin/sh
exec ${LISPS_DIR}/cmucl-21a/bin/lisp "\$@"
EOF
chmod +x ${LISPS_DIR}/bin/cmucl-21a
# Note, that this is already a fully functional lisp now
popd
bin/build.sh -C "" -o "cmucl-21a"
bin/make-dist.sh -I ${LISPS_DIR}/cmucl-dev/ linux-4/
cat <<EOF > ${LISPS_DIR}/bin/cmucl-dev
#!/bin/sh
exec ${LISPS_DIR}/cmucl-dev/bin/lisp "\$@"
EOF
chmod +x ${LISPS_DIR}/bin/cmucl-dev
popd

5.0.5 ECL (Embeddable Common Lisp)

Requires
gcc, make
./configure --prefix=${LISPS_DIR}/ecl-dev/
make && make install
ln -s $LISPS_DIR/ecl-dev/bin/ecl ${LISPS_DIR}/bin/ecl-dev

5.0.6 JSCL (Java Script Common Lisp)

Requires
Conforming CL implementation, web browser, nodejs
Notes
Doesn't provide LOAD yet (no filesystem), but author confirmed that this will be implemented (virtual filesystem on the browser and the physical one on the nodejs).
mkdir ${LISPS_DIR}/jscl-dev
pushd jscl
./make.sh

# Run in the console (node-repl)
cp jscl.js repl-node.js ${LISPS_DIR}/bin/jscl-dev
cat <<EOF > ${LISPS_DIR}/bin/jscl-dev
#!/bin/sh
exec node ${LISPS_DIR}/jscl-dev/repl-node.js
EOF
chmod +x ${LISPS_DIR}/bin/jscl-dev

# Run in the web browser (optional)
cp jscl.js repl-web.js jquery.js jqconsole.min.js jscl.html style.css \
   ${LISPS_DIR}/jscl-dev/
# replace surf with your favourite browser supporting JS
cat <<EOF > ${LISPS_DIR}/bin/jscl-dev-browser
#!/bin/sh
exec surf ${LISPS_DIR}/jscl-dev/jscl.html
EOF
chmod +x ${LISPS_DIR}/bin/jscl-dev-browser

popd

5.0.7 GCL (GNU Common Lisp)

Requires
gcc, make
# Doesn't work both with head and the release, luckily it works with
# the next pre-release branch
git checkout Version_2_6_13pre
./configure --prefix=${LISPS_DIR}/gcl-2.6.13-pre
make && make install
ln -s ${LISPS_DIR}/gcl-2.6.13-pre/bin/gcl ${LISPS_DIR}/bin/gcl-2.6.13-pre

5.0.8 MKCL (Man-Kai Common Lisp)

Requires
gcc, make
pushd mkcl
./configure --prefix=${LISPS_DIR}/mkcl-dev
make && make install
ln -s ${LISPS_DIR}/mkcl-dev/bin/mkcl ${LISPS_DIR}/bin/mkcl-dev
popd

5.0.9 SBCL (Steel Bank Common Lisp)

Requires
ANSI-compliant CL implementation
Notes
  • Lisp has to close on EOF in top-level (CMUCL doesn't do that),
  • ECL has some bug regarding Lisp-to-C compiler apparently triggered by the SBCL compilation - don't use it here,
  • we could use precompiled SBCL like with the CMUCL, but let's exploit the fact, that we can compile from the C-bootstrapped implementation (we'll use already built clisp-dev),
  • it is advised to run the script in fast terminal (like xterm) or in the terminal multiplexer and to detach it - SBCL compilation process is very verbose,
  • if you build SBCL on Windows, consider using MinGW to preserve POSIX compatibility.
pushd sbcl
export GNUMAKE=make
./make.sh "clisp"
INSTALL_ROOT=${LISPS_DIR}/sbcl-dev ./install.sh
cat <<EOF > ${LISPS_DIR}/bin/sbcl-dev
#!/bin/sh
SBCL_HOME=${LISPS_DIR}/sbcl-dev/lib/sbcl exec ${LISPS_DIR}/sbcl-dev/bin/sbcl "\$@"
EOF
chmod +x ${LISPS_DIR}/bin/sbcl-dev
popd

5.0.10 WCL

Requires
tcsh, gcc, git
Notes
very incomplete implementation
pushd wcl
REV=`git rev-parse HEAD`
sed -i -e "s/WCL_VERSION = \"3.0.*$/WCL_VERSION = \"3.0-dev (git-${REV})\"/" CONFIGURATION
LD_LIBRARY_PATH=`pwd`/lib make rebuild
mkdir ${LISPS_DIR}/wcl-dev
cp -a bin/ lib/ doc/ ${LISPS_DIR}/wcl-dev/
cat <<EOF > ${LISPS_DIR}/bin/wcl-dev
#!/bin/sh
LD_LIBRARY_PATH=${LISPS_DIR}/wcl-dev/lib exec ${LISPS_DIR}/wcl-dev/bin/wcl "\$@"
EOF
chmod +x ${LISPS_DIR}/bin/wcl-dev
popd

5.0.11 XCL

Requires
gcc
Notes
last commit in 2011
pushd xcl
mkdir ${LISPS_DIR}/xcl-dev
XCL_HOME=${LISPS_DIR}/xcl-dev make
cp -a clos compiler lisp COPYING README xcl ${LISPS_DIR}/xcl-dev
# This will build in XCL_HOME, even if run in source directory
./xcl <<EOF
(rebuild-lisp)
EOF

ln -s ${LISPS_DIR}/xcl-dev/xcl ${LISPS_DIR}/bin/xcl-dev
popd

6 Portability libraries

It is important to know the difference between the language standard, implementation-specific extensions and the portability libraries. The language standard is something you can depend on in any conforming implementation.

Sometimes it's just not enough. You may want to do ** serializethreading*, or to *data, which is very hard to express (or even impossible) in the language provided by the standard. That's where the implementation-specific extensions kick in. Why are they called "implementation-specific"? Because the API may be different between implementations - reaching consensus is a hard thing1.

The most straightforward approach I can imagine is to reach for the documentation of the Common Lisp implementation you are currently using and to use the API provided by this implementation. I dare you not to do that! It's definitely the easiest thing to do at first, but mind the consequences. You lock yourself, and your users in the implementation you prefer. What if you want to run it on the JVM or to make it a shared library? Nope, you're locked-in.

"What can I do then?" - you may ask. Before I answer this question, I'll tell you how many people do it (or did it in the past) - they used read-time conditionals directly in the code. Something like the following:

(defun my-baz ()
  #+sbcl                        (sb-foo:do-baz-thing 'quux)
  #+ccl                         (ccl:baz-thing       'quux)
  #+(and ecl :baz-thing)        (ext:baz             'quux)
  #+abcl                        (ext:baz             'quux)
  #+(and clisp :built-with-baz) (ext:baz-thingie     'quux)
  #-(or sbcl ccl ecl abcl clisp)
  (error "Your implementation isn't supported. Fix me!"))

If the creator felt more fancy and had some extra time, they put it in the package my-app-compat. It's all great, now your application works on all supported implementations. If somebody wants theirs implementation to work, send the creator a patch, who incorporates it into the code and voila, everything works as desired.

We have one problem however. Libraries tend to depend on one another. There is also a lot of software which uses features beyond the ANSI specification (it's all good, programmers need these!). Do you see code duplication everywhere? How many times does a snippet above have to be copy-pasted, or rewritten from scratch? It's not black magic after all. APIs between ad-hoc implementations don't exactly match, covered CL implementations differ…

So you quickload your favorite library which depends on 10 other libraries which implement BAZ functionality in theirs own unique way, with a slightly different API on the unsupported implementation - that's why we have my-baz abstraction after all, right? Now, to make it work, a user has to:

  1. Find which of the ten libraries don't work (not trivial!),
  2. find and clone the repositories (we want to use git for patches),
  3. fix each one of them (grep helps!) and commit the changes,
  4. push the changes to your own forked repository and create a pull request (or send a diff to the mailing list) - *ten times*,
  5. voila, you're done, profit, get rich, grab a beer.

It's a lot of work which the user probably won't bothered to do. They will just drop the task, choose another implementation or hack their own code creating the Yet Another Baz Library for the implementations he cares for reinventing the wheel once more. It's a hacker's mortal sin.

I'm going to tell you now what is the Right Thing™ here. Of course you are free to disagree. When you feel that there is a functionality you need which isn't covered by the standard you should

  1. Look if there is a library which provides it.

    You may ask on IRC, the project's mailing list, check out the CLiki, do some research on the web. Names sometimes start with trivial-*, but it's not a rule. In other words: do your homework.

  2. If you can't find such a library, create one.

    And by creating such a library I mean comparing the API proposed by at least two CL implementations (three would be optimal IMHO), carefully designing your own API which covers the functionality (if it's trivial, this should be easy) and implementing it in your library.

    Preferably (if possible) add a fallback implementation for implementations not covered (with the appropriate warning, that it may be inefficient or not complete in one way or another).

    It may be worth reading the Maintaining Portable Lisp Programs paper written by Christophe Rhodes.

  3. Write beautiful documentation.

    A CL implementation docs may be very rough. It takes time to write them and programmers tend to prioritize code over the documentation. It's really bad, but it's very common for the documentation to be incomplete or outdated.

    Document your library, describe what it does, how to use it. Don't be afraid of the greatness! People will praise you, success will come, world will be a better place. And most importantly, your library will be useful to others.

  4. Publish the library.
  5. Make that library your project's dependency.

I know it's not easy, but in the long term it's beneficial. I guarantee you that. That's how the ecosystem grows. Less duplication, more cooperation - pure benefit.

Some people don't follow this path. They didn't think it through, or maybe they did and decided that keeping the dependency list minimal is essential to their project, or were simply lazy and hacked their own solution. There are also some old projects which exported a number of features being a very big portability library and an application at the same time (ACL-compat, McCLIM and others). What to do then?

If it's a conscious decision of the developer (who doesn't want to depend on /anything/), you can do nothing but provide a patch adding your own implementation to the supported list. It's their project, their choice, we have to respect that.

But before doing that you may simply ask if they have something against plugging these hacks with the proper portability library. If they don't - do it, everybody will benefit.

There are a few additional benefits of the presented portability library approach for the implementations itself. Having these internal details in one place makes it more probable that your implementation is already supported. If the library has a bug it's easier to fix it in one place. Also, if the CL implementation changes it's API, it's easy to propagate changes to the corresponding portability libraries. New CL implementation creators have a simplified task of making their work usable with existing libraries.

It is worth noting, that creating such library paves the way to the new quasi-standard functionalities. For instance Bordeaux Threads has added recently CONDITION-WAIT function, which isn't implemented on all implementations. It is a very good stimulus to add it. This is how library creators may have real impact on the implementation creators decisions about what to implement next.

6.1 Portability layer highlights

Here are some great projects helping CL implementations be part of a more usable ecosystem. Many of these are considered being part of the de-facto standard:

bordeaux-threads
Provides thread primitives, locks and conditionals
cl-store
Serializing and deserializing CL objects from streams
cffi
Foreign function interface (accessing foreign libraries)
closer-mop
Meta-object protocol - provides it's own closer-common-lisp-user package (redefines for instance defmethod)
usocket
TCP/IP and UDP/IP socket interface.
osicat
Osicat is a lightweight operating system interface for Common Lisp on POSIX-like systems, including Windows
cl-fad
Portable pathname library
trivial-garbage
trivial-garbage provides a portable API to finalizers, weak hash-tables and weak pointers
trivial-features
trivial-features ensures consistent *FEATURES* across multiple Common Lisp implementations
trivial-gray-streams
trivial-gray-streams system provides an extremely thin compatibility layer for gray streams
external-program
external-program enables running programs outside the Lisp process

There are many other very good libraries which span multiple implementations. Some of them have some drawbacks though.

For instance IOlib is a great library, but piggy-backs heavily on UN*X - if you develop for many platforms you may want to consider other alternatives..

UIOP is also a very nice set of utilities, but isn't documented well, does too many things at once and tries to deprecate other actively maintained projects - that is counterproductive and socially wrong. I'd discourage using it.

There are a few arguments supporting UIOP's state - it is a direct dependency of ASDF, so it can't (or doesn't want to) depend on other libraries, but many utilities are needed by this commonly used system definition library. My reasoning here is as follows: UIOP goes beyond ASDF's requirements and tries to make actively maintained projects obsolete. Additionally it works only on supported implementations even for features which may be implemented portably.

6.2 UIOP discussion

I'm aware that my opinion regarding UIOP may be a bit controversial. I've asked the library author and a few other people for feedback which I'm very grateful for. I'm publishing it here to keep opinions balanced.

6.2.1 Fare Rideau

Dear Daniel,

while there is a variety of valid opinions based on different interests and preferences, I believe your judgment of UIOP is based on incorrect premises.

First, I object to calling UIOP "not well documented". While UIOP isn't the best documented project around, all its exported functions and variables have pretty decent DOCSTRINGs, and there is at least one automatic document extractor, HEΛP, that can deal with the fact that UIOP is made of many packages, and extract the docstrings into a set of web pages, with a public heλp site listed in the UIOP README.md. The fact that some popular docstring extractors such as quickdocs can't deal with the many packages that UIOP creates with its own uiop:define-package doesn't mean that UIOP is less documented than other projects on which these extractors work well, it's a bug in these extractors.

Second, regarding the deprecation of other projects: yes, UIOP does try to deprecate other projects, but (a) it's a good thing, and (b) I don't know that any of the projects being deprecated is "actively maintained". It's a good thing to try to deprecate other lesser libraries, as I've argued in my article Consolidating Common Lisp libraries: whoever writes any library should work hard so it will deprecate all its rivals, or so that a better library will deprecate his and all rivals (such as optima deprecating my fare-matcher). That's what being serious about a library is all about. As for the quality of the libraries I'm deprecating, one widely-used project the functionality of which is completely covered by UIOP is cl-fad. cl-fad was a great improvement in its day, but some of its API is plain broken (e.g. the :directories argument to its walk-directory function has values with bogus names, while its many pathname manipulation functions get things subtly wrong in corner cases), and its implementation not quite as portable as UIOP (that works on all known actively used implementations). There is no reason whatsoever to ever choose cl-fad over UIOP for a new project. Another project is trivial-backtrace. I reproduced most of its functionality, except in a more stable, more portable way (to every single CL implementation). The only interface I didn't reproduce from it is map-backtrace, which is actually not portable in trivial-backtrace (only for SBCL and CCL), whereas serious portable backtrace users will want to use SLIME's or SLY's API, anyway. As for external-program, a good thing it has for it is some support for asynchronous execution of subprocesses; but it fails to abstract much over the discrepancies between implementations and operating systems, and is much less portable than uiop:run-program (as for trivial-shell, it just doesn't compete).

UIOP is also ubiquitous in a way that other libraries aren't: all implementations will let you (require "asdf") out of the box at which point you have UIOP available (exception: mostly dead implementations like Corman Lisp, GCL, Genera, SCL, XCL, may require you to install ASDF 3 on top of their code; still they are all supported by UIOP, whereas most portability libraries don't even bother with any of them). This ubiquity is important when writing scripts. Indeed, all the functionality in UIOP is so basic that ASDF needed it at some point - there is nothing in UIOP that wasn't itself required by some of ASDF's functionality, contrary to your claim that "UIOP goes beyond ASDF's requirements" (exception: I added one function or two to match the functionality in cl-fad, such as delete-directory-tree which BTW has an important safeguard argument :validate; but even those functions are used if not by ASDF itself, at least by the scripts used to release ASDF itself). I never decided "hey, let's make a better portability library, for the heck of it". Instead, I started making ASDF portable and robust, and at some point the portability code became a large chunk of ASDF and I made it into its own library, and because ASDF is targetting 16 different implementations and has to actually work on them, this library soon became much more portable, much more complete and much more robust than any other portability library, and I worked hard to achieve feature parity with all the libraries I was thereby deprecating.

Finally, a lot of the functionality that UIOP offers is just not offered by any other library, much less with any pretense of universal portability.

6.2.2 David Gu

For the documentation thing, I really think Quickdocs could do a better job. The bug #24 stated that problem, however, it's remain to be solved. I will check this out if I have free time recently.

I use UIOP a lot in my previous company, the reason is simple and maybe a little naive: my manager didn't want to involve too many add-ons in the software. UIOP is shipped together with ASDF, it's really "convenient", and its robustness is the final reason why I will stick to it. If people understand how UIOP came out in the history from ASDF2 to ASDF3, I think people will understand why it's acting like deprecating several other projects - that's not the original idea of it.

But anyway, I really learned a lot from this post and also the comments. In my opinion, avoid reinventing the wheels is the right idea and directions for this community. So from that perspective, I support @fare's idea "It's a good thing to try to deprecate other lesser libraries". Including this article and along with Maintaining Portable Lisp Programs and @fare's Consolidating Common Lisp Libraries, we should let more people involved in this topic.

Footnotes:

1

If you are Common Lisp implementer and plan to add a feature beyond ANSI specification, please consider writing a proposal and submitting it to Common Lisp Document Repository. It will make everybody's life easier.

15 Jun 2016 12:00am GMT

12 Jun 2016

feedPlanet Lisp

Michael Malis: Building Fizzbuzz in Fractran from the Bottom Up

In this post, I am going to show you how to write Fizzbuzz in the programming language Fractran. If you don't know, Fractran is an esoteric programming language. That means it is extraordinary difficult to write any program in Fractran. To mitigate this difficultly, instead of writing Fizzbuzz in raw Fractran, what we are going to do is build a language that compiles to Fractran, and then write Fizzbuzz in that language.

This post is broken up into three parts. The first part covers what Fractran is and a way of understanding what a Fractran program does. Part 2 will go over the foundation of the language we will build and how it will map to Fractran. Finally, in Part 3, we will keep adding new features to the language until it becomes easy to write Fizzbuzz in it.


Part 1: Understanding Fractran

Before we can start writing programs in Fractran, we have to first understand what Fractran is. A Fractran program is represented as just a list of fractions. To execute a Fractran program, you start with a variable N=2. You then go through the list of fractions until you find a fraction F, such that N*F is an integer. You then set N=N*F and go back to the beginning of the list of fractions. You keep repeating this process until there is no fraction F such that N*F is an integer.

Since there is no way to print anything with the regular Fractran rules, we are going to add one additional rule on top of the ordinary ones. In addition to the list of fractions, each program will have a mapping from numbers to characters representing the "alphabet" of the program. After multiplying N by F, whenever the new N is a multiple of one of the numbers in the alphabet, that will "print" the character that the number maps to. I have written a function, run-fractran, which implements this version of Fractran and included it here. It takes a list of fractions and an alphabet as an alist and executes the program.

Let's walk through a simple example. Let's say we have the following Fractran program:

9/2, 1/5, 5/3

with the alphabet 5->'a'. To run this program, we start with N=2. We then go through the list fractions until we find a fraction F such that N*F is an integer. On this first step, F becomes 9/2, since N*F = 2 * 9/2 = 9 which is an integer. We then set N to N*F so that N now becomes 9. Repeating this process again, we get F=5/3 and N=N*F=15. Since the number 5 is in the alphabet, and N is now a multiple of 5, we output the character that 5 maps to, 'a'. If we keep repeating these steps, we eventually reach a point where N=1 and we have outputted the string "aa". Since 1 times any of the fractions does not result in an integer, the program terminates with the output "aa".

At this point, you may be thinking that writing any program in Fractran is nearly impossible. The truth is that there is a simple trick you can use that makes it much easier program Fractran. All you need to do is look at the prime factorization of all of the numbers. Let's see what the above Fractran program looks like if we convert every number into a tuple (a,b,c) where a is the how many times 2 divides the number, b is how many times 3 does, and c is how many times 5 does. The program then becomes:

(0, 2, 0) / (1, 0, 0)
(0, 0, 0) / (0, 0, 1)
(0, 0, 1) / (0, 1, 0)

We also have the tuple (0,0,1) mapping to 'a' for our alphabet. We start with N = (1,0,0). If you don't know, multiplying two numbers is the same as adding the counts of each prime factors, and division is the same as subtracting the counts. For example, 2 * 6 = (1,0,0) + (1,1,0) = (2,1,0) = 12. With this way of looking at the program, finding a fraction F such that N*F is an integer becomes finding a "fraction" F such that each element in the tuple N is greater than or equal to the corresponding element in the tuple in the denominator of F. Once we find such F, instead of multiplying N by it, you subtract from each element of N the corresponding value in the denominator of F (equivalent to dividing by the denominator), and add the corresponding value in the numerator (equivalent to multiplying by the numerator). Executing the program with this interpretation proceeds as follows.

We start with N = (1,0,0). Since every value in N is greater than or equal to their corresponding values in the denominator of the first fraction, we subtract every value in the first denominator and then add every value in the numerator to get N = (1,0,0) - (1,0,0) + (0,2,0) = (0,2,0). Repeating this again, F becomes the third fraction. Subtracting the denominator and adding the numerator gets us N = (0,1,1). Then since every value in N is greater than or equal to their corresponding element in (0,0,1), we print 'a'. The program continues, just like it did for the original Fractran program.

Basically we can think of every prime number as having a "register" which can take on non-negative integer values. Each fraction is an instruction that operates on some of the registers. You can interpret a fraction as saying if the current value of each register is greater than or equal to the the value specified by the denominator (the number of times the prime for that register divides the denominator), you subtract from the registers all of the values in the denominator, add all the values specified in the numerator (the number of times the prime for each register divides the numerator), and then jump back to the first instruction. Otherwise, if any register is less than the value specified in the denominator, continue to the next fraction. For example, the fraction 9/2 can be translated into the following pseudocode:

;; If the register corresponding to the prime number 2 
;; is greater or equal to 1
if reg[2] >= 1
  ;; Decrement it by 1 and increment the register 
  ;; corresponding to 3 by 2. 
  reg[2] = reg[2] - 1
  reg[3] = reg[3] + 2
  goto the beginning of the program
;; Otherwise continue with the rest of the program.

Although programming Fractran is still difficult, this technique suddenly makes writing Fizzbuzz in Fractran tractable.


Part 2: Compiling to Fractran

For our compiler, we are going to need to generate a lot of primes. To do so, we will use a function, new-prime, which will generate a different prime each time it is called.1

(defun prime (n)
  "Is N a prime number?"
  (loop for i from 2 to (isqrt n)
        never (multiple n i)))

(defparameter *next-new-prime* nil)

(defun new-prime ()
  "Returns a new prime we haven't used yet."
  (prog1 *next-new-prime*
    (setf *next-new-prime*
          (loop for i from (+ *next-new-prime* 1)
                if (prime i)
                  return i))))

So now that we've got new-prime, we've we can start figuring out how we are going to compile to Fractran. The first detail we will need to figure out is how to express control flow in Fractran. In other words, we need a way to specify which fractions will execute after each other fractions. This is a problem because after a fraction executes, you always jump back to the first fraction.

Expressing control flow actually winds up being surprisingly easy. For each fraction we can designate a register. Then, we only execute a fraction if its register is set. It is easy to have a fraction conditionally execute depending on whether its register is set by using the trick we are using to interpret a Fractran program. All we need to do is multiply the denominator of each fraction by the prime for the register of that fraction. This way, we will pass over a fraction unless its register is set. Also, all we need to do to specify which fraction should execute after a given fraction is to multiply the numerator of the given fraction by the prime of the register for the next fraction. By doing this, after a fraction executes, it will set the register of the next fraction.

In order to keep track of the primes for the current fraction and for the next fraction, we will have two global variables. The first will be the prime number for the current instruction, and the second will be the prime number for the next instruction:

(defparameter *cur-inst-prime* nil)
(defparameter *next-inst-prime* nil)

We will also need a function advance which will advance the values of the variables once we move on to the next instruction.

(defun advance ()
  (setf *cur-inst-prime* *next-inst-prime*
        *next-inst-prime* (new-prime)))

Now that we've got a way of expressing control flow, we can start planning out what the language we will build will look like. From this point on, I am going to call the language we are building, Lisptran. An easy way we represent a Lisptran program is as just a list of expressions. We can have several different kinds of expressions each of which does something different.

The simplest kind of expression we will want is an inline fraction. If a Lisptran expression is just a fraction, we can just add that fraction to the Fractran program being generated.

Another kind of expression that would be useful are labels. Whenever a Lisptran expression is a Lisp symbol, we can interpret that as a label. Each label will be converted into that fraction that is the prime of the next instruction after the label divided by the prime of the label. This way we can jump to the instruction after the label by setting the register for the label. In order to make keeping track of the primes of labels easy, we are going to keep a hash-table, *lisptran-labels*, mapping from labels to the primes for those labels. We will also have a function prime-for-label, which will lookup the prime for a label or assign a new prime if one hasn't been assigned yet:

(defparameter *lisptran-labels* nil)

(defun prime-for-label (label)
  (or (gethash label *lisptran-labels*)
      (setf (gethash label *lisptran-labels*)
            (new-prime))))

One last kind of expression that will be useful are macro calls. A macro call will be a list whose first element is the name of a macro followed by a list of arbitrary Lisp expressions (The expressions don't have to be Fractran expressions. They can be interpreted however the macro wants them to be.). In order to compile a macro call, we will lookup the function associated with the macro, and call it on the expressions in the rest of the macro call. That function should then return a list of Lisptran expressions which will then be compiled in place of the macro call. After that we just continue compiling the new code generated by the macro expansion.

To keep track of the definitions of macros, we will keep a hash-table *lisptran-macros*, which will map from the name of the macro to the function for that macro. In order to make defining Lisptran macros easy, we can create a Lisp macro deftran, that works in a similar way to defmacro. When defining a macro with deftran, you are really just defining a function which will take the expressions in the macro call, and return a list of Lisptran instructions to be compiled in its place. Here is the definition for deftran:

(defparameter *lisptran-macros* (make-hash-table))

(defmacro deftran (name args &body body)
  "Define a Lisptran macro."
  `(setf (gethash ',name *lisptran-macros*)
         (lambda ,args ,@body)))

And that's all of the different kinds of expressions we will need in Lisptran.

Although we now have all of the expressions we need, there are a few more pieces of the compiler we need to figure out. For example, we still haven't figured out how we are going to represent variables yet. Ultimately this is trivial. We can just assign a register to every variable and keep a mapping from variable names to primes in the same way we have the mapping for labels:

(defparameter *lisptran-vars* nil)

(defun prime-for-var (var)
  (or (gethash var *lisptran-vars*)
      (setf (gethash var *lisptran-vars*)
            (new-prime))))

One last piece of the compiler we need to figure out is how we are going to represent the alphabet of the program. One way we can do this is just represent the characters in our alphabet as variables. The alphabet of a program could just be all of the variables that have characters for names and the primes of the registers for those variables. By doing it this way, we can print a character by just incrementing and then immediately decrementing a variable! Here is code that can be used to obtain the alphabet from *lisptran-vars*:

(defun alphabet (vars)
  "Given a hash-table of the Lisptran variables to primes, 
   returns an alist representing the alphabet."
  (loop for var being the hash-keys in vars 
        using (hash-value prime)
        if (characterp var)
          collect (cons var prime)))

Now that we can express control flow, variables, and macros, we have everything we need to write the actual Lisptran to Fractran compiler:

(defun assemble (insts)
  "Compile the given Lisptran program into Fractran. 
   Returns two values. The first is the Fractran program 
   and the second is the alphabet of the program."
  (let* ((*cur-prime* 2)
         (*cur-inst-prime* (new-prime))
         (*next-inst-prime* (new-prime))
         (*lisptran-labels* (make-hash-table))
         (*lisptran-vars* (make-hash-table)))
    (values (assemble-helper insts)
            (alphabet *lisptran-vars*))))

(defun assemble-helper (exprs)
  (if (null insts)
      '()
      (let ((expr (car exprs))
            (rest (cdr exprs)))
        (cond
          ;; If it's a number, we just add it to the 
          ;; Fractran  program and compile the rest 
          ;; of the Lisptran program
          ((numberp expr)
           (cons expr (assemble-helper rest)))

          ;; If it's a symbol, we divide the prime for 
          ;; the next instruction by the prime for the 
          ;; label.
          ((symbolp expr)
           (cons (/ *cur-inst-prime* 
                    (prime-for-label expr))
                 (assemble-helper rest)))

          ;; Otherwise it's a macro call. We look up the 
          ;; macro named by the first symbol in the 
          ;; expression and call it on the rest of the 
          ;; rest of the expressions in the macro call. 
          ;; We then append all of the instructions 
          ;; returned by it to the rest of the program 
          ;; and compile that.
          (:else
            (let ((macrofn (gethash (car inst)
                                    *lisptran-macros*)))
              (assemble-helper (append (apply macrofn
                                              (cdr inst))
                                       rest))))))))

The function assemble takes a Lisptran program and returns two values. It returns the generated Fractran program and the alphabet of that program. assemble first initializes all of the global variables for the program and then goes to assemble-helper which recursively processes the Lisptran program according to the specification above. Using the function run-fractran that I mentioned above, we can write a function that will execute a given Lisptran program as follows:

(defun run-lisptran (insts)
  "Run the given Lisptran program."
  (multiple-value-call #'run-fractran (assemble insts)))

Part 3: Building Lisptran

Now that we've completed the core compiler, we can start adding actual features to it. From here on out, we will not touch the core compiler. All we are going to do is define a couple Lisptran macros. Eventually we will have enough macros such that programming Lisptran seems like programming a high level assembly language.

The first operations we are going should define are basic arithmetic operations. For example, addition. In order to add addition to Lisptran, we can define a macro addi, which stands for add immediate. Immediate just means that we know what number we are adding at compile time. The macro addi will take a variable and a number, and will expand into a fraction which will add the given number to the register for the variable. In this case, the denominator for the fraction will just be the prime for the current instruction (execute this instruction when that register is set) and the numerator will be the prime for the next instruction (execute the next instruction after this one) times the prime for the variable raised to the power of the number we are adding (add the immediate to the register). Here is what the definition for addi looks like:

(deftran addi (x y)
  (prog1 (list (/ (* *next* (expt (prime-for-var x) y))
                  *cur*))
    (advance)))

With are also going to want an operation that performs subtraction. It's a bit tricky, but we can implement a macro subi (subtract immediate) in terms of addi, since adding a number is the same as adding the negative of that number:2

(deftran subi (x y) `((addi x ,(- y))))

Now that we've got some macros for performing basic arithmetic, we can start focusing on macros that allow us to express control flow. The first control flow macro we will implement is >=i (jump if greater than or equal to immediate). In order to implement >=i, we will have it expand into three fractions. The first fraction will test if the variable is greater or equal to the immediate. If the test succeeds, we will then advance to the second fraction which will restore the variable (since when a test succeeds, all of the values from the denominator are decremented from the corresponding registers), and then jump to the label passed in to >=i. If the test fails, we will fall through to the third fraction which will just continue onto the next fraction after that.

The denominator of the first fraction will be the prime for current instruction (execute the instruction if that register is set) times the prime for the register raised to the power of the constant (how we test that the register is greater than or equal to the immediate) and the numerator will be the prime for the second instruction (so we go to the second instruction if the test succeeds). The second fraction is just the prime for the label passed into >=i (so we jump to wherever the label designates) divided the prime for that instruction. Lastly, the denominator of the third fraction is the prime for the current instruction (so we fall through to it if the test in the first fraction fails), and the numerator is just the prime for the next instruction so that we continue to that if the test fails:

(deftran >=i (var val label)
  (prog1 (let ((restore (new-prime)))
           (list (/ restore
                    (expt (prime-for-var var) val)
                    *cur-inst-prime*)
                 (/ (* (prime-for-label label)
                       (expt (prime-for-var var) val))
                    restore)
                 (/ *next-inst-prime* *cur-inst-prime*)))
    (advance)))

Believe it or not, but after this point, we won't need to even think about fractions anymore. Lisptran now has enough of a foundation that all of the further macros we will need can be expressed in terms of addi, subi and >=i. The only two functions that actually need to be implemented in terms of Fractran are addi and >=i. That means no more thinking about Fractran. From here on out, all we have is Lisptran!

We can easily define unconditional goto in terms of >=i. Since all of the registers start at 0, we can implement goto as greater than or equal to zero. We use the Lisp function gensym to generate a variable without a name so that the variable doesn't conflict with any other Lisptran variables:

(deftran goto (label) `((>=i ,(gensym) 0 ,label)))

Then through a combination of >=i and goto, we can define <=i:

(deftran <=i (var val label)
  (let ((gskip (gensym))) 
    `((>=i ,var (+ ,val 1) ,gskip)
      (goto ,label)
      ,gskip)))

Now that we have several macros for doing control flow, we can start building some utilities for printing. As mentioned previously printing a character is the same as incrementing the variable with the character as its name and then immediately decrementing it:

(deftran print-char (char)
  `((addi ,char 1)
    (subi ,char 1)))

Then if we want to write a macro that prints a string, it can just expand into a series of calls to print-char, each of which prints a single character in the string:

(deftran print-string (str)
  (loop for char across str
        collect `(print-char ,char)))

We are also going to need a function to print a number. Writing this with the current state of Lisptran is fairly difficult since we haven't implemented several utilities such as mod yet, but we can start by implementing a macro print-digit that prints the value of a variable that is between 0 and 9. We can implement it, by having it expand into a series of conditions. The first one will check if the variable is less than or equal to zero. If so it will print the character zero and jump past the rest of the conditions. Otherwise it falls through to the next condition which tests if the variable is less than or equal to one and so on. We don't have to manually write the code for print-digit because we can use Lisp to generate the code for us:

(deftran print-digit (var)
  (loop with gend = (gensym)
        for i from 0 to 9
        for gprint = (gensym)
        for gskip = (gensym)
        append `((<=i ,var ,i ,gprint)
                 (goto ,gskip)
                 ,gprint
                 (print-char ,(digit-char i))
                 (goto ,gend)
                 ,gskip)
        into result
        finally (return `(,@result ,gend))))

At this point, now that we have macros for performing basic arithmetic, basic control flow, and printing, we can start writing some recognizable programs. For example here is a program that prints the numbers from zero to nine:

(start
 (>=i x 10 end)
 (print-digit x)
 (print-char #\newline)
 (addi x 1)
 (goto start)
 end)

If you are curious I have included the Fractran program generated by this Lisptran program here. It's hard to believe that the above Lisptran program and the Fractran program are equivalent. They look completely different!

Now that we have a bunch of low level operations, we can start building some higher level ones. You may not have thought of it, but instructions don't need to just have flat structure. For example, now that we have goto, we can use it to define while loops (just like in Loops in Lisp):

(deftran while (test &rest body)
  (let ((gstart (gensym))
        (gend (gensym)))
    `((goto ,gend)
      ,gstart
      ,@body
      ,gend
      (,@test ,gstart))))

In order to implement while, we are assuming that all predicates take labels as their last argument which is where they will jump to if the predicate succeeds. Now that we have while loops, we can start writing some much more powerful macros around manipulating variables. Here's two useful ones, one that sets a variable to zero, and one that copies the value in one variable to another:

(deftran zero (var)
  `((while (>=i ,var 1)
      (subi ,var 1))))

(deftran move (to from)
  (let ((gtemp (gensym)))
    `((zero ,to)
      (while (>=i ,from 1)
        (addi ,gtemp 1)
        (subi ,from 1))
      (while (>=i ,gtemp 1)
        (addi ,to 1)
        (addi ,from 1)
        (subi ,gvar 1)))))

For move, we first have to decrement the number we are moving from and increment a temporary variable. Than we restore both the original variable and the variable we are moving the value to at the same time.

With all of these macros, we can finally start focusing on macros that are actually relevant to Fizzbuzz. One operation that is absolutely going to be necessary for Fizzbuzz is mod. We can implement a macro modi by repeatedly subtracting the immediate until the variable is less than the immediate.

(deftran modi (var val)
  `((while (>=i ,var ,val)
      (subi ,var ,val))))

We only need one more real feature before we can start writing Fizzbuzz. We are going to need a way of printing numbers. In order to print an arbitrary number, we are going to need a way of doing integer division. We can implement a macro divi by repeatedly subtracting the immediate until the variable is less than the immediate and keeping track of the number of times we've subtracted the immediate.

(deftran divi (x y)
  (let ((gresult (gensym)))
    `((zero ,gresult)
      (while (>=i ,x ,y)
        (addi ,gresult 1)
        (subi ,x ,y))
      (move ,x ,gresult))))

Now for the final macro we will need. A macro for printing numbers. Actually, we are going to cheat a little. Printing numbers winds up being pretty difficult since you have to print the digits from left to right, but you can only look at the lowest digit at a time. To make things easier, we are only to write a macro that is able to print two digit numbers. We won't need to print 100 since "buzz" will be printed instead.

(deftran print-number (var)
  (let ((gtemp (gensym))
        (gskip (gensym)))
    `((move ,gtemp ,var)
      (divi ,gtemp 10)
      (>=i ,gtemp 0 ,gskip)
      (print-digit ,gtemp)
      ,gskip
      (move ,gtemp ,var)
      (modi ,gtemp 10)
      (print-digit ,gtemp)
      (print-char #\newline))))

Now our language is sufficiently high enough that Fizzbuzz is going to be practically as easy as it will get. Here is an implementation of Fizzbuzz in Fractran.

((move x 1)
 (while (<=i x 100)
   (move rem x)
   (modi rem 15)
   (<=i rem 0 fizzbuzz)

   (move rem x)
   (modi rem 3)
   (<=i rem 0 fizz)

   (move rem x)
   (modi rem 5)
   (<=i rem 0 buzz)

   (print-number x)
   (goto end)

   fizzbuzz
   (print-string "fizzbuzz")
   (goto end)

   fizz
   (print-string "fizz")
   (goto end)

   buzz
   (print-string "buzz")
   (goto end)

   end
   (addi x 1)))

I've also included the generated Fractran program here and included all of the full source code for this blog post here.

I find it absolutely amazing that we were able to build a pretty decent language by repeatedly adding more and more features on top of what we already had. To recap, we implemented a basic arithmetic operation (addi) in terms of raw Fractran and then defined a second (subi) in terms of that. From there we defined three macros for doing control flow (>=i, goto, <=i), with the second two being defined in terms of the first. Then we were then able to define macros for printing (print-char, print-string, print-digit). At this point we had all of the low level operations we needed so we could start implement while loops (while), a high level control flow construct. With while loops, we were able to define several macros for manipulating variables (zero, move). With these new utilities for manipulating variables we could define more advanced arithmetic operations (modi, divi). Then with these new operations we were able to define a way to print an arbitrary two digit number (print-number). Finally, using everything we had up to this point, we were able to write Fizzbuzz. It's just incredible that we could make a language by always making slight abstractions on top of the operations we already had.

The post Building Fizzbuzz in Fractran from the Bottom Up appeared first on Macrology.

12 Jun 2016 12:59am GMT

10 Jun 2016

feedPlanet Lisp

CL Test Grid: quicklisp 2016-05-31

The difference between this and previous months:


Grouped by lisp implementation first and then by library:
https://common-lisp.net/project/cl-test-grid/ql/quicklisp-2016-05-31-diff.html

Grouped by library first and then by lisp impl:
https://common-lisp.net/project/cl-test-grid/ql/quicklisp-2016-05-31-diff2.html

(Both reports show the same data, just arranged differently)

This time the diff is smaller than the past month. There are some regressions, some improvements.

If you're interested in some particular failure or need help investigating something, just ask in the comments or on the mailing list.

10 Jun 2016 1:44am GMT

08 Jun 2016

feedPlanet Lisp

Wimpie Nortje: Daemonizing Common Lisp services.

Update (2016-06-09):

During a discussion on Reddit I realised that the issues discussed in this post are not relevant to the systemd init system.

At the time of writing my servers run Ubuntu 14.04 which uses Upstart as the init system. Upstart detaches the tty of the started services. That is the cause of the issues discussed below.

Most Linux distributions seem to migrate to systemd as the init system. systemd does not detach the tty, it binds stdin to /dev/null and stdout and stderr to a logging stream.

If your OS uses systemd you can turn a foreground application into a service without using GNU Screen or the daemon package.

Deploying a Common Lisp server application in production requires that it runs as a daemon.

A Unix init system like Upstart or systemd can be used to ensure that the service is always running. In this case the application can daemonize itself or it can run in the foreground and let the init system take care of daemonization.

It is usually easier to create an init script for a self-daemonizing application but then the source code gets more complicated.

When the application runs in the foreground the init script is slightly more tricky but the application is easier to debug.

Self-daemonizing

The procedure to daemonize an application is well documented but most of these documentation is focused on C. To implement the complete procedure in Common Lisp can become a mission in itself because it involves forking and opening and closing standard IO stream at the right times.

The daemon package implements all the necessary steps and provides a trivial API for daemonizing a program.

Though daemon is easy to use, all the forking makes it difficult to get backtraces when something goes wrong. The application then appears to be hanging while it is actually waiting in the debugger for input but the debugger can't be used because standard IO is closed.

When a service daemonizes itself it is advisable to devise some method to interact with the application while it is waiting in the debugger. One of the swank libraries could be useful.

Run in foreground

Foreground applications can be used as-is for a service but it can not be used as the main application instantiated by the init system.

During the daemonizing process the init system closes all the standard IO streams. This causes the REPL to exit and the application to end.

GNU Screen can be used to keep the standard IO open for a daemonized service. However, when an error occurs one is in much the same situation as with a self-daemonizing application. The application appears to hang because it is waiting in the debugger but the debugger can't easily be reached.

Screen has options to log standard IO to a file. Debugging then consists of killing the service and working through the logged backtrace. This is not ideal but it has the advantage of not introducing any code complexity in order to make a daemon.

Like in the self-daemonizing case, a swank library can be used to get live debugging back.

Comparing the options

Self-daemonizing Foreground
Daemonizing responsibility Common Lisp code Unix init system
Tool daemon library GNU Screen
Init script complexity Less complex More complex
Code complexity Increased complexity Same complexity as normal program
Debugging options Embedded swank server Logged data or embbeded swank server

08 Jun 2016 12:00am GMT

07 Jun 2016

feedPlanet Lisp

Zach Beane: Lisp stuff on YouTube

If you want to see some neat videos, subscribe to dto, Baggers, and WarWeasle on YouTube. They all regularly post neat graphical stuff done in Common Lisp.

If you know any more people I should follow on YouTube, let me know.

07 Jun 2016 12:37pm GMT

03 Jun 2016

feedPlanet Lisp

Lispjobs: Secure Outcomes, Contract Common Lisp programmer

Secure Outcomes builds and provides digital livescan fingerprinting systems for use by law enforcement, military, airports, schools, Fortune 500s, etc.

All of our systems are constructed in Common Lisp.

We are looking for a contract CL developer located anywhere in the world that can build software for us.

Strong CL/LW background, of course, but also knowledge of foreign function interfacing etc needed.

Work full or part time.

See what we do at http://www.secureoutcomes.net

Resumes to lisp@secureoutcomes.net. No calls please.


03 Jun 2016 6:32pm GMT

Timofei Shatrov: All you need is PROGV

I have never seen PROGV in use - Erik Naggum

Common Lisp is very, very old. Tagbody and progv, anyone? - Hacker News user pwnstigator

I haven't written anything on this blog lately, mostly because of lack of time to work on side projects and consequently the lack of Lisp things to talk about. However recently I've been working on various improvements to my Ichiran project, and here's the story of how I came to use the much maligned (or rather, extremely obscure) special operator PROGV for the first time.

Ichiran is basically a glorified Japanese dictionary (used as the backend for the web app ichi.moe) and it heavily depends on a Postgres database that contains all the words, definitions and so on. The database is based on a dump of an open JMdict dictionary, which is constantly updated based on the users' submissions.

Well, the last time I generated the database from this dump was almost a year ago, and I wanted to update the definitions for a while. However this tends to break the accuracy of my word segmenting algorithm. For this reason I want to keep the old and the new database at the same time and be able to run the whatever code with either of the databases.

I'm using Postmodern to access the database, which has a useful macro named with-connection. If I have a special variable *connection* and consistently use (with-connection *connection* ...) in my database-accessing functions then I can later call


(let ((*connection* '("foo" "bar" "baz" "quux")))
  (some-database-accessing-function))

and it will use connection ("foo" "bar" "baz" "quux") instead of the default one. I can even encapsulate it as a macro

(defmacro with-db (dbid &body body)
  `(let ((*connection* (get-spec ,dbid)))
     (with-connection *connection*
       ,@body)))

(dbid and get-spec are just more convenience features, so that I can refer to the connection by a single keyword instead of a list of 4 elements).

So far so good, but there's a flaw with this approach. For performance reasons, some of the data from the database is stored in certain global variables. For example I have a variable *suffix-cache* that contains a mapping between various word suffixes and objects in the database that represent these suffixes. Obviously if I run something with a different connection, I want to use *suffix-cache* that's actually suitable for this connection.

I created a simple wrapper macro around defvar that looks like this:

(defvar *conn-vars* nil)

(defmacro def-conn-var (name initial-value &rest args)
  `(progn
     (defvar ,name ,initial-value ,@args)
     (pushnew (cons ',name ,initial-value) *conn-vars* :key 'car)))

Now with-db can potentially add new dynamic variable bindings together with *connection* based on the contents of *conn-vars*. It's pretty trivial to add the new bindings at the macro expansion time. However that poses another problem: now all the conn-vars need to be defined before with-db is expanded. Moreover, if I introduce a new conn-var, all instances of with-db macro must be recompiled. This might be not a problem for something like a desktop app, but my web app usually runs for months without being restarted, with new code being hot-swapped into the running image. I certainly don't need the extra hassle of having to recompile everything in a specific order.

Meanwhile I had the definition of let opened in the Hyperspec, and there was a link to progv at the bottom. I had no idea what it does, and thinking that my Lisp has gotten rusty, clicked through to refresh my memory. Imagine my surprise when I found that 1) I have never used this feature before and 2) it was exactly what I needed. Indeed, if I can bind dynamic variables at runtime, then I don't need to re-expand the macro every time the set of these variables changes.

The final code ended up being pretty messy, but it worked:

(defvar *conn-var-cache* (make-hash-table :test #'equal))

(defmacro with-db (dbid &body body)
  (alexandria:with-gensyms (pv-pairs var vars val vals iv key exists)
    `(let* ((*connection* (get-spec ,dbid))
            (,pv-pairs (when ,dbid
                         (loop for (,var . ,iv) in *conn-vars*
                            for ,key = (cons ,var *connection*)
                            for (,val ,exists) = (multiple-value-list (gethash ,key *conn-var-cache*))
                            collect ,var into ,vars
                            if ,exists collect ,val into ,vals
                            else collect ,iv into ,vals
                            finally (return (cons ,vars ,vals))))))
       (progv (car ,pv-pairs) (cdr ,pv-pairs)
         (unwind-protect
              (with-connection *connection*
                ,@body)
           (loop for ,var in (car ,pv-pairs)
              for ,key = (cons ,var *connection*)
              do (setf (gethash ,key *conn-var-cache*) (symbol-value ,var))))))))

Basically the loop creates a pair of list of variables and list of their values (no idea why progv couldn't have accepted an alist or something). The values are taken from *conn-var-cache* which takes the pairing of variable name and connection spec as the key. Then I also add an unwind-protect to save the values of the variables that might have changed within the body back into the cache. Note that this makes nested with-db's unreliable! The fix is possible, and left as an exercise to the reader. Another problem is that dynamic variables bindings don't get passed into new threads, so no threads should be spawned within the with-db macro.

And this is how I ended up using progv in production. This probably dethrones displaced array strings as the most obscure feature in my codebase. Hopefully I'll have more things to write about in the future. Until next time!

03 Jun 2016 9:12am GMT

01 Jun 2016

feedPlanet Lisp

Quicklisp news: May 2016 Quicklisp dist update now available

New projects:

Updated projects: 3d-vectors, backports, binfix, bit-smasher, blackbird, caveman2-widgets, cepl, cepl.camera, cepl.devil, cl-ana, cl-autowrap, cl-bloom, cl-cairo2, cl-charms, cl-dot, cl-erlang-term, cl-gamepad, cl-geometry, cl-hash-table-destructuring, cl-html5-parser, cl-i18n, cl-jpeg, cl-liballegro, cl-libssh2, cl-mediawiki, cl-mongo, cl-mpi, cl-ohm, cl-pango, cl-project, cl-qrencode, cl-redis, cl-sdl2, cl-selenium, cl-store, clack, closer-mop, coleslaw, croatoan, deeds, dissect, djula,esrap, exscribe, femlisp, fiasco, firephp, flare, form-fiddle, glkit, glop, graph, halftone, helambdap, hl7-parser, hu.dwim.partial-eval, hu.dwim.quasi-quote, hu.dwim.util, kenzo, legit, lisp-interface-library, lisp-namespace, lisp-unit2, mcclim, oclcl,pathname-utils, pzmq, qt-libs, qtools, qtools-ui, query-fs, queues, rcl, restas, rtg-math, rutils, scalpl, sdl2kit, serapeum, simple-tasks, sketch, skitter, slime, snakes, stp-query, stringprep, stumpwm, temporal-functions, trivia, trivial-backtrace, trivial-benchmark, trivial-string-template, trivialib.bdd, ubiquitous, utilities.print-tree, varjo, verbose, vgplot, weblocks, weblocks-stores, xhtmlambda, zs3.

To get this update, use: (ql:update-dist "quicklisp")

You didn't miss it -- there wasn't a fundraiser in April. Or May, either. I've got my fingers crossed to see one soon, but I'm not sure exactly when it might happen. Stay tuned!

01 Jun 2016 1:54pm GMT

22 May 2016

feedPlanet Lisp

McCLIM: Old news list

For posterity we are publishing the archival news:

22 May 2016 1:00am GMT

McCLIM: New website

We are happy to announce that the McCLIM website has been refreshed. All broken links has been replaced and the infrastructure has been moderated into something easier to maintain.

Website comparison

On the left is the old version while on the right is the current design. Website is responsive, has the RSS stream and all the goods coleslaw provides.

You may expect frequent improvements to the codebase in the near future - stay tuned!

22 May 2016 1:00am GMT