fidzu : LISP

Reworked the menu structure for Hemlock Apps on the train last night. Nothing significant, just a minor change to storage. Also, the first implementation built static menus and there's a need for some menus to be dynamic, so I sketched a solution to make them dynamic if desired.

Gave some thought to walking ASDF definition files and parsing lisp code to extract content for presentation. Parsing the dot asd file should be pretty straightforward, or so it seems at first blush. Parsing lisp code has been done, so I'm going to grovel through sources to find code that I can crib. Might ask on c.l.l first.

Last night, I wrote some code using interactive development technique. Built a small app to use for doc finding and viewing. I kind of want my docs to be easy to find, access and edit. Plus, I want to be able to see which files were edited - when and why. I spent too much time grepping for a file that I hadn't edited in a while and couldn't remember which directory it lived in and decided to start to address that morass of insanity.

Man, was it weird to work incrementally. Just stubbed and built functions line-by-line. Thought of what something did and made sure each part did what it was supposed to before any other code was written. This is really easy in Lisp because the functional model supports it well. Plus the interactive nature of the Lisp environment (REPL) favors rapid development.

Actual coding went like this: Built the app framework, with a couple of menus. No content in any functions except the basic Hemlock code to launch the app. Ran it. It brought up a window with menus. Menus were there, but were connected to stub functions. Verified the menus were selectable and didn't report an error on selection.

Then, gradually added content. I started by adding print commands when I began editing a function. Then, verified that, when the function is called, the print call output was correct. Then, converted the print call into a function call. Repeated the test until the code did what it was supposed to.

Learned the concept from The Mythical Man-Month by Fred Brooks (Harlan Mills - incremental development.) Problem with software is it's just easy to riff off a function or two, then start debugging. It doesn't seem like a problem, because the logic is "out there" when it's written in one fell swoop.

That's how I've written code for a long time. And I've got the furrowed brow from debugging the mess and carving in functionality that was needed IMMEDIATELY after "finishing".

But last night it became clear that building code incrementally reduces stress. I felt like I was cheating - wasn't I supposed to iterate over the logic a gazillion times until I randomly, hopefully, poorly finished the algorithm?

Apparently not.

When I finished for the night, I left code in a working state - knowing that the code that was there was stable, too. It was also well-documented. Too weird.

And one function is going to replace a heavily used directory traversing algorithm I'd written a long time ago that I'd been hesitant to rework because it worked, sort of, kind of, well enough. In ten minutes it was rewritten. Best part is it makes a hack I've been using even easier to use. Kind of amazing how quickly a mole-hill becomes a mountain.

I'm going to write a more detailed post on incremental development, what I did and how it progressed. It was so much fun to do, that it'd be interesting to see if that sense of fun comes across in an article.

23 Jul 2008 1:08pm GMT

Build some menus and the next thing you know, you're supporting them for the rest of your life...

Feature request, huh? Well, that didn't take long...was using the new menubar/menus scheme with Ed Vecto commands, inserting Vecto commands into a lisp function when I decided to extend the Ed Vecto app to add my Vecto extensions and customizations.

Ok, so now the Hemlock apps menus would need to be open so another app could append a menu to a menubar. Not a problem. (Could be more than append. I'm taking a kiss approach to coding these days, so I'll let the interactions define the scope.)

Before adding the code to modify a defmenubar, I built some defmenu/definsert forms for my vecto libs. Got bored walking my code a line at a time, copying and pasting function calls and their args. So I added definsert to the set of menus and made inserting definsert calls easier. But it was still tedious.

That got me thinking about how nice it would be if there was a menu-builder app that would parse my lisp forms for the files in an asdf package. The app would, after parsing, make the forms available (with args) so I could just pick and choose which lisp forms to make available to add to a menubar or menu or associate with a function (eg. definsert.)

I'll do some sketching on the train today and see how much work that'd be to build.

22 Jul 2008 4:36pm GMT

I guess everyone has heard of "the editor wars" (usually simplified to "emacs vs. vi(m)".

So have I. I have one editor I use almost daily, but there are three editors I can actuall use (well, OK, two of them are more "editor families" than distinct editors).

First off, I self-identify as an emacs user. I've used one or another emacs as my primary editing environment for the last 20 years. A new editor would have to be pretty darn astoundingly good for me to change that.

However, if you don't have a favourite editor already, I suggest you try out several different editors. Use them for at least 3-4 weeks and spend a day or two coming to grips with your editing environment. When it comes to editing comfort, first impressions can actually lure you into ditching something that is good and instead make you choose something that is less good. After you become sufficiently used to your editing environment it will be hard to change.

So, why did I end up with emacs-like editors (one of the "editor families")? Primarily thanks to the fact that emacs had a built-in tutorial. Secondarily becaiuse of emacs lisp. I've always liked lisp and lisp-like languages (something that may be obvious, looking back through my Advogato diary entries).

I said I was proficient with three different editors, I better explain taht too. I can (and do) use vi (and vi- heritage) editor(s) (primarily nvi and vim, with a distinct preference of nvi over vim), because I used to earn my living by being a unix sysadmin, occasionally being sent out to clients. You can't really expect to be able to install and compile emacs whenever you hit a new site, it just wastes time. So, learn to use the tools available.

I can also use ed. Admittedly, I mostly use ed as part of shell scripting, but that's because it can trivially be driven via stdin (build edit command using echo, pipe these into ed filename and make sure to finish with wq, stick this in a loop and you're set to do things in sequence, with externally kept state; not pretty, but it does the job).

Somewhat amusingly, I actually used vi before I used emacs, but never really continued with vi (and from there derived editors). I don't know if I'd kept with vi, had there been a good in-editor tutorial (or at least a tutorial docuiment I dcould've played around with). It felt like a struggle, initially. I had some rudimentary ed when I started with vi, but the latter was sufficiently different to cause impedance mismatch between what I knew and the capabilities available in the new environment.

Emacs, on the other hand, was nothing like ed, came with in- editor tutorials, in-editor documentation and a lisp environment to boot. I was willing to invest the necessary time to becxome proficient and have, as they say, never looked back really hard.

Clearly, I have a favourite editor, but I don't think there is such a thing as "one true editor, best for all possible things". I guess the fact that my .emacs contains (setq wq "Emacs, not vi!") to be sufficient proof of that...

22 Jul 2008 2:51pm GMT

Yesterday, while writing up a function, I used a combination of anaphoric macro and menu selection. A short time ago, I'd changed this same idiom from this:

(let ((res (select-menu items)))

  (when res

     (menu-item-dispatch res)))

To this:

(awhen (select-menu items)

  (menu-item-dispatch it))

And I looked at the anaphoric form and said, that's not (awhen (select-menu ...)) that's (amenu ...)

(defmacro amenu (items &body; body)

  `(awhen (select-menu ,items)

     ,@body))



(amenu items

   (menu-item-dispatch it))

That's better.

22 Jul 2008 1:38pm GMT

It took more time to write the last blog entry than it did to make the menus work. Added all the Ed Vecto menus and tested them. All forms insert fine - with a couple of exceptions; the forms that have keywords or body parameters aren't parsed yet. Going to add my Vecto extensions and charting calls.

Added some lisp forms and ran into limitations. The menubar/menu/insert forms work for simple, straightforward data entry. More complex forms don't work.

For some Lisp forms, eg. defclass, it'll be necessary to construct a dialog interface that provides the means to grow forms when there can be multiple entries. Or, for more complex structures like loop, the interface will have to accommodate the myriad options.

Already have an app/interface (data entry code) for Lisp forms. The app simply makes it easy to select a lisp function/macro call for insertion at the point without args. While it speeds up data entry, only a few forms have the extra juice (custom functions with interface code) to add more structure to a call. Will reconcile the new definsert and defmenu's with Lisp forms and, where needed, extend the interface to make inserting complex forms easy.

Won't update all the Lisp forms right away. Will bring over those that I frequently use and gradually bring the rest in over time.

22 Jul 2008 5:34am GMT

hmmm. It's a bit dusty in here.

After being poked by a few people to revive this thing, I thought I'd give it a shot. For various uninteresting reasons, the original purpose (of this blog) was not realized.

Anyway, I think I'll use to do a couple of things. One is as a place to put/discuss snippits of code, etc. that people ask me for. Another to discuss issues I've run into, and lastly, if I can find the time for a couple of lispy projects that are deeply on the back burner at the moment, I'll probably discuss them here. Wish I had more time for messing about with lisp at the moment, but such is life.

I'll try and keep it at least somewhat lisp-related, but I've also got a couple of (simple) maths topics I keep getting asked about, so may well put primers here, with code, if I can figure out a decent way to put some maths symbols in blogger (anyone know a good approach)....


Ok, so just to have some lisp content, something I was asked for the other day (and next time I'll just link here): Here is a rejection sampling algorithm for generating Gaussian samples. There are faster methods, but speed is fine on sbcl with the given declares (some of which may be unneeded, I wrote this ages ago). Should work fine anywhere, but good output is dependent a decent implementation of random. I won't bother describing the algorithm just now. Output is easily mapped to any other Gaussian you might want, this basic function doesn't do that for you.

(defun randn (&optional; (state *random-state*))

  "returns two double-floats sampled from zero-mean, variance 1.0 

   Gaussian distribution using polar Box-Muller method"

  (declare (optimize speed))

  (loop with w of-type (double-float 0.0d0)

     for x1 of-type (double-float 0.0d0) = (- (* 2 (random 1.0d0 state)) 1)

     for x2 of-type (double-float 0.0d0) = (- (* 2 (random 1.0d0 state)) 1)

     for s  of-type (double-float 0.0d0) = (+ (* x1 x1) (* x2 x2))

     while (>= s 1)

     finally 

       (setf w (sqrt (the (double-float 0.0d0) (/ (* -2 (log s)) s))))

       (return (values (* x1 w) (* x2 w)))))

Often you want an array of these, so you can easily mimic a matlab-like randn function and take advantage of the need to sample two at a time:

(defun randn-array (dims &optional; (state *random-state*)) 

  (declare (optimize speed))

  (loop with res = (make-array dims :element-type 'double-float)

     with bound = (- (array-total-size res) 1) 

     for n below bound

     do (multiple-value-bind (a b) (randn)

          (setf (row-major-aref res n) a (row-major-aref res (incf n)) b))

     finally 

       (when (evenp bound)

         (setf (row-major-aref res n) (randn)))

       (return res)))

With sbcl this takes about 1/3 of a second to generate a million element array, so I've never bothered tweaking it further. I haven't done any careful analysis of the output, but it inherits from the Mersenne-twister a decent underlying distribution. Here's the sort of thing you get plotting it against itself, pretty reasonable:



[update: as evan points out in comments, you probably want an optional state argument to this. My fault for typing this up quickly this morning; the version I actually use is part of a package with some dependencies, so I cleaned it up --- a bit too much! Updated now]

21 Jul 2008 5:39pm GMT

In my last post, I documented some interface code that I was playing with for Hemlock, the Common Lisp Editor. I'd written two macros, 'definsert and 'defchoice for the purpose of creating functions that provided a means to quickly insert lisp forms at the point while making it possible to present custom options for selection on a per form basis.

Friday night, I had a little time to play with the code and thought it was a good first-pass. I didn't get much time to code this weekend, but I kept thinking about what I'd written.

Saturday, I gave some thought to creating a set of menu items for the Hemlock Vecto editor/utility (I'm calling Ed Vecto.) It was easy to add code to definsert to construct a menu. But, the menu was just a blob that screamed for better structure (categories.)

On top of the weak menu structure, defchoice and definsert were disjoint and defchoice wouldn't work if more than one slot in a definsert needed customization. Better to use the same approach as defstruct or defclass and create a set of slots with customizable behavior, eg. initform/initarg forms. That way, defchoice could be eliminated.

Want the means to make a menu selection and dispatch to a function - in this case to perform data collection and insertion of a form.

While Hemlock apps had menus, the structure wasn't well defined - simply trees. Now was as good a time as any to update the menu structure. Created defmenubar and defmenu. Defmenubar defines the menubar categories. Defmenu maps categories to subcategories.

The basic structure for menus and insert commands looks like:

(defmenubar canvas graphics-state paths text) ; Vecto categories.

(defmenu canvas with-canvas clear-canvas save-png save-png-file) ; Vecto canvas funcs.



(definsert with-canvas (&key width height) &body body) ; this form isn't parsed yet.

(definsert clear-canvas)

(definsert save-png file)

(definsert save-png-stream stream)



(defmenu graphics-state set-line-cap set-fill ...) ; more options available.

(definsert set-line-cap (style :initform '(:miter :bevel :round)))

(definsert set-fill (color :initform choose-color))

Not sure about whether the keyword :initform will be used, but it's a reasonable placeholder for this article.

Will have to provide a mapping from defmenu slots to the definsert forms - probably use a keyword in defmenu (since defmenu creates the two elements of a menu the name and dispatch function and definsert creates the function I want to call when a menu is selected in Ed Vecto.) Adding a keyword option will make defmenu look like:

(defmenu graphics-state (:prefix 'insert) set-line-cap set-fill)

The new additions will make Hemlock apps cleaner and eliminate the trees I've been relying on for menus. Wanted to address Hemlock app menus for a while now, so it's good to take a shot at cleaning them up. Hopefully, this'll be a decent approach.

Will be upgrading Vecto to the latest revision (1.3.1) and adding the new calls to Ed Vecto (and playing with the new features - gradients and text-paths.) After testing Ed Vecto, will get back to the charting app.

21 Jul 2008 4:23am GMT

I started a new job a little over a week ago wherein I have to write Java code, mostly writing tests in JUnit. I will state up front that I do not have anything against Java. I think it's a decently designed language considering when it was designed and its target audience. I should also say that I'm impressed with Eclipse. I doubt my experience with Java would be as enjoyable without it. Kudos to the Eclipse team. That said, I sorely miss the qualities of Lisp when trying to do my work.

Now, I haven't done extensive work with Java yet since I am still getting up to speed on many things at my new place of employment. Thus, it may be the case I don't know how to do certain things effectively in Java/JUnit. Still, here are some things I really miss in Lisp when moving to Java.

Special variables: Debugging in the system I'm using involves changing various static constants in classes. This is fine if you have write access to things, but when you don't or when obtaining it may cause conflicts with the version control software, it's a pain. I do not like that it is a compile-time constant. (let ((*debug* t)) ...) would be so much nicer.

CLOS and its MOP: You don't realize the power of generic functions until you don't have them. Some of the code I'm working on is old and contains, shall we say, questionable design decisions. (To be fair, the old code would never pass current code reviews, but it's now production code, so we can't change it.) Testing it is problematic because I have to extend existing (mock!) classes to get at the objects I need to examine at the right time. Adding an around method to a particular method would be much cleaner. Even better, adjusting the methods programatically while debugging would be great.

Multiple inheritance: The desire to use multiple inheritance in Java may pass as I get more accustomed to Java, but certain common methods would be better served if implemented as a mixin or something similar. Right now, there's more code repetition than there should be, in my opinion. However, it can't be factored out into separate classes very easily.

Overall, what I dislike about using Java is the inability to poke and prod a running system to learn something about how it works. Sure, you can use a debugger with breakpoints, but when an error occurs, I can't drop down to Java code and start making calls. It's the interaction that's lovely about Lisp and Java just doesn't have it in the same way. Partial information is powerful. Interacting with a failure is powerful. Being able to navigate the stack after an exception has been thrown while the system is still loaded is powerful. They are powerful because they provide information that may otherwise be hard to obtain. Instead of tracing through code, I can interact with data. The latter is considerably more productive, in my experience.

20 Jul 2008 1:59pm GMT

Did you ever want to recreate the source form of a package, to see what state it is in currently? With defpackage-form, you can!

Then again, I seem to fiddle too much with packages lately. In particular, currently I am experimenting with a new work flow of Lisp package management.

Package Forms during Development

When writing new Lisp code, I start with just the essentials of a new package in my current Emacs buffer:

(defpackage #:foo
  (:use #:cl))

(This is basically what I get with C-x C-r P RET RET RET.)

During development, I find it distracting to remind myself of keeping the package form up-to-date. When I decide to import new symbols or shadow others I manipulate the package object directly, instead of updating the above DEFPACKAGE form and reevaluating it: (import 'bar:baz), (shadow 'qux), (use-package 'cl-fred), etc.. All this can be automated with a few editor key bindings, and the effects are visible in my working environment immediately.

At this point, I seldomly bother with export lists, because I might change names around later on, and also I am working mostly in the package.

At the end of the hacking session, I can just evaluate (defpackage-form 'foo) to get the current state of package FOO conveniently as a DEFPACKAGE form, which I can then use to replace the initial stub. Eventually, if the code grows big enough to split it up, I move the package form to a separate file packages.lisp.

At least in the first iteration, the export list is likely still incomplete, so I can choose to include all symbols whose home package is the package I am working on, nicely sorted by name. Then I remove those symbols which are not meant for exporting. Like that, it is easy to bulk export many symbols at once. As a common convention, symbols starting with % are meant to be internal. They can be omitted from the export list automatically.

FORMAT Controls, the Ultimate Line Noise

Package forms print neatly (and by that I mean closer to how I would layout them manually) with the following additions:

(defun pprint-defpackage (stream defpackage-form)
  (format stream "~:<~W~^ ~3I~:_~W~^~1I~@{~:@_~:<~W~^ ~:I~@_~@{~W~^ ~_~}~:>~}~:>"
          defpackage-form))

(set-pprint-dispatch '(cons (member defpackage))
                     'pprint-defpackage)

Current versions of SBCL (newer than 1.0.17.36) come with this style included already. I also toyed with a two-column layout, but it is harder to edit.

Incremental Updates

Incremental updates of package forms are a little trickier, though. The generated form could, for example, be compared to the original with diff-sexp. I have not automated that stage much yet. If there are only few additions, it is probably easier to just add them manually.

Otherwise, I could imagine an Emacs before-save-hook which checks whether the export list of a package matches the DEFPACKAGE form in packages.lisp (or wherever it is stored-this could be figured out with source locations), and some more automation and integration with Emacs.

Very unfortunately, some loss of information occurs on the round trip from package form to object and back:

• Reader conditionals inside DEFPACKAGE forms cannot be recreated easily. Neither can comments, for what it's worth.
• There appears to be no way to figure out which package a symbol was actually imported from. For example, if we export CL:NIL from package FOO and then import FOO:NIL into some other package BAR, it will appear the same as if CL:NIL was imported directly into BAR.
• The default :USE list (i.e., when none is specified) is implementation dependent, which naturally can cause some implementation dependent packages to show up in this list.

20 Jul 2008 11:23am GMT

In my last post, I wrote a macro 'definsert to insert a lisp form in Hemlock. At first, I used it only for Vecto commands, but now it's working for some lisp forms, too. That's a nice bonus.

Later, I was working with the Hemlock Vecto app. I wrote all the definsert's for all the Vecto commands. There were two functions that took a single keyword argument out of a set of three choices. Seemed to make sense to present the keyword choices when inserting those two commands so as not to have to type them. What to do, what to do...

Ooh, ooh! I know, write a macro that creates a function that handles a dialog-style selection mechanism such that, when an insert-foo command is evaluated, it'll first check to see if a 'menu-foo function exists. If it does, it'll call that function instead of asking the default set of questions (use default-args, new var names, or enter values.)

(defmacro defmenu (name &rest items)

 `(defun (intern (format nil "~:@(~menu-~A~) ()

    (select-menu-item ',items)))

Saweet! Now, the commands providing custom UI choices look like:

(definsert set-line-join style)

(defmenu set-line-join :miter :bevel :round)

So now, when my ui calls 'insert-set-line-join, the app finds the 'menu-set-line-join function, calls it and presents the equivalent of a set of radio buttons with the three choices '(:miter :bevel :round). Select one and the following is inserted at the point:

(set-line-join :miter)

Similar problem with colors. Only a little different because colors come in rgb and rgba. Plus, there are color tables (I currently have two - x11-rgb and another color palette from a paint company.) Soon, there will be palettes composed of complementary colors. But I digress.

Early on, while working with Vecto, I wrote the methods 'set-stroke, 'set-fill and 'to-alpha to dispatch on types '(RGB RGBA String Symbol) and wrap the set-*-rgb and set-*-rgba functions from Vecto. That way, I can pollute my code with color names as strings if I want to. [Note to self, get rid of the strings.] The idea is that colors with names can be called by said name, and colors that are one-offs can use rgb/a values:

(set-fill 'red)

(set-fill "RED")

(set-stroke 'darkslateblue)

So, the 'defmenu declaration, in the colors case, will have to call a function. The declaration will look like:

(definsert set-fill color)

(defmenu #'choose-color)

This change requires 'defmenu to check if the args variable holds a function. If it does, it's funcalled. Otherwise, the default case holds:

(defmacro defmenu (name &rest items)

 `(defun (intern (format nil "~:@(~menu-~A~) ()

    (if (functionp ,(car items))

        (funcall ,(car items))

        (select-menu-item ',items)))

The function 'choose-color will present '(rgb rgba) as UI choices. Regardless of the choice, the color table is selected, then the color name. If the color is to be converted to alpha, the alpha channel value is entered. Finally, the collected data is formatted and output at the point as either an rgb color or call to convert the rgb color to rgba:

(set-fill "RED")

(set-fill (to-alpha "RED" 0.4))

Now to build the high-level interface. While it may seem odd to provide one interface option via 'defmenu, for the way I'm working with Hemlock it's fine.

19 Jul 2008 4:03am GMT