17 Aug 2016
Shareablee engineers grapple with some of the hardest challenges in social media analytics and big data. We are involved in every aspect of the product development process from brainstorming to coding and deployment. Products that we create drive decision-making at some of the world's most influential brands such as NFL, ESPN, Facebook, HBO, Ebay, Subway and Samsung.
- Build features and modules that our customers fall in love with
- Work on high volume data collection, storage, and transformation
- Develop reusable code and libraries for future use
- Optimize various processes for maximum speed and scalability
- Develop and maintain data collectors and aggregators
- Collaborate with product managers and team leads to build the right thing
- Collaborate with directors and senior developers to build things the right way
- 2+ years of work experience as a software developer.
- A degree in computer science or a related field.
- Proficiency in one or more of the following:
- Clojure, and/or another functional language
- Python or a similar high-level general purpose language
- Experience with (or very strong interest to learn) the following frameworks and tools: Storm, Hadoop, Cassandra, PostgreSQL, Redis, RabbitMQ, Kafka, Django, Elasticsearch
- Basic understanding of distributed computing paradigms
- Experience with Amazon Web Services, Big Data processing and Map/Reduce
- Integration of multiple data sources and databases into one system
- Data migration, transformation, and scripting
- Implement automated testing platforms and unit tests
- Proficiency with code versioning tools such as Git
- The ability to map continuous learning to every aspect of your life
- Humble confidence
Please send resumes and cover letter to email@example.com
17 Aug 2016 7:33pm GMT
16 Aug 2016
We are delighted and overwhelmed by the support that many of you have shown for our effort to crowdfund the maintenance and continued development of McCLIM. Never in our wildest dreams had we imagined that we would reach our highest monthly goal of 2000 USD in only a few days.
In addition to our regular maintenance and programmed improvements, at this level of funding, we will be able to post so-called "bounties", i.e., specific sums of cash for solving specific problems. Once posted, they will be available at this URL:
Like we mentioned in our initial appeal for contributions, this campaign has an important side-effect, beyond that of providing a budget for improvements, namely that of creating new excitement around McCLIM. This excitement will ultimately result in contributions in the form of code, thereby multiplying the importance of the direct monetary support.
In order to make our work on this project as transparent as possible, we will do our utmost to provide monthly reports on our progress, and we will provide details on how the cash was put to work.
We sincerely hope to make sufficient progress, sufficiently fast, that you will consider additional contributions, in the form of cash or code, in the future.
Robert Strandh and Daniel Kochmański
16 Aug 2016 1:00am GMT
12 Aug 2016
McCLIM is currently the only free native toolkit for developing GUI applications in Common Lisp. A while ago, I took over the maintenance, because I did not want to see McCLIM abandoned.
But since I have many other projects going, I decided to hire Daniel Kochmański part time (25%) to do some of the urgent work. This experience turned out so well that I would like for him to continue this work. Not only am I very pleased with the quality of Daniel's work, but I was also very surprised about the additional excitement that his work generated.
While I could continue funding Daniel myself for a while, I can not do so indefinitely. For that reason, I decided that we should try crowdfunding. Steady funding at the 25% level for a year or more would allow us to address some of the issues that often come up, such as the numerous remaining defects, the lack of modern visual appearance, and more.
If you are interested in seeing this happen, I strongly urge you to consider contributing to this project. Our target is modest. We would like to obtain at least 600 USD per month for at least one year. Even a small monthly contribution from a significant number of people can make this happen.
To contribute, please visit https://salt.bountysource.com/teams/mcclim and follow the instructions.
12 Aug 2016 1:00am GMT
11 Aug 2016
Setting up the basic environment
We assume that you already have configured the Common Lisp environment (including Quicklisp) and that you know the basics of the Common Lisp programming. The necessary systems to load are
clim-listener, you may load them with the Quicklisp. After that launch the listener in a separate thread:
(ql:quickload '(mcclim clim-listener)) (clim-listener:run-listener :new-process t)
Finding your way around CLIM
CLIM is the global minimum in graphics toolkits. Geometry, and means of abstracting it. Switch the listener into the CLIM-INTERNALS package to get started. Type
(in-package climi) in the
Evaluate the 4 following forms in
Slime REPL, then call
(cos-animation) in the
Listener REPL to demonstrate CLIM's animation capabilities. You cannot evaluate (cos-animation) in the
Slime REPL, as
*standard-output* is bound to the its output stream which isn't a
sheet, and thus cannot be drawn on.
(in-package climi) (defparameter *scale-multiplier* 150 "try modifying me while running!") (defparameter *sleep-time* 0.0001 "modify and eval to speed or slow the animation, set to `nil' to stop") (defun cos-animation () (let* ((range (loop for k from 0 to (* 2 pi) by 0.1 collect k)) ; length of 62 (idx 0) (record (updating-output (*standard-output*) (loop for x from (nth idx range) to (+ (nth idx range) (* 2 pi)) by 0.01 with y-offset = 150 for x-offset = (- 10 (* *scale-multiplier* (nth idx range))) for y-value = (+ y-offset (* *scale-multiplier* (cos x))) for x-value = (+ x-offset (* *scale-multiplier* x)) do (draw-point* *standard-output* x-value y-value :ink +green+ :line-thickness 3))))) (loop while *sleep-time* do (progn (sleep *sleep-time*) (if (= 61 idx) (setq idx 0) (incf idx)) (redisplay record *standard-output*)))))
If you want to stop the animation, issue in the
(setf *sleep-time* nil)
If it wasn't already obvious, you can plot w/e.
(CLIM-LISTENER::DRAW-FUNCTION-FILLED-GRAPH #'tanh :min-x (- 0 pi pi) :max-x pi :min-y -1.1 :max-y 1.5 :ink +blue+)
Drawning class hierarchy
,clear output history in the
Listener REPL and RET to clear the screen.
"," indicates that you are activating a command. Try typing comma, then C-/ to activate completion. C-c C-c to dismiss.
Children of the class
CLIMI::SHEET can be drawn on using arbitrary geometry. Try
(clim-listener::com-show-class-subclasses 'sheet) in the
Listener REPL to view the subclasses of it.
Commands and presentations
The name COM-whatever indicates that the function in question is a clim command, which you can define in the
Slime REPL like so,
(in-package clim-listener) ;;; Runme! We will need these in a moment. (dolist (image-name '("mp-avatar.png" "vulpes-avatar.png" "stas-avatar.png" "suit-avatar.png" "rainbow-dash-avatar.png" "chaos-lord-avatar.png")) (uiop:run-program (format nil "curl https://common-lisp.net/project/mcclim/static/media/tutorial-1/~A --output /tmp/~A" image-name image-name))) (define-listener-command (com-ls :name t) ((path 'string)) (clim-listener::com-show-directory path))
,ls /tmp/ -- then
,display image <SPACE> and click on one of the displayed paths to supply it as an argument. At the core of CLIM is the notion of a presentation. Objects have presentation methods, ie, some arbitrary rendering geometry, and when PRESENT'd on the screen CLIM remembers the type. Thus one can supply objects of the appropriate types as arguments to a command simply by clicking on them. Read about
CLIMI::DEFINE-COMMAND in the specification to learn more. Let's define our first presentation method.
Intermixing S-expressions with the presentation types
Evaluate the forms below in the
(in-package climi) (defvar lords '("mircea_popescu" "asciilifeform" "ben_vulpes")) (defclass wot-identity () ((name :accessor name :initarg :name :initform nil) (avatar :accessor avatar :initarg :avatar :initform nil))) (defmethod lord? ((i wot-identity)) (member (name i) lords :test #'string=)) (define-presentation-type wot-identity ()) (defun make-identity (name avatar-pathname) (make-instance 'wot-identity :name name :avatar avatar-pathname)) (defparameter *identities* (mapcar (lambda (l) (apply #'make-identity l)) '(("mircea_popescu" #P"/tmp/mp-avatar.png") ("ben_vulpes" #P"/tmp/vulpes-avatar.png") ("asciilifeform" #P"/tmp/stas-avatar.png") ("Suit" #P"/tmp/suit-avatar.png") ("RainbowDash" #P"/tmp/rainbow-dash-avatar.png") ("ChaosLord" #P"/tmp/chaos-lord-avatar.png")))) (define-presentation-method present (object (type wot-identity) stream (view textual-view) &key acceptably) (declare (ignorable acceptably)) (multiple-value-bind (x y) (stream-cursor-position stream) (with-slots (name avatar) object (draw-pattern* stream (climi::make-pattern-from-bitmap-file avatar :format :png) (+ 150 x) (+ 30 y)) (draw-text* stream name (+ 153 x) (+ 167 y) :ink +black+ :text-size 20) (draw-text* stream name (+ 152 x) (+ 166 y) :ink (if (lord? object) +gold+ +blue+) :text-size 20)) (setf (stream-cursor-position stream) (values x (+ 200 y))) object)) (defun eval-and-then-call-me-in-the-listener () (let* ((n 8) (sheet *standard-output*)) (labels ((gen (i) (let* ((out-and-start '(f x))) (loop for k from 0 to i do (setq out-and-start (apply #'append (mapcar (lambda (s) (case s ;; (x '(y f + f f + + x)) ;; (y '(y f + f f x - - f f x)) (x '(+ y f + f f + y y +)) (y '(f - y f f x f f)) )) out-and-start)))) (remove-if (lambda (sym) (member sym '(x y) :test 'eq)) out-and-start)))) (let* ((x 300) (y 300) (new-x x) (new-y y) (a 1) (step 15)) (loop for r in (gen n) do (progn (cond ((= a 1) (setq new-y (+ step y))) ((= a 2) (setq new-x (- x step))) ((= a 3) (setq new-y (- y step))) ((= a 4) (setq new-x (+ step x)))) (case r (f (clim:draw-line* sheet x y new-x new-y :ink clim:+blue+ :line-thickness 6 :line-cap-shape :round) (setq x new-x y new-y)) (- (setq a (if (= 1 a) 4 (1- a)))) (+ (setq a (if (= 4 a) 1 (1+ a)))) (t nil))))) (let* ((x 300) (y 300) (new-x x) (new-y y) (a 1) (step 15)) (loop for r in (gen n) do (progn (cond ((= a 1) (setq new-y (+ step y))) ((= a 2) (setq new-x (- x step))) ((= a 3) (setq new-y (- y step))) ((= a 4) (setq new-x (+ step x)))) (case r (f (clim:draw-line* sheet x y new-x new-y :ink clim:+white+ :line-thickness 2 :line-cap-shape :round) (setq x new-x y new-y)) (- (setq a (if (= 1 a) 4 (1- a)))) (+ (setq a (if (= 4 a) 1 (1+ a)))) (t nil))))))))
(dolist (i *identities*) (present i)) at the CLIM Listener.
(lord? at the listener and then clicking on one of the identities. Add a closing paren and RET. Notice how objects can be seamlessly intermixed with S-expressions. If this example fails for you it may be that you have not recent enough version of McCLIM.
Unripe fruits. The future isn't what it used to be - some assembly required.
(CLIM-DEMO::DEMODEMO)(available with system
The essential machinery of a 'live' GUI builder
Navigator (essentially an extended `apropos')
11 Aug 2016 1:00am GMT
19 Jul 2016
Thanks to Masayuki Takagi, a not-so-obvious bug was fixed in the basic unification machinery of CL-UNIFICATION. This led to the addition of a couple of new utility functions and some other cleanups (hopefully).
19 Jul 2016 10:58am GMT
12 Jul 2016
As part of yet another yak-shaving quest I've come to write a library that I only now realise I've been missing for a while: an extensible and generic iteration macro.
Now, you might be familiar with the Iterate library that is supposed to fill some of the same niches as For does. Despite being available for a very long time, Iterate is not very commonplace as far as I've been able to tell. Most people still use the Common Lisp standard
map* variants- I do too.
I've tried to get into iterate multiple times, but I never really was able to get along with it, especially when it came to figuring out how to extend it for further constructs. Now, naturally this may just be my problem, but nevertheless I'm bold enough to consider that enough justification for me to go ahead and write my own attempt at a solution for the problem. I won't elaborate why I don't like iterate here as I believe there isn't much constructive or useful input to be gained from doing so.
Instead I will try to illustrate what For does, why it does it, and how it does it. A good part of that is already covered in the documentation but I will allow myself to be a bit more prosaic rather than declarative here. So let's dive in and have a look at the most simple of loops- an infinite one!
The main idea behind For in contrast to Loop and Iterate is to mirror the structure of
let. As such we always have a list of bindings and a body. Unlike
let however, every variable is followed by a symbol that describes what kind of binding it is- how it is initialised, stepped, how if at all it terminates the loop, and whether it delivers a return value. So let's take a look at something a bit more sophisticated.
(for ((i ranging 1 10)) (print i))
Here we see an actual binding in action. We bind a variable
i using the
ranging type with the arguments
10. As probably expected this will go through the numbers 1 to 10 and print each of them. Bindings can accept any number and kind of argument they want to:
(for ((i ranging 1 10 :by 2)) (print i))
And now it will step by twos. Stunning.
Similar to Loop, For can of course also iterate over various sequences and other objects. Out of the box iteration bindings for lists, vectors, hash-tables, and packages is provided. Also just like loop we can accumulate values in various ways. Here too we support the same things as Loop does, namely collecting, appending, nconcing, counting, summing, maximising, minimising.
Additionally however, For provides a generic iterator mechanism for the cases where you do not really know or care what type your sequence is. This can also be used to update the sequence in-place if doing so makes sense. Let's see a practical example of converting a generic kind of sequence into a list:
(for ((item over my-sequence) (list collecting item)))
This will function without any work required from you for lists, vectors, arrays, streams, wild pathnames, packages, and hash-tables. It can also be extended to be able to iterate over any kind of sequence you might want by writing a new iterator class and the implementation for three simple methods.
Sometimes it's also necessary to terminate the loop according to some condition, and a binding does not seem like the correct place to put this kind of constraint. This is why in addition to bindings we have clauses that can appear in the body of the For.
(for ((i from 0)) (while (< i 10)) (print i))
This is nice and can easily be implemented by a macrolet. At least that's what I went for until I started trying to wrap my brain around the problem of return values. Some clauses like
thereis would like to return a value- in this case whether the test has succeeded at all.
In the case of bindings where we have full control over the expressions and literals we can easily transform them however we want. This allows bindings to establish forms that wrap around the For body, add return values, and so on. However, in the case of clauses implemented through a macrolet the expansion happens within the body and at a different time. It is thus impossible¹ for the clause macro to communicate that it would like to hook into the mechanism surrounding the body. We could do it at run-time of course, but that would mean run-time consing and unnecessary tests every iteration- way too costly.
So, unfortunately I had to retract that idea and instead go for a minimal code-walking. It is so minimal that I don't know if it can even really be called that. What For now does is look through each item in its body, test whether it is a cons with a symbol in its car that refers to a clause, and if so call the clause expander function for that. This allows us to give clauses the same amount of power as bindings have and fixes the issues we had before. As you can see however there is a cost associated with it. In order to avoid full-blown code-walking (something understandably frowned upon) we can only recognise literal top-level For body expressions as clauses.
Nevertheless I think this is a small price to pay. The amount of times I would want to use a clause within another form seems very minimal to me at this point in time. Who knows though, I may come to eat my words at a later date.
Another thing worth mentioning I think is the actual extension mechanism of For itself. As per usual for my systems there's varying levels of support to help you, but you can always ignore them and just get full control so you can define exactly what's going on how.
So- the lowest we can go is defining a clause or binding function directly. We can define the above
while clause like that simply enough. After all, all it needs to do is expand to a test that ends the loop if passed instead of the clause form.
(define-direct-clause while (form) (values NIL `(unless ,form (end-for)))
Generally if we think about what an iteration is about, we can distinguish three sections: an initialisation that introduces some values and initialises them, a body section that is executed on every step, and an end section that determines a return value. This is reflected in the three values that a binding or clause function must return- a form to wrap around the rest of the For block, a form to evaluate every step, and an optional form to evaluate as a return value if we think we have data that would be useful to return. To illustrate the return value we'll also look at the
returning clause, which is useful if we have a non-standard value we'd like to give back, or if we want to force the primary value to something else.
(define-direct-clause returning (form) (values NIL NIL form))
All return values from bindings and clauses are gathered together into a single
values form at the end of the For. This allows us to have multiple
collect bindings or combine an accumulation with a clause and things like that. In general it's just convenient to allow the loop to return multiple values.
Most bindings and clauses outside of the most primitive ones will want to establish some kind of helper variables around the loop to keep, say, the head and tail of the list being accumulated. In order to provide this conveniently the
&aux arguments in the lambda-list of the next definition macros are rewritten such that their value inside the definition body is a gensym and it automatically expands to a
let that binds the gensym to the specified value. Thus writing our
collecting binding becomes very simple:
(define-form-binding collecting (var form &aux (head (cons NIL NIL)) (tail head)) `(setf ,tail (setf (cdr ,tail) (cons ,form NIL)) ,var (cdr ,head)))
tail are bound to fresh gensyms in the body, so they insert gensyms into our backquote expression. Simultaneously the definition takes care of the first return value for us by constructing an appropriate
let* form that binds the gensym contained in
(cons NIL NIL) and
head respectively. This form is then wrapped around the rest of the For so that the variables are available within the body.
Finally, often times we also know that all of the arguments passed to the binding need to only be evaluated once before the loop. To make this convenient we have
define-value-binding which treats the actual binding arguments similar to how the
&aux arguments work. Using this, defining something like
across becomes trivial as well:
(define-value-binding across (var vector &aux (i -1) (length (length vector))) `(if (= ,length (incf ,i)) (end-for) (update ,var (aref ,vector ,i))))
But just as mentioned before, if you don't trust the system to do this for you or simply don't like it, you can always return to the low-level definition macros and do the plumbing yourself.
Given that For allows you to both expand into body forms, surrounding forms, and return value forms, I think it is safe to say that pretty much every feature you might need to express in an iterator can be done and without much to write either. Take a look at the definitions of the standard bindings and clauses to get a feel for it.
Finally I'd like to take a look at the previously mentioned iterator system that For bundles with it. As stated, there's very little you need to do to add support for a new data type. Subclass
iterator, add methods for
make-iterator and you're done. With that in place, you can directly go ahead and use the
over binding to go through your sequence. If it makes sense you can also add support for
(setf current), enabling you to use the
updating binding which permits setting the current element in the sequence as well.
All in all I hope that I've figured out some good solutions to the problems presented by an extensible looping construct. Since the system is still very young, I don't really have too much experience with it myself yet and can't make any grand claims like this being the "be all end all iteration macro" or whatever. But it doesn't have to be that either. It solves the problem that drove me into this direction, as well as a few other ones on top of that, so I'm fine with it being what it is. If I've managed to convince you to look at For to see if it fits into your toolbelt, then I would already have achieved much more than I initially set out to do.
 This is not quite true, as pointed out to me by Mark Cox. It is indeed possible to make macros communicate with a bit of ingenuity. Relevant to this are the COMPILER-LET-CONFUSION Issue in the CLHS, and an example he was kind enough to write up to illustrate it.
12 Jul 2016 4:18pm GMT
10 Jul 2016
Zach's Querying plists blog post showcases a neat little querying DSL for plists. I couldn't shake the feeling that it looked an awful lot like pattern matching. I've often been impressed by optima, but I barely get to use it, so I thought I should try and see what querying plists looked like using pattern matching.
Here's what I came up with. Zach's example
(query-plists '(:and (:= :first-name "Zach") (:= :state "ME") (:not (:= :last-name "Beane"))) *people*)
(remove-if-not (lambda-match ((plist :first-name "Zach" :state "ME" :last-name (not "Beane")) t)) *people*)
It turned out more succinct than I initially expected! Also, it's trivially adaptable to other kinds of objects. E.g., given the following class:
(defclass person () ((first-name :initarg :first-name) (last-name :initarg :last-name) (state :initarg :state)))
all we have to do is swap
plist with the class name
person and we're all set:
(remove-if-not (lambda-match ((person :first-name "Zach" :state "ME" :last-name (not "Beane")) t)) *people*)
We can't quite define something exactly like Zach's
query-plists because, AFAICT, optima's patterns are not first-class objects but perhaps we can cheat a little bit.
;; naming things is hard. :-/ (defmacro matchp (pattern) `(lambda-match (,pattern t))) (defun filter (predicate list) (remove-if-not predicate list)) (filter (matchp (plist :first-name "Zach" :state "ME" :last-name (not "Beane"))) *people*)
Making this equally succinct when the query criteria are not constant is a challenge for another day and makes it clear that
matchp is a lousy abstraction. ;-)
10 Jul 2016 11:39pm GMT
07 Jul 2016
For normal, "serious" data, I like to stick things in a database and uses its query system to full effect. But sometimes I have a temporary need to work on a bunch of temporary data in plist form, and want to easily poke through it. Here's the latest version of something I find myself doing pretty often for that:
(defun compile-plist-query (query) (labels ((callfun (object) (lambda (fun) (funcall fun object))) (compile-= (keyword value) (lambda (plist) (equal (getf plist keyword) value))) (compile-and (funs) (lambda (plist) (every (callfun plist) funs))) (compile-or (funs) (lambda (plist) (some (callfun plist) funs))) (compile-not (fun) (lambda (plist) (not (funcall fun plist))))) (let ((operator (first query)) (operands (rest query))) (ecase operator (:= (compile-= (first operands) (second operands))) (:and (compile-and (mapcar #'compile-plist-query operands))) (:or (compile-or (mapcar #'compile-plist-query operands))) (:not (compile-not (compile-plist-query (first operands)))))))) (defun query-plists (query plists) (remove-if-not (compile-plist-query query) plists))
With the definitions above, I can use any combination of logic for querying the plists for particular property values:
(query-plists '(:and (:= :first-name "Zach") (:= :state "ME") (:not (:= :last-name "Beane"))) *people*)
This is similar to the MP3 query system described in Practical Common Lisp, but a little simpler and self-contained, suitable for random-ish plist structure.
07 Jul 2016 4:06pm GMT
03 Jul 2016
Nice idea from Matt Emerson:
I don't know if anyone will find it useful, but as an experiment, I've decided to try holding IRC "office hours" on #ccl on irc.freenode.net.
So, on Tuesday, July 5, from 10:00 am to 11:00 am and 4:00 pm to 5:00 pm Eastern time (that's 14:00 to 15:00 and 20:00 to 21:00 UTC), I'll be available in the channel to talk about whatever you want, as long as it relates to CCL somehow.
If you don't have an IRC client, you can use http://webchat.freenode.net. Enter your preferred nickname (I use "rme" for myself, for example) in the nickname field, and use "#ccl" for the channels field. Leave the "Auth to services" checkbox unchecked.
03 Jul 2016 12:32am GMT
29 Jun 2016
Clozure CL's default prompt is "? ". You can customize this by setting ccl:*listener-prompt-format* to a format control of your choice.
Note that a format control "string" can be a function. Here's an example that makes the prompt contain the current package name.
(defun prompt-formatter (stream break-level) (princ (package-name *package*) stream) (if (plusp break-level) (format stream " ~d > " break-level) (write-string "> " stream)))
To start using this, do
(setq ccl:*listener-prompt-format* #'prompt-formatter)
29 Jun 2016 11:28pm GMT