24 Nov 2020

feedPlanet Lisp

Michał Herda: Goodbye, Hexstream

I am saddened that I need to write this post, but I need to make a public confession.

After Jean-Philippe Paradis, a Common Lisp programmer better known online as Hexstream, requested me to review his "extensive contributions" to the Common Lisp ecosystem, he seems to have disliked my reply so much that he has declared me the single biggest threat to the Common Lisp community right now.

(A gist copy of the review is here for people who would rather avoid browsing the full issue.)

The review has appeared after yet another discussion thread on GitHub - originally about implementations of Clojurelike arrow macros in Common Lisp - has been derailed by Hexstream in the traditional way in which he derailed many [1] other[2] GitHub[3] discussions[4]: asserting as a logical fact that his preferences take precedence over other people's preferences, aggressively calling out other people for questioning this state of matters, and finally playing the victim card of being silenced, censored, and tortured by a so-called Common Lisp Mafia.

Unlike during the past few times, this time I have decided not to give up posting. On the contrary, I have spend a considerable amount of my personal time (including one all-nighter) to actually respond to every single post of Hexstream, analyze it, take it apart into individual claims that he is making, and refute every single false point that I could find to the best of my ability using the full extent of my available tools.

After several posts of increasing angriness exchanged with Hexstream, in which discussion I have once again tried to coerce him into changing his course and stop being an aggressive offender towards members of the Common Lisp community, and after being explicitly invited to analyze Hexstream's contribution to the Common Lisp community in a tweet of his, I replied to his request with an analysis of the public data collected from GitHub, Quicklisp and Hexstream's public CV. Hexstream has announced multiple times that he is proud of this information and there is nothing to hide there; no, quite the contrary. Hence, I felt welcome to use it and see for myself what kinds of prominent contributions of his I must have missed.

It seems that my analysis of that data was not well-received; Hexstream disappeared with a mere "see you in 2021" comment, stating that he has projects with higher priorities to work on at the moment, and simply replied on GitHub that "my posts contain countless factual, logical and other errors". Afterwards, his Twitter contained this.

I did have a fair amount of respect left for phoe before today, but after he said I am not a Common Lisp expert and that I am a fraud, based on malicious deliberately superficial

Welp.

            (with-irony "

          

It seems to me that I must have thought the unthinkable. (How could I have said that he is not a Common Lisp expert and a fraud? How was it even possible!?) Moreover, I then dared to say it aloud. Worst of all, I even backed it all with solid, concrete, data-based evidence that cannot be immediately refuted as a mere opinion and requires some serious figuring out of how to turn it around so that the Common Lisp Mafia is guilty for all the facts that I've noted.

All of a sudden, after posting this single post, I have become the main threat to the whole Common Lisp community, declared impossible to directly and indirectly fund in an ethical manner, and then proclaimed to require immediate medical attention of psychiatric nature.

Oh goodness. I assume that the analysis must have been way too short for his liking. I regret that I have not found the time to go into his GitHub issues in detail...

            ")

          

So, Hexstream. If you're reading this, I hope that my review serves as a proper wake-up call for you to actually see that your behavior is off and needs adjustment in order for other people to actually consider you acceptable in the Common Lisp community. If it does not, I have done everything to actually try and help you as a fellow Common Lisp hacker. I can, and will, do no more in this matter, and will instead do everything to protect the people I respect, like, and cooperate with from your destructive influence.

You are planning to launch some kind of Common Lisp Revival 2020 Fundraiser soon. I would like to tell you that I consider you to be the wrong person to launch one: not even for any of the aforementioned reasons, but for the reason that to you, Common Lisp seems to be a completely different language than it is to me. Based on the above review that you have requested me to do, it seems that you perceive Common Lisp as a strictly single-player language where you have to struggle against countless feats and enemies on Twitter, GitHub, and wherever else, in order to produce anything of even the smallest value after grand feats and massive effort to struggle against censorship.

On the contrary, I know many people who consider Common Lisp to be a multiplayer language where people support one another, are eager to help each other, share knowledge, indulge in fascinating projects that would be tough to indulge in with other languages and, best of all, are not hostile towards one another at the smallest hint of suspicion. Some of those people form the Common Lisp Foundation that, in my opinion, should take over any kind of Common Lisp revival fundraisers.

Obviously, all other reasons from my analysis why you are not entitled to represent the Common Lisp community as head of such a fundraiser still apply. And they are much more damning than the worldview issue above.

  • Your claimed commercial expertise in Common Lisp is void.
  • Your fifteen years of overall experience in Lisp have no basis in actual code.
  • Your projects larger than micro-utilities have been so poor that, as you claim, you have disposed of them yourself.
  • Your micro-utilities do not have a single dependent in the main Quicklisp distribution and they do not show signs of actual use by programmers.
  • Your documentation projects are generally not acceptable in the Common Lisp community because they are encumbered by the implicit unbearable personality of their author.
  • You have not contributed a single line of code to any GitHub repository hosted by anyone else throughout your eight and a half years of presence on GitHub and fifteen years of overall programming experience that you claim to have.
  • You derail GitHub conversations with offensive and aggressive comments, indulge in Twitter rants containing more offensive and aggressive comments, and tie them together with your personal website containing even more offensive and aggressive comments.
  • You repeatedly defame various honored and respected members of the Common Lisp community, including Rainer Joswig, Michael Fiano, Daniel Kochmański, Stas Boukarev, and Zach Beane. And, I guess, me.
  • Oh, about Zach! have I mentioned https://xach.exposed ?

And to top it all, after the above analysis was posted, instead of fulfilling my hopes and responding to this critique of your Lisp merit by indulging in meritocratic discussion about your technical contributions to the Common Lisp ecosystem, you instead immediately announced that I require psychiatric help.

For completeness, I do have to admit: you have been popularizing crowdfunding among Lispers and achieved visible success there, with multiple authors and repositories adopting various means of crowdfunding (GitHub Sponsors, Patreon, LiberaPay) thanks to your efforts and suggestions. This is the one single thing that I can unambiguously consider a net positive coming from you. That's all.

Other than that, I do have to repeat what I have said at the end of my analysis. You try to pose as a Common Lisp expert. No, with all of the above I have no reasons to claim that you are one. Your expertise is hollow. Your experience seems false. You pretend to be someone you are not. You are a scam, Hexstream, and I am saddened and torn that I need to speak these words because I sincerely wish you were not.


The earliest Lisp commit that I was able to find in my GitHub repositories is from November 2015. That is exactly five years ago. In 2015, I was getting frustrated over Emacs keybindings. In 2015, you were "exposing" Zach Beane. Through these five years, I was learning Lisp to the best of my ability. Through these five years, you were doing I have no idea what. I can only guess based on what I see.

And I see Twitter rants. I see GitHub issue derailments. I see self-announced policies that contradict one another. I see tiny Lisp libraries with zero users. I see no other Lisp code of yours. I see no code of yours in any other GitHub repositories. I see big claims backed by nothing. I see an image of a Common Lisp expert that is so fragile that it falls into pieces after a brief glance.

Seriously, what were you doing with your life during these years? Researching ethics? Verifying the boundaries set by Twitter and GitHub moderation teams? Fighting for your life while the Common Lisp Mafia caged you and demanded a ransom of 20,000,000 US parentheses for your freedom?

I simply cannot comprehend it. And I do feel sorry for you, since most likely neither can you.


If you are still reading, please answer one question that I will ask at the end of this block of text. I will attempt to be somewhat honest regarding myself in the topic of my own impact on the Common Lisp community, as I see it. No boasting too much, not being too humble. Let's try it.

I have attempted to complete the Common Lisp UltraSpec which I talked about at an European Lisp Symposium one time and then failed miserably at this task after grossly misestimating it. I have implemented package-local nicknames in Clozure Common Lisp and then used the momentum from that work to make a portability library for package-local nicknames. I have managed to rewrite and optimize the somewhat famed split-sequence system commonly used in the Common Lisp ecosystem. I have managed to overhaul the even more famed Lisp Koans by rewriting them almost from scratch and fixing multiple compliance errors. I have successfully convinced Massachusetts Institute of Technology to release the Common Lisp WordNet interface under a permissive license (which took only half a year of pinging people via mail) and fixed it up as appropriate. I have written a utility suite for managing protocols and test cases with some documentation that I am proud of even after two whole years. I wrote an implementation of Petri nets in Common Lisp that seems either to work fine or not to be used at all, because I do not get much attention from it; still, I've tested it (hopefully) well enough to be useful in the general case. I recently wrote the fastest priority queue available in Common Lisp after someone mentioned that the ones on Quicklisp are too slow. I then ended up miserably failing at rewriting the Common Lisp arrows system, which resulted in a different system with a tutorial for arrows that I have received several thanks for. And then there's some smaller libraries that might not be all that mentionworthy.

I have been hosting the Online Lisp Meeting series which have met general acclaim and popularity and are considered a worthy continuous extension of the ideas of the European Lisp Symposium - even if, in my opinion, they contain a bit too much CL content, compared to the ELS ideals and statistics. The eleventh installment is bound to happen this week, where I will speak for the second time - again about the topic of control flow and condition systems. I already have two more talks queued up and we plan on going until the next European Lisp Symposium, which will most likely eat up all of the available talks and then some. (Maybe some of the rejected papers will sublimate as OLM videos though?... I sincerely hope so! ELS recently had to reject papers not because they were bad, but because they had an already full schedule.)

With help of countless people helping me on various stages of the book lifecycle and with support from Apress Publishing, I have managed to release the book The Common Lisp Condition System along with a pair of accompanying Common Lisp systems, the larger Portable Condition System and the smaller trivial-custom-debugger, plus a release of source code from the book and a free online appendix to the book containing content that did not make it inside in time. I have also proven that the condition system can be easily implemented in a non-Lisp, which is Java, and I will talk about this in extent to the WebAssembly committee to ensure that WASM has all the necessary functionalities to ensure that Common Lisp can be efficiently implemented on WASM.

Finally, I made some art once. I think it did not sting anybody's eyes too hard. Or that it's strictly Lisp-related too much... but hey, it's CL implementations, and the Lisp Lizard.

I think I am generally tolerated and maybe even enjoyed in my community as a Common Lisp programmer, despite my occasional outbursts of frustration and outright stupidity. I try to be available on Reddit, IRC, Discord, and in private messages for all sorts of support that I am capable of providing. I try to teach other people the way I was taught when I was starting out. Whenever I notice that I should apologize and make amends because I fucked up somewhere (e.g. in the recent Quickdocs issue), I do try my best to be sorry and amend my behavior as appropriate, and I try to welcome other people's remarks and inegrate them into my behavior as appropriate; I think it helps other people tolerate my behavior when I'm not easily tolerable.

And, well, you know, there is this single person in my environment who just keeps on smearing shit on people in my vicinity, but I don't think I care anymore; this person has willingly made so many enemies by now, that they are ignored by many, confronted by few (who actually have some time to spare), and hell, I even got some most unexpected people to cheer me on in my attempts to actually try and confront this guy and his bullshit excuses for repeatedly setting fires in the Common Lisp world.

But, yeah, anyway. You still consider that it's me who needs psychiatric help. Is that right?


So, Hexstream, this is a goodbye. Thank you for the unique chance to train my patience, persistence, and insistence. I assure you that it has not gone to waste, and I assure you that I will remember it for the rest of my life.

Since you do not seem to want to change your behavior in the slightest, then I wish you to stay on your current course and not change in the slightest so you may see for yourself where it leads you. The faster you slide into irrelevance because of your current choice, the healthier the Common Lisp community will be.

(And I mean the real Common Lisp community, containing more than just a single person who's purely accidentally named Jean-Philippe.)

Bye. I don't think I will miss you much, even though I adore the technical thought behind some of your libraries. And if I encounter you again on the Internet, be prepared to once again meet the side of me that has long run out of spare chances to give you anymore.

24 Nov 2020 1:24pm GMT

23 Nov 2020

feedPlanet Lisp

Vsevolod Dyomkin: The Common Lisp Condition System Book

Several months ago I had a pleasure to be one of the reviewers of the book The Common Lisp Condition System (Beyond Exception Handling with Control Flow Mechanisms) by Michał Herda. I doubt that I have contributed much to the book, but, at least, I can express my appreciation in the form of a reader review here.

My overall impression is that the book is very well-written and definitely worth reading. I always considered special variables, the condition system, and multiple returns values to be the most underappreciated features of Common Lisp, although I have never imagined that a whole book may be written on these topics (and even just two of them). So, I was pleasantly flabbergasted.

The book has a lot of things I value in good technical writing: a structured and logical exposition, detailed discussions of various nuances, a subtle sense of humor, and lots of Lisp. I should say that reading the stories of Tom, Kate, and Mark was so entertaining that I wished to learn more about their lives. I even daydreamt (to use the term often seen throughout the book) about a new semi-fiction genre: stories about people who behave like computer programs. I guess a book of short stories containing the two from this book and the story of Mac from "Practical Common Lisp" can already be initialized. "Anthropomorphic Lisp Tales"...

So, I can definitely recommend reading CLCS to anyone interested in expanding their Lisp knowledge and general understanding of programming concepts. And although I can call myself quite well versed with the CL condition system, I was also able to learn several new tricks and enrich my understanding. Actually, that is quite valuable as you never know when one of its features could become handy to save your programming day. In my own Lisp career, I had several such a-ha moments and continue appreciating them.

This book should also be relevant to those, who have a general understanding of Lisp, but are compelled to spend their careers programming in inferior languages: you can learn more about one of the foundations of interactive programming and appreciate its value. Perhaps, one day you'll have access to programming environments that focus on this dimension or you'll be able to add elements of interactivity to your own workflow.

As for those who are not familiar with Lisp, I'd first start with the classic Practical Common Lisp.

So, thanks to Michał for another great addition to my virtual Lisp books collection. The spice mush flow, as they say...

23 Nov 2020 12:41pm GMT

16 Nov 2020

feedPlanet Lisp

Michał Herda: Damn Fast Priority Queue: a speed-oriented priority queue implementation

I think I have accidentally outperformed all of the Quicklisp priority queue implementations. Enter Damn Fast Priority Queue.

Detailed description and benchmarks are available on the GitHub repository. It seems that my implementation is consistently an order of magnitude faster than most of the other priority heaps (with Pileup being the runner-up, only being about 3-4x slower than DFPQ).

16 Nov 2020 10:18pm GMT

15 Nov 2020

feedPlanet Lisp

Michał Herda: Cafe Latte - a condition system in Java

I've more or less finished Cafe Latte - an implementation of Common Lisp dynamic variables, control flow operators, and condition system in plain Java.

It started out as a proof that a condition system can be implemented even on top of a language that has only automatic memory management and a primitive unwinding operator (throw), but does not have dynamic variables or non-local returns by default.

It should be possible to use it, or parts of it, in other projects, and its source code should be readable enough to understand the underlying mechanics of each Lisp control flow operator.

15 Nov 2020 10:26pm GMT

07 Nov 2020

feedPlanet Lisp

Nicolas Hafner: Closing in on Production - November Kandria Update

header
October somehow flew by really quickly for me. It's already November, and we're nearing the end of the year, too. Just thinking about that is making me reminiscent, but I'll have to hold off on doing my yearly wrap-up for another two months! Who knows, a lot more can still happen in that time. Last month marked another release for Kandria, and this month marked the start of Kandria being an actual team effort!

I'm really glad that it's no longer just me working on things. Fred already introduced himself in the last monthly, and by now he has already started work and delivered some really great stuff:

new light attack player idle

As a result, the game already feels a lot more fun to play. The step up from the combat animations I had made early in the year is huge!

We're still not done with it though, there's a few more moves missing, and a lot more left to adjust and fine-tune of course. We'll also have to get started on some real enemy designs soon and implement those to have some interesting encounters to test things with.

I can now also finally announce the third team member, Tim White, who'll be working on characters, story, and dialogue for the game:

Hey there! I'm Tim, a games writer from the UK. I've been in the industry for ten years now (where did the time go?!), and have been lucky enough to work at Jagex on Transformers Universe, and most recently with Brightrock Games on War for the Overworld and an unannounced game.

Kandria jumped out the screen at me straight away, with its detailed world and story, custom-made dev tools, and strong creative and artistic direction. I also have a real soft spot for post-apocalyptic worlds, and the ethics surrounding artificial life. Applying was a no brainer, and I can't wait to start!

You can find Tim on Twitter at @TimAlanWhite, or on the official Kandria Discord.

Both Tim and Fred will be giving quick updates on what's happening in the weekly newsletter from now on. The newsletter has now also been moved away from Mailchimp to my own mailing list service called Courier. I'm glad to finally have made the switch, freeing me from Mailchimp's slow and clunky interface!

On the engine side, I reworked the lighting and background systems to allow changing the lighting and parallax background to fit the current environment. As part of this I also changed the shadow casting to work properly so that it no longer contains the weird corner case glitches it used to.

I also had to make some fixes to the animation system to make it more capable and to make it less of a hassle to use when animations are changed or added. Previously the tooling there would easily mess up your data.

Then, in order to prepare for Tim, I reworked the quest system to be much easier to manage and control, and added a couple of additional features that should be very useful to control branching. To test it I made some quick draft animations for Fi and jotted her down in the test level.

fi

She'll now comment on things you can find throughout the level.

I also wrote a bunch of documentation to help Tim and Fred get set up and running with the game, introduced some very useful tooling like hot-reloading to make it faster to iterate on animations and textures, and improved the editor, especially for the in-game animation properties.

With all of this now in, we are very, very close to ending post-production. There's a few not-so-small things that I still need to do, like an animation system for the UI that I started working on yesterday, and one very nasty bug that popped up on Windows systems with surround sound configured. Still, with all of this in mind, I think we're well on track for the vertical slice release in March.

I hope there'll be a 0.0.4 demo release by the end of this month, which will be the last public demo until the vertical slice 0.1.0 demo. After that- I don't know yet how things will go. A lot about the game is going to become much clearer in the coming months as we decide on stuff like the core plot and work out the first area of the game for the vertical slice.

Aside from putting out whatever fires Fred and Tim stumble across this month, I'll be focusing on two things: first, fix surround sound on Windows. This is important to me as having the game crash and burn because of something so... tangential, is really terrible. Second, implement a UI animation system. The UI toolkit I'm using, Alloy, does not currently have a way to animate things. This is fine for tools and other UI like that, but in games you really want to spruce things up by tweening and animating to make your UI more interesting to look at. That's the last major addition to Alloy that's needed to have everything we need.

If time permits, I'll also work on some more platforming challenge levels to give the 0.0.4 demo some more content.

Anyway, I'm really happy to have a team together now, and I'm very excited to see how quickly things develop from here! To be fair, I'm also quite a bit worried what with being, I suppose, my own boss now, and the responsibilities that brings. I suppose time will tell whether I can figure out a good schedule and manage things well. For now I'm cautiously optimistic.

Alright, back to thinking about the animation system now, and see you next month, or next week if you're on the mailing list!

07 Nov 2020 2:01pm GMT

01 Nov 2020

feedPlanet Lisp

Alexander Artemenko: sphinxcontrib-cldomain

This is an add-on to the Sphinx documentation system which allows using of information about Common Lisp packages in the documentation.

Initially, Sphinx was created for Python's documentation and now it is widely used not only for python libraries but also for many other languages.

Sphinx uses reStructured text markup language which is extensible. You can write your own extensions in Python to introduce new building blocks, called "roles".

sphinxcontrib-cldomain consists of two parts. The first part is a python extension to the Sphinx which adds an ability to render documentation for CL functions, methods and classes. The second - a command-line docstring extractor, written in CL.

Initially, cldomain was created by Russell Sim, but at some moment I've forked the repository to port it to the newer Sphinx, Python3 and Roswell.

The coolest feature of the cldomain is its ability to mix handwritten documentation with docstring. The second - cross-referencing. You can link between different docstrings and chapters of the documentation.

Today I will not show you any code snippets. Instead, I've created an example repository with a simple Common Lisp system and documentation:

https://cl-doc-systems.github.io/sphinxcontrib-cldomain/

This example includes a GitHub workflow to update the documentation on a push to the main branch and can be used as a skeleton for you own libraries.

The main thing I dislike in Sphinx and cldomain is the Python :( Other cons are the complexity of the markup and toolchain setup.

In the next few posts, I'll review a few other documentation tools for Common Lisp and try to figure out if they can replace Sphinx for me.

I think we as CL community must concentrate our efforts to improve the documentation level of our software and choosing the best setup which can be recommended for everybody is the key.

01 Nov 2020 12:43am GMT

30 Oct 2020

feedPlanet Lisp

ABCL Dev: ABCL 1.8.0

Under the gathering storms of the Fall 2020, we are pleased to release ABCL 1.8.0 as the Ninth major revision of the implementation.

This Ninth Edition of the implementation now supports building and running on the recently released openjdk15 platform. This release is intended as the last major release to support the openjdk6 openjdk7, and openjdk8 platforms, for with abcl-2.0.0 we intend to move the minimum platform to openjdk11 or better in order to efficiently implement atomic memory compare and swap operations.

With this release, the implementation of the EXT:JAR-PATHNAME and EXT:URL-PATHNAME subtypes of cl:PATHNAME has been overhauled to the point that arbitrary references to ZIP archives within archives now work for read-only stream operations (CL:PROBE-FILE CL:TRUENAME, CL:OPEN, CL:LOAD, CL:FILE-WRITE-DATE, CL:DIRECTORY, and CL:MERGE-PATHNAMES). The previous versions of the implementation relied on the ability for java.net.URL to open streams of an archive within an archive, behavior that was silently dropped after Java 5, and consequently hasn't worked on common platforms supported by the Bear in a long time. The overhaul of the implementation restores the feasibility of accessing fasls from within jar files. Interested parties may examine the ASDF-JAR contrib for a recipe for packaging and accessing such artifacts. Please consult the "Beyond ANSI: Pathnames" Section 4.2 of the User Manual for further details for how namestrings and components of PATHNAME objects have been revised.

A more comprehensive list of CHANGES is available with the source.





30 Oct 2020 11:34am GMT

28 Oct 2020

feedPlanet Lisp

Alexander Artemenko: cl-pdf

This is the library for PDF generation and parsing.

Today I'm too lazy to provided step by step examples, but I have a real task to do with this library.

Some time ago I've read the article about productivity which recommended to print a "life calendar". This calendar should remind you: "The life is limited and the time's price is high."

The calendar is a grid where every box is one week of you life. The article suggested to buy a poster with the calendar, but I don't want to wait for a parcel with the poster! I want to print it now!

And here is where cl-pdf comes on the scene!

I wrote this simple function to generate the poster of A1 format:

(defun render (&optional (filename "life.pdf"))
  (flet ((to-ppt (size-in-mm)
           (/ size-in-mm 1/72 25.4)))
    (let* ((width (to-ppt 594))       ;; This is A1 page size in mm
           (height (to-ppt 841))
           (margin-top (to-ppt 70))
           (margin-bottom (to-ppt 30))
           (span  (to-ppt 2))
           (year-weeks 52)
           (years 90)
           (box-size (/ (- (- height (+ margin-top margin-bottom))
                            (* span (1- years)))
                        years))
           (boxes-width (+ (* box-size year-weeks)
                            (* span (1- year-weeks))))
           (boxes-height (+ (* box-size years)
                             (* span (1- years))))
           ;; horizontal margin depends on box size,
           ;; because we need to center them
           (margin-h (/ (- width boxes-width)
                        2))
           (box-radius (/ box-size 3))
           (helvetica (pdf:get-font "Helvetica")))
      (pdf:with-document ()
        (pdf:with-page (:bounds (rutils:vec 0 0 width height))
          ;; For debug
          ;; (pdf:rectangle margin-h margin-bottom
          ;;                boxes-width
          ;;                boxes-height
          ;;                :radius box-radius)
          (loop for year from 0 below years
                do (loop for week from 0 below year-weeks
                         for x = (+ margin-h (* week (+ box-size span)))
                         for y = (+ margin-bottom (* year (+ box-size span)))
                         do (pdf:rectangle x y box-size box-size :radius box-radius)))
          
          ;; The title
          (pdf:draw-centered-text
           (/ width 2)
           (+ margin-bottom
               boxes-height
               ;; space between text and boxes in mm
               (to-ppt 15))
           "LIFE CALENDAR"
           helvetica
           ;; font-size in mm
           (to-ppt 30))

          ;; Labels for weeks
          (let ((font-size
                  ;; We want labels to be slightly smaller than boxes
                  (* box-size 2/3)))
            (pdf:draw-right-text
             (+ margin-h
                 (/ box-size 4))
             (+ margin-bottom
                 boxes-height
                 ;; space between text and boxes in mm
                 (to-ppt 10))
             "Weeks of the year"
             helvetica
             font-size)
            
            (loop for week below year-weeks
                  do (pdf:draw-centered-text
                      (+ margin-h
                          (/ box-size 2)
                          (* week (+ box-size span)))
                      (+ margin-bottom
                          boxes-height
                          ;; space between text and boxes in mm
                          (to-ppt 3))
                      (rutils:fmt "~A" (1+ week))
                      helvetica
                      font-size))

            ;; Labels for years
            (pdf:with-saved-state
              (pdf:translate
               (- margin-h
                   (to-ppt 10))
               (- (+ margin-bottom
                      boxes-height)
                   (/ box-size 4)))
              (pdf:rotate 90)
              (pdf:draw-left-text
               0 0
               "Years of your life"
               helvetica
               font-size))
            
            (loop for year below years
                  do (pdf:draw-left-text
                      (- margin-h
                          ;; space between text and boxes in mm
                          (to-ppt 3))
                      (+ margin-bottom
                          (/ box-size 4)
                          (* year (+ box-size span)))
                      (rutils:fmt "~A" (- years year))
                      helvetica
                      font-size))

            ;; The Question.
            (pdf:draw-left-text
             (- width margin-h)
             (- margin-bottom
                 (to-ppt 10))
             "Is this the End?"
             helvetica
             (* font-size 2))
            
            (pdf:close-and-stroke)))
        (pdf:write-document filename)))))

Here is how result will look like:

The PDF can be downloaded here.

This program demonstrates a few features of cl-pdf:

There are a lot more features but all of them aren't documented, only several examples :(

GitHub shows 4 forks with some patches. And some of them are turned into a pull-request, but maintainer is inactive on the GitHub since 2019 :(

28 Oct 2020 8:49pm GMT

26 Oct 2020

feedPlanet Lisp

Alexander Artemenko: cl-async-await

This library implements the async/await abstraction to make it easier to write parallel programs.

Now we'll turn "dexador" http library calls into async and will see if we can parallel 50 requests to the site which response in 5 seconds.

To create a function which can return a delayed result, a "promise", we have to use cl-async-await:defun-async:

POFTHEDAY> (cl-async-await:defun-async http-get (url &rest args)
             (apply #'dexador:get url args))

Now let's call this function. When called it returns a "promise" object not the real response from the site:

POFTHEDAY> (http-get "https://httpbin.org/delay/5")
#<CL-ASYNC-AWAIT:PROMISE Not awaited>

Now we can retrieve the real result, using cl-async-await:await function:

POFTHEDAY> (cl-async-await:await *)
"{
  \"args\": {}, 
  \"data\": \"\", 
  \"files\": {}, 
  \"form\": {}, 
  \"headers\": {
    \"Accept\": \"*/*\", 
    \"Content-Length\": \"0\", 
    \"Host\": \"httpbin.org\", 
    \"User-Agent\": \"Dexador/0.9.14 (SBCL 2.0.8); Darwin; 19.5.0\", 
    \"X-Amzn-Trace-Id\": \"Root=1-5f9732d6-148ee9a305fab66c26a2dbfd\"
  }, 
  \"origin\": \"188.170.77.131\", 
  \"url\": \"https://httpbin.org/delay/5\"
}
"
200 (8 bits, #xC8, #o310, #b11001000)
#<HASH-TABLE :TEST EQUAL :COUNT 7 {1002987DE3}>
#<QURI.URI.HTTP:URI-HTTPS https://httpbin.org/delay/5>
#<CL+SSL::SSL-STREAM for #<FD-STREAM for "socket 192.168.43.216:49762, peer: 35.170.225.136:443" {10085B0BF3}>>

If we look a the promise object again, we'll see it has a state now:

POFTHEDAY> **
#<CL-ASYNC-AWAIT:PROMISE (VALUES {
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "0", 
    "Host": "httpbin.org", 
    "User-Agent": "Dexador/0.9.14 (SBCL 2.0.8); Darwin; 19.5.0", 
    "X-Amzn-Trace-Id": "Root=1-5f9732d6-148ee9a305fab66c26a2dbfd"
  }, 
  "origin": "188.170.77.131", 
  "url": "https://httpbin.org/delay/5"
}

  200 #<HASH-TABLE :TEST EQUAL :COUNT 7 {1002987DE3}>
  https://httpbin.org/delay/5
  #<SSL-STREAM for #<FD-STREAM for "socket 192.168.43.216:49762, peer: 35.170.225.136:443" {10085B0BF3}>>) >

Ok, it is time to see if we can retrieve results from this site in parallel. To make it easier to test speed, I'll wrap all code into the separate function.

The function returns the total number of bytes in all 50 responses:

POFTHEDAY> (defun do-the-test ()
             (let ((promises
                     (loop repeat 50
                           collect (http-get "https://httpbin.org/delay/5"
                                             :use-connection-pool nil
                                             :keep-alive nil))))
               ;; Now we have to fetch results from our promises.
               (loop for promise in promises
                     for response = (cl-async-await:await
                                     promise)
                     summing (length response))))

POFTHEDAY> (time (do-the-test))
Evaluation took:
  6.509 seconds of real time
  2.496912 seconds of total run time (1.672766 user, 0.824146 system)
  38.36% CPU
  14,372,854,434 processor cycles
  1,519,664 bytes consed
  
18300

As you can see, the function returns in 6.5 seconds instead of 250 seconds! This means cl-async-await works!

The only problem I found was this concurrency issue:

https://github.com/j3pic/cl-async-await/issues/3

But probably it is only related to Dexador.

26 Oct 2020 8:48pm GMT

23 Oct 2020

feedPlanet Lisp

Alexander Artemenko: parseq

With this library, you can write parsers to process strings, lists and binary data!

Let's take a look at one of the examples. It is a parser for the dates from RFC 5322. This format is used in email messages:

Thu, 13 Jul 2017 13:28:03 +0200

Parser consist of rules, combined in different ways. We'll go through the parser's parts one by one.

This simple rule matches one space character:

POFTHEDAY> (parseq:defrule FWS ()
               #\space)

;; It matches if string contains one space
POFTHEDAY> (parseq:parseq 'FWS
                          " ")
#\ 
T

;; But not on string from many spaces:
POFTHEDAY> (parseq:parseq 'FWS
                          "   ")
NIL
NIL

;; And of cause not on some other string
POFTHEDAY> (parseq:parseq 'FWS
                          "foo")
NIL
NIL

The next rule we need is the rule to parse hours, minutes and seconds. These parts have two digits and we'll use rep expression to specify how many digits matches the rule:

POFTHEDAY> (parseq:defrule hour ()
               (rep 2 digit))

POFTHEDAY> (parseq:parseq 'hour
                          "15")
(#\1 #\5)
T

See, this rule returns digits as the list! But to make it useful, we need the integer. Parseq rules support different kinds of transformations. They can are optional and can be specified like this:

;; This transformation will return as the string instead of list:
POFTHEDAY> (parseq:defrule hour ()
               (rep 2 digit)
             (:string))

POFTHEDAY> (parseq:parseq 'hour
                          "15")
"15"
T

;; Now we'll add a transformation from string to integer:
POFTHEDAY> (parseq:defrule hour ()
               (rep 2 digit)
             (:string)
             (:function #'parse-integer))

POFTHEDAY> (parseq:parseq 'hour
                          "15")
15 (4 bits, #xF, #o17, #b1111)
T

We'll define the minute and second rules the same way.

The next rule matches the abbreviated day of the week. It combines other rules or terms using or expression:

POFTHEDAY> (parseq:defrule day-of-week ()
               (or "Mon" "Tue" "Wed"
                   "Thu" "Fri" "Sat"
                   "Sun"))

POFTHEDAY> (parseq:parseq 'day-of-week
                          "Friday")
NIL
NIL

POFTHEDAY> (parseq:parseq 'day-of-week
                          "Fri")
"Fri"
T

;; The same way we define a rule for month abbrefiation
POFTHEDAY> (parseq:defrule month ()
               (or "Jan" "Feb" "Mar" "Apr"
                   "May" "Jun" "Jul" "Aug"
                   "Sep" "Oct" "Nov" "Dec"))

A little bit complex rule is used for matching timezone. Timezone is a string from 4 digits prefixed by plus or minus sign. We'll combine this knowledge using or/and expressions and will use option :string to get results as a single string:

POFTHEDAY> (parseq:defrule zone ()
               (and (or "+" "-")
                    (rep 4 digit))
             (:string))

POFTHEDAY> (parseq:parseq 'zone
                          "0300")
NIL
NIL

POFTHEDAY> (parseq:parseq 'zone
                          "+0300")
"+0300"
T

POFTHEDAY> (parseq:parseq 'zone
                          "-0300")
"-0300"
T

Now let's return to the time of day parsing. According to the RFC, seconds part is optional. Parseq has an expression ? to match optional rules.

Here is how a rule matching the time of day will look like:

POFTHEDAY> (parseq:defrule time-of-day ()
               (and hour
                    ":"
                    minute
                    (? (and ":" second))))

POFTHEDAY> (parseq:parseq 'time-of-day
                          "10:31:05")
(10 ":" 31 (":" 5))
T

To make the rule return only digits we have to use :choose transform. Choose extracts from results by index. You can specify index as an integer or as a list if you need to extract the value from the nested list:

POFTHEDAY> (parseq:defrule time-of-day ()
               (and hour
                    ":"
                    minute
                    (? (and ":" second)))
             (:choose 0 2 '(3 1)))

POFTHEDAY> (parseq:parseq 'time-of-day
                          "10:31:05")
(10 31 5)

;; Seconds are optional because of ? expression:
POFTHEDAY> (parseq:parseq 'time-of-day
                          "10:31")
(10 31 NIL)
T

;; This (:choose 0 2 '(3 1)) is equivalent to:
POFTHEDAY> (let ((r '(10 ":" 31 (":" 5))))
             (list (elt r 0)
                   (elt r 2)
                   (elt (elt r 3)
                        1)))
(10 31 5)

Another interesting transformation rule is :flatten. It is used to "streamline" result having nested structure and used in this rule which matches both time of day and timezone:

;; Without flatten we'll get nested lists:
POFTHEDAY> (parseq:defrule time ()
               (and time-of-day FWS zone)
             (:choose 0 2))

POFTHEDAY> (parseq:parseq 'time
                          "10:31 +0300")
((10 31 NIL) "+0300")

POFTHEDAY> (parseq:defrule time ()
               (and time-of-day FWS zone)
             (:choose 0 2)
             (:flatten))

;; Pay attention, :flatten removes nils:
POFTHEDAY> (parseq:parseq 'time
                          "10:31 +0300")
(10 31 "+0300")
T

Now, knowing how rules are combined and data is transformed, you will be able to read rest rules yourself:

POFTHEDAY> (parseq:defrule day ()
               (and (? FWS)
                    (rep (1 2) digit)
                    FWS)
             (:choose 1)
             (:string)
             (:function #'parse-integer))

POFTHEDAY> (parseq:defrule year ()
               (and FWS
                    (rep 4 digit)
                    FWS)
             (:choose 1)
             (:string)
             (:function #'parse-integer))

POFTHEDAY> (parseq:defrule date ()
               (and day month year))

(parseq:defrule date-time ()
    (and (? (and day-of-week ","))
         date
         time)
  (:choose '(0 0) 1 2)
  (:flatten))

Another cool Parseq's feature is an ability to debug parser execution. Now I'll turn on this debug mode and parse a string:

POFTHEDAY> (parseq:trace-rule 'date-time :recursive t)

POFTHEDAY> (parseq:parseq 'date-time
                          "Thu, 13 Jul 2017 13:28:03 +0200")
1: DATE-TIME 0?
 2: DAY-OF-WEEK 0?
 2: DAY-OF-WEEK 0-3 -> "Thu"
 2: DATE 4?
  3: DAY 4?
   4: FWS 4?
   4: FWS 4-5 -> #\ 
   4: FWS 7?
   4: FWS 7-8 -> #\ 
  3: DAY 4-8 -> 13
  3: MONTH 8?
  3: MONTH 8-11 -> "Jul"
  3: YEAR 11?
   4: FWS 11?
   4: FWS 11-12 -> #\ 
   4: FWS 16?
   4: FWS 16-17 -> #\ 
  3: YEAR 11-17 -> 2017
 2: DATE 4-17 -> (13 "Jul" 2017)
 2: TIME 17?
  3: TIME-OF-DAY 17?
   4: HOUR 17?
   4: HOUR 17-19 -> 13
   4: MINUTE 20?
   4: MINUTE 20-22 -> 28
   4: SECOND 23?
   4: SECOND 23-25 -> 3
  3: TIME-OF-DAY 17-25 -> (13 28 3)
  3: FWS 25?
  3: FWS 25-26 -> #\ 
  3: ZONE 26?
  3: ZONE 26-31 -> "+0200"
 2: TIME 17-31 -> (13 28 3 "+0200")
1: DATE-TIME 0-31 -> ("Thu" 13 "Jul" 2017 13 28 3 "+0200")

("Thu" 13 "Jul" 2017 13 28 3 "+0200")
T

We can improve this parser by using :function transformation to return a local-time:timestamp. First, let's redefine rule for matching the month and make it return the month number:

POFTHEDAY> (parseq:defrule january  () "Jan" (:constant 1))
POFTHEDAY> (parseq:defrule february () "Feb" (:constant 2))
POFTHEDAY> (parseq:defrule march    () "Mar" (:constant 3))
POFTHEDAY> (parseq:defrule april    () "Apr" (:constant 4))
POFTHEDAY> (parseq:defrule may      () "May" (:constant 5))
POFTHEDAY> (parseq:defrule june     () "Jun" (:constant 6))
POFTHEDAY> (parseq:defrule july     () "Jul" (:constant 7))
POFTHEDAY> (parseq:defrule august   () "Aug" (:constant 8))
POFTHEDAY> (parseq:defrule september () "Sep" (:constant 9))
POFTHEDAY> (parseq:defrule october  () "Oct" (:constant 10))
POFTHEDAY> (parseq:defrule november () "Nov" (:constant 11))
POFTHEDAY> (parseq:defrule december () "Dec" (:constant 12))

POFTHEDAY> (parseq:defrule month ()
               (or january february march april
                   may june july august
                   september october november december))

POFTHEDAY> (parseq:parseq 'month "Sep")
9 (4 bits, #x9, #o11, #b1001)
T

Next, we need to reimplement the rule matching a timezone to make it return local-time:timezone.

We'll be using an advanced technique of binding variables to pass value from one rule to another, because I want to store the timezone as a string and to parse it's hour and minute parts simultaneously.

To accomplish this task, we have to divide or timezone matching rule into two. The first rule will match it as a string of sign and four digits. Then it will save the result into an external variable and exit with a nil result to give a chance to execute the second rule:

POFTHEDAY> (parseq:defrule zone-as-str ()
               (and (or #\+ #\-)
                    (rep 4 digit))
             (:string)
             (:external zone-as-str)
             ;; Save the value into a variable:
             (:lambda (z)
               (setf zone-as-str z))
             ;; and just exit:
             (:test (z)
               (declare (ignore z))
               nil))

Now we'll redefine our zone rule to call zone-as-str first and then to parse the same text again, this time as hours and minutes. As the final step, it creates a local-time:timezone object:

POFTHEDAY> (parseq:defrule zone ()
               (or zone-as-str
                   (and (or #\+ #\-)
                        hour
                        minute))
             (:let zone-as-str)
             (:lambda (sign hour minute)
               (local-time::%make-simple-timezone
                zone-as-str
                zone-as-str
                ;; This is an offset in seconds:
                (+ (* (ecase sign
                        (#\+ 1)
                        (#\- -1))
                      hour
                      3600)
                   (* minute 60)))))

;; Here is the execution trace:
POFTHEDAY> (parseq:parseq 'zone
                          "+0300")
1: ZONE 0?
 2: ZONE-AS-STR 0?
 2: ZONE-AS-STR -|
 2: HOUR 1?
 2: HOUR 1-3 -> 3
 2: MINUTE 3?
 2: MINUTE 3-5 -> 0
1: ZONE 0-5 -> #<LOCAL-TIME::TIMEZONE +0300>
#<LOCAL-TIME::TIMEZONE +0300>
T

Now we need to redefine the original date-time rule, to create local-time:timestamp as the result:

POFTHEDAY> (parseq:parseq 'date-time
                          "Thu, 13 Jul 2017 13:28:03 +0200")
("Thu" 13 7 2017 13 28 3 #<LOCAL-TIME::TIMEZONE +0200>)
T

POFTHEDAY> (parseq:defrule date-time ()
               (and (? (and day-of-week ","))
                    date
                    time)
             (:choose '(1 2)                       ; year
                      '(1 1)                       ; month
                      '(1 0)                       ; day
                      '(2 0)                       ; hour
                      '(2 1)                       ; minute
                      '(2 2)                       ; second
                      '(2 3))                      ; timezone
             
             (:lambda (year month day hour minute second timezone)
               (local-time:encode-timestamp
                0             ; nanoseconds
                (or second 0) ; secs are optional
                minute
                hour
                day
                month
                year
                :timezone (or timezone
                              local-time:*default-timezone*))))

POFTHEDAY> (parseq:parseq 'date-time
                          "Thu, 13 Jul 2017 13:28:03 +0200")
@2017-07-13T14:28:03.000000+03:00
T

I've got a different value for the time because local-time prints timestamp in my timezone which is UTC+3.

The cool feature of the Parseq is its ability to work with any data, including binary. This way it can be used to parse binary formats.

As an example of parsing binary data, Parseq includes this parser rules for working with PNG image format:

https://github.com/mrossini-ethz/parseq/blob/master/examples/png.lisp

There are other interesting features. Please, read the docs to learn more.

If you are aware of other parsing libraries which worth to be written about, let me know in the comments.

23 Oct 2020 8:47pm GMT