02 Feb 2023
Planet Grep
Xavier Mertens: This Blog Has 20 Years!
Twenty years ago… I decided to start a blog to share my thoughts! That's why I called it "/dev/random". How was the Internet twenty years ago? Well, they were good things and bad ones…
With the years, the blog content evolved, and I wrote a lot of technical stuff related to my job, experiences, tools, etc. Then, I had the opportunity to attend a lot of security conferences and started to write wrap-ups. With COVID, fewer conferences and no more reviews. For the last few months, I'm mainly writing diaries for the Internet Storm Center therefore, I publish less private stuff here, and just relay the content published on the ISC website. If you have read my stuff for a long time (or even if you are a newcomer), thank you very much!
A few stats about the site:
- 2056 articles
- 20593 pictures
- 5538 unique visitors to the RSS feed in the last 30 days
- 85000 hits/day on average (bots & attacks included ?)
I know that these numbers might seem low for many of you but I'm proud of them!
The post This Blog Has 20 Years! appeared first on /dev/random.
02 Feb 2023 8:30pm GMT
Koen Vervloesem: How to stop brltty from claiming your USB UART interface on Linux
Today I wanted to program an ESP32 development board, the ESP-Pico-Kit v4, but when I connected it to my computer's USB port, the serial connection didn't appear in Linux. Suspecting a hardware issue, I tried another ESP32 board, the ESP32-DevKitC v4, but this didn't appear either, so then I tried another one, a NodeMCU ESP8266 board, which had the same problem. Time to investigate...
The dmesg
output looked suspicious:
[14965.786079] usb 1-1: new full-speed USB device number 5 using xhci_hcd [14965.939902] usb 1-1: New USB device found, idVendor=10c4, idProduct=ea60, bcdDevice= 1.00 [14965.939915] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [14965.939920] usb 1-1: Product: CP2102 USB to UART Bridge Controller [14965.939925] usb 1-1: Manufacturer: Silicon Labs [14965.939929] usb 1-1: SerialNumber: 0001 [14966.023629] usbcore: registered new interface driver usbserial_generic [14966.023646] usbserial: USB Serial support registered for generic [14966.026835] usbcore: registered new interface driver cp210x [14966.026849] usbserial: USB Serial support registered for cp210x [14966.026881] cp210x 1-1:1.0: cp210x converter detected [14966.031460] usb 1-1: cp210x converter now attached to ttyUSB0 [14966.090714] input: PC Speaker as /devices/platform/pcspkr/input/input18 [14966.613388] input: BRLTTY 6.4 Linux Screen Driver Keyboard as /devices/virtual/input/input19 [14966.752131] usb 1-1: usbfs: interface 0 claimed by cp210x while 'brltty' sets config #1 [14966.753382] cp210x ttyUSB0: cp210x converter now disconnected from ttyUSB0 [14966.754671] cp210x 1-1:1.0: device disconnected
So the ESP32 board, with a Silicon Labs, CP2102 USB to UART controller chip, was recognized, and it was attached to the /dev/ttyUSB0
device, as it should normally do. But then suddenly the brltty
command intervened and disconnected the serial device.
I looked up what brltty is doing, and apparently this is a system daemon that provides access to the console for a blind person using a braille display. When looking into the contents of the package on my Ubuntu 22.04 system (with dpkg -L brltty
), I saw a udev rules file, so I grepped for the product ID of my USB device in the file:
$ grep ea60 /lib/udev/rules.d/85-brltty.rules ENV{PRODUCT}=="10c4/ea60/*", ATTRS{manufacturer}=="Silicon Labs", ENV{BRLTTY_BRAILLE_DRIVER}="sk", GOTO="brltty_usb_run"
Looking at the context, this file shows:
# Device: 10C4:EA60 # Generic Identifier # Vendor: Cygnal Integrated Products, Inc. # Product: CP210x UART Bridge / myAVR mySmartUSB light # BrailleMemo [Pocket] # Seika [Braille Display] ENV{PRODUCT}=="10c4/ea60/*", ATTRS{manufacturer}=="Silicon Labs", ENV{BRLTTY_BRAILLE_DRIVER}="sk", GOTO="brltty_usb_run"
So apparently there's a Braille display with the same CP210x USB to UART controller as a lot of microcontroller development boards have. And because this udev rule claims the interface for the brltty daemon, UART communication with all these development boards isn't possible anymore.
As I'm not using these Braille displays, the fix for me was easy: just find the systemd unit that loads these rules, mask and stop it.
$ systemctl list-units | grep brltty brltty-udev.service loaded active running Braille Device Support $ sudo systemctl mask brltty-udev.service Created symlink /etc/systemd/system/brltty-udev.service → /dev/null. $ sudo systemctl stop brltty-udev.service
After this, I was able to use the serial interface again on all my development boards.
02 Feb 2023 8:30pm GMT
Frederic Descamps: MySQL 8.0.32: thank you for the contributions
The latest MySQL release has been published on January 17th, 2023. MySQL 8.0.32 contains some new features and bug fixes. As usual, it also contains contributions from our great MySQL Community.
I would like to thank all contributors on behalf of the entire Oracle MySQL team !
MySQL 8.0.32 contains patches from Facebook/Meta, Alexander Reinert, Luke Weber, Vilnis Termanis, Naoki Someya, Maxim Masiutin, Casa Zhang from Tencent, Jared Lundell, Zhe Huang, Rahul Malik from Percona, Andrey Turbanov, Dimitry Kudryavtsev, Marcelo Altmann from Percona, Sander van de Graaf, Kamil Holubicki from Percona, Laurynas Biveinis, Seongman Yang, Yamasaki Tadashi, Octavio Valle, Zhao Rong, Henning Pöttker, Gabrielle Gervasi and Nico Pay.
Here is the list of the above contributions and related bugs, we can see that for this release, our connectors got several contributions, always a good sign of their increasing popularity.
We can also notice the return of a major contributor: Laurynas Biveinis!
Connectors
Connector / NET
- #74392 - Support to use (Memory-)Stream for bulk loading data - Alexander Reinert
- #108837 - Fix unloading issues - Gabriele Gervasi
Connector / Python
- #81572 - Allow MySQLCursorPrepared.execute() to accept %(key_name)s in operations and dic - Luke Weber
- #81573 - Add MySQLCursorPreparedDict option - Luke Weber
- #82366 - Waning behaviour improvements - Vilnis Termanis
- #89345 - Reduce callproc roundtrip time - Vilnis Termanis
- #90862 - C extension - Fix multiple reference leaks - Vilnis Termanis
- #96280 - prepared cursor failed to fetch/decode result of varbinary columns - Naoki Someya
- #103488 - Stubs (.pyi) for type definition for connection and cursor objects - Maxim Masiutin
- #108076 - If extra init_command options are given for the Django connector, load them - Sander van de Graaf
- #108733 - python connector will return a date object when time is 00:00:00 - Zhao Rong
Connector / J
- #104954 - MysqlDataSource fails to URL encode database name when constructing JDBC URL - Jared Lundell
- #106252 - Connector/J client hangs after prepare & execute process with old version server - Zhe Huang
- #106981 - Remove superfluous use of boxing - Andrey Turbanov
- #108414 - Malformed packet generation for
COM_STMT_EXECUTE
- Seongman Yang - #108419 - Recognize "ON DUPLICATE KEY UPDATE" in "INSERT SET" Statement - Yamasaki Tadashi
- #108814 - Fix name of relocation POM file - Henning Pöttker
Connector / C++
- #108652 - Moving a local object in a return statement prevents copy elision - Octavio Valle
Clients & API
- C API (client library) Fix sha256_password_auth_client_nonblocking - Facebook
- #105761 - mysqldump make a non-consistent backup with -single-transaction option - Marcelo Altmann
- #108861 - Fix typo in dba.upgradeMetadata() error message (Shell AdminAPI) - Nico Pay
Replication
- Fix race between binlog sender heartbeat timeout - Facebook
InnoDB and Clone
- #106616 (private) - 8.0 upgrade (from 5.6) crashes with Assertion failure - Rahul Malik
- #107854 (private) - Assertion failure: dict0mem.h - Marcelo Altmann
- #108111 - Garbled UTF characters in SHOW ENGINE INNODB STATUS - Kamil Holubicki
- #108317 - clone_os_copy_file_to_buf partial read handling completely broken - Laurynas Biveinis
Optimizer
- #104934 (private) - Statement crash - Casa Zhang
- #107633 - Fixing a type-o, should be "truncate" - Dimitry Kudryavtsev
If you have patches and you also want to be part of the MySQL Contributors, it's easy, you can send Pull Requests from MySQL's GitHub repositories or send your patches on Bugs MySQL (signing the Oracle Contributor Agreement is required).
Thank you again to all our contributors !
02 Feb 2023 8:30pm GMT
Planet Debian
Matthew Garrett: Blocking free API access to Twitter doesn't stop abuse
In one week from now, Twitter will block free API access. This prevents anyone who has written interesting bot accounts, integrations, or tooling from accessing Twitter without paying for it. A whole number of fascinating accounts will cease functioning, people will no longer be able to use tools that interact with Twitter, and anyone using a free service to do things like find Twitter mutuals who have moved to Mastodon or to cross-post between Twitter and other services will be blocked.
There's a cynical interpretation to this, which is that despite firing 75% of the workforce Twitter is still not profitable and Elon is desperate to not have Twitter go bust and also not to have to tank even more of his Tesla stock to achieve that. But let's go with the less cynical interpretation, which is that API access to Twitter is something that enables bot accounts that make things worse for everyone. Except, well, why would a hostile bot account do that?
To interact with an API you generally need to present some sort of authentication token to the API to prove that you're allowed to access it. It's easy enough to restrict issuance of those tokens to people who pay for the service. But, uh, how do the apps work? They need to be able to communicate with the service to tell it to post tweets, retrieve them, and so on. And the simple answer to that is that they use some hardcoded authentication tokens. And while registering for an API token yourself identifies that you're not using an official client, using the tokens embedded in the clients makes it look like you are. If you want to make it look like you're a human, you're already using tokens ripped out of the official clients.
The Twitter client API keys are widely known. Anyone who's pretending to be a human is using those already and will be unaffected by the shutdown of the free API tier. Services like movetodon.org do get blocked. This isn't an anti-abuse choice. It's one that makes it harder to move to other services. It's one that blocks a bunch of the integrations and accounts that bring value to the platform. It's one that hurts people who follow the rules, without hurting the ones who don't. This isn't an anti-abuse choice, it's about trying to consolidate control of the platform.
comments
02 Feb 2023 10:21am GMT
John Goerzen: Using Yggdrasil As an Automatic Mesh Fabric to Connect All Your Docker Containers, VMs, and Servers
Sometimes you might want to run Docker containers on more than one host. Maybe you want to run some at one hosting facility, some at another, and so forth.
Maybe you'd like run VMs at various places, and let them talk to Docker containers and bare metal servers wherever they are.
And maybe you'd like to be able to easily migrate any of these from one provider to another.
There are all sorts of very complicated ways to set all this stuff up. But there's also a simple one: Yggdrasil.
My blog post Make the Internet Yours Again With an Instant Mesh Network explains some of the possibilities of Yggdrasil in general terms. Here I want to show you how to use Yggdrasil to solve some of these issues more specifically. Because Yggdrasil is always Encrypted, some of the security lifting is done for us.
Background
Often in Docker, we connect multiple containers to a single network that runs on a given host. That much is easy. Once you start talking about containers on multiple hosts, then you start adding layers and layers of complexity. Once you start talking multiple providers, maybe multiple continents, then the complexity can increase. And, if you want to integrate everything from bare metal servers to VMs into this - well, there are ways, but they're not easy.
I'm a believer in the KISS principle. Let's not make things complex when we don't have to.
Enter Yggdrasil
As I've explained before, Yggdrasil can automatically form a global mesh network. This is pretty cool! As most people use it, they join it to the main Yggdrasil network. But Yggdrasil can be run entirely privately as well. You can run your own private mesh, and that's what we'll talk about here.
All we have to do is run Yggdrasil inside each container, VM, server, or whatever. We handle some basics of connectivity, and bam! Everything is host- and location-agnostic.
Setup in Docker
The installation of Yggdrasil on a regular system is pretty straightforward. Docker is a bit more complicated for several reasons:
- It blocks IPv6 inside containers by default
- The default set of permissions doesn't permit you to set up tunnels inside a container
- It doesn't typically pass multicast (broadcast) packets
Normally, Yggdrasil could auto-discover peers on a LAN interface. However, aside from some esoteric Docker networking approaches, Docker doesn't permit that. So my approach is going to be setting up one or more Yggdrasil "router" containers on a given Docker host. All the other containers talk directly to the "router" container and it's all good.
Basic installation
In my Dockerfile, I have something like this:
FROM jgoerzen/debian-base-security:bullseye
RUN echo "deb http://deb.debian.org/debian bullseye-backports main" >> /etc/apt/sources.list && \
apt-get --allow-releaseinfo-change update && \
apt-get -y --no-install-recommends -t bullseye-backports install yggdrasil
...
COPY yggdrasil.conf /etc/yggdrasil/
RUN set -x; \
chown root:yggdrasil /etc/yggdrasil/yggdrasil.conf && \
chmod 0750 /etc/yggdrasil/yggdrasil.conf && \
systemctl enable yggdrasil
The magic parameters to docker run
to make Yggdrasil work are:
--cap-add=NET_ADMIN --sysctl net.ipv6.conf.all.disable_ipv6=0 --device=/dev/net/tun:/dev/net/tun
This example uses my docker-debian-base images, so if you use them as well, you'll also need to add their parameters.
Note that it is NOT necessary to use --privileged
. In fact, due to the network namespaces in use in Docker, this command does not let the container modify the host's networking (unless you use --net=host
, which I do not recommend).
The --sysctl
parameter was the result of a lot of banging my head against the wall. Apparently Docker tries to disable IPv6 in the container by default. Annoying.
Configuration of the router container(s)
The idea is that the router node (or more than one, if you want redundancy) will be the only ones to have an open incoming port. Although the normal Yggdrasil case of directly detecting peers in a broadcast domain is more convenient and more robust, this can work pretty well too.
You can, of course, generate a template yggdrasil.conf
with yggdrasil -genconf
like usual. Some things to note for this one:
- You'll want to change
Listen
to something likeListen: ["tls://[::]:12345"]
where 12345 is the port number you'll be listening on. - You'll want to disable the
MulticastInterfaces
entirely by just setting it to[]
since it doesn't work anyway. - If you expose the port to the Internet, you'll certainly want to firewall it to only authorized peers. Setting
AllowedPublicKeys
is another useful step. - If you have more than one router container on a host, each of them will both
Listen
and act as a client to the others. See below.
Configuration of the non-router nodes
Again, you can start with a simple configuration. Some notes here:
- You'll want to set
Peers
to something likePeers: ["tls://routernode:12345"]
whererouternode
is the Docker hostname of the router container, and 12345 is its port number as defined above. If you have more than one local router container, you can simply list them all here. Yggdrasil will then fail over nicely if any one of them go down. Listen
should be empty.- As above,
MulticastInterfaces
should be empty.
Using the interfaces
At this point, you should be able to ping6
between your containers. If you have multiple hosts running Docker, you can simply set up the router nodes on each to connect to each other. Now you have direct, secure, container-to-container communication that is host-agnostic! You can also set up Yggdrasil on a bare metal server or VM using standard procedures and everything will just talk nicely!
Security notes
Yggdrasil's mesh is aggressively greedy. It will peer with any node it can find (unless told otherwise) and will find a route to anywhere it can. There are two main ways to make sure your internal comms stay private: by restricting who can talk to your mesh, and by firewalling the Yggdrasil interface. Both can be used, and they can be used simultaneously.
By disabling multicast discovery, you eliminate the chance for random machines on the LAN to join the mesh. By making sure that you firewall off (outside of Yggdrasil) who can connect to a Yggdrasil node with a listening port, you can authorize only your own machines. And, by setting AllowedPublicKeys
on the nodes with listening ports, you can authenticate the Yggdrasil peers. Note that part of the benefit of the Yggdrasil mesh is normally that you don't have to propagate a configuration change to every participatory node - that's a nice thing in general!
You can also run a firewall inside your container (I like firehol
for this purpose) and aggressively firewall the IPs that are allowed to connect via the Yggdrasil interface. I like to set a stable interface name like ygg0
in yggdrasil.conf
, and then it becomes pretty easy to firewall the services. The Docker parameters that allow Yggdrasil to run are also sufficient to run firehol.
Naming Yggdrasil peers
You probably don't want to hard-code Yggdrasil IPs all over the place. There are a few solutions:
- You could run an internal DNS service
- You can do a bit of scripting around Docker's
--add-host
command to add things to /etc/hosts
Other hints & conclusion
Here are some other helpful use cases:
- If you are migrating between hosts, you could leave your reverse proxy up at both hosts, both pointing to the target containers over Yggdrasil. The targets will be automatically found from both sides of the migration while you wait for DNS caches to update and such.
- This can make services integrate with local networks a lot more painlessly than they might otherwise.
This is just an idea. The point of Yggdrasil is expanding our ideas of what we can do with a network, so here's one such expansion. Have fun!
Note: This post also has a permanent home on my webiste, where it may be periodically updated.
02 Feb 2023 4:18am GMT
Dirk Eddelbuettel: RInside 0.2.18 on CRAN: Maintenance
A new release 0.2.18 of RInside arrived on CRAN and in Debian today. This is the first release in ten months since the 0.2.17 release. RInside provides a set of convenience classes which facilitate embedding of R inside of C++ applications and programs, using the classes and functions provided by Rcpp.
This release brings a contributed change to how the internal REPL is called: Dominick found the current form more reliable when embedding R on Windows. We also updated a few other things around the package.
The list of changes since the last release:
Changes in RInside version 0.2.18 (2023-02-01)
The random number initialization was updated as in R.
The main REPL is now running via 'run_Rmainloop()'.
Small routine update to package and continuous integration.
My CRANberries also provides a short report with changes from the previous release. More information is on the RInside page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page, or to issues tickets at the GitHub repo.
If you like this or other open-source work I do, you can now sponsor me at GitHub.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
02 Feb 2023 12:17am GMT
23 Jan 2023
Planet Lisp
Nicolas Martyanoff: Custom Common Lisp indentation in Emacs
While SLIME is most of the time able to indent Common Lisp correctly, it will sometimes trip on custom forms. Let us see how we can customize indentation.
In the process of writing my PostgreSQL client in Common Lisp, I wrote a READ-MESSAGE-CASE
macro which reads a message from a stream and execute code depending on the type of the message:
(defmacro read-message-case ((message stream) &rest forms)
`(let ((,message (read-message ,stream)))
(case (car ,message)
(:error-response
(backend-error (cdr ,message)))
(:notice-response
nil)
,@forms
(t
(error 'unexpected-message :message ,message)))))
This macro is quite useful: all message loops can use it to automatically handle error responses, notices, and signal unexpected messages.
But SLIME does not know how to indent READ-MESSAGE-CASE
, so by default it will align all message forms on the first argument:
(read-message-case (message stream)
(:authentication-ok
(return))
(:authentication-cleartext-password
(unless password
(error 'missing-password))
(write-password-message password stream)))
While we want it aligned the same way as HANDLER-CASE
:
(read-message-case (message stream)
(:authentication-ok
(return))
(:authentication-cleartext-password
(unless password
(error 'missing-password))
(write-password-message password stream)))
Good news, SLIME indentation is defined as a list of rules. Each rule associates an indentation specification (a S-expression describing how to indent the form) to a symbol and store it as the common-lisp-indent-function
property of the symbol.
You can obtain the indentation rule of a Common Lisp symbol easily. For example, executing (get 'defun 'common-lisp-indent-function)
(e.g. in IELM or with eval-expression
) yields (4 &lambda &body)
. This indicates that DEFUN
forms are to be indented as follows:
- The first argument of
DEFUN
(the function name) is indented by four spaces. - The second argument (the list of function arguments) is indented as a lambda list.
- The rest of the arguments are indented based on the
lisp-body-indent
custom variable, which controls the indentation of the body of a lambda form (two spaces by default).
You can refer to the documentation of the common-lisp-indent-function
Emacs function (defined in SLIME of course) for a complete description of the format.
We want READ-MESSAGE-CASE
to be indented the same way as HANDLER-CASE
, whose indentation specification is (4 &rest (&whole 2 &lambda &body))
(in short, an argument and a list of lambda lists). Fortunately there is a way to specify that a form must be indented the same way as another form, using (as <symbol>)
.
Let us first define a function to set the indentation specification of a symbol:
(defun g-common-lisp-indent (symbol indent)
"Set the indentation of SYMBOL to INDENT."
(put symbol 'common-lisp-indent-function indent))
Then use it for READ-MESSAGE-CASE
:
(g-common-lisp-indent 'read-message-case '(as handler-case))
While it is in general best to avoid custom indentation, exceptions are sometimes necessary for readability. And SLIME makes it easy.
23 Jan 2023 6:00pm GMT
18 Jan 2023
Planet Lisp
TurtleWare: Method Combinations
Table of Contents
- Introduction
- Defining method combinations - the short form
- Defining method combinations - the long form
- Conclusions
Update [2023-01-23]
Christophe Rhodes pointed out that "The Hooker" method combination is not conforming because there are multiple methods with the same "role" that can't be ordered and that have different qualifiers:
Note that two methods with identical specializers, but with different qualifiers, are not ordered by the algorithm described in Step 2 of the method selection and combination process described in Section 7.6.6 (Method Selection and Combination). Normally the two methods play different roles in the effective method because they have different qualifiers, and no matter how they are ordered in the result of Step 2, the effective method is the same. If the two methods play the same role and their order matters, an error is signaled. This happens as part of the qualifier pattern matching in define-method-combination.
http://www.lispworks.com/documentation/HyperSpec/Body/m_defi_4.htm
So instead of using qualifier patterns we should use qualifier predicates. They are not a subject of the above paragraph because of its last sentence (there is also an example in the spec that has multiple methods with a predicate). So instead of
(define-method-combination hooker ()
(... (hook-before (:before*)) ...) ...)
the method combination should use:
(defun hook-before-p (method-qualifier)
(typep method-qualifier '(cons (eql :before) (cons t null))))
(define-method-combination hooker ()
(... (hook-before hook-before-p) ...) ...)
and other "hook" groups should also use predicates.
Another thing worth mentioning is that both ECL and SBCL addressed issues with the qualifier pattern matching and :arguments since the publication of this blog post.
Introduction
Method combinations are used to compute the effective method for a generic function. An effective method is a body of the generic function that combines a set of applicable methods computed based on the invocation arguments.
For example we may have a function responsible for reporting the object status and each method focuses on a different aspect of the object. In that case we may want to append all results into a list:
(defgeneric status (object)
(:method-combination append))
(defclass base-car ()
((engine-status :initarg :engine :accessor engine-status)
(wheels-status :initarg :wheels :accessor wheels-status)
(fuel-level :initarg :fuel :accessor fuel-level))
(:default-initargs :engine 'ok :wheels 'ok :fuel 'full))
(defmethod status append ((object base-car))
(list :engine (engine-status object)
:wheels (wheels-status object)
:fuel (fuel-level object)))
(defclass premium-car (base-car)
((gps-status :initarg :gps :accessor gps-status)
(nitro-level :initarg :nitro :accessor nitro-level))
(:default-initargs :gps 'no-signal :nitro 'low))
(defmethod status append ((object premium-car))
(list :gps (gps-status object)
:nitro (nitro-level object)))
CL-USER> (status (make-instance 'premium-car))
(:GPS NO-SIGNAL :NITRO LOW :ENGINE OK :WHEELS OK :FUEL FULL)
CL-USER> (status (make-instance 'base-car))
(:ENGINE OK :WHEELS OK :FUEL FULL)
The effective method may look like this:
(append (call-method #<method status-for-premium-car>)
(call-method #<method status-for-base-car> ))
Note that append
is a function so all methods are called. It is possible to use other operators (for example a macro and
) and then the invocation of particular methods may be conditional:
(and (call-method #<method can-repair-p-for-premium-car>)
(call-method #<method can-repair-p-for-base-car> ))
Defining method combinations - the short form
The short form allows us to define a method combination in the spirit of the previous example:
(OPERATOR (call-method #<m1>)
(call-method #<m2>)
...)
For example we may want to return as the second value the count of odd numbers:
(defun sum-and-count-odd (&rest args)
(values (reduce #'+ args)
(count-if #'oddp args)))
(define-method-combination sum-and-count-odd)
(defclass a () ())
(defclass b (a) ())
(defclass c (b) ())
(defgeneric num (o)
(:method-combination sum-and-count-odd)
(:method sum-and-count-odd ((o a)) 1)
(:method sum-and-count-odd ((o b)) 2)
(:method sum-and-count-odd ((o c)) 3)
(:method :around ((o c))
(print "haa!")
(call-next-method)))
(num (make-instance 'b)) ;; (values 3 1)
(num (make-instance 'c)) ;; (values 6 2)
Note that the short form supports also around methods. It is also important to note that effective methods are cached, that is unless the generic function or the method combination changes, the computation of the effective method may be called only once per the set of effective methods.
Admittedly these examples are not very useful. Usually we operate on data stored in instances and this is not a good abstraction to achieve that. Method combinations are useful to control method invocations and their results. Here is another example:
(defmacro majority-vote (&rest method-calls)
(let* ((num-methods (length method-calls))
(tie-methods (/ num-methods 2)))
`(prog ((yes 0) (no 0))
,@(loop for invocation in method-calls
append `((if ,invocation
(incf yes)
(incf no))
(cond
((> yes ,tie-methods)
(return (values t yes no)))
((> no ,tie-methods)
(return (values nil yes no))))))
(error "we have a tie! ~d ~d" yes no))))
(define-method-combination majority-vote)
(defclass a () ())
(defclass b (a) ())
(defclass c (b) ())
(defclass d (c) ())
(defgeneric foo (object param)
(:method-combination majority-vote)
(:method majority-vote ((o a) param) nil)
(:method majority-vote ((o b) param) t)
(:method majority-vote ((o c) param) t)
(:method majority-vote ((o d) param) nil))
(foo (make-instance 'a) :whatever) ; (values nil 0 1)
(foo (make-instance 'b) :whatever) ; #<error tie 1 1>
(foo (make-instance 'c) :whatever) ; (values t 2 0)
(foo (make-instance 'd) :whatever) ; #<error tie 2 2>
Defining method combinations - the long form
The long form is much more interesting. It allows us to specify numerous qualifiers and handle methods without any qualifiers at all.
The Hooker
Here we will define a method combination that allows us to define named hooks that are invoked before or after the method. It is possible to have any number of hooks for the same set of arguments (something we can't achieve with the standard :before
and :after
auxiliary methods):
(defun combine-auxiliary-methods (primary around before after)
(labels ((call-primary ()
`(call-method ,(first primary) ,(rest primary)))
(call-methods (methods)
(mapcar (lambda (method)
`(call-method ,method))
methods))
(wrap-after (the-form)
(if after
`(multiple-value-prog1 ,the-form
,@(call-methods after))
the-form))
(wrap-before (the-form)
(if before
`(progn
,@(call-methods before)
,the-form)
the-form))
(wrap-around (the-form)
(if around
`(call-method ,(first around)
(,@(rest around)
(make-method ,the-form)))
the-form)))
(wrap-around (wrap-after (wrap-before (call-primary))))))
(define-method-combination hooker ()
((normal-before (:before))
(normal-after (:after)
:order :most-specific-last)
(normal-around (:around))
(hook-before (:before *))
(hook-after (:after *)
:order :most-specific-last)
(hook-around (:around *))
(primary () :required t))
(let ((around (append hook-around normal-around))
(before (append hook-before normal-before))
(after (append normal-after hook-after)))
(combine-auxiliary-methods primary around before after)))
With this we may define a generic function and associated methods similar to other functions with an extra feature - we may provide named :before
, :after
and :around
methods. Named auxiliary methods take a precedence over unnamed ones. Only after that the specialization is considered. There is one caveat - PCL
-derived CLOS
implementations (clasp
, cmucl
, ecl
, sbcl
) currently ( ) have a bug preventing wildcard qualifier pattern symbol *
from working. So better download ccl
or wait for fixes. Here's an example for using it:
;;; The protocol.
(defgeneric note-buffer-dimensions-changed (buffer w h)
(:method (b w h)
(declare (ignore b w h))
nil))
(defgeneric change-dimensions (buffer w h)
(:method-combination hooker))
;;; The implementation of unspecialized methods.
(defmethod change-dimensions :after (buffer w h)
(note-buffer-dimensions-changed buffer w h))
;;; The stanard class.
(defclass buffer ()
((w :initform 0 :accessor w)
(h :initform 0 :accessor h)))
;;; The implementation for the standard class.
(defmethod change-dimensions ((buffer buffer) w h)
(print "... Changing the buffer size ...")
(setf (values (w buffer) (h buffer))
(values w h)))
(defmethod note-buffer-dimensions-changed ((buffer buffer) w h)
(declare (ignore buffer w h))
(print "... Resizing the viewport ..."))
;;; Some dubious-quality third-party code that doesn't want to interfere with
;;; methods defined by the implementation.
(defmethod change-dimensions :after system (buffer w h)
(print `(log :something-changed ,buffer ,w ,h)))
(defmethod change-dimensions :after my-hook ((buffer buffer) w h)
(print `(send-email! :me ,buffer ,w ,h)))
CL-USER> (defvar *buffer* (make-instance 'buffer))
*BUFFER*
CL-USER> (change-dimensions *buffer* 10 30)
"... Changing the buffer size ..."
"... Resizing the viewport ..."
(LOG :SOMETHING-CHANGED #<BUFFER #x30200088220D> 10 30)
(SEND-EMAIL! :ME #<BUFFER #x30200088220D> 10 30)
10
30
The Memoizer
Another example (this time it will work on all implementations) is optional memoization of the function invocation. If we define a method with the qualifier :memoize
then the result will be cached depending on arguments. The method combination allows also "normal" auxiliary functions by reusing the function combine-auxiliary-methods
from the previous section.
The function ensure-memoized-result
accepts the following arguments:
test
: compare generationsmemo
: a form that returns the current generationcache-key
: a list composed of a generic function and its argumentsform
: a form implementing the method to be called
When the current generation is nil
that means that caching is disabled and we remove the result from the cache. Otherwise we use the test
to compare the generation of a cached value and the current one - if they are the same, then the cached value is returned. Otherwise it is returned.
(defparameter *memo* (make-hash-table :test #'equal))
(defun ensure-memoized-result (test memo cache-key form)
`(let ((new-generation ,memo))
(if (null new-generation)
(progn
(remhash ,cache-key *memo*)
,form)
(destructuring-bind (old-generation . cached-result)
(gethash ,cache-key *memo* '(nil))
(apply #'values
(if (,test old-generation new-generation)
cached-result
(rest
(setf (gethash ,cache-key *memo*)
(list* new-generation (multiple-value-list ,form))))))))))
The method with the qualifier :memoize
is used to compute the current generation key. When there is no such method then the function behaves as if the standard method combination is used. The method combination accepts a single argument test, so it is possible to define different predicates for deciding whether the cache is up-to-date or not.
(define-method-combination memoizer (test)
((before (:before))
(after (:after) :order :most-specific-last)
(around (:around))
(memoize (:memoize))
(primary () :required t))
(:arguments &whole args)
(:generic-function function)
(let ((form (combine-auxiliary-methods primary around before after))
(memo `(call-method ,(first memoize) ,(rest memoize)))
(ckey `(list* ,function ,args)))
(if memoize
(ensure-memoized-result test memo ckey form)
form)))
Now let's define a function with "our" method combination. We will use a counter to verify that values are indeed cached.
(defparameter *counter* 0)
(defgeneric test-function (arg &optional opt)
(:method-combination memoizer eql))
(defmethod test-function ((arg integer) &optional opt)
(list* `(:counter ,(incf *counter*)) arg opt))
CL-USER> (test-function 42)
((:COUNTER 1) 42)
CL-USER> (test-function 42)
((:COUNTER 2) 42)
CL-USER> (defmethod test-function :memoize ((arg integer) &optional (cache t))
(and cache :gen-z))
#<STANDARD-METHOD TEST-FUNCTION :MEMOIZE (INTEGER)>
CL-USER> (test-function 42)
((:COUNTER 3) 42)
CL-USER> (test-function 42)
((:COUNTER 3) 42)
CL-USER> (test-function 42 nil)
((:COUNTER 4) 42)
CL-USER> (test-function 42)
((:COUNTER 3) 42)
CL-USER> (test-function 43)
((:COUNTER 5) 43)
CL-USER> (test-function 43)
((:COUNTER 5) 43)
CL-USER> (defmethod test-function :memoize ((arg (eql 43)) &optional (cache t))
(and cache :gen-x))
#<STANDARD-METHOD TEST-FUNCTION :MEMOIZE ((EQL 43))>
CL-USER> (test-function 43)
((:COUNTER 6) 43)
CL-USER> (test-function 43)
((:COUNTER 6) 43)
CL-USER> (test-function 42)
((:COUNTER 3) 42)
Conclusions
Method combinations are a feature that is often overlooked but give a great deal of control over the generic function invocation. The fact that ccl
is the only implementation from a few that I've tried which got method combinations "right" doesn't surprise me - I've always had an impression that it shines in many unexpected places.
18 Jan 2023 12:00am GMT
16 Jan 2023
Planet Lisp
Nicolas Martyanoff: ANSI color rendering in SLIME
I was working on the terminal output for a Common Lisp logger, and I realized that SLIME does not interpret ANSI escape sequences.
This is not the end of the world, but having at least colors would be nice. Fortunately there is a library to do just that.
First let us install the package, here using use-package
and straight.el
.
(use-package slime-repl-ansi-color
:straight t)
While in theory we are supposed to just add slime-repl-ansi-color
to slime-contribs
, it did not work for me, and I add to enable the minor mode manually.
If you already have a SLIME REPL hook, simply add (slime-repl-ansi-color-mode 1)
. If not, write an initialization function, and add it to the SLIME REPL initialization hook:
(defun g-init-slime-repl-mode ()
(slime-repl-ansi-color-mode 1))
(add-hook 'slime-repl-mode-hook 'g-init-slime-repl-mode)
To test that it works as intended, fire up SLIME and print a simple message using ANSI escape sequences:
(let ((escape (code-char 27)))
(format t "~C[1;33mHello world!~C[0m~%" escape escape))
While it is tempting to use the #\Esc
character, it is part of the Common Lisp standard; therefore we use CODE-CHAR
to obtain it from its ASCII numeric value. We use two escape sequences, the first one to set the bold flag and foreground color, and the second one to reset display status.
If everything works well, should you see a nice bold yellow message:
16 Jan 2023 6:00pm GMT
15 Jan 2023
FOSDEM 2023
Call for volunteers
With FOSDEM just around the corner, it is time for us to enlist your help. Every year, an enthusiastic band of volunteers make FOSDEM happen and make it a fun and safe place for all our attendees. We could not do this without you. This year we again need as many hands as possible, with the buildup (starting Friday at noon), heralding during the conference and Cleanup (on Sunday evening). No need to worry about missing lunch. Food will be provided. Would you like to be part of the team that makes FOSDEM tick? Sign up here! You could舰
15 Jan 2023 11:00pm GMT
07 Dec 2022
FOSDEM 2023
Accepted stands for FOSDEM 2023
With great pleasure (and an apology for missing the deadline by seven days), we can announce the following projects will have a stand at FOSDEM 2023 (4 & 5th of February). This is the list of stands (in no particular order): Eclipse Foundation FOSSASIA Matrix.org Foundation Software Freedom Conservancy CentOS and RDO FreeBSD Project Free Software Foundation Europe Realtime Lounge Free Culture Podcasts Open Culture Foundation + COSCUP Open Toolchain Foundation Open UK and Book Signing Stand The Apache Software Foundation The Perl/Raku Foundation PostgreSQL GNOME KDE GitLab Homebrew Infobooth on amateur radio (hamradio) IsardVDI Jenkins Fluence La Contre-Voie舰
07 Dec 2022 11:00pm GMT
12 Nov 2022
FOSDEM 2023
Presentations - Call for Participation
We now invite proposals for presentations. FOSDEM offers open source and free software developers a place to meet, share ideas and collaborate. Renowned for being highly developer-oriented, the event brings together some 8000+ geeks from all over the world. The twenty-third edition will take place on Saturday 4th and Sunday 5th February 2023 at the usual location, ULB Campus Solbosch in Brussels. It will also be possible to participate online. Developer Rooms For more details about the Developer Rooms, please refer to the Calls for Papers that are being added to https://fosdem.org/2023/news/2022-11-07-accepted-developer-rooms/ when they are issued. Main Tracks Main舰
12 Nov 2022 11:00pm GMT