05 Feb 2025

feedPlanet Debian

Reproducible Builds: Reproducible Builds in January 2025

Welcome to the first report in 2025 from the Reproducible Builds project!

Our monthly reports outline what we've been up to over the past month and highlight items of news from elsewhere in the world of software supply-chain security when relevant. As usual, though, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website.

Table of contents:

  1. reproduce.debian.net
  2. Two new academic papers
  3. Distribution work
  4. On our mailing list…
  5. Upstream patches
  6. diffoscope
  7. Website updates
  8. Reproducibility testing framework

reproduce.debian.net

The last few months saw the introduction of reproduce.debian.net. Announced at the recent Debian MiniDebConf in Toulouse, reproduce.debian.net is an instance of rebuilderd operated by the Reproducible Builds project. Powering that is rebuilderd, our server designed monitor the official package repositories of Linux distributions and attempt to reproduce the observed results there.

This month, however, we are pleased to announce that in addition to the existing amd64.reproduce.debian.net and i386.reproduce.debian.net architecture-specific pages, we now build for a three more architectures (for a total of five) - arm64 armhf and riscv64.


Two new academic papers

Giacomo Benedetti, Oreofe Solarin, Courtney Miller, Greg Tystahl, William Enck, Christian Kästner, Alexandros Kapravelos, Alessio Merlo and Luca Verderame published an interesting article recently. Titled An Empirical Study on Reproducible Packaging in Open-Source Ecosystem, the abstract outlines its optimistic findings:

[We] identified that with relatively straightforward infrastructure configuration and patching of build tools, we can achieve very high rates of reproducible builds in all studied ecosystems. We conclude that if the ecosystems adopt our suggestions, the build process of published packages can be independently confirmed for nearly all packages without individual developer actions, and doing so will prevent significant future software supply chain attacks.

The entire PDF is available online to view.


In addition, Julien Malka, Stefano Zacchiroli and Théo Zimmermann of Télécom Paris' in-house research laboratory, the Information Processing and Communications Laboratory (LTCI) published an article asking the question: Does Functional Package Management Enable Reproducible Builds at Scale?.

Answering strongly in the affirmative, the article's abstract reads as follows:

In this work, we perform the first large-scale study of bitwise reproducibility, in the context of the Nix functional package manager, rebuilding 709,816 packages from historical snapshots of the nixpkgs repository[. We] obtain very high bitwise reproducibility rates, between 69 and 91% with an upward trend, and even higher rebuildability rates, over 99%. We investigate unreproducibility causes, showing that about 15% of failures are due to embedded build dates. We release a novel dataset with all build statuses, logs, as well as full diffoscopes: recursive diffs of where unreproducible build artifacts differ.

As above, the entire PDF of the article is available to view online.


Distribution work

There as been the usual work in various distributions this month, such as:


On our mailing list…

On our mailing list this month:


Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:


diffoscope

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 285, 286 and 287 to Debian:

In addition, fridtjof added support for the ASAR .tar-like archive format. [][][][] and lastly, Vagrant Cascadian updated diffoscope in GNU Guix to version 285 [][] and 286 [][].


strip-nondeterminism is our sister tool to remove specific non-deterministic results from a completed build. This month version 1.14.1-1 was uploaded to Debian unstable by Chris Lamb, making the following the changes:


Website updates

There were a large number of improvements made to our website this month, including:


Reproducibility testing framework

The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In January, a number of changes were made by Holger Levsen, including:

In addition:

Lastly, both Holger Levsen [] and Vagrant Cascadian [] performed some node maintenance.


If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

05 Feb 2025 11:49am GMT

04 Feb 2025

feedPlanet Debian

Dominique Dumont: Azure API throttling strikes back

Hi

In my last blog, I explained how we resolved a throttling issue involving Azure storage API. In the end, I mentioned that I was not sure of the root cause of the throttling issue.

Even though we no longer had any problem in dev and preprod cluster, we still faced throttling issue with prod. The main difference between these 2 environments is that we have about 80 PVs in prod versus 15 in the other environments. Given that we manage 1500 pods in prod, 80 PVs does not look like a lot. 🤨

To continue the investigation, I've modified k8s-scheduled-volume-snapshotter to limit the number of snaphots done in a single cron run (see add maxSnapshotCount parameter pull request).

In prod, we used the modified snapshotter to trigger snapshots one by one.

Even with all previous snapshots cleaned up, we could not trigger a single new snapshot without being throttled🕳. I guess that, in the cron job, just checking the list of PV to snapshot was enough to exhaust our API quota. 😒

Azure doc mention that a leaky bucket algorithm is used for throttling. A full bucket holds tokens for 250 API calls, and the bucket gets 25 new tokens per second. Looks like that not enough.🐌

I was puzzled 😵‍💫 and out of ideas 😶.

I looked for similar problems in AKS issues on GitHub where I found this comment that recommend using useDataPlaneAPI parameter in the CSI file driver. That was it! 😃

I was flabbergasted 🤯 by this parameter: why is CSI file driver able to use 2 APIs ? Why is one on them so limited ? And more importantly, why is the limited API the default one ?

Anyway, setting useDataPlaneAPI: "true" in our VolumeSnapshotClass manifest was the right solution. This indeed solved the throttling issue in our prod cluster. ⚕

But not the snaphot issue 😑. Amongst the 80 PV, I still had 2 snaphots failing.🦗

Fortunately, the error was mentioned in the description of the failed snapshots: we had too many (200) snapshots for these shared volumes.

What ?? 😤 All these snapshots were cleaned up last week.

I then tried to delete these snaphots through azure console. But the console failed to delete these snapshot due to API throttling. Looks like Azure console is not using the right API. 🤡

Anyway, I went back to the solution explained in my previous blog, I listed all snapshots with az command. I indeed has a lot of snaphots, a lot of them dated Jan 19 and 20. There was often a new bogus snaphot created every minute.

These were created during the first attempt at fixing the throttling issue. I guess that even though CSI file driver was throttled, a snaphot was still created in the storage account, but the CSI driver did not see it and retried a minute later💥. What a mess.

Anyway, I've cleaned up again these bogus snapshot 🧨, and now, snaphot creation is working fine 🤸🏻‍♂️.

For now.

All the best.

04 Feb 2025 1:23pm GMT

Paul Wise: FLOSS Activities January 2025

Focus

This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes

Sponsors

All work was done on a volunteer basis.

04 Feb 2025 2:43am GMT

31 Jan 2025

feedFOSDEM 2025

FOSDEM Treasure Hunt Update – Signs Stolen, But the Hunt Continues!

Treasure hunters, we have an update! Unfortunately, some of our signs have been removed or stolen, but don't worry-the hunt is still on! To ensure everyone can continue, we will be posting all signs online so you can still access the riddles and keep progressing. However, there is one exception: the 4th riddle must still be heard in person at Building H, as it includes an important radio message. Keep your eyes on our updates, stay determined, and don't let a few missing signs stop you from cracking the code! Good luck, and see you at Infodesk K with舰

31 Jan 2025 11:00pm GMT

30 Jan 2025

feedPlanet Lisp

Neil Munro: Ningle Tutorial 3: Static Files Middleware

Contents

Introduction

Welcome back to this tutorial series, in this chapter we will be looking at the topic of static files, before we begin, we need to come to an understanding on just what static files are. Static files are files that do not need to be further processed by your application; images, videos, css, JavaScript files all generally do not need to be processed by your web applications, and thus can simply be served as is. Files that must be processed by your web application (like the templates from the previous tutorial) typically need further processing by your web application and thus are not static files, and could be called dynamic files (although this term isn't really used).

While developing an application there's often a requirement to de-couple these static files from the main application code, you might want to serve these separately in production and many web servers help you do this for performance reasons (in fact NGINX is very good at this), however you might not need the scalability locally while you are working on your application, and so Ningle has a middleware module to serve these static files when you don't need a dedicated static file server.

Another, more practical consideration of serving static files is that if you don't have a way to serve these files for you, you would have to write a controller for every image, or css file in your application, this wont scale well at all, and you'll spend most of the time writing code to serve files rather than building your application. Static file management allows you to serve a directory of files and reference them by path, but you must set it up correctly in the first place.

Note: We will be using djula, as we did before, however as of 2025-01-15 this has not yet been merged into quicklisp, you may need to clone the latest djula from github into your quicklisp/local-projects directory to gain access to the latest version needed for this tutorial.

Introducing Middleware

In reality Ningle deligates the responsibility of serving static files to the underlying lack package by way of the lack middleware. There are a number of different lack middleware modules available by default and throughout this tutorial we will look at most (if not all) of them.

In most web frameworks (Ningle included) middleware runs between the request being accepted and the code in your controller running. It is similar to a controller in that it has access to the request and response objects, and it may pass its response onto the next middleware function or a controller, it depends on what the middleware function is written to do.

In the case of static files here, the end goal will be that a request for a file will come to your webserver, and the static middleware module will run before any controllers, and if the static resource is found, the middleware function will return a response and with our not-found method, if the url couldn't be found, our not-found method runs instead.

Simple Middleware Example

To illustrate how this works in practice, we will write a piece of custom middleware that will add a new variable to the request environment, which we will then extract in a controller and display in a template, we'll use a number that gets incremented each time the middleware is run. In effect we will implement a hit counter in middleware!

Please note: This will not actually be used in the tutorial overall and serves only as a guide for how to write custom middleware generally, please follow this section to complete your understanding and feel free to include it (if you wish), but it will not be featured in the accompanying github code or used anywhere else in this tutorial.

In our main application code we define an app objects, under this we will define a new variable to track our count.

(defvar *app* (make-instance 'ningle:app))
(defvar *count* 0)

Now in order to take advantage of using middleware we must restructure how we built the ningle app, you may recall writing a start function that looked like the following.

(defun start (&key (server :woo) (address "127.0.0.1") (port 8000))
    (clack:clackup
     *app*
     :server server
     :address address
     :port port))

We will need to edit this and introduce the idea of a lack builder. This is a way of building an application with more capabilities. Instead of simply passing our *app* object to the clackup function, we instead wrap our *app* object in the lack builder function which allows us to plug in middleware.

(clack:clackup
     (lack.builder:builder *app*)
     :server server
     :address address
     :port port)

It may not be immediately obvious, but where previously the first argument to clackup was our *app* object, we instead call lack.builder.builder passing in *app*, it is in this builder call that we will hook in our middleware. Before we can do that however, we must write some middleware!

Above our start function I will write our middleware function:

(defun hit-counter-middleware (app)
  (lambda (env)
    (setf (getf env :hit-counter) (incf *count*))
    (funcall app env)))

This is all it needs, we need to define a function that first accepts a ningle application object, and it returns a function (a lambda in this instance) that accepts the env (the request environment), because there may be a chain of middleware functions that potentially terminate with our controller, the lambda must return the result of calling the next middleware function with the app and environment.

Within the body of the lambda, however, we are free to begin doing whatever we want!

In this example, we only do one thing, we add a new entry into the environment and assign it to be the incremented (incf) value of *count* with this line (setf (getf env :hit-counter) (incf *count*)).

We next must edit the controller to retrieve this stored value and render it into the template (which means we'll also need to edit the template).

Thankfully editing our controller is easy, we need only add a new keyword argument to the render-template* function.

(setf (ningle:route *app* "/")
      (lambda (params)
        (let ((user  (list :username "NMunro"))
              (posts (list (list :author (list :username "Bob")  :content "Experimenting with Dylan" :created-at "2025-01-24 @ 13:34")
                           (list :author (list :username "Jane") :content "Wrote in my diary today" :created-at "2025-01-24 @ 13:23"))))
          (djula:render-template* "index.html" nil :title "Home"
                                                   :user user
                                                   :posts posts
                                                   :hit-counter (getf (lack/request:request-env ningle:*request*) :hit-counter)))))

The only addition is the :hit counter (getf (lack/request:request-env ningle:*request*) :hit-counter) line. This will retrieve the :hit-counter value from the request environment.

In our index.html template, in the div with the class="container", we will add the following:

    <div class="row">
        <div class="col-12">
            <h4>Hits</h4>
            <p>{{ hit-counter }}</p>
        </div>
    </div>

The last thing we must do is return to the lack.builder section of our start function and hook the middleware into the app.

(lack.builder:builder #'hit-counter-middleware *app*)

It must be included before *app* as the hit-counter-middleware will be wrapping our application and run before anything in our app does. As this tutorial (and your own applications) grow, this line and the different middleware modules will change as requirements do.

If you save and load the project, you should see that there is a div in your template that updates a count every time the page is refreshed. At this point you may notice that the counter is incremented by 2 each time, this is not a mistake, this is because your web browser will request the page itself, and a favicon.ico file (and hit the not-found controller).

For clarity here is the edited main.lisp file:

(defpackage ningle-tutorial-project
  (:use :cl)
  (:export #:start
           #:stop))

(in-package ningle-tutorial-project)

(defvar *app* (make-instance 'ningle:app))
(defvar *count* 0)

(setf (ningle:route *app* "/")
      (lambda (params)
        (let ((user  (list :username "NMunro"))
              (posts (list (list :author (list :username "Bob")  :content "Experimenting with Dylan" :created-at "2025-01-24 @ 13:34")
                           (list :author (list :username "Jane") :content "Wrote in my diary today" :created-at "2025-01-24 @ 13:23"))))
          (djula:render-template* "index.html" nil :title "Home"
                                                   :user user
                                                   :posts posts
                                                   :hit-counter (getf (lack/request:request-env ningle:*request*) :hit-counter)))))

(defmethod ningle:not-found ((app ningle:<app>))
    (declare (ignore app))
    (setf (lack.response:response-status ningle:*response*) 404)
    (djula:render-template* "error.html" nil :error "Not Found"))

(defun hit-counter-middleware (app)
  (lambda (env)
    (setf (getf env :hit-counter) (incf *count*))
    (funcall app env)))

(defun start (&key (server :woo) (address "127.0.0.1") (port 8000))
    (djula:add-template-directory (asdf:system-relative-pathname :ningle-tutorial-project "src/templates/"))
    (clack:clackup
        (lack.builder:builder #'hit-counter-middleware *app*)
     :server server
     :address address
     :port port))

(defun stop (instance)
    (clack:stop instance))

Understanding how to write custom middleware is very important, and I hope that this has served as a good foundation, however, as mentioned at the beginning of this section we will not be using this piece of custom middleware in our application. You are free to include it if you wish, but it will not feature in the companion code in github.

Aceesslog Middleware

Now that we have discussed what middleware is, work it does, how it works, and how to implement it, we will look at some of the middleware modules included in lack which ningle therefore has access to.

We will start with what is known as accesslog middleware, it's a very simple piece of middleware that just logs requests as they come in.

As we did in the previous section, we must adjust the lack.builder line, however, this time we do not need to write any function, the middleware that comes with lack uses some simplified syntax.

(defun start (&key (server :woo) (address "127.0.0.1") (port 8000))
    (djula:add-template-directory (asdf:system-relative-pathname :ningle-tutorial-project "src/templates/"))
    (clack:clackup
        (lack.builder:builder :accesslog *app*)
     :server server
     :address address
     :port port))

If you recompile and run your application, and view the terminal output, you will see information about the incoming requests to the web server.

This is a simple example, but it highlights an important distinction that the bundled lack middleware isn't a reference to a function, it's a keyword, as we will see in the next section, they can be a little more complicated than just a keyword, but this particular piece of middleware, it is just a keyword to turn it on. Other pieces of middleware may be a list that include configuration options, if needed.

Static Files Middleware

What we would like to do, when we write our templates is to be able to tell our template that a file is a static file and must be served from the static location. We will need to use a special djula tag to inform our templates that a file is a static file, which may seem a little counter intuitive, however, if for some reason we need to change where static files are served from (for example we may initially host them locally, but then switch to s3 or cloudflare or something), we'd have to go through all our templates changing the url, whereas using static file middleware, we'd set up a base once, and if we need to change it, we change it in one place and then our templates wouldn't need to change at all.

While this sounds like a lot of work, remarkably, it isn't!

There's only really three steps to setting up static file handling in Ningle!

As we are using djula (and a reminder quicklisp may not yet have the latest version of djula, you may need to use git to clone it into your quicklisp/local-projects), we must configure djula to be aware of where our static files will be mounted. So, just as we added a template directory, we must also add a static directory, in our example this is in the start function:

(djula:add-template-directory (asdf:system-relative-pathname :ningle-tutorial-project "src/templates/"))
(djula:set-static-url "/public/")

This second line is the one we have added, when we use the static tag later on, it will know to use "/public/" as our static path.

NOTE: Be mindful to ensure there's a trailing slash when calling set-static-url!

The second thing we must do is hook into the lack static middleware.

(lack.builder:builder :accesslog
                      (:static
                       :root (asdf:system-relative-pathname :ningle-tutorial-project "src/static/")
                       :path "/public/")
                      *app*)

Mentioned previously, some middleware setup will be lists, in this instance, a list where the first item is a keyword naming the lack middleware module to use (this will be a common pattern with other lack middleware) and then any arguments that the middleware module uses. In this case, we need to define where on our host file system we will be storing our static files, this is the :root argument and we specify that relative to our project, static files will be stored in /src/static and we need to have these mounted on a path which is exactly what the :path argument does, we will hide the physical location of our static files (good security) and state that they're available behind "/public/".

For clarity, this is what the start function should look like:

(defun start (&key (server :woo) (address "127.0.0.1") (port 8000))
    (djula:add-template-directory (asdf:system-relative-pathname :ningle-tutorial-project "src/templates/"))
    (djula:set-static-url "/public/")
    (clack:clackup
      (lack.builder:builder :accesslog
                            (:static
                             :root (asdf:system-relative-pathname :ningle-tutorial-project "src/static/")
                             :path "/public/")
                            *app*)
     :server server
     :address address
     :port port))

The final thing we need to do is, in our templates, use the static tag to load a given static file. In the base.html file, you might want to display an image. You can use whatever image you like, but if you want to use the one I've created, you can use this.

You should put this file (or the image of your choice) in the src/static/images/ directory (and create it, if you have not), I have called the image logo.jpg and have stored it in src/static/logo.jpg. This will exposed it as /public/images/logo.jpg and from here we can place these into our templates.

<img src='{% static "images/logo.jpg" %}' alt='Logo'>

If you save, reload, and view this project in your web browser, you should see the image rendered as you might expect. Inspecting the page you will see that the src attribute will be src="/public/images/logo.jpg". The image is being served without writing having to write a controller, and is served from the root you defined.

Tidying Up

Now that we have the ability to serve images, css etc, we might want to take this time to writing some css (although I personally hate writing CSS), and making the site look good. Although it is beyond this tutorial to teach bootstrap or other css frameworks (although I will use bootstrap), I will be using bootstrap to make my site look a little nicer, you can refer to the github code to see exactly what I have done regarding frontend styling.

There is something I will do to help our application look a little better...

I will create a nicer looking error page that will take advantage of our new staticfiles middleware, so the contents of src/templates/error.html will be:

{% extends "base.html" %}

{% block content %}
    <div class="container">
        <div class="row">
            <div class="col-12">
                <h1>{{ error }}</h1>
                <img src="{% static "images/lua.jpg" %}" alt="A picture of a dog looking sad and confused" class="error-404">
            </div>
        </div>
    </div>
{% endcontent %}

I will save this photo to src/static/images/lua.jpg.

And in the main.lisp file, I will modify the not-found method:

(defmethod ningle:not-found ((app ningle:<app>))
    (declare (ignore app))
    (setf (lack.response:response-status ningle:*response*) 404)
    (djula:render-template* "error.html" nil :error "Not Found"))

I have also edited the controller for the index page:

(setf (ningle:route *app* "/")
      (lambda (params)
        (let ((user  (list :username "NMunro"))
              (posts (list (list :author (list :username "Bob")  :content "Experimenting with Dylan" :created-at "2025-01-24 @ 13:34")
                           (list :author (list :username "Jane") :content "Wrote in my diary today" :created-at "2025-01-24 @ 13:23"))))
          (djula:render-template* "index.html" nil :title "Home"
                                                   :user user
                                                   :posts posts))))

In my frontend I have edited the html to include a created-at attribute to the posts and included it as we did before with the post author and content:

<h5 class="card-title">{{ post.author.username }}</h5>
<p class="card-text">{{ post.content }}</p>
<p class="text-muted small">Posted on: {{ post.created-at }}</p>

The exact styling I leave up to you, but I wanted to be clear that there is a small content change to the html.

Conclusion

To recap, after working your way though this tutorial you should be able to:

Github

The link for this tutorial is available here.

Resources

30 Jan 2025 11:55pm GMT

29 Jan 2025

feedFOSDEM 2025

Join the FOSDEM Treasure Hunt!

Are you ready for a challenge? We're hosting a treasure hunt at FOSDEM, where participants must solve six sequential riddles to uncover the final answer. Teamwork is allowed and encouraged, so gather your friends and put your problem-solving skills to the test! The six riddles are set up across different locations on campus. Your task is to find the correct locations, solve the riddles, and progress to the next step. No additional instructions will be given after this announcement, it's up to you to navigate and decipher the clues! To keep things fair, no hints or tips will be given舰

29 Jan 2025 11:00pm GMT

27 Jan 2025

feedPlanet Lisp

Zach Beane: Maxima in the browser with ECL and wasm

Via Raymond Toy on the ecl mailing list, Maxima in the browser.

27 Jan 2025 4:37pm GMT

26 Jan 2025

feedFOSDEM 2025

Introducing Lightning Lightning Talks

The regular FOSDEM lightning talk track isn't chaotic enough, so this year we're introducing Lightning Lightning Talks (now with added lightning!). Update: we've had a lot of proposals, so submissions are now closed! Thought of a last minute topic you want to share? Got your interesting talk rejected? Has something exciting happened in the last few weeks you want to talk about? Get that talk submitted to Lightning Lightning Talks! This is an experimental session taking place on Sunday afternoon (13:00 in k1105), containing non-stop lightning fast 5 minute talks. Submitted talks will be automatically presented by our Lightning舰

26 Jan 2025 11:00pm GMT

18 Jan 2025

feedPlanet Lisp

Joe Marshall: Valid Use Case for Copilot

Our compay proides us with GitHub copilot, which is yet another example of an "AI" engine. I've reviewed it before and I wasn't particulary impressed. It writes mediocre code.

However, I found a use case where it really shines. I was setting up the blank project repositories for our upcoming hackathon. This involved creating a bunch of otherwise empty repositories with a customize README.md file. The simple solution was to automate this with an ad hoc shell script.

I don't make a practice of writing shell scripts. I write a few every now and then, but I don't live and breathe bash. It takes me a while because I forget the flags. I have to look up the exact syntax for things like the find command. Then I have to fight with the shell to get the quoting and escaping right. It's a tedious chore. No doubt I would become adept at it if it were something I did daily, but it's not.

So I decided to ask GitHub copilot to write some scripts for me. I gave it my problem statement: start with a file with a list of repository and description pairs. For each pair, look for the subdirectory with that name and descend into it. Use the sed command to modify the README.md file in the subdirectory to add the repository discription. Then use git to commit and push the change.

Copilot gave this a couple of seconds thought and came up with this script:

while IFS= read -r line; do
  dir_name=$(echo "$line" | cut -d' ' -f1)
  description=$(echo "$line" | cut -d' ' -f2-)
  if [ -d "$dir_name" ]; then
    sed -i "s/\[\[github_repo_description\]\]/$description/g" "$dir_name/README.md"
  else
    echo "Directory $dir_name does not exist."
  fi
done < descriptions.txt

find . -type d -name "hack*" -exec sh -c '
  cd "$0"
  if [ -f README.md ]; then
    git add README.md
    git commit -m "Update description."
    git push
  fi
' {} \;       

It isn't a script that'd I'd write home about, but it got the job done. I ran the script and had the repositories set up in couple of minutes. I could have written the script myself, but it would have take me longer. I had forgotten the options to the find command. I had forgotten the arguments to the cut command. I would have had to look them up. Copilot saved me that time.

A co-worker of mine questioned the engineering tradeoff of using a resource hog like generative AI to write crappy, throwaway shell scripts. From the standpoint of an indiviual developer, though, this is the first use case for copilot that I've where it actualy saved me time and effort.

18 Jan 2025 5:38am GMT