## 22 Feb 2019

### Planet Debian

#### Dirk Eddelbuettel: RVowpalWabbit 0.0.13: Keeping CRAN happy

Another small RVowpalWabbit package update brings us version 0.0.13. And just like Rblpapi yesterday, we have a new RVowpalWabbit update to cope with staged installs which will be a new feature of R 3.6.0. No other changes were made No new code or features were added.

We should mention once more there is a newer, but not on CRAN, package rvw thanks to the excellent GSoC 2018 and beyond work by Ivan Pavlov (who was mentored by James and myself) so if you are into Vowpal Wabbit from R go check it out.

More information is on the RVowpalWabbit page. Issues and bugreports should go to the GitHub issue tracker.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

22 Feb 2019 12:42pm GMT

#### Dirk Eddelbuettel: Rblpapi 0.3.8: Keeping CRAN happy

A minimal maintenance release of Rblpapi, now at version 0.3.9, arrived on CRAN earlier today. Rblpapi provides a direct interface between R and the Bloomberg Terminal via the C++ API provided by Bloomberg (but note that a valid Bloomberg license and installation is required).

This is the ninth release since the package first appeared on CRAN in 2016. It accomodates a request by CRAN / R Core to cope with staged installs which will be a new feature of R 3.6.0. No other changes were made (besides updating a now-stale URL at Bloomberg in a few spots and other miniscule maintenance). However, a few other changes have been piling up at the GitHub repo so feel free to try that version too. Details of this release below:

#### Changes in Rblpapi version 0.3.9 (2019-02-20)

• Add 'StagedInstall: no' to DESCRIPTION to accomodate R 3.6.0.

Courtesy of CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the Rblpapi page. Questions, comments etc should go to the issue tickets system at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

22 Feb 2019 2:28am GMT

## 21 Feb 2019

### Planet Debian

#### Iustin Pop: QEMU SCSI Tape passthrough

Just a note to myself since I searched the internet for a long time to make this workâ€¦

I needed to test a really trivial change to the scsi tape driver code, but didn't want to reboot an actual machine that has a tape drive. So I thought, maybe QEMU can pass-through an arbitrary device?

Indeed it can, but all the examples one finds are either

• referring to libvirt or openstack or another random virtualisation tool, and their xml config files;
• and most likely referring to disks/block devices, for which there are simpler ways to do things.

After many fruitless searches, finally found a reddit comment that shows how it is done for, well, (cdrom) disks again, but in a generic way. A bit of massaging results in:

$qemu â€¦ \ -drive id=footape,if=none,format=raw,readonly=off,file=/dev/sgX \ -device virtio-scsi-pci,id=scsi0 \ -device scsi-generic,bus=scsi0.0,drive=footape  Note 1: The special note is that tapes shouldn't be addressed by their regular devices (stX/nstX), since that would actually open the tape and hangs without a tape. You need to find the scsi generic device for it (via sg_map), and use that as the file argument to the drive parameter. Note 2: Of course, if running qemu unprivileged, don't forget permissions on the 'sg' device. After that, all good. Almost, that is: while mt status and mt eject work, at least for my tape drive, mt tell said (in the virtual machine): # mt tell /dev/nst0: Input/output error # dmesg â€¦ [ 128.799466] st 1:0:0:0: [st0] Sense Key : Illegal Request [current] [ 128.800172] st 1:0:0:0: [st0] Add. Sense: Invalid field in cdb  Well, even with that, it was good enough for me, so thought to write this down. And yes, I ejected the tape from the VM. 21 Feb 2019 11:22pm GMT #### Bits from Debian: Infomaniak Platinum Sponsor of DebConf19 We are very pleased to announce that Infomaniak has committed to support DebConf19 as a Platinum sponsor. "Infomaniak is proud to support the annual Debian Developers' Conference", said Marc Oehler, Chief Operating Officer at Infomaniak. "The vast majority of our hostings work using Debian and we share this community's values: promoting innovation whilst ensuring that security, transparency and user freedom remains top priority." Infomaniak is Switzerland's largest web-hosting company, also offering backup and storage services, solutions for event organizers, live-streaming and video on demand services. It wholly owns its datacenters and all elements critical to the functioning of the services and products provided by the company (both software and hardware). With this commitment as Platinum Sponsor, Infomaniak contributes to make possible our annual conference, and directly supports the progress of Debian and Free Software helping to strengthen the community that continues to collaborate on Debian projects throughout the rest of the year. Thank you very much Infomaniak, for your support of DebConf19! ## Become a sponsor too! DebConf19 is still accepting sponsors. Interested companies and organizations may contact the DebConf team through sponsors@debconf.org, and visit the DebConf19 website at https://debconf19.debconf.org. 21 Feb 2019 2:45pm GMT ## 19 Feb 2019 ### Planet Debian #### Vincent Sanders: A very productive weekend I just hosted a NetSurf Developer weekend which is an opportunity for us to meet up and make use of all the benefits of working together. We find the ability to plan work and discuss solutions without loosing the nuances of body language generally results in better outcomes for the project. Due to other commitments on our time the group has not been able to do more than basic maintenance activities in the last year which has resulted in the developer events becoming a time to catch up on maintenance rather than making progress on features. Because of this the July and November events last year did not feel terribly productive, there were discussions about what we should be doing and bugs considered but a distinct lack of commuted code. As can be seen from our notes this time was a refreshing change. We managed to complete a good number of tasks and actually add some features while still having discussions, addressing bugs and socialising. We opened on the Friday evening by creating a list of topics to look at over the following days and updating the wiki notes. We also reviewed the cross compiler toolchains which had been updated to include the most recent releases for things like openssl, curl etc. As part of this review we confirmed the decision to remove the Atari platform from active support as its toolchain builds have remained broken for over two years with no sign of any maintainer coming forward. While it is a little sad to see a platform be removed it has presented a burden on our strained resources by requiring us to maintain a CI worker with a very old OS using tooling that can no longer be replicated. The tooling issue means a developer cannot test changes locally before committing so testing changes that affected all frontends was difficult. Saturday saw us clear all the topics from our list which included: • Fixing a bug preventing compiling our reference counted string handling library. • Finishing the sanitizer work started the previous July • Fixing several bugs in the Framebuffer frontend installation. • Making the Framebuffer UI use the configured language for resources. The main achievement of the day however was implementing automated system testing of the browser. This was a project started by Daniel some eight years ago but worked on by all of us so seeing it completed was a positive boost for the whole group. The implementation consisted of a frontend named monkey. This frontend to the browser takes textural commands to perform operations (i.e. open a window or navigate to a url) and generates results in a structured text format. Monkey is driven by a python program named monkeyfarmer which runs a test plan ensuring the results are as expected. This allows us to run a complete browsing session in an automated way, previously someone would have to manually build the browser and check the tests by hand. This manual process was tedious and was rarely completed across our entire test corpus generally concentrating on just those areas that had been changed such as javascript output. We have combined the monkey tools and our test corpus into a CI job which runs the tests on every commit giving us assurance that the browser as a whole continues to operate correctly without regression. Now we just have the task of creating suitable plans for the remaining tests. Though I remain hazy as to why, we became inordinately amused by the naming scheme for the tools. We rounded the Saturday off by going out for a very pleasant meal with some mutual friends. Sunday started by adding a bunch of additional topics to consider and we made good progress addressing these. We performed a bug triage and managed to close several issues and commit to fixing a few more. We even managed to create a statement of work of things we would like to get done before the next meetup. My main achievement on the Sunday was to add WEBP image support. This uses the Google libwebp library to do all the heavy lifting and adding a new image content handler to NetSurf is pretty straightforward. 19 Feb 2019 9:54pm GMT #### Sylvain Beucler: RenPyWeb - Ren'Py in your HTML5 web browser I like the Ren'Py project, a popular game engine aimed at Visual Novels - that can also be used as a portable Python environment. One limitation was that it required downloading games, while nowadays people are used to Flash- or HTML5- based games that play in-browser without having to (de)install. Can this fixed? While maintaining compatibility with Ren'Py's several DSLs? And without rewriting everything in JavaScript? Can Emscripten help? While this is a Python/Cython project? After lots of experimenting, and full-stack patching/contributing, it turns out the answer is yes! Live demo: https://renpy.beuc.net/ At last I finished organizing and cleaning-up, published under a permissive free software / open source license, like Python and Ren'Py themselves. Python port: https://www.beuc.net/python-emscripten/python/dir?ci=tip Build system: https://github.com/renpy/renpyweb Development in going on, consider supporting the project! https://www.patreon.com/Beuc 19 Feb 2019 6:38pm GMT #### Reproducible builds folks: Reproducible Builds: Weekly report #199 Here's what happened in the Reproducible Builds effort between Sunday February 10th and Saturday February 16th 2019: • strip-nondeterminism is our tool that post-processes files to remove known non-deterministic output. This week, Chris Lamb adjusted its behaviour to deduplicate hardlinks via stat(2) before processing to avoid issues when handling files in parallel; as the per-filetype handlers are yet currently guaranteed to be atomic, one process could temporarily truncate a file which can cause errors in other processes operating on the "same" file under a different pathname. This was thus causing package build failures in packages that de-duplicate hardlinks in their build process such as the Debian Administrator's Handbook (#922168). • There was a brief update from the Debian Ruby maintainers on whether the language might need to strip -fdebug-prefix-map from the tools used to build extensions. • On our mailing list, Holger Levsen re-raised a question regarding uploading the "official" .buildinfo files to buildinfo.debian.net. • On Tuesday 26th February Chris Lamb will speak at Speck&Tech 31 "Open Security" on Reproducible Builds in Trento, Italy. • Jelle van der Waa fixed some spelling mistakes on the reproducible-builds.org project website. [â€¦] • 6 Debian package reviews were added, 4 were updated and 16 were removed in this week, adding to our knowledge about identified issues. ## diffoscope development diffoscope is our in-depth "diff-on-steroids" utility which helps us diagnose reproducibility issues in packages. This week: • Chris Lamb: • Add support for comparing .crx Chrome browser extensions. (#41) • Add support for comparing MP3 and files with similar metadata. (#43) • Replace the literal xxd(1) output (!)(!) in tests/data/hello.wasm with its binary equivalent (#47) and ensure both WebAssembly test data files are actually unique. (#42) • Catch tracebacks when mounting invalid filesystem images under guestfs. [â€¦] • Fix tests when using Ghostscript 9.20 vs 9.26 for Debian stable and for stable with the security repositories enabled. [â€¦][â€¦] • Temporarily drop ubuntu-devel from internal test matrix due to a linux-firmware package installation issue. [â€¦] • Ed Maste: • Include relocations in objdump disassembly. (#48) • Graham Christensen: • Clarify "no file-specific differences" message when we fallback to a binary diff. (!19) • Mattia Rizzolo: • Make test_ps.test_text_diff pass with Ghostscript version 9.26. [â€¦] In addition, Vagrant Cascadian updated diffoscope in GNU Guix [â€¦] and went on to upload disorderfs [â€¦] and trydiffoscope [â€¦] too. ## Packages reviewed and fixed, and bugs filed ## Test framework development We operate a comprehensive Jenkins-based testing framework that powers tests.reproducible-builds.org. • Hans-Christoph Steiner: • Set the LANG and LC_ALL environment variables for F-Droid builds to workaround an unsolved issue in Java/Gradle. [â€¦][â€¦] • Modernise some dependencies. [â€¦] • Node maintenance. [â€¦] • Holger Levsen: • Mattia Rizzolo: This week's edition was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists. 19 Feb 2019 12:03pm GMT ## 18 Feb 2019 ### Planet Debian #### Russ Allbery: INN 2.6.3 INN 2.6.3 has been released. This is a bug fix and minor feature release over INN 2.6.2, and the upgrade should be painless. The main ISC downloads page will be updated shortly; in the meantime, you can download the new release from ftp.isc.org or my personal INN pages. The latter also has links to the full changelog and the other INN documentation. The big change in this release is support for Python 3. Embedded Python filtering and authentication hooks for innd and nnrpd can now use version 3.3.0 or later of the Python interpreter. Python 2.x is still supported (2.3.0 or later). Also fixed in this release are several issues with TLS: fixed selection of elliptic curve selection, a new configuration parameter to fine-tune the cipher suites with TLS 1.3, and better logging of failed TLS session negotiation. This release also returns the error message from Python and Perl filter hook rejects to CHECK and TAKETHIS commands and fixes various other, more minor bugs. As always, thanks to Julien Ã‰LIE for preparing this release and doing most of the maintenance work on INN! 18 Feb 2019 10:30pm GMT #### Chris Lamb: Book Review: Painting the Sand Painting the Sand (2017) Kim Hughes Staff Sargeant Kim Hughes is a bomb disposal operator in the British Army undergoing a gruelling six-month tour of duty in Afghanistan during which he defuses over 100 improvised explosives devices, better known as IEDs. Cold opening in the heat of the desert, it begins extremely strongly with a set piece of writing that his editor should be proud of. The book contains colourful detail throughout and readers will quickly feel like "one of the lads" with the shibboleth military lingo and acronyms. Despite that, this brisk and no-nonsense account - written in that almost-stereotypical squaddie's tone of voice - is highly accessible and furthermore is refreshing culturally-speaking given the paucity of stories in the zeitgest recounting British derring-do in "Afghan", to adopt Hughes' typical truncation of the country. However, apart from a few moments (such as when the Taliban adopt devices undetectable with a metal detector) the tension deflates slowly rather than mounting like a lit fuse. For example, I would have wished for a bit more improvised suspense and drama when they were being played, trapped or otherwise manipulated by the Taliban for once, and only so much of that meekness can be attributed to Hughes' self-effacing and humble manner. Indeed, the book was somewhat reminiscent of The Secret Barrister in that the ending is a little too quick for my taste, rushing through "current" events and becoming a little too self-referential in parts, this comparison between the two words being aided by both writers having somewhat of a cynical affect about them. One is left with the distinct impression that Hughes went to Afghan a job. He gets it done with minimal fuss and no unnecessary nonsense â€¦ and that's what exactly what he does as a memorialist. 18 Feb 2019 6:43pm GMT #### Raphaël Hertzog: Freexianâ€™s report about Debian Long Term Support, January 2019 Like each month, here comes a report about the work of paid contributors to Debian LTS. ### Individual reports In January, about 204.5 work hours have been dispatched among 13 paid contributors. Their reports are available: • Abhijith PA did 12 hours (out of 12 hours allocated). • Antoine BeauprÃ© did 9 hours (out of 20.5 hours allocated, thus keeping 11.5h extra hours for February). • Ben Hutchings did 24 hours (out of 20 hours allocated plus 5 extra hours from December, thus keeping one extra hour for February). • Brian May did 10 hours (out of 10 hours allocated). • Chris Lamb did 18 hours (out of 18 hours allocated). • Emilio Pozuelo Monfort did 42.5 hours (out of 20.5 hours allocated + 25.25 extra hours, thus keeping 3.25 extra hours for February). • Hugo Lefeuvre did 20 hours (out of 20 hours allocated). • Lucas Kanashiro did 5 hours (out of 4 hours allocated plus one extra hour from December). • Markus Koschany did 20.5 hours (out of 20.5 hours allocated). • Mike Gabriel did 10 hours (out of 10 hours allocated). • Ola Lundqvist did 4.5 hours (out of 8 hours allocated + 6.5 extra hours, thus keeping 8 extra hours for February, as he also gave 2h back to the pool). • Roberto C. Sanchez did 10.75 hours (out of 20.5 hours allocated, thus keeping 9.75 extra hours for February). • Thorsten Alteholz did 20.5 hours (out of 20.5 hours allocated). ### Evolution of the situation In January we again managed to dispatch all available hours (well, except one) to contributors. We also still had one new contributor in training, though starting in February Adrian Bunk has become a regular contributor. But: we will lose another contributor in March, so we are still very much looking for new contributors. Please contact Holger if you are interested to become a paid LTS contributor. The security tracker currently lists 40 packages with a known CVE and the dla-needed.txt file has 42 packages needing an update. ### Thanks to our sponsors New sponsors are in bold. No comment | Liked this article? Click here. | My blog is Flattr-enabled. 18 Feb 2019 2:05pm GMT #### Thomas Lange: Netplan support in FAI The new version FAI 5.8.1 now generates the configuration file for Ubuntu's netplan tool. It's a YAML description for setting up the network devices, replacing the /etc/network/interfaces file. The FAI CD/USB installation image for Ubuntu now offers two different variants to be installed, Ubuntu desktop and Ubuntu server without a desktop environment. Both are using Ubuntu 18.04 aka Bionic Beaver. FAI 5.8.1 also improves UEFI support for network installations. UEFI boot is still missing for the ISO images. The FAI ISO images are available from [1]. The FAIme build service [2] for customized cloud and installation images also uses the newest FAI version. 18 Feb 2019 1:34pm GMT ## 17 Feb 2019 ### Planet Debian #### Niels Thykier: Making debug symbols discoverable and fetchable Michael wrote a few days ago about the experience of debugging programs on Debian. And he is certainly not the only one, who found it more difficult to find debug symbols on Linux systems in general. But fortunately, it is a fixable problem. Basically, we just need a service to map a build-id to a downloadable file containing that build-id. You can find the source code to my (prototype) of such a dbgsym service on salsa.debian.org. It exposes one API endpoint, "/api/v1/find-dbgsym", which accepts a build-id and returns some data about that build-id (or HTTP 404 if we do not know the build-id). An example: $ curl --silent http://127.0.0.1:8000/api/v1/find-dbgsym/5e625512829cfa9b98a8d475092658cb561ad0c8/ | python -m json.tool
{
"package": {
"architecture": "amd64",
"build_ids": [
],
"checksums": {
},
"https://snapshot.debian.org/archive/debian-debug/20161028T101957Z/pool/main/6/6tunnel/6tunnel-dbgsym_0.12-1_amd64.deb"
],
"name": "6tunnel-dbgsym",
"version": "1:0.12-1"
}
}


Notice how it includes a download URL and a SHA256 checksum, so with this you can download the package containing the build-id directly from this and verify the download. The sample_client.py included in the repo does that and might be a useful basis for others interested in developing a client for this service.

To seed the database, so it can actually answer these queries, there is a bulk importer that parses Packages files from the Debian archive (for people testing: the ones from debian-debug archive are usually more interesting as they have more build-ids).

## Possible improvements

• Have this service deployed somewhere on the internet rather than the loopback interface on my machine.
• The concept is basically distribution agnostic (Michael's post in fact links to a similar service) and this could be a standard service/tool for all Linux distributions (or even just organizations). I am happy to work with people outside Debian to make the code useful for their distribution (including non-Debian based distros).
• The prototype was primarily based around Debian because it was my point of reference (plus what I had data for).
• The bulk importer could (hopefully) be faster on new entries.
• The bulk importer could import the download urls as well, so we do not have to fetch the relevant data online the first time when people are waiting.
• Most of the django code / setup could probably have been done better as this has been an excuse to learn django as well.

Patches and help welcome.

## Kudos

This prototype would not have been possible without python3, django, django's restframework, python's APT module, Postgresql, PoWA and, of course, snapshot.debian.org (including its API).

17 Feb 2019 9:03pm GMT

#### Birger Schacht: Sway in experimental

A couple of days ago the 1.0-RC2 version of Sway, a Wayland compositor, landed in Debian experimental. Sway is a drop in replacement for the i3 tiling window manager for wayland. Drop in replacement means that, apart from minor adaptions, you can reuse your existing i3 configuration file for Sway. On the Website of sway you can find a short introduction video that shows the most basic concepts of using Sway, though if you have worked with i3 you will feel at home soon.

In the video the utility swaygrab is mentioned, but this tool is not part of Sway anymore. There is another screenshot tool now though, called grim which you can combine with the tool slurp if you want to select regions for screenshots. The video also mentions swaylock, which is a screen locking utility similar to i3lock. It was split out of the main Sway release a couple of weeks ago but there also exists a Debian package by now. And there is a package for swayidle, which is a idle management daemon, which comes handy for locking the screen or for turning of your display after a timeout. If you need clipboard manager, you can use wl-clipboard. There is also a notification daemon called mako (the Debian package is called mako-notifier and is in NEW) and if you don't like the default swaybar, you can have a look at waybar (not yet in Debian, see this RFS). If you want to get in touch with other Sway users there is a #sway IRC channel on freenode. For some tricks setting up Sway you can browse the wiki.

If you want to try Sway, beware that is is a release candiate and there are still bugs. I'm using Sway since a couple of month and though i had crashes when it still was the 1.0-beta.1 i hadn't any since beta.2. But i'm using a pretty conservative setup.

Sway was started by Drew DeVault who is also the upstream maintainer of wlroots, the Wayland compositor library Sway is using and who some might now from his sourcehut project (LWN Article). He also just published an article about Wayland misconceptions. The upstream of grim, slurp and mako is Simon Ser, who also contributes to sway. A lot of thanks for the Debian packaging is due to nicoo who did most of the heavy lifting and to Sean for having patience when reviewing my contributions. Also thanks to Guido for maintaining wlroots!

17 Feb 2019 7:52pm GMT

#### Andrew Cater: Debian 9.8 released : Desktop environments and power settings

Debian 9.8 - the latest update to Debian Stretch - was released yesterday. Updated installation media can be found at the Debian CD Netnstall page, for example, at As part of the testing, I was using a very old i686 laptop which powers down at random intervals because the battery is old. It tends to suspend almost immediately. I found that the power management settings for the Cinnamon desktop were hart to find: using a Mate disktop allowed me to find the appropriate settings in the Debian menu layout much more easily Kudos to Steve McIntyre and Andy Simpkins (amongst others) for such a good job producing and testing the Debian CDs

17 Feb 2019 3:12pm GMT

## 16 Feb 2019

### Planet Debian

#### Jonathan Dowland: embedding Haskell in AsciiDoc

I'm a fan of the concept of Literate Programming (I explored it a little in my Undergraduate Dissertation a long time ago) which can be briefly (if inadequately) summarised as follows: the normal convention for computer code is by default the text within a source file is considered to be code; comments (or, human-oriented documentation) are exceptional and must be demarked in some way (such as via a special symbol). Literate Programming (amongst other things) inverts this. By default, the text in a source file is treated as comments and ignored by the compiler, code must be specially delimited.

Haskell has built-in support for this scheme: by naming your source code files .lhs, you can make use of one of two conventions for demarking source code: either prefix each source code line with a chevron (called Bird-style, after Richard Bird), or wrap code sections in a pair of delimiters \begin{code} and \end{code} (TeX-style, because it facilitates embedding Haskell into a TeX-formatted document).

For various convoluted reasons I wanted to embed Haskell into an AsciiDoc-formatted document and I couldn't use Bird-style literate Haskell, which would be my preference. The AsciiDoc delimiter for a section of code is a line of dash symbols, which can be interleaved with the TeX-style delimiters:

------------
\begin{code}
next a = if a == maxBound then minBound else succ a
\end{code}
------------


Unfortunately the Tex-style delimiters show up in the output once the AsciiDoc is processed. Luckily, we can swap the order of the AsciiDoc and Literate-Haskell delimiters, because the AsciiDoc ones are treated as a source-code comment by Haskell and ignored. This moves the visible TeX-style delimiters out of the code block, which is a minor improvement:

\begin{code}
------------
next a = if a == maxBound then minBound else succ a
------------
\end{code}


We can disguise the delimiters outside of the code block further by defining an empty AsciiDoc macro called "code". Macros are marked up with surrounding braces, leaving just stray \begin and \end tokens in the text. Towards the top of the AsciiDoc file, in the pre-amble:

= Document title
Document author
:code:


This could probably be further improved by some AsciiDoc markup to change the style of the text outside of the code block immediately prior to the \begin token (perhaps make the font 0pt or the text colour the same as the background colour) but this is legible enough for me, for now.

The resulting file can be fed to an AsciiDoc processor (like asciidoctor, or intepreted by GitHub's built-in AsciiDoc formatter) and to a Haskell compiler. Unfortunately GitHub insists on a .adoc extension to interpret the file as AsciiDoc; GHC insists on a .lhs extension to interpret it as Literate Haskell (who said extensions were semantically meaningless these daysâ€¦). So I commit the file as .adoc for GitHub's benefit and maintain a local symlink with a .lhs extension for my own.

Finally, I am not interested in including some of the Haskell code in my document that I need to include in the file in order for it to work as Haskell source. This can be achieved by changing from the code delimiter to AsciiDoc comment delimeters on the outside:

////////////
\begin{code}
utilityFunction = "necessary but not interesting for the document"
\end{code}
////////////


You can see an example of a combined AsciiDoc-Haskell file here (which is otherwise a work in progress):

16 Feb 2019 10:50pm GMT

#### Vasudev Kamath: Note to Self: Growing the Root File System on First Boot

These 2 are the use cases I came across for expanding root file system.

1. RaspberryPi images which comes in smaller image size like 1GB which you write bigger size SD cards like 32GB but you want to use full 32GB of space when system comes up.
2. You have a VM image which is contains basic server operating system and you want to provision the same as a VM with much larger size root file system.

My current use case was second but I learnt the trick from 1, that is the RaspberryPi3 image spec by Debian project.

Idea behind the expanding root file system is first expanding the root file system to full available size and then run resize2fs on the expanded partition to grow file system. resize2fs is a tool specific for ext2/3/4 file system. But this needs to be done before the file system is mounted.

Here is my modified script from raspi3-image-spec repo. Only difference is I've changed the logic of extracting root partition device to my need, and of course added comments based on my understanding.

#!/bin/sh

# Just extracts root partition and removes partition number to get the device
# name eg. /dev/sda1 becomes /dev/sda
roottmp=$(lsblk -l -o NAME,MOUNTPOINT | grep '/$')
rootpart=/dev/${roottmp%% */} rootdev=${rootpart%1}

# Use sfdisk to extend partition to all available free space on device.
flock $rootdev sfdisk -f$rootdev -N 2 <<EOF
,+
EOF

sleep 5

# Wait for all pending udev events to be handled

sleep 5

# detect the changes to partition (we extended it).
flock $rootdev partprobe$rootdev

# remount the root partition in read write mode
mount -o remount,rw $rootpart # Finally grow the file system on root partition resize2fs$rootpart

exit 0fs


raspi3-image-spec uses sytemd service file to execute this script just before any file system is mounted. This is done by a making service execute before local-fs.pre target. From the man page for systemd.special

local-fs.target
systemd-fstab-generator(3) automatically adds dependencies of type Before= to all mount units that refer to local mount points for this target unit. In addition, it adds dependencies of type Wants= to this target unit for those mounts listed in /etc/fstab that have the auto mount option set.

Service also disables itself on executing to avoid re-runs on every boot. I've used the service file from raspi3-image-spec as is.

## Testing with VM

raspi3-image-spec is well tested, but I wanted to make sure this works with my use case for VM. Since I didn't have any spare physical disks to experiment with I used kpartx with raw file images. Here is what I did

1. Created a stretch image using vmdb2 with grub installed. Image size is 1.5G
2. I created another raw disk using fallocate of 4G size.
3. I created a partition on 4G disk.
4. Loop mounted the disk and wrote 1G image on it using dd
5. Finally created a VM using virt-install with this loop mounted device as root disk.

Below is my vmdb configuration yml again derived from raspi3-image-spec one with some modifications to suit my needs.

# See https://wiki.debian.org/RaspberryPi3 for known issues and more details.
steps:
- mkimg: "{{ output }}"
size: 1.5G

- mklabel: msdos
device: "{{ output }}"

- mkpart: primary
device: "{{ output }}"
start: 0%
end: 100%
tag: /

- kpartx: "{{ output }}"

- mkfs: ext4
partition: /
label: RASPIROOT

- mount: /

- unpack-rootfs: /

- debootstrap: stretch
mirror: http://localhost:3142/deb.debian.org/debian
target: /
variant: minbase
components:
- main
- contrib
- non-free
unless: rootfs_unpacked

# TODO(https://bugs.debian.org/877855): remove this workaround once
# debootstrap is fixed
- chroot: /
shell: |
echo 'deb http://deb.debian.org/debian buster main contrib non-free' > /etc/apt/sources.list
apt-get update
unless: rootfs_unpacked

- apt: install
packages:
- ssh
- parted
- dosfstools
- linux-image-amd64
tag: /
unless: rootfs_unpacked

- grub: bios
tag: /

- cache-rootfs: /
unless: rootfs_unpacked

- shell: |
echo "experimental" > "${ROOT?}/etc/hostname" # '..VyaTFxP8kT6' is crypt.crypt('raspberry', '..') sed -i 's,root:[^:]*,root:..VyaTFxP8kT6,' "${ROOT?}/etc/shadow"

sed -i 's,#PermitRootLogin prohibit-password,PermitRootLogin yes,g' "${ROOT?}/etc/ssh/sshd_config" install -m 644 -o root -g root fstab "${ROOT?}/etc/fstab"

install -m 644 -o root -g root eth0 "${ROOT?}/etc/network/interfaces.d/eth0" install -m 755 -o root -g root rpi3-resizerootfs "${ROOT?}/usr/sbin/rpi3-resizerootfs"
install -m 644 -o root -g root rpi3-resizerootfs.service "${ROOT?}/etc/systemd/system" mkdir -p "${ROOT?}/etc/systemd/system/systemd-remount-fs.service.requires/"
ln -s /etc/systemd/system/rpi3-resizerootfs.service "${ROOT?}/etc/systemd/system/systemd-remount-fs.service.requires/rpi3-resizerootfs.service" install -m 644 -o root -g root rpi3-generate-ssh-host-keys.service "${ROOT?}/etc/systemd/system"
mkdir -p "${ROOT?}/etc/systemd/system/multi-user.target.requires/" ln -s /etc/systemd/system/rpi3-generate-ssh-host-keys.service "${ROOT?}/etc/systemd/system/multi-user.target.requires/rpi3-generate-ssh-host-keys.service"
rm -f ${ROOT?}/etc/ssh/ssh_host_*_key* root-fs: / # Clean up archive cache (likely not useful) and lists (likely outdated) to # reduce image size by several hundred megabytes. - chroot: / shell: | apt-get clean rm -rf /var/lib/apt/lists # TODO(https://github.com/larswirzenius/vmdb2/issues/24): remove once vmdb # clears /etc/resolv.conf on its own. - shell: | rm "${ROOT?}/etc/resolv.conf"
root-fs: /


I could not run with vmdb2 installed from Debian archive, so I cloned raspi3-image-spec and used vmdb2 submodule from it. And here are rest of commands used for testing the script.

fallocate -l 4G rootdisk.img
# Create one partition with full disk
sfdisk -f rootdisk.img <<EOF
,+
EOF
kpartx -av rootdisk.img # mounts on /dev/loop0 for me
dd if=vmdb.img of=/dev/loop0
sudo virt-install --name experimental --memory 1024 --disk path=/dev/loop0 --controller type=scsi,model=virtio-scsi --boot hd --network bridge=lxcbr0


Once VM booted I could see the root file system is 4G of size instead of 1.5G it was after using dd to write image on to it. So success!.

16 Feb 2019 4:43pm GMT