I frequently see a pattern in image build/refresh scripts where a set of packages is installed, and then all packages are updated:

apt update
apt install -y pkg1 pkg2 pkg2
apt dist-upgrade -y

While it's not much, this results in redundant work. For example reading/writing package database, potentially running triggers (man-page refresh, ldconfig, etc). The internal package dependency resolution stuff isn't actually different: "install" will also do upgrades of needed packages, etc. Combining them should be entirely possible, but I haven't found a clean way to do this yet.

The best I've got so far is:

apt update
apt-cache dumpavail | dpkg --merge-avail -
(for i in pkg1 pkg2 pkg3; do echo "$i install") | dpkg --set-selections
apt-get dselect-upgrade

This gets me the effect of running "install" and "upgrade" at the same time, but not "dist-upgrade" (which has slightly different resolution logic that'd I'd prefer to use). Also, it includes the overhead of what should be an unnecessary update of dpkg's database. Anyone know a better way to do this?

© 2020, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.

30 Oct 2020 7:07pm GMT

These days, the internal workings of Linux applications involve many different moving parts. Sometimes, it can be rather difficult to debug them when things go wrong or run slower than expected. Tracing an application's execution is one way of understanding potential issues without diving into the source code. To this end, we wrote an app-tracing tool called etrace, designed to detect performance bottlenecks and runtime issues in snaps.

In this article, we will be taking a look at etrace with an overview of the basic functionality of etrace, and highlight its usage through several representative examples.

Etrace at a glance

Etrace is a generic tracing application, useful for three broad measurement and debugging purposes:

• How much time it takes for an application to display a (graphical/UI) window on a screen.
• The sequence of tasks created and executed by the main program during its runtime.
• The list of files accessed during a program's execution.

These metrics can be used to debug potential problems in snaps, understand what a snap is trying to do, or find performance bottlenecks in a snap. Of course, it also works with native Linux packages or any executable program as well, albeit with slightly reduced out of the box functionality - it doesn't reinstall the native package for example - but can still trace the application and measure how long it takes to display a window.

Basic usage

Etrace is available as a snap - first we have to install it. Because etrace is used to run arbitrary programs, including other snaps and even traditional linux packages, it needs system-wide permissions via classic confinement, which can be accepted by using the -classic flag when running the below command. To install etrace:

snap install etrace --candidate --classic 

Note that currently, etrace only works with X11 systems.

The first use case of etrace is to measure how long it takes for a graphical snap application to display a window on a screen. We included this functionality in etrace partly because there were not any sufficiently effective tools or options to achieve this available out there. With etrace, it is very easy to do this.

Let's start out with a simple snap, gnome-calculator, and run it in a loop 10 times to see how much time this execution takes. Please note you need to have gnome-calculator installed - snap install gnome-calculator. Here we use the -no-trace option because we don't want the full tracing stack, we just want etrace to measure how long it takes to launch - we will get into the full tracing capabilities later on.

$ etrace --repeat=10 exec --use-snap-run --no-trace  gnome-calculator --cmd-stderr=/dev/null
Total startup time: 1.531152957s
Total startup time: 513.948576ms
Total startup time: 512.980061ms
Total startup time: 515.576753ms
Total startup time: 508.354472ms
Total startup time: 515.734329ms
Total startup time: 508.414271ms
Total startup time: 514.258788ms
Total startup time: 508.407346ms
Total startup time: 511.950964ms

When you run this command, you should see the gnome-calculator window show up very briefly 10 times. If this is the first time you have run gnome-calculator on this boot, it may be slower for reasons we explain in another blog post. The numbers shown indicate elapsed time it takes for the application to display a window. For non-graphical applications, it will be the time it takes before the application exits by itself.

This use case itself is great if you want to check snap startup time. Snap developers can use this first etrace functionality to help optimize their startup times and compare different application settings. Etrace also supports JSON output for integration with other scripts as well as CI, build systems for testing and metrics collection/data gathering.

Useful Snap Options

As mentioned earlier, snap applications may be slower to start on the very first execution due to system caches not being set up. Re-running the application multiple times will usually erase this one-time cost, but when measuring how long it takes for a snap to start, we want to specifically look at this worst case scenario when data is not cached in memory and the system has to do the maximum amount of work before the user sees the application. To accommodate this frequent use-case with etrace, we have two options which make this much easier. The first is the ability to reinstall the snap:

--reinstall-snap 

This option will remove the snap (and thus delete its root-owned data in system directories, but not user-data), and then reinstall the snap. This clears out system caches like fonts, internal databases, and other data that is typically owned by root, but does not delete data that is specific to a user, such as browsing history.

The second option allows you to handle user data:

--clean-snap-user-data

This flag will create a snapshot of the snap before deleting the user data. As mentioned, an example of user data would be a Web browser profile.

With both of these flags specified in the etrace command we clear more of the system's cache and thus see more consistent startup times for gnome-calculator on the same machine:

$ etrace --repeat=10 exec --use-snap-run --no-trace  gnome-calculator --reinstall-snap --clean-snap-user-data --cmd-stderr=/dev/null
Total startup time: 1.524554867s
Total startup time: 1.531155857s
Total startup time: 1.526059853s
Total startup time: 1.533270808s
Total startup time: 1.535193187s
Total startup time: 1.538566123s
Total startup time: 1.529733654s
Total startup time: 1.528315879s
Total startup time: 1.53689303s
Total startup time: 1.023516385s

Here, we see that the startup times are slower than before as well as more consistent, due to the elimination of the caching that the system was doing in between runs. It is also worth noting that in the previous example, the startup was much slower than all the other ones because it was the first execution of gnome-calculator in this desktop session.

If your application requires additional things to be done before/after starting it up either to test a specific scenario or to clear out more caching that might be taking place, etrace supports running generic scripts via the -prepare-script and -restore-script options as well as -prepare-script-args and -restore-script-args.

Tracing task execution

The second use case for etrace is to check what tasks or processes a snap runs during its execution. This is useful to examine the snap's behavior and troubleshoot snaps that do run well, fail to launch or provide no meaningful output.

While etrace normally will automatically close the graphical window created by the program, sometimes it's necessary to not have this behavior, for example to test a specific action in the application or when working with a command line application which does not create a window. For these use cases, there is the -no-window-wait option to instruct etrace to wait for the program to exit by itself.

Here we use etrace exec with full tracing turned on (by omitting the -no-trace option):

$ etrace exec telegram-desktop --no-window-wait
56 exec calls during snap run:
     Start   Stop     Elapsed       Exec
     0       95417    95.417022ms   /snap/bin/telegram-desktop
     14991   20267    5.276918ms    /usr/lib/snapd/snap-seccomp
     38522   39649    1.127004ms    /usr/lib/snapd/snap-device-helper
     40294   41350    1.055955ms    /usr/lib/snapd/snap-device-helper
     41984   43035    1.051902ms    /usr/lib/snapd/snap-device-helper
     43688   44741    1.053094ms    /usr/lib/snapd/snap-device-helper
...
     82329   83477    1.148939ms    /usr/lib/snapd/snap-device-helper
     88778   93460    4.682064ms    snap-update-ns
     95417   100613   5.196094ms    /usr/lib/snapd/snap-exec
     100613  212749   112.13684ms   /snap/telegram-desktop/1708/bin/desktop-launch
     105275  107645   2.36988ms     /usr/bin/date
     115309  118616   3.30615ms     /usr/bin/getent
...
     120239  122471   2.232074ms    /usr/bin/md5sum
     192968  196316   3.347873ms    /usr/bin/head
     199725  203120   3.395795ms    /usr/bin/ln
     204533  207864   3.331899ms    /usr/bin/rm
     208199  211477   3.277063ms    /usr/bin/ln
     212749  6000720  5.787970066s  /snap/telegram-desktop/1708/usr/bin/telegram-desktop
Total time:  6.000720024s
Total startup time: 6.008373172s

This is a table showing the total set of tasks that were run during the program's full execution, including the start time and end time (in microseconds) as well as the total elapsed time for the task execution. This may help in debugging problems with broken snaps, for example an intermediate shell script may be misbehaving before the final program is executed, and this would be apparent to the snap developer who has an expectation of what the the snap should be doing in the normal case.

Tracing file accesses

The third use-case of etrace is to see what files are accessed by a snap at runtime. For example, classic snaps, since they are not confined, may access any file on a user's system and thus can sometimes run into bugs where they are reading the 'wrong' file from the host and not those shipped with the snap itself, the way that snaps are designed to work. This third functionality is also helpful if you're trying to understand the full scope of a program's execution. It can also be used to get an explicit list of files that a graphical snap tries to access before displaying a window.

To do this, use the file subcommand. Here we will look at a simple program, hello-world, which accesses a small number of files (many snaps end up accessing hundreds of files):

$ ./etrace file --use-snap-run hello-world --no-window-wait
Hello World!
6 files accessed during snap run:
     Filename                            Size (bytes)
     /dev/pts/0                          0
     /etc/ld.so.cache                    157250
     /home/user/                         4096
     /lib/x86_64-linux-gnu/libc-2.23.so   
     /lib/x86_64-linux-gnu/libc.so.6     2030544
     /snap/hello-world/29/bin/echo       31

Total startup time: 87.038215ms

This will show us the files that were accessed as well as their sizes. There are options to also show what program accessed which file, filter the files by their location, or filter the files by what programs accessed those files, similar to the lsof command. Etrace also supports a rich JSON output format with details such as when the file was accessed, what syscall accessed the file and others.

Summary

Hopefully you find this blog post and etrace useful the next time you need to measure how long snaps take to startup, or debug problems with your snap applications, or even just peek at what is going on under the hood of your favorite snap.

Etrace is undergoing active development and welcomes contributions! We will be posting more about etrace in the coming future with some more use cases and stories. As always, come join us on the snapcraft forum if you have comments or suggestions.

Photo by Steve Harvey on Unsplash.

30 Oct 2020 1:42pm GMT

Adoptámos um novo animal de estimação, numa semana repleta de animação com o encontro mensal da Comundade Ubuntu Portugal, Hacktoberfest e o lançamento Groovy Gorilla, aqui fica mais um episódio no vosso podcast preferido.

Já sabem: oiçam, subscrevam e partilhem!

• https://www.powerpc-notebook.org/pt
• https://letsencrypt.org/docs/rate-limits
• https://github.com/InterruptorPt/ate-onde-chega-cultura
• https://www.youtube.com/watch?v=MQoZTcUsnlw
• https://interruptor.pt/artigos/interruptor-x-hacktoberfest
• https://podes.pt/programa
• https://www.opensourcelisbon.com

Apoios

Podem apoiar o podcast usando os links de afiliados do Humble Bundle, porque ao usarem esses links para fazer uma compra, uma parte do valor que pagam reverte a favor do Podcast Ubuntu Portugal.
E podem obter tudo isso com 15 dólares ou diferentes partes dependendo de pagarem 1, ou 8.
Achamos que isto vale bem mais do que 15 dólares, pelo que se puderem paguem mais um pouco mais visto que têm a opção de pagar o quanto quiserem.

Se estiverem interessados em outros bundles não listados nas notas usem o link https://www.humblebundle.com/?partner=PUP e vão estar também a apoiar-nos.

Atribuição e licenças

Este episódio foi produzido por Diogo Constantino e Tiago Carrondo e editado por Alexandre Carrapiço, o Senhor Podcast.

A música do genérico é: "Won't see it comin' (Feat Aequality & N'sorte d'autruche)", por Alpha Hydrae e está licenciada nos termos da [CC0 1.0 Universal License](https://creativecommons.org/publicdomain/zero/1.0/).

Este episódio e a imagem utilizada estão licenciados nos termos da licença: Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0), cujo texto integral pode ser lido aqui. Estamos abertos a licenciar para permitir outros tipos de utilização, contactem-nos para validação e autorização.

29 Oct 2020 10:45pm GMT

I know I'm very late, but I really wanted to talk about this year Google Summer of Code.

For the third year in a row, I was a mentor. And this year I have a huge deception to share. I'm really sad. This week, I'v received the GSoC 2011 t-shirt. They sent me the wrong size. XXXL. I can use it as a sleeping bag with my girlfriend. I'm really disappointed.

Hopefully, GSoC is not only about receiving a t-shirt. It is also about mentoring a student.

Nearly two years ago, I started working on a complete refactoring of GTG. The code was a mess, with a lot of duplicate everywhere, with two bugs appearing while you were trying to solve one, etc.

I abstracted the structure we were using in several places and started to write a library to handle those "Acyclical Directed Graphs". As usual, it appeared that development was taking longer than expected. Weeks turned into months. Then, when it started to look good, I discovered that I forgot one critical point: thread-awareness. I felt hopeless.

Because I didn't had the motivation to do that heavy work, I proposed it as a Summer of Code project to a very motivated student: Izidor Matušov^[1].

Words doesn't do any justice to the excellent work that Izidor did this summer. He's simply awesome. Some students are goods because they have previous experience. Izidor kicks asses. He learns so quickly, he's so assertive. The work was even harder than what we anticipated. But he managed to achieve everything, including feeding me with cookies at the Desktop Summit, where we met and had an awesome hacking week.

As much as I'm deceipted about the t-shirt, I'm delighted about the work achieved this summer. Izidor now knows GTG nearly as much as I do. He's taking initiatives, like organizing an online GTG hackfest on November 26th^[2] and he's a bug-answering machine.

Dear Google, GNOME foundation and Lanedo^[3], I would like to thank you. Thanks to your support:

1. I received a worthless piece of clothes that travelled half of the world in order to clean my cat's dirtiness.
2. GTG 0.2.9 should be released before the end of the year
3. GTG gained a new co-maintainer
4. I gained a new friend. And it probably worth everything else.

Congratulations, co-maintainer Izidor. And welcome to the community!

06 Nov 2011 6:23pm GMT

Part 1 : Introduction
Part 2 : Home is Desktop
Part 3 : There's no tray icon in GNOME !
Part 4 : Kill The Double Click

When I started installing the best desktop possible for Marie and Jean, we were still in the GNOME 2.X era. GNOME 3 solved my previous concerns. No in the way I envisioned it, but solved them anyway. No more desktop icons, no more tray icons.

But now that I'm introducing Marie and Jean to GNOME 3, I still have some concerns. And one of that main concern is the infamous double-click!

Do you remember? Jean is a very brilliant mind, even though he never used a computer during his whole life. As a reasoning scientist, he was trying to find the logic behind my teaching.

During one of our first lesson, "Using the mouse", the conversation went like this:
- How do I know if I have to click or double click?
- Well, you double-click on icon and simple click on links and buttons.
- How do I know what is a button or an icon?
- …

Since that time, I've tried many times to find a logic behind single or double clicking. There is not. You have to learn it by experience. And it is totally, utterly pointless.

I also realized that a single click was something really hard for Jean. Achieving to click on a given point without moving the mouse is really hard for older people. Then, ask them to click twice, with a completely arbitrary speed, without moving the mouse, not to quickly, not to slowly. Impossible.

Marie, on her side, was double-clicking everywhere. And, surprisingly, it works most of the time.

So, why do we have double-click in some places? Because we want to be able to select an item without "activating" it. How often does it happen? Never for Jean. Very rarely for Marie.

To summarize, we are making the most frequent action very hard to nearly impossible in order to allow a very rare action?

I tried to disable completely the double-clicking in Nautilus.

Do you know what?

It works. Even for me. I had chronic pain in my hand and disabling double-click was a relief. I explained to Marie to never double-click anymore. She's still double-clicking from time to time but everything works even better than before. Jean was eventually able to launch a file from within Nautilus.

Selection of one or multiple file with single mouse click

What about selection of files? I explained to Marie to draw a square with the mouse. And, yes, she found that absolutely intuitive. The only drawback I found so far was the inconsistency with lists, where double-clicking is still required. Marie called me one day because she tried to play a specific song in Rhythmbox. It wasn't working. I realize that she had to double-click on the song. "But you told me to never double click anymore!". Sorry Marie.

I'm myself incredibly frustrated by any system that requires double-click. Why do we still have double-click by default in GNOME3?


Part 1 : Introduction
Part 2 : Home is Desktop
Part 3 : There's no tray icon in GNOME !
Part 4 : Kill The Double Click


Picture by Dave Dugdale

05 Nov 2011 12:17pm GMT

Si vous deviez me décrire en deux mots, nul doute que fourbe et profiteur vous viendraient spontanément à la bouche. Paresseux, parasite et inutile suivraient de près. Et j'en suis fier. J'en ai même fait mon mode de vie.

Ma technique est simple mais éprouvée. Je croise un inconnu dans la rue à l'air affable. Tenez, prenez ce jeune homme à l'allure dynamique. Il s'appelle Jean, c'est ma prochaine victime. Il ne se doute encore de rien mais j'irai dormir dans le lit de sa femme tout en vidant son frigo.

Au premier abord, je fais le numéro du sympa-sociable, les circonstances m'ont conduit dans la rue, où j'ère sans but précis, mais je ne me plains pas, je ne quémande rien, au contraire, je refuse tout geste de pitié trop ostentatoire. J'ai ma fierté.

Lorsque Jean se propose de m'emmener manger à la maison, juste pour la soirée, je fais d'abord mine de ne pas être intéressé. Mais mes yeux acquiescent et Jean, en rigolant, insiste, me forçant presqu'à le suivre. Inutile de vous dire que c'est ce que j'attendais mais la victime doit croire qu'elle a l'initiative, c'est primordial.

Martine, la femme de Jean, n'est que moyennement contente de cet imprévu. Qu'à cela ne tienne, je fais mon charmeur, je séduis tout en ayant l'air de ne pas vouloir déranger. Je fais également un peu le pitre pour la dérider.

Et ça marche. Avant la fin de la soirée, elle discutera avec moi plus qu'avec Jean lui-même, ce dernier étant parfaitement inconscient du destin de proie que je lui réserve. De manière indirecte, je fais comprendre que je n'ai nul part où aller. Jean et Martine n'ont pas le cœur de me renvoyer seul dans le froid de la nuit. Ils se proposent donc de m'héberger, juste pour une nuit. Tandis que je m'installe confortablement sur le sofa, j'entends Martine descendre l'escalier. Elle est en déshabillé, prête à aller au lit.

- « Bonne nuit ! » me lance-t-elle avec un sourire innocent avant de remonter dare-dare dans sa chambre.

Je ricane. Je n'ai même pas eu besoin de répondre. Une seule soirée me suffit. Homme ou femme, nul ne me résiste. Je suis comme ça moi.

Bien entendu, le « seulement pour une nuit » se prolongera. Je commencerai doucement à faire comprendre mes goûts précis, envoyant Jean au supermarché afin de m'acheter ce que je souhaite. Lorsqu'elle rentre du travail, Martine a à peine un regard pour Jean. Elle se rue à l'intérieur pour voir comment je vais. Pendant ce temps-là, je me prélasse sur le canapé, je me balade un peu. Avec mon air faussement négligent, j'ai pris soin de casser quelques bibelots auxquels ils tenaient beaucoup, par pure cruauté.

Lorsque Jean partit quelques jours dans sa famille à l'étranger, je n'hésitai pas: je me glissai une nuit dans le lit de Martine, sans même lui demander, sans même m'annoncer. Elle prit un air faussement surpris mais je sais qu'elle n'attendait que cela. Elles sont toutes les mêmes. Jean nous a surpris en rentrant plus tôt. Cela ne lui a pas plu. Il m'a dit qu'il m'avait sorti de la rue, qu'il n'acceptait pas cela.

Par méchanceté, j'ai répondu en déféquant sur la moquette du salon. Il a pu tout nettoyer. Il n'était vraiment pas content mais Martine a fini par le convaincre de me garder et d'exercer le moindre de mes désirs.

Il faut dire qu'ils sont vraiment bien mes deux esclaves. Je dors dans leur lit, ils me nourrissent, nettoient sans que je n'aie besoin de faire attention à rien. Quoi que je fasse, ils me regardent avec un air attendri et me trouvent adorable. Même au milieu de la nuit, il suffit que je me mette à miauler pour qu'ils s'enquièrent immédiatement de mes besoins.

Des esclaves aussi dociles, c'est rare. Je vais les garder encore quelques temps.

28 Oct 2011 4:59pm GMT