16 Feb 2017
Knowing that collectd is a daemon that collects system and applications metrics and that Gnocchi is a scalable timeseries database, it sounds like a good idea to combine them together. Cheery on the cake: you can easily draw charts using Grafana.
While it's true that Gnocchi is well integrated with OpenStack, as it comes from this ecosystem, it actually works standalone by default. Starting with the 3.1 version, it is now easy to send metrics to Gnocchi using collectd.
What we'll need to install to accomplish this task is:
How you install them does not really matter. If they are packaged by your operating system, go ahead. For Gnocchi and collectd-gnocchi, you can also use pip:
# pip install gnocchi[file,postgresql] […] Successfully installed gnocchi-3.1.0 # pip install collectd-gnocchi Collecting collectd-gnocchi Using cached collectd-gnocchi-1.0.1.tar.gz […] Installing collected packages: collectd-gnocchi Running setup.py install for collectd-gnocchi ... done Successfully installed collectd-gnocchi-1.0.1
The detailed installation procedure for Gnocchi is detailed in the documentation. It among other things explains which flavors are available - here I picked PostgreSQL and the file driver to store the metrics.
Gnocchi is simple to configure and is again documented. The default configuration file is
/etc/gnocchi/gnocchi.conf - you can generate it with
gnocchi-config-generator if needed. However, it also possible to specify another configuration file by appending the
--config-file option to any command line
In Gnocchi's configuration file, you need to set the
indexer.url configuration option to point an existing PostgreSQL database and set
storage.file_basepath to an existing directory to store your metrics (the default is
/var/lib/gnocchi). That gives something like:
url = postgresql://root:p4assw0rd@localhost/gnocchi
file_basepath = /var/lib/gnocchi
Once done, just run the
gnocchi-upgrade command to initialize the index and storage.
Collectd provides a default configuration file that loads a bunch of plugin by default, that will meter all sort of metrics on your computer. You can check the documentation online to see how to disable or enable plugins.
As the collectd-gnocchi plugin is written in Python, you'll need to enable the Python plugin and load the collectd-gnocchi module:
That is enough to enable the storage of metrics in Gnocchi.
Running the daemons
Once everything is configured, you can launch
gnocchi-metricd and the
$ gnocchi-metricd 2017-01-26 15:22:49.018 15971 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed. […] # In another terminal $ gnocchi-api --port 8041 […] STARTING test server gnocchi.rest.app.build_wsgi_app Available at http://127.0.0.1:8041/ […]
Once that's done, you can start
$ collectd # Or to run in foreground with a different configuration file: # $ collectd -C collectd.conf -f
If you have any problem launchding colllectd, check syslog for more information: there might be an issue loading a module or plugin.
If no error are printed, then everythin's working fine and you soon should see gnocchi-api printing some requests such as:
127.0.0.1 - - [26/Jan/2017 15:27:03] "POST /v1/resource/collectd HTTP/1.1" 409 113 127.0.0.1 - - [26/Jan/2017 15:27:03] "POST /v1/batch/resources/metrics/measures?create_metrics=True HTTP/1.1" 400 91
Enjoying the result
Once everything runs, you can access your newly created resources and metric by using the gnocchiclient. It should have been installed as a dependency of collectd_gnocchi, but you can also install it manually using
pip install gnocchiclient.
If you need to specify a different endpoint you can use the
--endpoint option (which default to http://localhost:8041). Do not hesitate to check the
--help option for more information.
$ gnocchi resource list --details +---------------+----------+------------+---------+----------------------+---------------+----------+----------------+--------------+---------+-----------+ | id | type | project_id | user_id | original_resource_id | started_at | ended_at | revision_start | revision_end | creator | host | +---------------+----------+------------+---------+----------------------+---------------+----------+----------------+--------------+---------+-----------+ | dd245138-00c7 | collectd | None | None | dd245138-00c7-5bdc- | 2017-01-26T14 | None | 2017-01-26T14: | None | admin | localhost | | -5bdc-94f8-26 | | | | 94f8-263e236812f7 | :21:02.297466 | | 21:02.297483+0 | | | | | 3e236812f7 | | | | | +00:00 | | 0:00 | | | | +---------------+----------+------------+---------+----------------------+---------------+----------+----------------+--------------+---------+-----------+ $ gnocchi resource show collectd:localhost +-----------------------+-----------------------------------------------------------------------+ | Field | Value | +-----------------------+-----------------------------------------------------------------------+ | created_by_project_id | | | created_by_user_id | admin | | creator | admin | | ended_at | None | | host | localhost | | id | dd245138-00c7-5bdc-94f8-263e236812f7 | | metrics | interface-en0@if_errors-0: 5d60f224-2e9e-4247-b415-64d567cf5866 | | | interface-en0@if_errors-1: 1df8b08b-555a-4cab-9186-f9b79a814b03 | | | interface-en0@if_octets-0: 491b7517-7219-4a04-bdb6-934d3bacb482 | | | interface-en0@if_octets-1: 8b5264b8-03f3-4aba-a7f8-3cd4b559e162 | | | interface-en0@if_packets-0: 12efc12b-2538-45e7-aa66-f8b9960b5fa3 | | | interface-en0@if_packets-1: 39377ff7-06e8-454a-a22a-942c8f2bca56 | | | interface-en1@if_errors-0: c3c7e9fc-f486-4d0c-9d36-55cea855596a | | | interface-en1@if_errors-1: a90f1bec-3a60-4f58-a1d1-b3c09dce4359 | | | interface-en1@if_octets-0: c1ee8c75-95bf-4096-8055-8c0c4ec8cd47 | | | interface-en1@if_octets-1: cbb90a94-e133-4deb-ac10-3f37770e32f0 | | | interface-en1@if_packets-0: ac93b1b9-da71-4876-96aa-76067b35c6c9 | | | interface-en1@if_packets-1: 2f8528b2-12ae-4c4d-bec7-8cc987e7487b | | | interface-en2@if_errors-0: ddcf7203-4c49-400b-9320-9d3e0a63c6d5 | | | interface-en2@if_errors-1: b249ea42-01ad-4742-9452-2c834010df71 | | | interface-en2@if_octets-0: 8c23013a-604e-40bf-a07a-e2dc4fc5cbd7 | | | interface-en2@if_octets-1: 806c1452-0607-4b56-b184-c4ffd48f52c0 | | | interface-en2@if_packets-0: c5bc6103-6313-4b8b-997d-01930d1d8af4 | | | interface-en2@if_packets-1: 478ae87e-e56b-44e4-83b0-ed28d99ed280 | | | load@load-0: 5db2248d-2dca-401e-b2e2-bbaee23b623e | | | load@load-1: 6f74ac93-78fd-4a74-a47e-d2add487a30f | | | load@load-2: 1897aca1-356e-4791-907f-512e516992b5 | | | memory@memory-active-0: 83944a85-9c84-4fe4-b471-1a6cf8dce858 | | | memory@memory-free-0: 0ccc7cfa-26a5-4441-a15f-9ebb2aa82c6d | | | memory@memory-inactive-0: 63736026-94c4-47c5-8d6f-a9d89d65025b | | | memory@memory-wired-0: b7217fd6-2cdc-4efd-b1a8-a1edd52eaa2e | | original_resource_id | dd245138-00c7-5bdc-94f8-263e236812f7 | | project_id | None | | revision_end | None | | revision_start | 2017-01-26T14:21:02.297483+00:00 | | started_at | 2017-01-26T14:21:02.297466+00:00 | | type | collectd | | user_id | None | +-----------------------+-----------------------------------------------------------------------+ % gnocchi metric show -r collectd:localhost load@load-0 +------------------------------------+-----------------------------------------------------------------------+ | Field | Value | +------------------------------------+-----------------------------------------------------------------------+ | archive_policy/aggregation_methods | min, std, sum, median, mean, 95pct, count, max | | archive_policy/back_window | 0 | | archive_policy/definition | - timespan: 1:00:00, granularity: 0:05:00, points: 12 | | | - timespan: 1 day, 0:00:00, granularity: 1:00:00, points: 24 | | | - timespan: 30 days, 0:00:00, granularity: 1 day, 0:00:00, points: 30 | | archive_policy/name | low | | created_by_project_id | | | created_by_user_id | admin | | creator | admin | | id | 5db2248d-2dca-401e-b2e2-bbaee23b623e | | name | load@load-0 | | resource/created_by_project_id | | | resource/created_by_user_id | admin | | resource/creator | admin | | resource/ended_at | None | | resource/id | dd245138-00c7-5bdc-94f8-263e236812f7 | | resource/original_resource_id | dd245138-00c7-5bdc-94f8-263e236812f7 | | resource/project_id | None | | resource/revision_end | None | | resource/revision_start | 2017-01-26T14:21:02.297483+00:00 | | resource/started_at | 2017-01-26T14:21:02.297466+00:00 | | resource/type | collectd | | resource/user_id | None | | unit | None | +------------------------------------+-----------------------------------------------------------------------+ $ gnocchi measures show -r collectd:localhost load@load-0 +---------------------------+-------------+--------------------+ | timestamp | granularity | value | +---------------------------+-------------+--------------------+ | 2017-01-26T00:00:00+00:00 | 86400.0 | 3.2705004391254193 | | 2017-01-26T15:00:00+00:00 | 3600.0 | 3.2705004391254193 | | 2017-01-26T15:00:00+00:00 | 300.0 | 2.6022800611413044 | | 2017-01-26T15:05:00+00:00 | 300.0 | 3.561742940080275 | | 2017-01-26T15:10:00+00:00 | 300.0 | 2.5605337960379466 | | 2017-01-26T15:15:00+00:00 | 300.0 | 3.837517851142473 | | 2017-01-26T15:20:00+00:00 | 300.0 | 3.9625948392427883 | | 2017-01-26T15:25:00+00:00 | 300.0 | 3.2690042162698414 | +---------------------------+-------------+--------------------+
As you can see, the command line works smoothly and can show you any kind of metric reported by collectd. In this case, it was just running on my laptop, but you can imagine it's easy enough to poll thousands of hosts with collectd and Gnocchi.
Bonus: charting with Grafana
Grafana configuration screen
You can then create a new dashboard by filling the forms as you wish. See this other screenshot for a nice example:
Charts of my laptop's load average
I hope everything is clear and easy enough. If you have any question, feel free to write something in the comment section!
16 Feb 2017 8:53am GMT
13 Feb 2017
Probably the biggest news of the last two weeks is that Boris's native HDMI audio driver is now on the mailing list for review. I'm hoping that we can get this merged for 4.12 (4.10 is about to be released, so we're too late for 4.11). We've tested stereo audio so far, no compresesd audio (though I think it should Just Work), and >2 channel audio should be relatively small amounts of work from here. The next step on HDMI audio is to write the alsalib configuration snippets necessary to hide the weird details of HDMI audio (stereo IEC958 frames required) so that sound playback works normally for all existing userspace, which Boris should have a bit of time to work on still.
I've also landed the vc4 part of the DSI driver in the drm-misc-next tree, along with a fixup. Working with the new drm-misc-next trees for vc4 submission is delightful -- once I get a review, I can just push code to the tree and it will magically (through Daniel Vetter's pull requests and some behind-the-scenes tools) go upstream at the appropriate time. I am delighted with the work that Daniel has been doing to make the DRM subsystem more welcoming of infrequent contributors and reducing the burden on us all of participating in the Linux kernel process.
In 3D land, the biggest news is that I've fixed a kernel oops (producing a desktop lockup) when CMA returns out of memory. Unfortunately, VC4 doesn't have an MMU, so we require that memory allocations for graphics be contiguous (using the Contiguous Memory Allocator), and to make things worse we have a limit of 256MB for CMA due to an addressing bug of the V3D part, so CMA returning out of memory is a serious and unfortunately frequent problem. I had a bug where a CMA failure would put the failed allocation into the BO cache, so if you asked for a BO of the same size soon enough, you'd get the bad BO back and crash. I've been unable to construct a good minimal testcase for it, but the patch is on the mailing list and in the rpi-4.4.y tree now.
I've also fixed a bug in the downstream tree's "firmware KMS" mode (where I use the closed source firmware for display, but open source 3D) where fullscreen 3D rendering would get a single frame displayed and then freeze.
In userspace, I've fixed a bit of multisample rendering (copies of MSAA depth buffers) that gets more of our regression testing working, and worked on some potential rasterization problems (the texwrap regression tests are failing due to texture filtering troubles, and I'm not sure if we're actually doing anything wrong or not because we're near the cutoff for the "how accurate does the filtering have to be?").
Coming up, I'm looking at fixing a cursor updates bug that Michael Zoran has found, and fixing up the DSI panel driver so that it can hopefully get into 4.12. Also, after a recent discussion with Eben, I've realized that we can actually scan out tiled buffers from the GPU, so we may get big performance wins for un-composited X if I can have Mesa coordinate with the kernel on producing tiled buffers.
13 Feb 2017 10:19pm GMT
10 Feb 2017
libinput has a couple of features that 'automagically' work on touchpads such as disable-while-typing and the lid switch triggered disabling of touchpads and disabling the touchpad when an external mouse is plugged in . But not all of these features make sense on all touchpads. For example, an Apple Magic Trackpad doesn't need disable-while-typing because unless you have a creative arrangement of input devices , the touchpad won't be where your palm is likely to hit it. Likewise, a Logitech T650 connected over a unifying receiver shouldn't get disabled when the laptop lid closes.
For this to work, libinput has some code to figure out whether a touchpad is internal or external. Initially we had some code to detect this but eventually moved this to the ID_INPUT_TOUCHPAD_INTEGRATION property now set by udev's hwdb (systemd 231 and later). Having it in the hwdb makes it quite trivial to override locally where the current rules are insufficient (and until the hwdb is fixed, thanks for filing a bug). We still have the fallback code though in case the tag is missing. On a sufficiently modern distribution, udevadm info /sys/class/input/event4 for your touchpad device node should show something like ID_INPUT_TOUCHPAD_INTEGRATION=internal.
So for any feature that libinput adds for touchpads, we only enable it where it makes sense. That's why your external touchpad doesn't trigger disable-while-typing or the lid switch.
 ok, I admit, this is something we should've left to the client, but now we have the feature.
 yes, I'm sure there's at least one person out there that uses the touchpad upside down in front of the keyboard and is now angry that libinput doesn't allow arbitrary rotation of the device combined with configurable dwt. I think of you every night I cry myself to sleep.
10 Feb 2017 12:27am GMT
06 Feb 2017
Last week-end, I was in Brussels, Belgium for the 2017 edition of the FOSDEM, one of the greatest open source developer conference.
This year, I decided to propose a talk about Gnocchi which was accepted in the Python devroom. The track was very wlell organized (thanks to Stéphane Wirtel) and I was able to present Gnocchi to a room full of Python developers!
I've explained why we created Gnocchi and how we did it, and finally briefly explained how to use it with the command-line interface or in a Python application using the SDK.
06 Feb 2017 10:54am GMT
03 Feb 2017
Seems that there was a rift in the spacetime that sucked away the video of my LCA talk, but the awesome NextDayVideo team managed to pull it back out. And there's still the writeup and slides available.
03 Feb 2017 12:00am GMT
02 Feb 2017
It's always difficult to know when to release, and we really wanted to do it earlier. But it seems that each week more awesome work was being done in Gnocchi, so we kept delaying it while having no pressure to push it out.
A photo posted by Julien Danjou (@juldanjou) on Jan 22, 2017 at 5:43am PST
I've made my own gnocchis to celebrate!
But now that the OpenStack cycle is finishing, even Gnocchi does not strictly follow it, it seemed to be a good time to cut the leash and leave this release be.
There are again some major new changes coming from 3.0. The previous version 3.0 was tagged in October and had 90 changes merged from 13 authors since 2.2. This 3.1 version have 200 changes merged from 24 different authors. This is a great improvement of our contributor base and our rate of change - even if our delay to merge is very low. Once again, we pushed usage of release notes to document user visible changes, and they can be read online.
Therefore, I am going to summary quickly the major changes:
The REST API authentication mechanism has been modularized. It's now simple to provide any authentication mechanism for Gnocchi as a plugin. The default is now a HTTP basic authentication mechanism that does not implement any kind of enforcement. The Keystone authentication is still available, obviously.
Batching has been improved and can now create metrics on the fly, reducing the latency needed when pushing measures to non-existing metrics. This is leveraged by the collectd-gnoccchi plugin for example.
The performance of Carbonara based backend has been largely improved. This is not really listed as a change as it's not user-visible, but an amazing work of profiling and rewriting code from Pandas to NumPy has been done. While Pandas is very developer-friendly and generic, using NumPy directly offers way more performance and should decrease
gnocchi-metricdCPU usage by a large factor.
The storage has been split into two parts: the storage of incoming new measures to be processed, and the storage and archival of aggregated metrics. This allows to use e.g. file to store new measures being sent, and once processed store them into e.g. Ceph. Before that change, all the new measures had to go into Ceph. While there's no specific driver yet for incoming measures, it's easy to envision a driver for systems like Redis or Memcached.
I will write more about some of these new features in the upcoming weeks, as they are very interesting for Gnocchi's users.
We are planning to run a scalability test and benchmarks using the ScaleLab in a few weeks if everything goes as planned. I will obviously share the result here, but we also submitted a talk for the next OpenStack Summit in Boston to present the results of our scalability and performance tests - hoping the session will be accepted.
I will also be talking about Gnocchi this Sunday at FOSDEM.
We don't have a very determined roadmap for Gnocchi during the next weeks. Sure we do have a few ideas on what we want to implement, but we are also very easily influenced by the requests of our user: therefore feel free to ask for anything!
02 Feb 2017 8:34pm GMT
01 Feb 2017
I merged a patchset from James Ye today to add support for switch events to libinput, specifically: lid switch events. This feature is scheduled for libinput 1.7.
First, what are switches and how are they different so keys? A key's state is transient with a neutral state of "key is up". The state itself is expected to change frequently. Switches don't always have a defined logical neutral state and the state changes only infrequently. This requires different handling in applications and thus libinput exposes a new interface (and capability) for switches.
The interface itself is trivial. A switch event has two properties, the switch type (e.g. "lid") and the switch state (on/off). See the libinput-debug-events source code for a simple code to print the state and type.
In libinput, we generally try to restrict ourselves to the cases we know how to handle. So in the first iteration, we'll support a single switch event: the lid switch. This is the toggle that changes when you close the lid on your laptop.
But libinput uses this internally too: touchpads are disabled automatically whenever the lid is closed. Indeed, this functionally was the main motivation for this patchset. On a number of devices, we get ghost touches when the lid is closed. Even though the touchpad is unreachable by the user interference with the screen still causes events, moving the pointer in unexpected ways and generally being a nuisance. Some trackpoints suffer from the same issue. But now that libinput knows about the lid switch it can transparently disable the touchpad whenever the lid is closed and thus discard the events.
Lid switches on some devices are unreliable. There are some devices where the lid is permanently closed and other devices where the lid can be closed, but we'll never see the open event. So we change behaviour based on a few factors. After all, no-one likes a dysfunctional touchpad because the lid switch is broken (if you do, seek help). For one, whenever we detect keyboard events while in logically closed state we'll assume that the lid is open after all and adjust state accordingly. Unless the lid switch is reliable, we don't sync the initial state. That's annoying for those who start libinput in closed mode, but it filters out all devices that set the lid switch to "on" and then never change again. On the Surface 3 devices we go even further: we know those devices needs a bit of hand-holding. So whenever we detect activity on the keyboard, we also write the EV_SW/SW_LID state to the device node, thus updating the kernel to be correct again (and thus help everyone else who may be listening).
The exact behaviours will likely change slightly over time as we have to deal with corner-cases one-by-one. But meanwhile, it's even easier for compositors to listen to switch events and users don't have to deal with ghost touches anymore. Many thanks to James Ye for implementing this.
01 Feb 2017 4:51am GMT
30 Jan 2017
Most of last week was spent switching my development environment over to a setup with no SD cards involved at all. This was triggered by yet another card failing, and I spent a couple of days off and on trying to recover it. I now have three scripts that build and swap my test environment between upstream 64-bit Pi3, upstream 32-bit Pi3, and downstream 32-bit Pi3, using just the closed source bootloader without u-boot or an SD card. Previously I was on Pi2 only (much slower for testing), and running downstream kernels was really difficult.
Once I got the new netboot system working, I tested and landed the NEON part of my tiling work (the big 208% download and 41% upload performance improvements). I'm looking forward to fixing up the clever tiling math parts soon.
I also tested and landed a few little compiler improvements. The nicest compiler improvement was turning on a switch that Marek added to gallium: We now run the GLSL compiler optimization loop exactly once (because it's required to avoid GLSL linker regressions), and rely on NIR to do the actual optimization after the GLSL linker has run. The GLSL IR is a terrible IR for doing optimization on (only a bit better than Mesa or TGSI IRs), and it's made worse by the fact that I wrote a bunch of its optimizations back before we had good datastructures available in Mesa and before we had big enough shaders that using good datastructures mattered. I'm horrified by my old code and can't wait to get it deleted (Timothy Arceri has been making progress on that front). Until we can actually delete it, though, cutting down the number of times we execute my old optimization passes should improve our compile times on complicated shaders.
Now that I have a good way to test the downstream kernel, I went ahead and made a giant backport of our current vc4 kernel code to the 4.9 branch. I hope the Foundation can get that branch shipped soon -- backporting to 4.9 is so much easier for me than old 4.4, and the structure of the downstream DT files makes it much clearer what there is left to be moved upstream.
Meanwhile, Michael Zoran has continued hacking on the staging VCHI code, and kernel reviewers were getting bothered by edits to code with no callers. Michael decided to solve that by submitting the old HDMI audio driver that feeds through the closed source firmware (I'm hoping we can delete this soon once Boris's work lands, though), and I pulled the V4L2 camera driver out of rpi-4.9.y and submitted that to staging as well. I unfortunately don't have the camera driver quite working yet, because when I modprobe it the network goes down. There are a ton of variables that have changed since the last time I ran the camera (upstream vs downstream, 4.10 vs 4.4, pi3 vs pi2), so it's going to take a bit of debugging before I have it working again.
Other news: kraxel from RH has resubmitted the SDHOST driver upstream, so maybe we can have wifi by default soon. Baruch Siach has submitted some fixes that I suspect get BT working. I've also passed libepoxy off to Emmanuele Bassi (long time GNOME developer) who has fixed it up to be buildable and usable on Linux again and converted it to the Meson build system, which appears to be really promising.
30 Jan 2017 8:13pm GMT
In order to read events and modify devices, libinput needs a file descriptor to the /dev/input/event node. But those files are only accessible by the root user. If libinput were to open these directly, we would force any process that uses libinput to have sufficient privileges to open those files. But these days everyone tries to reduce a processes privileges wherever possible, so libinput simply delegates opening and closing the file descriptors to the caller.
The functions to create a libinput context take a parameter of type struct libinput_interface. This is an non-opaque struct with two function pointers: "open_restricted" and "close_restricted". Whenever libinput needs to open or close a file, it calls the respective function. For open_restricted() libinput expects the caller to return an fd with the given flags.
In the simplest case, a caller can merely call open() and close(). This is what the debugging tools do (and the test suite). But obviously this means you have to run those as root. The main wayland compositors (weston, mutter, kwin, ...) instead forward the request to systemd-logind. That then opens the event node and returns the fd which is passed to libinput. And voila, the compositors don't need to run as root, libinput doesn't have to know how the fd is opened and everybody wins. Plus, logind will mute the fd on VT-switch, so we can't leak keyboard events.
In the X.org case it's a combination of the two. When the server runs with systemd-logind enabled, it will open the fd before the driver initialises the device. During the init stage, libinput asks the xf86-input-libinput driver to open the device node. The driver forwards the request to the server which simply returns the already-open fd. When the server runs without systemd-logind, the server opens the file normally with a standard open() call.
So in summary: you can easily run libinput without systemd-logind but you'll have to figure out how to get the required privileges to open device nodes. For anything more than a test or debugging program, I recommend using systemd-logind.
30 Jan 2017 12:00am GMT
27 Jan 2017
We're in the middle of the 1.7 development cycle and one of the features merged already is support for "wheel tilt", i.e. support for devices that don't have a separate horizontal wheel but instead rely on a tilt motion for horizontal event. Now, the way this is handled in the kernel is that the events are sent via REL_WHEEL (or REL_DIAL) so we don't actually need special code in libinput to handle tilt. But libinput tries to to make sense of input devices so the upper layers have a reliable base to build on - and that's why we need tilt-wheels to be handled.
For 'pointer axis' events (i.e. scroll events) libinput provides scroll sources. These specify how the scroll event was generated, allowing a caller to handle things accordingly. A finger-based scroll for example can trigger kinetic scrolling while a mouse wheel would not usually do so. The value for a pointer axis is also dependent on the scroll source - for continuous/finger based scrolling the value is in pixels. For a mouse wheel, the value is in degrees. This obviously doesn't work for a tilt event because degrees don't make sense in this context. So the new axis source is just that, an indicator that the event was caused by a wheel tilt rather than a rotation. Its value matches the default wheel rotation (i.e. 15 degrees) just to make use of it easier.
Of course, a device won't tell us whether it provides a proper wheel or just tilt. So we need a hwdb property and I've added that to systemd's repo. To make this work, set the MOUSE_WHEEL_TILT_HORIZONTAL and/or MOUSE_WHEEL_TILT_VERTICAL property on your hardware and you're off. Yay.
Patches for the wayland protocol have been merged as well, so this is/will be available to wayland clients.
27 Jan 2017 3:27am GMT
24 Jan 2017
I spent last week at linux.conf.au 2017 (the best open source conference), mostly talking to people and going to talks instead of acually hacking on vc4. I'd like to highlight a few talks you could go watch if you weren't there.
Jack Moffit gave an overview of what Servo is doing with Rust on parallel processing and process isolation in web engines. I've been hacking on this project in my spare time, and the community is delightful to work with. Check out his talk to see what I've been excited about recently.
I then spent the next session in the hallway chatting with him instead of going to Willy's excellent talk about how the Linux page cache works. One of the subjects of that talk makes me wonder: I've got these contiguous regions of GPU memory on VC4, could I use huge pages when we're mapping them for uploads to the GPU? Would they help enough to matter?
Dave Airlie gave a talk about the structure of the Vulkan drivers and his work with Bas on the Radeon Vulkan driver. It's awesome how quickly these two have been able to turn around a complete Vulkan driver, and it really validates our open source process and the stack we've built in Mesa.
Nathan Egge from Mozilla gave a talk on accelerating JPEG decode using the GPU. I thought it was going to be harder to do than he made it out to be, so it was pretty cool to see the performance improvements he got. The only sad part was that it isn't integrated into WebRender yet as far as I can see.
Karen Sandler talked about the issue of what happens to our copyrights when we die, from experience in working with current project maintainers on relicensing efforts. I think this is a really important talk for anyone who writes copyleft software. It actually softened my stance on CLAs (!) but is mostly a call to action for "Nobody has a good process yet for dealing with our copyrights after death, let's work on this."
And, finally, I think the most important talk of the conference to me was Daniel Vetter's Maintainers Don't Scale (no video yet, just a blog post). The proliferation of boutique trees in the kernel with "overworked" maintainers either ignoring or endlessly bikeshedding the code of outside contributors is why I would never work on the Linux kernel if I wasn't paid to do so. Daniel actually offers a way forward from the current state of maintainership in the kernel, with group maintainership (a familiar model in healthy large projects) to replace DRM's boutique trees and a reinforcement of the cooperative values that I think are why we all work on free software. I would love to be able to shut down my boutique DRM and ARM platform trees.
In VC4, I did still manage to write a couple of kernel fixes in response to a security report, and rebase DSI on 4.10 and respin some of its patches for resubmission. I'm still hoping for DSI in 4.11, but as usual it depends on DT maintainer review (no response for over a month). I also spent a lot of time with Keith talking about ARM cache coherency questions and trying experiments on my NEON tiling code.
While I was mostly gone last week, though, Boris has been working on finish off my HDMI audio branch. He now has a branch that has a seriously simplified architecture compared to what I'd talked about with larsc originally (*so* much cleaner than what I was modeling on), and he has it actually playing some staticy audio. He's now debugging what's causing the static, including comparing dumps of raw audio between VC4 and the firmware, along with the register settings of the HDMI and DMA engines.
24 Jan 2017 6:51pm GMT
20 Jan 2017
This is the write-up of my talk at LCA 2017 in Hobart. It's not exactly the same, because this is a blog and not a talk, but the same contents. The slides for the talk are here, and I will link to the video as soon as it is available. Update: Video is now uploaded.
Linux Kernel Maintainers
First let's look at how the kernel community works, and how a change gets merged into Linus Torvalds' repository. Changes are submitted as patches to mailing list, then get some review and eventually get applied by a maintainer to that maintainer's git tree. Each maintainer then sends pull request, often directly to Linus. With a few big subsystems (networking, graphics and ARM-SoC are the major ones) there's a second or third level of sub-maintainers in. 80% of the patches get merged this way, only 20% are committed by a maintainer directly.
Most maintainers are just that, a single person, and often responsible for a bunch of different areas in the kernel with corresponding different git branches and repositories. To my knowledge there are only three subsystems that have embraced group maintainership models of different kinds: TIP (x86 and core kernel), ARM-SoC and the graphics subsystem (DRM).
The radical change, at least for the kernel community, that we implemented over a year ago for the Intel graphics driver is to hand out commit rights to all regular contributors. Currently there are 19 people with commit rights to the drm-intel repository. In the first year of ramp-up 70% of all patches are now committed directly by their authors, a big change compared to how things worked before, and still work everywhere else outside of the graphics subsystem. More recently we also started to manage the drm-misc tree for subsystem wide refactorings and core changes in the same way.
I've covered the details of the new process in my Kernel Recipes talk "Maintainers Don't Scale", and LWN has covered that, and a few other talks, in their article on linux kernel maintainer scalability. I also covered this topic at the kernel summit, again LWN covered the group maintainership discussion. I don't want to go into more detail here, mostly because we're still learning, too, and not really experts on commit rights for everyone and what it takes to make this work well. If you want to enjoy what a community does who really has this all figured out, watch Emily Dunham's talk "Life is better with Rust's community automation" from last year's LCA.
What we are experts on is the Linux Kernel's maintainer model - we've run things for years with the traditional model, both as single maintainers and small groups, and now gained the outside perspective by switching to something completely different. Personally, I've come to believe that the maintainer model as implemented by the kernel community just doesn't scale. Not in the technical sense of big-O scalability, because obviously the kernel community scales to a rather big size. Much larger organizations, entire states are organized in a hierarchical way, the kernel maintainer hierarchy is not anything special. Besides that, git was developed specifically to support the Linux maintainer hierarchy, and git won. Clearly, the linux maintainer model scales to big numbers of contributors. Where I think it falls short is the constant factor of how efficiently contributions are reviewed and merged, especially for non-maintainer contributors. Which do 80% of all patches.
Cult of Busy
The first issue that routinely comes out when talking about maintainer topics is that everyone is overloaded. There's a pervasive spirit in our industry (especially in the US) hailing overworked engineers as heroes, with an entire "cult of busy" around. If you have time, you're a slacker and probably not worth it. Of course this doesn't help when being a maintainer, but I don't believe it's a cause of why the Linux maintainer model doesn't work. This cult of busy leads to burnout, which is in my opinion a prime risk when you're an open source person. Personally I've gone through a few difficult phases until I understood my limits and respected them. When you start as a maintainer for 2-3 people, and it increases to a few dozen within a couple of years, then getting a bit overloaded is rather natural - it's a new job, with a different set of responsibilities and I had no clue about a lot of things. That's no different from suddenly being a leader of a much bigger team anywhere else. A great talk on this topic is "What part of "… for life" don't you understand?" from Jacob Kaplan-Moss since it's by a former maintainer. It also contains a bunch of links to talks on burnout specifically. Ignoring burnout is not healthy, or not knowing about the early warning signs, it is rampant in our communities, but for now I'll leave it at that.
Boutique Trees and Bus Factors
The first issue I see is how maintainers usually are made: You scratch an itch somewhere, write a bit of code, suddenly a few more people find it useful, and "tag" you're the maintainer. On top, you often end up being stuck in that position "for life". If the community keeps growing, or your maintainer becomes otherwise busy with work&life, you have your standard-issue overloaded bottleneck.
That's the point where I think the kernel community goes wrong. When other projects reach this point they start to build up a more formal community structure, with specialized roles, boards for review and other bits and pieces. One of the oldest, and probably most notorious, is Debian with its constitution. Of course a small project doesn't need such elaborate structures. But if the goal is world domination, or at least creating something lasting, it helps when there's solid institutions that cope with people turnover. At first just documenting processes and roles properly goes a long way, long before bylaws and codified decision processes are needed.
The kernel community, at least on the maintainer side, entirely lacks this.
What instead most often happens is that a new set of ad-hoc, chosen-by-default maintainers start to crop up in a new level of the hierarchy, below your overload bottleneck. Because becoming your own maintainer is the only way to help out and to get your own features merged. That only perpetuates the problem, since the new maintainers are as likely to be otherwise busy, or occupied with plenty of other kernel parts already. If things go well that area becomes big, and you have another git tree with another overloaded maintainer. More often than not people move around, and accumulate small bits allover under their maintainership. And then the cycle repeats.
The end result is a forest of boutique trees, each covering a tiny part of the project, maintained by a bunch of notoriously overloaded people. The resulting cross-tree coordination issues are pretty impressive - in the graphics subsystem we fairly often end up with with simple drivers that somehow need prep patches in 5 different trees before you can even land that simple driver in the graphics tree.
Unfortunately that's not the bad part. Because these maintainers are all busy with other trees, or their work, or life in general, you're guaranteed that one of them is not available at any given time. Worse, because their tree has relatively little activity because it covers a small area, many only pick up patches once per kernel release, which means a built-in 3 month delay. That's all because each tree and area has just one maintainer. In the end you don't even need the proverbial bus to hit anyone to feel the pain of having a single point of failure in your organization - there's so many maintainer trees around that that absence always happens, and constantly.
Of course people get fed up trying to get features merged, and often the fix is trying to become a maintainer yourself. That takes a while and isn't easy - only 20% of all patches are authored by maintainers - and after the new code landed it makes it all worse: Now there's one more semi-absent maintainer with one more boutique tree, adding to all the existing troubles.
Checks and Balances
All patches merged into the Linux kernel are supposed to be reviewed, and rather often that review is only done by the maintainers who merges the patch. When maintainers send out pull requests the next level of maintainers then reviews those patch piles, until they land in Linus' tree. That's an organization where control flows entirely top-down, with no checks and balances to reign in maintainers who are not serving their contributors well. History of dicatorships tells us that despite best intentions, the end result tends to heavily favour the few over the many. As a crude measure for how much maintainers subject themselves to some checks&balances by their peers and contributors I looked at how many patches authored and committed by the same person (probably a maintainer) do not also carry a reviewed or acked tag. For the Intel driver that's less than 3%. But even within the core graphics code it's only 5%, and that covers the time before we started to experiment with commit rights for that area. And for the graphics subsystem overall the ratio is still only about 25%, including a lot of drivers with essentially just one contributor, who is always volunteered as the maintainer, and hence somewhat natural that those maintainers lack reviewers.
Outside of graphics only roughly 25% of all patches written by maintainers are reviewed by their peers - 75% of all maintainer patches lack any kind of recorded peer review, compared to just 25% for graphics alone. And even looking at core areas like
mm/ the ratio is only marginally better at about 30%. In short, in the kernel at large, peer review of maintainers isn't the norm.
And there's nothing outside of the maintainer hierarchy that could provide some checks and balance either. The only way to escalate disagreement is by starting a revolution, and revolutions tend to be long, drawn-out struggles and generally not worth it. Even Debian only recently learned that they lack a way to depose maintainers, and that maybe going maintainerless would be easier (again, LWN has you covered).
Of course the kernel is not the only hierarchy where there's no meaningful checks and balances. Professor at universities, and managers at work are in a fairly similar position, with minimal options for students or employers to meaningfully appeal decisions. But that's a recognized problem, and at least somewhat countered by providing ways to provide anonymous feedback, often through regular surveys. The results tend to not be all that significant, but at least provide some control and accountability to the wider masses of first-level dwellers in the hierarchy. In the kernel that amounts to about 80% of all contributions, but there's no such survey. On the contrary, feedback sessions about maintainer happiness only reinforce the control structure, with e.g. the kernel summit featuring an "Is Linus happy?" session each year.
Another closely related aspect to all this is how a project handles personal conflicts between contributors. For a very long time Linux didn't have any formal structures in this area either, with the only options available to unhappy people to either take it or leave it. Well, or usurping a maintainer with a small revolution, but that's not really an option. For two years we've now had the "Code of Conflict", which de facto just throws up its hands and declares that conflict are the normal outcome, essentially just encoding the status quo. Refusing to handle conflicts in a project with thousands of contributors just doesn't work, except that it results in lots of frustration and ultimately people trying to get away. Again, the lack of a poised board to enforce a strong code of conduct, independent of the maintainer hierarchy, is in line with the kernel community unwillingness to accept checks and balances.
Mesh vs. Hierarchy
The last big issue I see with the Linux kernel model, featuring lots of boutique trees and overloaded maintainer, is that it seems to harm collaboration and integration of new contributors. In the Intel graphics, driver maintainers only ever reviewed a small minority of all patches over the last few years, with the goal to foster direct collaboration between contributors. Still, when a patch was stuck, maintainers were the first point of contact, especially, but not only, for newer contributors. No amount of explaining that only the lack of agreement with the reviewer was the gating factor could persuade people to fully collaborate on code reviews and rework the code, tests and documentation as needed. Especially when they're coming with previous experience where code review is more of a rubber-stamp step compared to the distributed and asynchronous pair-programming it often resembles in open-source. Instead, new contributors often just ended up falling back to pinging maintainers to make a decision or just merge the patches as-is.
Giving all regular contributors commit rights and fully trusting them to do the right thing entirely fixed that: If the reviewer or author have commit rights there's no easy excuse anymore to involve maintainers when the author and reviewer can't reach agreement. Of course that requires a lot of work in mentoring people, making sure requirements for merging are understood and documented, and automating as much as possible to avoid screw ups. I think maintainers who lament their lack of review bandwidth, but also state they can't trust anyone else aren't really doing their jobs.
At least for me, review isn't just about ensuring good code quality, but also about diffusing knowledge and improving understanding. At first there's maybe one person, the author (and that's not a given), understanding the code. After good review there should be at least two people who fully understand it, including corner cases. And that's also why I think that group maintainership is the only way to run any project with more than one regular contributor.
On the topic of patch review and maintainers, there's also the habit of wholesale rewrites of patches written by others. If you want others to contribute to your project, then that means you need to accept other styles and can't enforce your own all the time. Merging first and polishing later recognizes new contributions, and if you engage newcomers for the polish work they tend to stick around more often. And even when a patch really needs to be reworked before merging it's better to ask the author to do it: Worst case they don't have time, best case you've improved your documentation and training procedure and maybe gained a new regular contributor on top.
A great take on the consequences of having fixed roles instead of trying to spread responsibilities more evenly is Alice Goldfuss' talk "Rock Stars, Builders, and Janitors: You're doing it wrong". I also think that rigid roles present a bigger bar for people with different backgrounds, hampering diversity efforts and in the spirit of Sarah Sharps post on what makes a good community, need to be fixed first.
Towards a Maintainer's Manifest
I think what's needed in the end is some guidelines and discussions about what a maintainer is, and what a maintainer does. We have ready-made licenses to avoid havoc, there's code of conducts to copypaste and implement, handbooks for building communities, and for all of these things, lots of conferences. Maintainer on the other hand you become by accident, as a default. And then everyone gets to learn how to do it on their own, while hopefully not burning too many bridges - at least I myself was rather lost on that journey at times. I'd like to conclude with a draft on a maintainer's manifest.
It's About the People
If you're maintainer of a project or code area with a bunch of full time contributors (or even a lot of drive-by contributions) then primarily you deal with people. Insisting that you're only a technical leader just means you don't acknowledge what your true role really is.
And then, trust them to do a good job, and recognize them for the work they're doing. The important part is to trust people just a bit more than what they're ready for, as the occasional challenge, but not too much that they're bound to fail. In short, give them the keys and hope they don't wreck the car too badly, but in all cases have insurance ready. And insurance for software is dirt cheap, generally a
git revert and the maintainer profusely apologizing to everyone and taking the blame is all it takes.
Recognize Your Power
You're a maintainer, and you have essentially absolute power over what happens to your code. For successful projects that means you can unleash a lot of harm on people who for better or worse are employed to deal with you. One of the things that annoy me the most is when maintainers engage in petty status fights against subordinates, thinly veiled as technical discussions - you end up looking silly, and it just pisses everyone off. Instead recognize your powers, try to stay on the good side of the force and make sure you share it sufficiently with the contributors of your project.
Accept Your Limits
At the beginning you're responsible for everything, and for a one-person project that's all fine. But eventually the project grows too much and you'll just become a dictator, and then failure is all but assured because we're all human. Recognize what you don't do well, build institutions to replace you. Recognize that the responsibility you initially took on might not be the same as that which you'll end up with and either accept it, or move on. And do all that before you start burning out.
Be a Steward, Not a Lord
I think one of key advantages of open source is that people stick around for a very long time. Even when they switch jobs or move around. Maybe the usual "for life" qualifier isn't really a great choice, since it sounds more like a mandatory sentence than something done by choice. What I object to is the "dictator" part, since if your goal is to grow a great community and maybe reach world domination, then you as the maintainer need to serve that community. And not that the community serves you.
Thanks a lot to Ben Widawsky, Daniel Stone, Eric Anholt, Jani Nikula, Karen Sandler, Kimmo Nikkanen and Laurent Pinchart for reading and commenting on drafts of this text.
20 Jan 2017 12:00am GMT
17 Jan 2017
In this blog post I promised I would get back to people who want to use the nvidia driver on an optimus laptop.
The set of xserver patches I blogged about last time have landed upstream and in Fedora 25 (in xorg-x11-server 1.19.0-3 and newer), allowing the nvidia driver packages to drop a xorg.conf snippet which will make the driver atuomatically work on optimus setups.
The negativo17.org nvidia packages now are using this, so if you install these, then the nvidia driver should just work on your laptop.
Note that you should only install these drivers if you actually have a supported (new enough) nvidia GPU. These drivers replace the libGL implementation, so installing them on a system without a nvidia GPU will cause things to break. This will be fixed soon by switching to libglvnd as the official libGL provider and having both mesa and the nvidia driver provide "plugins" for libglvnd. I've actually just completed building a new enough libglvnd + libglvnd enabled mesa for rawhide, so rawhide users will have libglvnd starting tomorrow.
17 Jan 2017 1:31pm GMT
16 Jan 2017
My day-to-day activities are still evolving around the Python programming language, as I continue working on the OpenStack project as part of my job at Red Hat. OpenStack is still the biggest Python project out there, and attract a lot of Python hackers.
Those last few years, however, things have taken a different turn for me when I made the choice with my team to rework the telemetry stack architecture. We decided to make a point of making it scale way beyond what has been done in the project so far.
I started to dig into a lot of different fields around Python. Topics you don't often look at when writing a simple and straight-forward application. It turns out that writing scalable applications in Python is not impossible, nor that difficult. There are a few hiccups to avoid, and various tools that can help, but it really is possible - without switching to another whole language, framework, or exotic tool set.
Working on those projects seemed to me like a good opportunity to share with the rest of the world what I learned. Therefore, I decided to share my most recent knowledge addition around distributed and scalable Python application in a new book, entitled The Hacker's Guide to Scaling Python (or Scaling Python, in short). The book should be released in a few months - fingers crossed.
And as the book is still a work-in-progress, I'll be happy to hear any remark, subject, interrogation or topic idea you might have or any particular angle you would like me to take in this book (reply in the comments section or shoot me an email). And if you'd like to get be kept updated on this book advancement, you can subscribe in the following form or from the book homepage.
The adventure of working on my previous book, The Hacker's Guide to Python, has been so tremendous and the feedback so great, that I'm looking forward releasing this new book later this year!
16 Jan 2017 4:00pm GMT
Last week I was on vacation, but the week before that I did some more work on figuring out Intel's Mesa CI system, and Mark Janes has started working on some better documentation for it. I now understand better how the setup is going to work, but haven't made much progress on actually getting a master running yet.
More fun, though, was finally taking a look at optimizing the tiled texture load/store code. This got started with Patrick Walton tweeting a link to a blog post on clever math for implementing texture tiling, given a couple of assumptions.
As with all GPUs these days, VC4 swizzles textures into a tiled layout so that when a cacheline is loaded from memory, it will cover nearby pixels in the Y direction as well as X, so that you are more likely to get cache hits for your neighboring pixels in drawing. And the tiling tends to be multiple levels, so that nearby cachelines are also nearby on the screen, reducing DRAM access latency.
For small textures on VC4, we have a single level of tiling: 4x4@32bpp blocks ("utiles") of raster order data, themselves arranged in up to a 4x4 block, and this mode is called LT. Once things go over that size, we go to T tiling, where we call a 4x4 LT block a subtile, and arrange subtiles either clockwise or counterclockwise order within a 2x2 block (a tile, which at this point is now 1kb of data), and tiles themselves are arranged left-to-right, then right-to-left, then left-to-right again.
The first thing I did was implement the blog post's clever math for LT textures. One of the nice features of the math trick is that it means I can do partial utile updates finally, because it skips around in the GPU's address space based on the CPU-side pixel coordinate instead of trying to go through GPU address space to update a 4x4 utile at a time. The downside of doing things this way is that jumping around in GPU address space means that our writes are unlikely to benefit from write combining, which is usually important for getting full write performance to GPU memory. It turned out, though, that the math was so much more efficient than what I was doing that it was still a win.
However, I found out that the clever math made reads slower. The problem is that, because we're write-combined, reads are uncached -- each load is a separate bus transaction to main memory. Switching from my old utile-at-a-time load using memcpy to the new code meant that instead of doing 4 loads using NEON to grab a row at a time, we were now doing 16 loads of 32 bits at a time, for what added up to a 30% performance hit.
Reads *shouldn't* be something that people do much, except that we still have some software fallbacks in X on VC4 ("core" unantialiased text rendering, for example, which I need to write the GLSL 1.20 shaders for), which involve doing a read/modify/write cycle on a texture. My first attempt at fixing the regression was just adding back a fast path that operates on a utile at a time if things are nicely utile-aligned (they generally are). However, some forced inlining of functions per cpp that I was doing for the unaligned case meant that the glibc memcpy call now got inlined back to being non-NEON, and the "fast" utile code ended up not helping loads.
Relying on details of glibc's implementation (their tradeoff for when to do NEON loads) and of gcc's implementation (when to go from memcpy calls to inlined 32-bits-at-a-time code) seems like a pretty bad idea, so I decided to finally write the NEON acceleration that Eben and I have talked about several times.
My first hope was that I could load a full cacheline with NEON's VLD. VLD1 only loads up to 1 "quadword" (16 bytes) at a time, so that doesn't seem like much help. VLD4 can load 64 bytes like we want, but it also turns AOS data into SOA in the process, and there's no corresponding "SOA-back-to-AOS store 8 or 16 bytes at a time" like we need to do to get things back into the CPU's strided representation. I tried VLD4+VST4 into a temporary, then doing my old untiling path on the cached temporary, but that still left me a few percent slower on loads than not doing any of this work at all.
Finally, I hit on using the VLDM instruction. It seems to be intended for stack loads/stores, but we can also use it to get 64 bytes of data in from memory untouched into NEON registers, and then I can use 4 (32bpp) or 8 (8 or 16bpp) VST1s to store it to the CPU side. With this, we get a 208.256% +/- 7.07029% (n=10) improvement to GetTexImage performance at 1024x1024. Doing the same NEON code for stores gave a 41.2371% +/- 3.52799% (n=10) improvement, probably mostly due to not calling into memcpy and having it go through its size/alignment-based memcpy path choosing process.
I'm not yet hitting full memory bandwidth, but this should be a noticeable improvement to X, and it'll probably help my piglit test suite runtime as well. Hopefully I'll get the code polished up and landed next week when I get back from LCA.
16 Jan 2017 10:28am GMT
11 Jan 2017
Hans de Goede: Xorg in Fedora-26 will use xorg-x11-drv-modesetting instead of -intel for all recent Intel GPUs
A while back Debian has switched to using the modesetting Xorg driver rather then the intel Xorg driver for Intel GPUs.
There are several good reasons for this, rather then repeating them I'm just going to point to the Debian announcement.
This blog post is to let all Fedora users know that starting with Fedora-26 / rawhide as of today, we are making the same change.
Note that the xorg-x11-drv-intel package has already been carrying a Fedora patch to not bind to the GPU on Skylake or newer, even before Debian announced this, this just makes the same change for older Intel GPUs.
For people who are using the now default GNOME3 on Wayland session, nothing changes, since Xwayland always uses glamor for X acceleration, just like the modesetting driver.
If you encounter any issues causes by this change, please file a bug in bugzilla.
The "for all recent Intel GPUs" in the subject of this blog post in practice means that we're making this change for gen4 and newer Intel GPUs.
11 Jan 2017 9:07am GMT