Under that name is a simple idea: making it easier to save, load, update and query objects in an object store.

I'm not the main developer for this piece of code, but contributed a large number of fixes to it, while porting a piece of code to it as a test of the API. Much of the credit for the design of this very useful library goes to Christian Hergert.

The problem

It's possible that you've already implemented a data store inside your application, hiding your complicated SQL queries in a separate file because they contain injection security issues. Or you've used the filesystem as the store and threw away the ability to search particular fields without loading everything in memory first.

Given that SQLite pretty much matches our use case - it offers good search performance, it's a popular thus well-documented project and its files can be manipulated through a number of first-party and third-party tools - wrapping its API to make it easier to use is probably the right solution.

The GOM solution

GOM is a GObject based wrapper around SQLite. It will hide SQL from you, but still allow you to call to it if you have a specific query you want to run. It will also make sure that SQLite queries don't block your main thread, which is pretty useful indeed for UI applications.

For each table, you would have a GObject, a subclass of GomResource, representing a row in that table. Each column is a property on the object. To add a new item to the table, you would simply do:

item = g_object_new (ITEM_TYPE_RESOURCE,
                     "column1", value1,
                     "column2", value2, NULL);
gom_resource_save_sync (item, NULL);

We have a number of features which try to make it as easy as possible for application developers to use gom, such as:

• Automatic table creation for string, string arrays, and number types as well as GDateTime, and transformation support for complex types (say, colours or images).
• Automatic database version migration, using annotations on the properties ("new in version")
• Programmatic API for queries, including deferred fetches for results

Currently, the main net gain in terms of lines of code, when porting SQLite, is the verbosity of declaring properties with GObject. That will hopefully be fixed by the GProperty work planned for the next GLib release.

The future

I'm currently working on some missing features to support a port of the grilo bookmarks plugin (support for column REFERENCES).

I will also be making (small) changes to the API to allow changing the backend from SQLite to a another one, such as XML, or a binary format. Obviously the SQL "escape hatches" wouldn't be available with those backends.

Don't hesitate to file bugs if there are any problems with the API, or its documentation, especially with respect to porting from applications already using SQLite directly. Or if there are bugs (surely, no).

Note that JavaScript support isn't ready yet, due to limitations in gjs.

¹: « SQLite don't hurt me, don't hurt me, no more »

17 Apr 2014 7:36am GMT

Things are moving forward for the Fedora Workstation project. For those of you who don't know about it, it is part of a broader plan to refocus Fedora around 3 core products with clear and distinctive usecase for each. The goal here is to be able to have a clear definition of what Fedora is and have something that for instance ISVs can clearly identify and target with their products. At the same time it is trying to move away from the traditional distribution model, a model where you primarily take whatever comes your way from upstream, apply a little duct tape to try to keep things together and ship it. That model was good in the early years of Linux existence, but it does not seem a great fit for what people want from an operating system today.

If we look at successful products MacOS X, Playstation 4, Android and ChromeOS the red thread between them is that while they all was built on top of existing open source efforts, they didn't just indiscriminately shovel in any open source code and project they could find, instead they decided upon the product they wanted to make and then cherry picked the pieces out there that could help them with that, developing what they couldn't find perfect fits for themselves. The same is to some degree true for things like Red Hat Enterprise Linux and Ubuntu. Both products, while based almost solely on existing open source components, have cherry picked what they wanted and then developed what pieces they needed on top of them. For instance for Red Hat Enterprise Linux its custom kernel has always been part of the value add offered, a linux kernel with a core set of dependable APIs.

Fedora on the other hand has historically followed a path more akin to Debian with a 'more the merrier' attitude, trying to welcome anything into the group. A metaphor often used in the Fedora community to describe this state was that Fedora was like a collection of Lego blocks. So if you had the time and the interest you could build almost anything with it. The problem with this state was that the products you built also ended up feeling like the creations you make with a random box of lego blocks. A lot of pointy edges and some weird looking sections due to needing to solve some of the issues with the pieces you had available as opposed to the piece most suited.

With the 3 products we are switching to a model where although we start with that big box of lego blocks we add some engineering capacity on top of it, make some clear and hard decisions on direction, and actually start creating something that looks and feels like it was made to be a whole instead of just assembled from a random set of pieces. So when we are planning the Fedora Workstation we are not just looking at what features we can develop for individual libraries or applications like GTK+, Firefox or LibreOffice, but we are looking at what we want the system as a whole to look like. And maybe most important we try our hardest to look at things from a feature/usecase viewpoint first as opposed to a specific technology viewpoint. So instead of asking 'what features are there in systemd that we can expose/use in the desktop being our question, the question instead becomes 'what new features do we want to offer our users in future versions of the product, and what do we need from systemd, the kernel and others to be able to do that'.

So while technologies such as systemd, Wayland, docker, btrfs are on our roadmap, they are not there because they are 'cool technologies', they are there because they provide us with the infrastructure we need to achieve our feature goals. And whats more we make sure to work closely with the core developers to make the technologies what we need them to be. This means for example that between myself and other members of the team we are having regular conversations with people such as Kristian Høgsberg and Lennart Poettering, and of course contributing code where possible.

To explain our mindset with the Fedora Workstation effort let me quickly summarize some old history. In 2001 Jim Gettys, one of the original creators of the X Window System did at talk a GUADEC in Sevile called 'Draining the Swamp'. I don't think the talk can be found online anywhere, but he outlined some of the same thoughts in this email reply to Richard Stallman some time later. I think that presentation has shaped the thinking of the people who saw it ever since, I know it has shaped mine. Jim's core message was that the idea that we can create a great desktop system by trying to work around the shortcomings or weirdness in the rest of the operating system was a total fallacy. If we look at the operating system as a collection of 100% independent parts, all developing at their own pace and with their own agendas, we will never be able to create a truly great user experience on the desktop. Instead we need to work across the stack, fixing the issues we see where they should be fixed, and through that 'drain the swamp'. Because if we continued to try to solve the problems by adding layers upon layers of workarounds and abstraction layers we would instead be growing the swamp, making it even more unmanageable. We are trying to bring that 'draining the swamp' mindset with us into creating the Fedora Workstation product.

With that in mind what is the driving ideas behind the Fedora Workstation? The Fedora Workstation effort is meant to provide a first class desktop for your laptop or workstation computer, combining a polished user interface with access to new technologies. We are putting a special emphasis on developers with our first releases, both looking at how we improve the desktop experience for developers, and looking at what tools we can offer to developers to let them be productive as quickly as possible. And to be clear when we say developers we are not only thinking about developers who wants to develop for the desktop or the desktop itself, but any kind of software developer or DevOPs out there.

The full description of the Fedora Workstation can be found here, but the essence of our plan is to create a desktop system that not only provides some incremental improvements over how things are done today, but which tries truly take a fresh look at how a linux desktop operating system should operate. The traditional distribution model, built up around software packages like RPM or Deb has both its pluses and minuses.
Its biggest challenge is probably that it creates a series of fiefdoms where a 3rd party developers can't easily target the system or a family of systems except through spending time very specifically supporting each one. And even once a developers decides to commit to trying to support a given system it is not clear what system services they can depend on always being available or what human interface design they should aim for. Solving these kind of issues is part of our agenda for the new workstation.

So to achieve this we have decided on a set of core technologies to build this solution upon. The central piece of the puzzle is the so called LinuxApps proposal from Lennart Poettering. LinuxApps is currently a combination of high level ideas and some concrete building blocks. In terms of the building blocks are technologies such as Wayland, kdbus, overlayfs and software containers. The ideas side include developing a permission system similar to what you for instance see Android applications employ to decide what rights a given application has and develop defined versioned library bundles that 3rd party applications can depend on regardless of the version of the operating system. On the container side we plan on expanding on the work Red Hat is doing with Docker and Project Atomic.

In terms of some of the other building blocks I think most of you already know of the big push we are doing to get the new Wayland display server ready. This includes work on developing core infrastructure like libinput, a new library for handling input devices being developed by Jonas Ådahl and our own Peter Hutterer. There is also a lot of work happening on the GNOME 3 side of things to make GNOME 3 Wayland ready. Jasper St.Pierre wrote up a great blog blog entry outlining his work to make GDM and the GNOME Shell work better with Wayland. It is an ongoing effort, but there is a big community around this effort as most recently seen at the West Cost Hackfest at the Endless Mobile office.

As I mentioned there is a special emphasis on developers for the initial releases. These includes both a set of small and big changes. For instance we decided to put some time into improving the GNOME terminal application as we know it is a crucial piece of technology for a lot of developers and system administers alike. Some of the terminal improvements can be seen in GNOME 3.12, but we have more features lined up for the terminal, including the return of translucency. But we are also looking at the tools provided in general and the great thing here is that we are able to build upon a lot of efforts that Red Hat is developing for the Red Hat product portfolio, like Software Collections which gives easy access to a wide range of development tools and environments. Together with Developers Assistant this should greatly enhance your developers experience in the Fedora Workstation. The inclusion of Software collections also means that Fedora becomes an even better tool than before for developing software that you expect to deploy on RHEL, as you can be sure that an identical software collection will be available on RHEL that you have been developing against on Fedora as software collections ensure that you can have the exact same toolchain and toolchain versions available for both systems.

Of course creating a great operating system isn't just about the applications and shell, but also about supporting the kind of hardware people want to use. A good example here is that we put a lot of effort into HiDPI support. HiDPI screens are not very common yet, but a lot of the new high end laptops coming out are using them already. Anyone who has used something like a Google Pixel or a Samsung Ativ Book 9 Plus has quickly come to appreciate the improved sharpness and image quality these displays brings. Due to the effort we put in there I have been very pleased to see many GNOME 3.12 reviews mentioning this work recently and saying that GNOME 3.12 is currently the best linux desktop for use with HiDPI systems due to it.

Another part of the puzzle for creating a better operating system is the software installation. The traditional distribution model often tended to try to bundle as many applications as possible as there was no good way for users to discover new software for their system. This is a brute force approach that assumes that if you checked the 'scientific researcher' checkbox you want to install a random collection of 100 applications useful for 'scientific researchers'. To me this is a symptom of a system that does not provide a good way of finding and installing new applications. Thanks to the ardent efforts of Richard Hughes we have a new Software Installer that keeps going from strength to strength. It was originally launched in Fedora 19, but as we move forward towards the first Fedora Workstation release we are enabling new features and adding polish to it. One area where we need to wider Fedora community to work with us is to increase the coverage of appdata files. Appdata files essentially contains the necessary metadata for the installer to describe and advertise the application in question, including descriptive text and screenshots. Ideally upstreams should come with their own appdata file, but in the case where they are not we should add them to the Fedora package directly. Currently applications from the GTK+ and GNOME sphere has relatively decent appdata coverage, but we need more effort into getting applications using other toolkits covered too.

Which brings me to another item of importance to the workstation. The linux community has for natural reasons been very technical in nature which has meant that some things that on other operating systems are not even a question has become defining traits on Linux. The choice of GUI development toolkits being one of these. It has been a great tool used by the open source community to shoot ourselves in the foot for many years now. So while users of Windows or MacOS X probably never ask themselves what toolkit was used to implement a given application, it seems to be a frequently asked one for linux applications. So we want to move away from it with the Workstation. So while we do ship the GNOME Shell as our interface and use GTK+ for developing tools ourselves, including spending time evolving the toolkit itself that does not mean we think applications written using for instance Qt, EFL or Java are evil and should be exorcised from the system. In fact if an application developer want to write an application for the linux desktop at all we greatly appreciate that effort regardless of what tools they decide to use to do so. The choice of development toolkits is a choice meant to empower developers, not create meaningless distinctions for the end user. So one effort we have underway is to work on the necessary theming and other glue code to make sure that if you run a Qt application under the GNOME Shell it feels like it belongs there, which also extends to if you are using accessibility related setups like the high contrast theme. We hope to expand upon that effort both in width and in depth going forward.

And maybe on a somewhat related note we are also trying to address the elephant in the room when it comes to the desktop and that is the fact that the importance of the traditional desktop is decreasing in favor of the web. A lot of things that you used to do locally on your computer you are probably just doing online these days. And a lot of the new things you have started doing on your computer or other internet capable device are actually web services as opposed to a local applications. The old Sun slogan of 'The Network is the Computer' is more true today than it has ever been before. So we don't believe the desktop is dead in any way or form, as some of the hipsters in the media like to claim, in fact we expect it to stay around for a long time. What we do envision though is that the amount of time you spend on webapps will continue to grow and that more and more of your computing tasks will be done using web services as opposed to local applications. Which is why we are continuing to deeply integrate the web into your desktop. Be that through things like GNOME Online accounts or the new Webapps that are introduced in Software installer. And as I have mentioned before on this blog we are also still working on trying to improve the integration of Chrome and Firefox apps into the desktop along the same lines. So while we want the desktop to help you use the applications you want to run locally as efficiently as possible, we also realize that you like us are living in a connected world and thus we need to help give you get easy access to your online life to stay relevant.

So there are of course a lot of other parts of the Fedora Workstation effort, but this has already turned into a very long blog post as it is so I leave the rest for later. Please feel free to post any questions or comments and I will try to respond.

16 Apr 2014 1:57pm GMT

The 2014 "Journées du Logiciel Libre" took place in Lyon like (almost) every year this past week-end. It's a francophone free software event over 2 days with talks, and plenty of exhibitors from local Free Software organisations. I made the 600 metres trip to the venue, and helped man the GNOME booth with Frédéric Peters and Alexandre Franke's moustache.

Our demo computer was running GNOME 3.12, using Fedora 20 plus the GNOME 3.12 COPR repository which was working pretty well, bar some teething problems.

We kept the great GNOME 3.12 video running in Videos, showcasing the video websites integration, and regularly demo'd new applications to passers-by.

The majority of people we talked to were pretty impressed by the path GNOME has taken since GNOME 3.0 was released: the common design patterns across applications, the iterative nature of the various UI elements, the hardware integration or even the online services integration.

The stand-out changes for users were the Maps application which, though a bit bare bones still, impressed users, and the redesigned Videos.

We also spent time with a couple of users dispelling myths about "lightness" of certain desktop environments or the "heaviness" of GNOME. We're constantly working on reducing resource usage in GNOME, be it sluggishness due to the way certain components work (with the applications binary cache), memory usage (cf. the recent gjs improvements), or battery usage (cf. my wake-up reduction posts). The use of gnome-shell using tablet-grade hardware for desktop machines shows that we can offer a good user experience on hardware that's not top-of-the-line.

We found a couple of bugs during demos, and promptly filed them in Bugzilla, or fixed them directly. In the future, we might want to run a stable branch version of GNOME Continuous to get fixes for embarrassing bugs quickly (such as a crash when enabling Zoom in gnome-shell which made an accessibility enthusiast tut at us).

GNOME and Rhône

14 Apr 2014 4:46pm GMT

When I started to write the landing page for The Hacker's Guide to Python, I wanted to try new things at the same time. I read about A/B testing a while ago, and I figured it was a good opportunity to test it out.

A/B testing

If you do not know what A/B testing is about, take a quick look at the Wikipedia page on that subject. Long story short, the idea is to serve two different version of a page to your visitors and check which one is getting the most success. When you found which version is better, you can definitely switch to it.

In the case of my book, I used that technique on the pre-launch page where people were able to subscribe to the newsletter. I didn't have a lot of things I wanted to test out on that page, so I just used that approach on the subtitle, being either "Learn everything you need to build a successful Python project" or "It's time you make the most out of Python".

Statistically, each version would be served half of the time, so both would get the same number of view. I then would build statistics about which page was attracting the most subscribers. With the results I would be able to switch definitively to that version of the landing page.

Technical design

My Web site, this Web site, is entirely static and served by Apache httpd. I didn't want to use any dynamic page, language or whatever. Mainly because I didn't want to have something else to install and maintain just for that on my server.

It turns out that Apache httpd is powerful enough to implement such a feature. There are different ways to build it, and I'm going to describe my choices here.

The first thing to pick is a way to balance the display of the page. You need to find a way so that if you get 100 visitors, around 50 will see the version A of your page, and around 50 will see the version B of the page.

You could use a random number generator, pick a random number for each visitor, and decides which page he's going to see. But it turns out that I didn't find a way to do that with Apache httpd at first sight.

My second thought was to pick the client IP address. But it's not such a good idea, because if you got visitors from, for example, people behind a company firewall, they are all going to be served the same page, so that kind of kills the statistics.

Finally, I picked time based balancing: if you visit the page on a second that is even, you get version A of the page, and if you visit the page on a second that is odd, you get version B. Simple, and so far nothing proves there are more visitors on even than odd seconds, or vice-versa.

The next thing is to always serve the same page to a returning visitor. I mean that if the visitor comes back later and get a different version, that's cheating. I decided the system should always serve the same page once a visitor "picked" a version. To do that, it's simple enough, you just have to use cookies to store the page the visitor has been attributed, and then use that cookie if he comes back.

Implementation

To do that in Apache httpd, I used the powerful mod_rewrite that is shipped with it. I put 2 files in the books directory, named either "the-hacker-guide-to-python-a.html" and "the-hacker-guide-to-python-b.html" that got served when you requested "/books/the-hacker-guide-to-python".

RewriteEngine On
RewriteBase /books
 
# If there's a cookie called thgtp-pre-version set,
# use its value and serve the page
RewriteCond %{HTTP_COOKIE} thgtp-pre-version=([^;])
RewriteRule ^the-hacker-guide-to-python$ %{REQUEST_FILENAME}-%1.html [L]
 
# No cookie yet and…
RewriteCond %{HTTP_COOKIE} !thgtp-pre-version=([^;]+)
# … the number of seconds of the time right now is even
RewriteCond %{TIME_SEC} [02468]$
# Then serve the page A and store "a" in a cookie
RewriteRule ^the-hacker-guide-to-python$ %{REQUEST_FILENAME}-a.html [cookie=thgtp-pre-version:a:julien.danjou.info,L]
 
# No cookie yet and…
RewriteCond %{HTTP_COOKIE} !thgtp-pre-version=([^;]+)
# … the number of seconds of the time right now is odd
RewriteCond %{TIME_SEC} [13579]$
# Then serve the page B and store "b" in a cookie
RewriteRule ^the-hacker-guide-to-python$ %{REQUEST_FILENAME}-b.html [cookie=thgtp-pre-version:b:julien.danjou.info,L]

With that few lines, it worked flawlessly.

Results

The results were very good, as it worked perfectly. Combined with Google Analytics, I was able to follow the score of each page. It turns out that testing this particular little piece of content of the page was, as expected, really useless. The final score didn't allow to pick any winner. Which also kind of proves that the system worked perfectly.

But it still was an interesting challenge!

06 Apr 2014 10:20pm GMT

During the wee hours of the morning, David Faure posted a new mime applications specification which will allow to setup per-desktop default applications, for example, watching films in GNOME Videos in GNOME, but DragonPlayer in KDE. Up until now, this was implemented differently in at least KDE and GNOME, even to the point that GTK+ applications would use the GNOME default when running on a KDE desktop, and vice-versa.

This is made possible using XDG_CURRENT_DESKTOP as implemented in gdm by Lars. This environment variable will also allow implementing a more flexible OnlyShowIn and NotShowIn desktop entry fields (especially for desktops like Unity implemented on top of GNOME, or GNOME Classic implemented on top of GNOME) and desktop-specific GSettings/dconf configurations (again, very useful for GNOME Classic). The environment variable supports applying custom configuration in sequence (first GNOME Classic then GNOME in that example).

Today, Ryan and David discussed the desktop file cache, making it faster to access desktop file data without hitting scattered files. The partial implementation used a custom structure, but, after many kdbus discussions earlier in the week, Ryan came up with a format based on serialised GVariant, the same format as kdbus messages (but implementable without implementing a full GVariant parser).

We also spent quite a bit of time writing out requirements for a filesystem notification to support some of the unloved desktop use cases. Those use cases are currently not supported by either inotify and fanotify.

That will end our face-to-face meeting. Ryan and David led a Lunch'n'Learn in the SUSE offices to engineers excited about better application integration in the desktops irrespective of toolkits.

Many thanks to SUSE for the accommodation as well as hosting the meeting in sunny Nürnberg. Special thanks to Ludwig Nussel for the morning biscuits :)

03 Apr 2014 7:30pm GMT

Wednesday, Mittwoch. Half of the hackfest has now passed, and we've started to move onto other discussion items that were on our to-do list.

We discussed icon theme related simplifications, especially for application developers and system integrators. As those changes would extend into bundle implementation, being pretty close to an exploded-tree bundle, we chose to postpone this discussion so that the full solution includes things like .service/.desktop merges, and Intents/Implements desktop keys.

David Herrman helped me out with testing some Bluetooth hardware (which might have involved me trying to make Mario Strikers Charged work in a Wii emulator on my laptop ;)

We also discussed a full-fledged shared inhibition API, and we agreed that the best thing to do would be to come up with an API to implement at the desktop level. The desktop could then proxy that information to other session- and/or system-level implementations.

David Faure spent quite a bit of time cleaning up after my bad copy/pasted build system for the idle inhibit spec (I copied a Makefile with "-novalidate" as an option, and the XML file was full of typos and errors). He also fixed the KDE implementation of the idle inhibit to match the spec.

Finally, I spent a little bit of time getting kdbus working on my machine, as this seemed to trigger the infamous "hidden cursor bug" without fail on every boot. Currently wondering why gnome-shell isn't sending any events at all before doing a VT switch and back.

Due to the Lufthansa strike, and the long journey times, tomorrow is going to be the last day of the hackfest for most us.

02 Apr 2014 10:28pm GMT

DisplayPort 1.2 Multi-stream Transport is a feature that allows daisy chaining of DP devices that support MST into all kinds of wonderful networks of devices. Its been on the TODO list for many developers for a while, but the hw never quite materialised near a developer.

At the start of the week my local Red Hat IT guy asked me if I knew anything about DP MST, it turns out the Lenovo T440s and T540s docks have started to use DP MST, so they have one DP port to the dock, and then dock has a DP->VGA, DP->DVI/DP, DP->HDMI/DP ports on it all using MST. So when they bought some of these laptops and plugged in two monitors to the dock, it fellback to using SST mode and only showed one image. This is not optimal, I'd call it a bug :)

Now I have a damaged in transit T440s (the display panel is in pieces) with a dock, and have spent a couple of days with DP 1.2 spec in one hand (monitor), and a lot of my hair in the other. DP MST has a network topology discovery process that is build on sideband msgs send over the auxch which is used in normal DP to read/write a bunch of registers on the plugged in device. You then can send auxch msgs over the sideband msgs over auxch to read/write registers on other devices in the hierarchy!

Today I achieved my first goal of correctly encoding the topology discovery message and getting a response from the dock:
[ 2909.990743] link address reply: 4
[ 2909.990745] port 0: input 1, pdt: 1, pn: 0
[ 2909.990746] port 1: input 0, pdt: 4, pn: 1
[ 2909.990747] port 2: input 0, pdt: 0, pn: 2
[ 2909.990748] port 3: input 0, pdt: 4, pn: 3

There are a lot more steps to take before I can produce anything, along with dealing with the fact that KMS doesn't handle dynamic connectors so well, should make for a fun tangent away from the job I should be doing which is finishing virgil.

I've ordered another DP MST hub that I can plug into AMD and nvidia gpus that should prove useful later, also for doing deeper topologies, and producing loops.

Also some 4k monitors using DP MST as they are really two panels, but I don't have one of them, so unless one appears I'm mostly going to concentrate on the Lenovo docks for now.

02 Apr 2014 12:02pm GMT

So the recent GNOME 3.12 release has gotten a very positive reception. Since I know that many members of my team has worked very hard on GNOME 3.12 I am delighted to see all the positive feedback the release is getting. And of course it doesn't hurt having it give us a flying start to the Fedora Workstation effort. Anyway, for the fun of it I tried putting together a set of press quotes, kinda like how they tend to do for computer game advertisements.

• "GNOME 3.12: Pixel perfect" "GNOME 3 has finally arrived" - The Register
• "It is the GNOME release I have been waiting for" - Linux Action Show
• "The Very Exciting GNOME 3.12 Has Been Released" - Phoronix.com
• "…. a milestone feature update for users …" - eweek.com
• "The design team has refined everything …" - omgubuntu.co.uk
• "One of the big Linux desktops is updated" - TheInquirer
• "High Resolution screens are best managed under Gnome 3.12″ - laptopspirit.fr
• "One of the most striking innovations.." - Heise.de
• "has resurrected what was once the darling of the Linux desktop" - TechRepublic.com

Some of the quotes might feel a little out of context, but as I said I did it for fun and if you end up spending time reading GNOME 3.12 articles to verify the quotes, then all the better

Also you should really check out the nice GNOME 3.12 release video that can be found on the GNOME 3.12 release page.

Anyway, I plan on doing a blog post about the Fedora Workstation effort this week and will talk a bit about how GNOME 3.12 and later fits into that.

02 Apr 2014 8:59am GMT

It has been a long time in the making, but I have finally cut a new release of the Transmageddon transcoder application. The code inside Transmageddon has seen some major overhaul as I have updated it to take advantage of new GStreamer APIs and features. New features in this release include:

• Support files with multiple audio streams, allowing you to transcode them to different codecs or drop them from the new file
• DVD ripping support. So know you can use your movie DVDs as input in Transmageddon, be aware though that you need to install things like lsdvd and the GStreamer dvdread plugin from gst-plugins-ugly for it to become available. And you probably also want libdvdcss installed to be able to transcode most movie DVDs.
• Another small feature of the release is that you can now set language information on files with one audio stream inside. I hope to extend this to also work with files that have multiple audio streams. If you rip a DVD with multiple audio streams Transmageddon will preserve the existing audio information, so in that case you shouldn't need to set the language metadata manually.
• Enabled VP9 support in the code.

There are some other smaller niceties too, like the use of blue default action buttons to match the GNOME 3 style better and I also switched to new icon designed by Jakub Steiner. There is also an appdata file now, which should make Transmageddon available in a nice way inside the new Fedora Software Installer'

Also there is now an Advanced section on the Transmageddon website explaining how you can create custom presets that allow you to do things like resize the video or change the bitrate of the audio.

And last, but not least here is a screenshot of the new version.

You can download the new version from the Transmageddon website, I will update the version in Fedora soon.

01 Apr 2014 7:43pm GMT

Today, Ryan carried on with writing the updated specification for startup notification.

David Faure managed to get Freedesktop.org specs updated on the website (thanks to Vincent Untz for some chmod'ing), and removed a number of unneeded items in the desktop file specification, with help from Jérôme.

I fixed a number of small bugs in shared-mime-info, as well as preparing for an 8-hour train ride.

Lars experimented with technics to achieve a high score at 2048, as well as discussing various specifications, such as the possible addition of an XDG_CURRENT_DESKTOP envvar. That last suggestion descended into a full-room eye-rolling session, usually when xdg-open code was shown.

01 Apr 2014 3:22pm GMT

So the release of the 3.14 linux kernel already happended and I'm a bit late for our regular look at what cool stuff will land in the 3.15 merge window for the Intel graphics driver.

The first big thing is Ben Widawsky's support for per-process address spaces. Since a long time we've already supported the per-process gtt page tables the hardware provides, but only with one address space. With Ben's work we can now manage multiple address spaces and switch between them, at least on Ivybridge and Haswell. Support for Baytrail and Broadwell for this feature is still in progress. This finally allows multiple different users to use the gpu concurrently without accidentally leaking information between applications. Unfortunately there have been a few issues with the code still so we had to disable this for 3.15 by default.

Another really big feature was the much more fine-grained display power domain handling and a lot of runtime power management infrastructure work from Imre and Paulo. Intel gfx hardware always had lots of automatic clock and power gating, but recent hardware started to have some explicit power domains which need to be managed by the driver. To do that we need to keep track of the power state of each domain, and if we need to switch something on we also need to restore the hardware state. On top of that every platform has it's own special set of power domains and how the logical pieces are split up between them also changes. To make this all manageable we now have an extensive set of display power domains and functions to handle them as abstraction between the core driver code and the platform specific power management backend. Also a lot of work has happened to allow us to reuse parts of our driver resume/suspend code for runtime power management. Unfortunately the patches merged into 3.15 are all just groundwork, new platforms and features will only be enabled in 3.16.

Another long-standing nuisance with our driver was the take-over from the firmware configuration. With the reworked modesetting infrastructure we could take over the output routing, and with the experimental i915.fastboot=1 option we could eschew the initial modeset. This release Jesse provided another piece of the puzzle by allowing the driver to inherit the firmware framebuffer. There's still more work to do to make fastboot solid and enable it by default, but we now have all the pieces in place for a smooth and fast boot-up.

There have been tons of patches for Broadwell all over the place. And we still have some features which aren't yet fully enabled, so there will be lots more to come. Other smaller features in 3.15 are improved support for framebuffer compression from Ville, again more work still left. 5.4GHz DisplayPort support, which is a required to get 4k working. Unfortunately most 4k DP monitors seem to expose two separate screens and so need a adriver with working MST (multi-stream support) which we don't yet support. On that topic: The i915 driver now uses the generic DP aux helpers from Thierry Redding. Having shared code to handle all the communication with DP sinks should help a lot in getting MST off the ground. And finally I'd like to highlight the large cursor support, which should be especially useful for high-dpi screens.


And of course there's been fixes and small improvements all over the place, as usual.

01 Apr 2014 11:03am GMT

I'm in Nürnberg this week for the Freedesktop Hackfest, aka the XDG Summit, aka the XDG Hackfest aka... :)

We started today with discussions about desktop actions, and how to implement them, such as whether showing specific "Edit" or "Share" sub-menus and how to implement them. We decided that that could be implemented through specific desktop keys which a file manager could use. This wasn't thought to be generally useful to require a specification for now.

The morning is stretching to discuss "splash screens". A desktop implementor running on low-end hardware is interested in having a placeholder window show up as soon as possible, in some cases even before the application has linked and the toolkit is available. This discussion is descending into slightly edge cases, such as text editors launching either new windows or new tabs depending on a number of variables.

Specific implementation options were discussed after a nice burrito lunch. We've decided that the existing X11 startup notification would be ported to D-Bus, using signals instead of X messages. Most desktop shells would support both versions for a while. Wayland clients that want startup notification would be required to use the D-Bus version of the specification. In parallel, we would start passing workspace information along with the DESKTOP_STARTUP_ID envvar/platform data.

Jérôme, David and I cleared up a few bugs in shared-mime-info towards the end of the day.

Many thanks to SUSE for the organisation, and accommodation sponsorship.

Update: Fixed a typo

31 Mar 2014 3:03pm GMT

Fifteen years ago today, March 29, 1999, I showed up at Sun's MPK29 building for my first day of work as a student intern. I was off-cycle from the normal summer interns, since I wasn't enrolled Spring semester but was going to finish my last two credits (English & Math) during the Summer Session at Berkeley. A friend from school convinced his manager at Sun to give me a chance, and after interviewing me, they offered to bring me on board for 3 months full-time, then dropping down to half-time when classes started.

I joined the Release Engineering team as a tools developer and backup RE in what was then the Power Client Desktop Software organization (to differentiate it from the Thin Client group) in the Workstation Products Group of Sun's hardware arm. The organization delivered the X Window System, OpenWindows, and CDE into Solaris, and was in the midst of two big projects for the Solaris 7 8/99 & 11/99 releases: a project called "SolarEZ" to make the CDE desktop more usable and to add support for syncing CDE calendar & mail apps with Palm Pilot PDAs; and the project to merge the features from X11R6.1 through X11R6.4 (including Xinerama, Display Power Management, Xprint, and LBX) into Sun's proprietary X11 fork, which was still based on the X11R6.0 release.

I started out with some simple bug fixes to learn the various build systems, before starting to try to write the scripts they wanted to simplify the builds. My first bug was to find out why every time they did a full build of Sun's X gate they printed a copy of the specs. (To save trees, they'd implemented a temporary workaround of setting the default printer queue on the build machines to one not connected to a printer, but then they had to go delete all the queued jobs every few days to free disk space.) After a couple days of digging, and learning far too much about Imake for my own good, I found the cause was a merge error in the Imake configuration specs. The TroffCmd setting for how to convert troff documents to PostScript had been resynced to the upstream setting of psroff, but the flags were still the ones for Sun's troff. These flags to Sun's troff generated a PostScript file on disk, while they made psroff send the PostScript to the printer - switching TroffCmd back to Solaris troff solved the issue, so I filed Sun bug 4227466 and made my first commit to the Solaris X11 code base.

Six months later, at the end of my internship, diploma in hand, Sun offered to convert my position to a regular full-time employee, and I signed on. (This is why I always get my anniversary recognition in September, since they don't count the internship.) Six months after that, the X11R6.4 project was finishing up and several of the X11 engineers decided to move on, making an opening for me to switch to the X11 engineering team as a developer. Like many junior developers joining new teams, I started out doing a lot of bug fixes, and some RFE's, such as integrating Xvfb & Xnest into Solaris 9.

A couple years in, Sun's attempts to reach agreement amongst all of the CDE co-owners to open source CDE had failed, resulting only in the not-really-open-source release of OpenMotif, so Sun chose to move on once again, to the GNOME Desktop. This required us to provide a lot of infrastructure in the X server and libraries that GNOME clients needed, and I got to take on some more challenging projects such as trying to port the Render extension from XFree86 to Xsun, assisting on the STSF font rendering project, adding wheel mouse support to Xsun, and working to ensure Xsun had the underlying support needed by GNOME's accessibility projects. Unfortunately, the results of some of those projects were not that good, but we learned a lot about how hard it was to retrofit Xsun with the newly reinvigorated X11 development work and how maintaining our own fork of all the shared code was simply slowing us down.

Then one fateful day in 2004, our manager was on vacation, so I got a call from the director of the Solaris x86 platform team, who had been meeting with video card vendors to decide what graphics to include in Sun's upcoming AMD64 workstations. The one consistent answer they'd gotten was that the time and cost to produce Solaris drivers went up and the feature set went down if they had to port to Xsun as well, instead of being able to reuse the XFree86 driver code they were already shipping for Linux. He asked what it would take to ship XFree86 on Solaris instead, and we discussed the options, then I talked with the other X engineers, and soon I was the lead of a project to replace the Solaris X server.

This was right at the time XFree86 was starting to come apart while the old X.Org industry consortium was trying to move X11 to an open development model, resulting in the formation of the X.Org Foundation. We chose to just go straight to Xorg, not XFree86, and not to fork as we'd done with Xsun, but instead to just apply any necessary changes as patches, and try to limit those to not changing core areas, so it was easier to keep up with new releases.

Thus, right at the end of the Solaris 10 development cycle we integrated the Xorg 6.8 X server for x86 platforms, and even made it the default for that release. We had no SPARC drivers ported yet, just all the upstream x86 drivers, so only provided Xsun in the SPARC release. The SPARC port, along with a 64-bit build for x64 systems, came in later Solaris 10 update releases. This worked out much better than our attempts to retrofit Xsun ever did.

Along with the Solaris 10 release, Sun announced that Solaris was going to be open sourced, and eventually as much of the OS source as we could release would be. But for the Solaris 10 release we only had time to add the Xorg server and the few libraries we needed for the new extensions. Most of the libraries and clients were still using the code from Sun's old X11R6 fork, and we needed to figure out which changes we could publish and/or contribute back upstream. We weren't ready to do this yet on the opening day of OpenSolaris.org, but I put together a plan for us to do so going forward, combing the tasks of migrating from our fork to a patched upstream code base, and the work of excising proprietary bits so it could be released as fully open source.

Due to my work with the X.Org open source community, my participation in the existing Solaris communities on Usenet and Yahoo Groups, and interest in X from the initial OpenSolaris participants, I was pulled into the larger OpenSolaris effort, culminating in serving 2 years on the OpenSolaris Governing Board. When the OpenSolaris distro effort ("Project Indiana") kicked off, I got involved in it to figure out how to integrate the X11 packages to it. Between driving the source tree changes and the distro integration work for X, I became the Tech Lead for the Solaris X Window System for the Solaris 11 release.

Of course, the best laid plans are always beset by changes in the surrounding world, and these were no different. While we finished the migration to a pure open-source X code base in November 2009, it was shortly followed by the closing of Oracle's acquisition of Sun, which eventually led to the shutdown of the OpenSolaris project. Since the X11 code is still open source and mostly external sources, it is still published on solaris-x11.java.net, alongside other open source code included in Solaris. So when Solaris 11 was released in November 2011, while the core OS code was closed again, it included the open source X11 code with the culmination of our efforts.

Solaris 11 also included a few bug fixes I'd found via experimenting with the Parfait static analysis tool created by Sun Labs. Since shortly after starting in the X11 group, I'd been handling the security bugs in X. Having tools to help find those seemed like a good idea, and that was reinforced when we used Coverity to find a privilege escalation bug in Xorg. When the Sun Labs team came to demo parfait to us, I decided to try it out, and found and fixed a number of bugs in X with it (though not in security sensitive areas, just places that could cause memory leaks or crashes in code not running with raised privileges).

After the Oracle acquisition, part of adopting Oracle's security assurance policies was using static analysis on our code base. With my experience in using static analyzers on X, I helped create the overall Solaris plan for using static analysis tools on our entire code base. This plan was accepted and I was asked to take on the role of leading our security assurance efforts across Solaris.

Another part of integrating Sun into Oracle was migrating all the bugs from Sun's bug database into Oracle's, so we could have a unified system, allowing our Engineered Systems teams to work together more productively, tracking bugs from the hardware through the OS up to the database & application level in a single system. Valerie Fenwick & Scott Rotondo had managed the Solaris bug tracking for years, and I'd worked with them, representing both the X11 and larger Desktop software teams. When Scott decided to move to the Oracle Database Security team, Valerie asked me to replace him, just as the effort to migrate the bugs was beginning. That turned into 2 years of planning, mapping, updating tools and processes, coordinating changes across the organization, traveling to multiple sites to train our engineers on the Oracle tools, and lots of communication to prepare our teams for the changes.

As we looked to the finish line of that work, planning was beginning for the Solaris 12 release. All of the above experiences led to me being named as the Technical Lead for the Solaris 12 release as a whole, covering the entire OS, coordinating and working with the tech leads for the individual consolidations. We're two years into that now, and while I can't really say much more yet than the official roadmap, I think we've got some solid plans in place for the OS.

So here it is now 15 years after starting at Sun, 14 after joining the X11 team. I still do a little work on X in my spare time (especially around the area of X security bugs), and officially am still in the X11 group, but most of my day is spent on the rest of Solaris now, between planning Solaris 12, managing our bug tracking, and ensuring our software is secure.

In those 15 years, I've learned a lot, accomplished a bit, and made a lot of friends. I've also:

• filed 2572 bugs & enhancement requests in the Sun/Oracle bug database, and fixed or closed 2155, assuming I got the bug queries correct.
• worked under 3 CEO's (Scott, Jonathan, Larry), and 4 managers, but more VP's and Directors than I can count thanks to Sun's love of regular reorgs.
• been part of the Workstation Products Group, the Webtop Applications group, the Software Globalization group, the User Experience Engineering group, the x64 Platform group, the Solaris Open Source group, and a few I've forgotten, again thanks to Sun's love of regular reorgs.
• been seen by my co-workers in a suit exactly 4 times (my initial job interview, and funerals for 3 colleagues we've lost over the years).

Where will I be in 15 more years? Hard to guess, but I bet it involves making sure everything is ready for Y2038, so I can retire in peace when that arrives.

For now, I offer my heartfelt thanks to everyone whose help made the last 15 years possible - none of it was done alone, and I couldn't have got here without a lot of you.

30 Mar 2014 3:35pm GMT

Those running Fedora Rawhide or GNOME 3.12 may have noticed that there is no Xorg.log file anymore. This is intentional, gdm now starts the X server so that it writes the log to the systemd journal. Update 29 Mar 2014: The X server itself has no capabilities for logging to the jornal yet, but no changes to the X server were needed anyway. gdm merely starts the server with a /dev/null logfile and redirects stdin/stderr to the journal.

Thus, to get the log file use journalctl, not vim, cat, less, notepad or whatever your $PAGER was before.

This leaves us with the following commands.

journalctl -e /usr/bin/Xorg 

Which would conveniently show something like this:

Mar 25 10:48:41 yabbi Xorg[5438]: (II) UnloadModule: "wacom"
Mar 25 10:48:41 yabbi Xorg[5438]: (II) evdev: Lenovo Optical USB Mouse: Close
Mar 25 10:48:41 yabbi Xorg[5438]: (II) UnloadModule: "evdev"
Mar 25 10:48:41 yabbi Xorg[5438]: (II) evdev: Integrated Camera: Close
Mar 25 10:48:41 yabbi Xorg[5438]: (II) UnloadModule: "evdev"
Mar 25 10:48:41 yabbi Xorg[5438]: (II) evdev: Sleep Button: Close
Mar 25 10:48:41 yabbi Xorg[5438]: (II) UnloadModule: "evdev"
Mar 25 10:48:41 yabbi Xorg[5438]: (II) evdev: Video Bus: Close
Mar 25 10:48:41 yabbi Xorg[5438]: (II) UnloadModule: "evdev"
Mar 25 10:48:41 yabbi Xorg[5438]: (II) evdev: Power Button: Close
Mar 25 10:48:41 yabbi Xorg[5438]: (II) UnloadModule: "evdev"
Mar 25 10:48:41 yabbi Xorg[5438]: (EE) Server terminated successfully (0). Closing log file.

The -e toggle jumps to the end and only shows 1000 lines, but that's usually enough. journalctl has a bunch more options described in the journalctl man page. Note the PID in square brackets though. You can easily limit the output to just that PID, which makes it ideal to attach to the log to a bug report.

journalctl /usr/bin/Xorg _PID=5438

Previously the server kept only a single backup log file around, so if you restarted twice after a crash, the log was gone. With the journal it's now easy to extract the log file from that crash five restarts ago. It's almost like the future is already here.

28 Mar 2014 11:54pm GMT

With the release of GNOME-Software 3.12, it's also time for another update about what is going on behind the scenes of Appstream, the project providing all the data needed to make software-centers work. So here is the long overdue update

If you have attended my FOSDEM talk or seen the slides, you know about the concept of "component metadata" to describe (almost) all software components which make up a Linux system, as well as their public interfaces they provide for users and other software to access. This metadata specification was originally designed as part of the Listaller project for use with the 3rd-party software installer.
But we already have the Appstream specification, which describes applications available on a distribution. And technically, applications are nothing but specialized components.
When I was asked about extending the Appstream spec with fonts and codecs, I decided to give up the Listaller components concept and merge it with the Appstream project, effectively creating a single source of metadata for all software in distributions.

Appstream and components were separate, because Appstream was designed just for application metadata shown in software-centers, while component-metadata was more technical and not user-visible. This separation, however, did not reflect reality. Software centers want to show things components were designed for (codecs, fonts, …), and application meta-information also can contain technical items not shown in a software-center, but useful for other functionality (e.g. for mimehandler finding). So, if you like clear naming of things (like I do), you can now think of the upcoming Appstream release 0.6 as Appstream not being a stream of application metadata, but rather metadata provided for use in applications.

Extending Appstream with components has a number of benefits:
First of all, users will soon get systems which are able to automatically install missing firmware, codecs, fonts and even libraries or Python-modules. This happens in a distro-agnostic way, so any application can request installations of missing components through Appstream and PackageKit without having to care about distribution details.

For distributors, you will get more information about the software upstream provides: Which public interfaces does it make available? What is the upstream's homepage? Summary? Short release-descriptions? Exposed library symbols? etc. This data was available before, of course, but we made it machine-readable and standardized now. This will also allow us to match software across distributions quickly, allowing efficient exchange of patches between distributions, and answering questions upstream has like "in which distribuions is my software available, and in which version?". Tracking upstream software is also easier now. Last but not least, the metadata format gives upstreams a small say in how their software gets packaged (= how it gets split / if there is a metapackage depending on the split parts).

For upstream authors, it is much easier to maintain just a few Appstream XML metafiles, instead of adding a Listaller component file as well (which was not XML). Most data for creating the metainfo files is already available, some can even be filled in automatically by the build-system.

With all the advantages this new system has comes a small disadvantage: These changes broke the backwards-compatibility of the Appstream spec, so to use Appstream data in version 0.6 or higher, you need to adjust your parsers. But of course you are using a library to access the data anyway, which already does that for you GNOME-Software 3.12 has almost complete support for the new (but still not released) Appstream 0.6 spec through it's libappstream-glib library. The libappstream library in Appstream's Git branch will get 0.6 support as well very soon. The only thing you need to do is adjust your code for the new library version, which broke it's API as well due to a rewrite in C (I hope that we can provide language bindings again very soon).
Soon, there will also be a Qt library for accessing Appstream, since some people expressed a demand for it (and working with GObject in Qt C++ code always feels a litle bit odd).
The 0.6 specification is still being fine-tuned to iron out some issues which might come up and mainly to improve the documentation and adjust all tools. It will be released in 2-3 weeks, and bring not only components support, but also lots of other small improvements and additions for better description of applications/components. In some cases, tag names have changed to have less ambiguous names.

Thanks to Richard Hughes for helping with the spec - our discussions about the Appstream spec are highly productive and in the end lead to a better specification. Also, the new 0.6 specification includes many extensions used in GNOME-Software by default now, which makes it easier for other distributions to provide them for GS.

The current draft for the Appstream 0.6 specifications can be found here. Some details might still change before the final release. As soon as the spec is final, there will be a ML announcement for distributors as well (watch the XDG distributions list), so there is enough time for distributors to future-proof their Appstream distro-data generators (I will also describe the changes more detailed then).

If you are confused now about the whole components stuff: Fear not! I will do some more detailed posts in a few weeks explaining upstream projects how to provide metadata properly (after the 0.6 release). I will also finish the wiki page for Appstream AppData in KDE projects, where we can then use 0.6-style data in KDE right away.

28 Mar 2014 12:48pm GMT

1 new GNOME Videos, 1 updated Bluetooth panel, 2 new thumbnailers, 9 grilo sources, and 1 major UPower rework.

I'm obviously very attached to the GNOME Videos UI changes, the first major UI rework in its 12-year existence.

26 Mar 2014 8:55pm GMT