19 May 2018

feedPlanet GNOME

Eisha Chen-yen-su: Internationalization of Fractal (3rd and last part)

"Tl;dr version": I've finished implementing the i18n of Fractal and I've submitted a first French translation of it. With some help from Daniel (my mentor) to complete the integration with the build system, so thank him for that! Here are my merge requests: https://gitlab.gnome.org/World/fractal/merge_requests/105 and https://gitlab.gnome.org/World/fractal/merge_requests/107.

I am going to detail a little bit what was done in order to achieve this.

Integrate gettext-rs to the project

I first added gettext-rs as one of the dependencies of Fractal (in this commit), as I have explained it in the previous articles. Then, I put the initialization of gettext-rs by asking it to look for the locale files in ./fractal-gtk/po (in this commit).

Wrap all the translatable strings in the Rust source files

Then I took quite some time to examine every strings in the source files of the crate fractal-gtk to wrap all messages that would end up in the GUI with gettext (it turned out that I didn't really need to use ngettext). Wrapping some strings in the format! macro was less obvious to deal with though. For instance:

secondary = format!("You've been invited to join to <b>{}</b> room by <b>{}</b>",
                    room_name, sender_name);

Couldn't be straightforwardly rewritten like this:

secondary = format!(gettext("You've been invited to join to <b>{}</b> room by <b>{}</b>"),
                    room_name, sender_name);

Because the first argument of format! needs to be a string literal, so I had to use this work around instead:

let sentence_template = gettext("You've been invited to join to <b>{room_name}</b> room by <b>{sender_name}</b>");
secondary = sentence_template.replace("{room_name}", room_name.as_str())
                             .replace("{sender_name}", sender_name.as_str());

I explicitly did what format! would have done here without having the constraint of using a string literal. Because in some languages, the place of {room_name} and {sender_name} can be reversed. See this commit for more details.

Adding the support of gettext within the build system

Next, we needed a way to automatically generate PO and POT files and move MO files to the right place for gettext-rs to read. So I was going to have the meson build system helping me for that. I first added a POTFILES.in file which lists all the files with translatable strings in them and a LINGUAS file which list the languages for which we want to have PO files. And I've added a meson.build file and updated to one in the project root to be able to generate the mentioned file: you can run `ninja -C _build fractal-pot` to generate a POT file and `ninja -C _build fractal-update-po` to generate/update PO files. See these commit for more details on: the POTFILE.in, the LINGUAS file and the meson.build files.

Once the ability to generate the locale files implemented, Daniel helped my with this commit that removes the hard-coded path to bind the text domain for gettext. And he added a Spanish translation (see this commit). I also submitted a French translation, see this commit and this one.

After my MR merged, my first task for GSoC is completed!! 🙂

19 May 2018 3:23pm GMT

Tobias Mueller: Talking at GPN 2018 in Karlsruhe, Germany

Similar to last year I managed to attend the Gulasch Programmier-Nacht (GPN) in Karlsruhe, Germany. Not only did I attend, I also managed to squeeze in a talk about PrivacyScore. We got the prime time slot on the opening day along with all the other relevant talks, including the Eurovision Song Contest, so we were not overly surprised that the audience had a hard time deciding where to go and eventually decided to attend talks which were not recorded. Our talk was recorded and is available here.

https://cdn.media.ccc.de/events/gpn/gpn18/webm-sd/gpn18-127-deu-Automatically_Assessing_Security_and_Privacy_Properties_of_Web_Sites_webm-sd.webm

Given the tough selection of the audience by the other talks, we had the people who were really interested. And that showed during the official Q&A as well as in the hallway track. We exchanged contacts with other interested parties and got a few excellent comments on the project.

Another excellent part of this year's GPN was the exhibition in the museum. As GPN takes places in a joint building belonging to the local media university as well as the superb art and media museum, the proximity to the artsy things allows for an interesting combination. This year, the open codes exhibition was not hosted in the ZKM, but GPN also took place in that exhibition. A fantastic setup. Especially with the GPN's motto being "digital naïves". One of the exhibition's pieces is an assembly robot's hand doing nothing else but writing a manifesto. Much like a disciplinary action for a school child. Except that the robot doesn't care so much. Yet, it's usefulness only expands to writing these manifestos. And the robot doesn't learn anything from it. I like this piece, because it makes me think about the actions we take hoping that they have a desired effect on something or someone but we actually don't know whether this is indeed the case.

I also like the Critical Engineering Manifesto being exhibited. I like to think about how the people who actual implement cetain technologies can be held responsible for the effects of it on individuals or the society. Especially with more and more "IoT" deployments where the "S" represents their security. It's easy to blame Facebook for "leaking" user profiles although it's in their Terms of Services, but it's harder to shift the blame for the smart milk sensor in your fridge invading my privacy by reporting how much I consume. We will have interesting times ahead of us.

An exhibit pointing out the beauty of algorithms and computation is a board that renders a Julia Set. That's wouldn't be so impressive in itself, but you can watch the machine actually compute the values. The exhibit has a user controllable speed regulator and an insight into the CPU as well as the higher level code. I think it's just an ingenious idea to enable the user to go full speed and see the captivating movements of the beautiful Julia set while also allowing the go super slow to investigate how this beauty is composed of relatively simple operations. Also, the slow execution itself is relatively boring. We get to see that we have to go very fast in order to be entertained. So fast that we cannot really comprehend what is going on.

I whole heartedly recommend visiting this exhibition. And the GPN, of course, too. It's a nice chaotic event with a particular flair. It's getting more and more crowded though, so better while the feeling lasts and doesn't get drowned by all the tourists.

19 May 2018 3:07pm GMT

Daniel García Moreno: Stickers in Riot

Yesterday I read a blog post about the new Riot.im Stickers. This is not a matrix.org feature, it's implemented as a widget, when you send an sticker you're sending a new event with the type "m.sticker", which is similar to the "m.image" event.


The matrix.org protocol is flexible so this is a good example of how to add new features to the clients that uses matrix without the need to change the protocol.

This is not a core feature because you can send images, but I think this is great and add a simple way to show reactions for the users, so as I was reading I thought that we can add this to Fractal, so I started to read how we can add support for this.

Reading the doc

The first thing to implement a feature is to read the specifications or the technical documentation so we can know what is needed... But there's no documentation yet about Stickers or widgets yet.

This is a problem because we can't implement a feature if we don't know what we should do. But free software give us a great opportunity when this happens, we've the Riot source code so we can look at the code and learn what they are doing.

Reading the code

Riot web is a javascript application that uses AJAX to communicate with different server APIs so the first thing that I did to start to understand the stickers thing was to open the firefox debugger and view how riot is communicating with the server.

From here I've learned that for stickers riot is asking to the scalar.vector.im server. But I don't understand the whole thing with the requests because riot does a lot of request to different APIs and I can't isolate the stickers thing easily.

To fill my understanding gap I go to the matrix-js-sdk and matrix-react-sdk and I did a quick grep to the source code looking for the API calls that I've view in firefox. With this I can understand the full stickers process.

Writing an example

To say that I know how this is working, it's not enough the code reading. To make sure that I've understood the whole process I need to write a simple program that does all the process and then I can say that I understand this.

So I started to write a simple python script using requests. This simple script does the request to the server and list all the stickers json so I can say that I'm able to communicate with the API.

Stickers in Fractal

After this small research I'm able to implement an initial sticker support for Fractal. I'll try to add a simple way to show and use stickers and a way to render stickers in the messages history.

If there's no secret problems we'll have a basic stickers support in Fractal soon.

19 May 2018 7:37am GMT

18 May 2018

feedPlanet GNOME

Marco Trevisan: Hello Planet GNOME!

Hey guys, although I've been around for a while hidden in the patches, some months ago (already!?!) I did my application to join the GNOME Foundation, and few days after - thanks to some anonymous votes - I got approved :), and thus I'm officially part of the family!

So, thanks again, and sorry for my late "hello" 🙂

18 May 2018 3:52pm GMT

Zeeshan Ali: Collabora and GStreamer spring in Sweden

Earlier this month, a few of us from Collabora, Olivier Crête, Nicolas Dufresne, George Kiagiadakis and I attended the GStreamer Spring Hackfest in Lund, Sweden. Hosted by Axis Communications (who uses GStreamer in their surveillance cameras for many years now), it was a great opportunity for the GStreamer community to touch base and work on open bugs and pet projects.



While I've been involved in the GStreamer project in the past, it was my first GStreamer hackfest. While a lot was achieved during the event, the most exciting outcomes were no doubt the closing of more than 350 bugs, and the agreement on a transition plan to move to GitLab.

Overall, the hackfest was very productive, with each member of our team managing to progress in their list of tasks while all taking part in bug triaging & cleaning in preparation of moving GStreamer's issue tracking to GitLab.

George spent time working on improving the new library API that is needed to introduce support for the non-interleaved audio layout, discussed a gst-rtsp-server issue with the Axis team, and merged all qt-gstreamer patches that were lying around in bugzilla and resolved all reported bugs, then declared it as unmaintained.

For his part, Nicolas participated in the planar audio format and split field interlaced video support work, started looking at adding per element latency tracing to GStreamer's existing latency tracer, and also discussed GStreamer CI, which will also move to GitLab to be able to run on pull requests also.

Olivier, during the first day, focused on the collective effort of reviewing all of the open bugs, managing to close a number of them while confirming and commenting on others. He also merged some outstanding patches he had (stay tuned for more details on those), and forward ported gst-validate for Android with the goal of running the CI on Android. He also merged a series of patches that enable bitcode embedding on the iOS target with the eventual goal of supporting tvOS as well.

As for myself, I mainly worked on (or rather started to work on) split-field interlacing support in GStreamer, adding relevant formats and modes in the GStreamer video library. In addition, as a Meson developer (Nirbheek Chauhan) was present, I took the opportunity to discuss with him the last bit of porting build system of Geoclue to Meson, a side project I've been working on. It helped me get it done faster but also helped Nirbheek find some issues in Meson and fix them!

All in all, my first GStreamer hackfest was an awesome experience (even though I was not feeling well). It was also very nice to hangout and socialize with old and new friends in the GStreamer community after a long time. Many thanks again to Axis for hosting us in their offices! See you at the GStreamer Conference this fall!

18 May 2018 2:59pm GMT

Felipe Borges: Boxes now supports RDP connections

Boxes has been the go-to option for easy virtual machine setups in GNOME for quite some time, but some people don't know that our beloved application can also act as a remote viewer.

The "Enter URL" option in the new machine assistant is how you get a new remote machine added to your collection. It supports addresses of Spice and VNC servers and oVirt and Libvirt brokers. You can also paste the URL of an operating system image (iso, img, qcow, etc…) and Boxes will download and boot it for you.

However, there is life out of our GNU/Linux boxes and we need to stay connected. Windows is extremely popular and it ships a RDP server by default, making the adoption of open alternatives a bit unhandy there.

Imagine you have clients running Windows that need your remote support, or you couldn't convince your family back home to switch to GNU/Linux, etc…

Now Boxes also supports RDP!

Boxes - screenshot

This feature is powered by FreeRDP. For convenience, I wrote a glib wrapper around the essential freerdp API so we can consume it via gobject-introspection in GNOME Boxes and others could reuse it for their own applications.

Heavily inspired in the gtk-vnc API, I decided to name it gtk-frdp. So original! 😉

If you are interested in writing a RDP client of your own, or maybe port an existent one to gtk-frdp, you can achieve it with a few lines of code such as below (I choose Python for legibility, but it could be any gobject-introspected language of your choice).

from gi.repository import Gtk, GtkFrdp

window = Gtk.Window () display = GtkFrdp.Display () window.add (display)
display.open_host ("192.168.0.1", 3389) display.username = "username" display.password = "password"
window.show_all () Gtk.main ()

That simple!

This and a few other cool features will be available in our next stable release, GNOME 3.30. Stay tuned!

18 May 2018 1:52pm GMT

Miguel de Icaza: Startup Improvements in Xamarin.Forms on Android

With Xamarin.Forms 3.0 in addition to the many new feature work that we did, we have been doing some general optimizations across the board, from compile times to startup times and wanted to share some recent results on the net effect on one of our larger sample apps.

These are the results when doing a cold start for the SmartHotel360 application on Android when compiled for 32bits (armeabi-v7a) on a Google Pixel (1st gen).

Release Release/AOT Release/AOT+LLVM
Forms 2.5.0 6.59s 1.66s 1.61s
Forms 3.0.0 3.52s 1.41s 1.38s

This is independent of the work that we are doing to improve Android's startup speed, that both brings additional benefits today, and will bring additional benefits in the future.

One of the areas that we are investing on for Android is to remove any dynamic code execution at startup to integrate with the Java runtime, instead all of this is being statically computed, similar to what we are doing on Mac and iOS where we completely eliminated reflection and code generation from startup.

18 May 2018 1:50pm GMT

Alexandre Franke: Fractal hackfest in Strasbourg

As Planet GNOME readers should be aware of by now 😉 we had a Fractal hackfest last week. I organised it with the help of Tobias and I want to thank all the parties that not only made it possible, but also made it an awesome event:

We were lucky to have Matthew from the core Matrix team with us for half of the event to answer our technical questions and help us steer our ship in the right direction.

I got involved with the discussions on various topics but as these have already been covered at length by the other attendees I will focus on some other stuff I managed to do. The first one is the one I'm the most happy about: I brought Quentin, a local student (sitting next to me on the picture), and he made his first GNOME contribution, which happened to be merge request !100 in Fractal 🎉. I also managed to write some code and make some progress on a few things. I landed the changes on generated avatars for users and rooms that don't have one set, and started working on a couple of other contributions which should be finished in the near future. Keep an eye on the commit log to see them as they happen! 😊 I'm happy with what we have achieved so far and I'm looking forward 🤩 to more good things coming soon.

Apart from the technical side of things, I also tried to act as a city guide and hope my guests liked the places I took them. I for sure had lots of fun hanging out with all those people!

18 May 2018 11:53am GMT

17 May 2018

feedPlanet GNOME

Rohit Kaushik: Working on GNOME To Do this Summer

A brief intro about myself
I am Rohit Kaushik (kaushik on IRC) from Delhi, India. I am currently pursuing B.E Computer Science at BITS Pilani, Goa. I am interested in Software Engineering, Machine Learning and Research. I usually spend my free time playing badminton, cricket or listening to music.
Last year, I worked on implementing Todoist for GNOME To Do and this time again I will be working on GNOME To Do, improving the two plugins that I wrote earlier and implementing newer features. I am grateful to GNOME and my mentor feaneron for giving me this opportunity.

About the Project

The project is aimed at improving todo.txt and Todoist error-free and support all the features of To Do. At the end of the project, we wish to have both the plugins in a usable and bug-free state. I am outlining the current list of task that I have planned to work on but any other suggestions are very much welcomed 🙂

Improvements to todo.txt plugin

Improvements to Todoist plugin

I will keep this blog updated with the work in progress. If you have suggestion, please feel free to add them in comments or send me a mail.

Looking forward to a great Summer.

Cheers!

17 May 2018 3:53pm GMT

Carlos Garnacho: Performance hackfest

Last evening I came back from the GNOME performance hackfest happening in Cambridge. There was plenty of activity, clear skies, and pub evenings. Here's some incomplete and unordered items, just the ones I could do/remember/witness/talk/overhear:

Overall, it was a nice and productive event. IMO having people with good knowledge both deep in the stack and wide in GNOME was determining, I hope we can repeat this feat again soon!

17 May 2018 10:50am GMT

16 May 2018

feedPlanet GNOME

Tobias Bernard: Banquets and Barbecues

tl;dr: We're splitting up Fractal into two separate apps: One to replace IRC, the other to replace Telegram.

This is an in-depth post on the thinking behind the split of the Fractal app, which was decided at the hackfest in Strasbourg last week. For more information about the hackfest, have a look at my other blog post.

1-1 woes

One of the biggest problems with Fractal at the moment is that 1-1 messaging is pretty terrible. Since the rooms in the sidebar are sorted by most recent activity, high-traffic public rooms (such as GNOME IRC channels) tend to drown out rooms with less traffic, such as 1-1s and small groups. This is problematic because the signal-to-noise ratio in 1-1 chats and small groups tends to be much higher than in high-traffic public rooms. This leaves the user constantly searching for the rooms they care about, while the rooms they don't care about are always at the top.

1-1 chats are quickly drowned out by high-traffic public rooms in the sidebar

One way to solve this problem is having a favorites group for "important" rooms. This is a feature Fractal has had for a while, and it does solve some of the problems with a room list sorted purely by recent activity. However, it only works well for rooms that are important over long periods of time, and needs to be managed manually. 1-1 chats are often brief, and there can be many of them in parallel. Putting them in favorites doesn't make sense in many cases, as it would balloon the size of the favorites group, and require lots of manual work when starting or ending a conversation.

The "obvious" solution would be doing what Riot does: Having a separate group of 1-1 rooms in the sidebar, and thereby keeping the 1-1 conversations in one consistent place. However, this creates more problems than it solves. In practice, it results in multiple groups of arbitrary length competing for real estate in the sidebar. If you have a lot of 1-1s, this means that you'll be able to see very few rooms (even when most of the 1-1s are old and not relevant at the moment). In Riot, this group is capped at 10 visible rooms by default, but that's still not great if you only need 2 of them at the moment. The category can be collapsed, but then you can't see which 1-1s have new messages, and it also means lots of busywork collapsing/expanding the group. Clearly this isn't an ideal solution, which is why we were very hesitant to go down this path.

Riot's separate 1-1 category doesn't really solve the problem, because old 1-1s take up a ton of vertical real estate when it's expanded

A way out?

As we were discussing this issue over the past few months, I started looking more closely at the way people use different messaging tools. One thing I found puzzling is that despite the fact that Matrix theoretically supports the use cases covered by popular apps like Whatsapp and Telegram, few people are actually using it to replace those apps. Instead, they use it to replace IRC and Slack.

Why? My theory is that most chat rooms fall in one of three categories:
Private Chats, which include 1-1s and small groups; Team Chats, which are larger, but still private and invite-only; and Public Rooms, which are basically like IRC.

Team Chats and Public Rooms share many characteristics: Both have relatively high amounts of traffic, and there's a lot of noise. The main difference is that Team Chats are private and the members rarely change (e.g. a company's internal Slack), while Public Rooms can be joined by anyone at any time, and there is no expectation of privacy (e.g. #gnome-hackers on IRC).

However, Private Chats have relatively little in common with the other two categories: They are low-traffic, and have little or no noise. This may sound like a small difference, but I think it's the reason why 1-1s suck in Fractal/Riot/IRC, and why people aren't using Matrix to replace Telegram.

The Banquet and the Barbecue

I've come to the conclusion that one app can't cover all the use cases that the Matrix protocol supports, and still provide competitive UX. If you design an app to deal with lots of high-traffic rooms (e.g. Riot as it is today), it will suck for 1-1s, so people will use something else for those. Similarly, Telegram is primarily designed for 1-1s and small groups, which is why it's a terrible experience if you have many high-traffic groups.

If we want Matrix to succeed as more than an IRC/Slack replacement we need multiple apps, each focusing on a distinct use case. For messaging, I think the most important distinction to make is between what I call the Banquet and the Barbecue.

Slack is one of the most widely used apps covering the Banquet use case

The Banquet is a big, loud place. There are tons of people, and you don't know many of them. Lots of things are happening all the time, and it's hard to keep track of everything. This is what Matrix is currently mostly used for. Slack, IRC, and Discord are also all in this category.

iMessage is a good example of an app focused on the Barbecue use case

The Barbecue is at the other end of the spectrum: It's a calm, private environment where friends, family, co-workers, and other acquaintances hang out. Conversations are mostly between 2 or 3 people, slow, and often very personal. Telegram, Whatsapp, iMessage, Facebook Messenger, and a myriad of other chat apps are optimized for this use case.

Fracturing Fractal

Now, what does this mean for Fractal? After a long discussion on Thursday, we decided to split up Fractal into two separate apps with different interfaces, each containing a subset of the user's Matrix rooms.

Exactly how rooms will be split between the two apps is not 100% clear yet. 1-1s are clearly Barbecue, public rooms are clearly Banquet, but private groups could go either way. For these cases we may need a way to explicitly move rooms between apps. The distinction should probably be part of the Matrix spec, so the intent for a room to be a Barbecue or Banquet room could be set when creating a room, and persist across devices.

The two apps will share practically all the internals, and even large parts of the interface. However, the split will allow us to do some things differently in each app to optimize the interfaces for the different use cases. Some of the changes we're considering are a bubble-style message view in the Barbecue app, and more room categories (such as low-priority) in the Banquet app's sidebar.

For more details on the split have a look at the blog posts by Daniel, Eisha, Julian, and Adrien.

Messages and Discussions

How exactly the apps will be branded (and what will happen to the Fractal name we all love) is still being decided, but there is some consensus to move to GNOME-style generic names. The Barbecue app will almost certainly be called "Messages". For the Banquet app there's less agreement, but my current favorite is "Discussions".

Early-stage mockups showing what the two different apps could look like

The Fractal brand will not go away though: We're thinking of keeping it around as the name of the community project that develops both GNOME Matrix apps, and/or using it for the backend powering both apps.

There are lots of details to be figured out in this transition, both from a design and an implementation perspective, but I'm very excited about this new direction. If you'd like to join the effort, come talk to us on Matrix.

Note: I have no illusions that this change will magically get everyone to leave Whatsapp/Telegram/iMessage and move to Matrix. In the short term, the goal is simply to make Matrix 1-1s a good experience. That said, if we ever want Matrix to make inroads with the general public, I think a move in this direction is an important precondition.

16 May 2018 7:07pm GMT

Andy Wingo: lightweight concurrency in lua

Hello, all! Today I'd like to share some work I have done recently as part of the Snabb user-space networking toolkit. Snabb is mainly about high-performance packet processing, but it also needs to communicate with management-oriented parts of network infrastructure. These communication needs are performed by a dedicated manager process, but that process has many things to do, and can't afford to make blocking operations.

Snabb is written in Lua, which doesn't have built-in facilities for concurrency. What we'd like is to have fibers. Fortunately, Lua's coroutines are powerful enough to implement fibers. Let's do that!

fibers in lua

First we need a scheduling facility. Here's the smallest possible scheduler: simply a queue of tasks and a function to run those tasks.

local task_queue = {}

function schedule_task(thunk)
   table.insert(task_queue, thunk)
end

function run_tasks()
   local queue = task_queue
   task_queue = {}
   for _,thunk in ipairs(queue) do thunk() end
end

For our purposes, a task is just a function that will be called with no arguments.

Now let's build fibers. This is easier than you might think!

local current_fiber = false

function spawn_fiber(fn)
   local fiber = coroutine.create(fn)
   schedule_task(function () resume_fiber(fiber) end)
end

function resume_fiber(fiber, ...)
   current_fiber = fiber
   local ok, err = coroutine.resume(fiber, ...)
   current_fiber = nil
   if not ok then
      print('Error while running fiber: '..tostring(err))
   end
end

function suspend_current_fiber(block, ...)
   -- The block function should arrange to reschedule
   -- the fiber when it becomes runnable.
   block(current_fiber, ...)
   return coroutine.yield()
end

Here, a fiber is simply a coroutine underneath. Suspending a fiber suspends the coroutine. Resuming a fiber runs the coroutine. If you're unfamiliar with coroutines, or coroutines in Lua, maybe have a look at the lua-users wiki page on the topic.

The difference between a fibers facility and just coroutines is that with fibers, you have a scheduler as well. Very much like Scheme's call-with-prompt, coroutines are one of those powerful language building blocks that should rarely be used directly; concurrent programming needs more structure than what Lua offers.

If you're following along, it's probably worth it here to think how you would implement yield based on these functions. A yield implementation should yield control to the scheduler, and resume the fiber on the next scheduler turn. The answer is here.

communication

Once you have fibers and a scheduler, you have concurrency, which means that if you're not careful, you have a mess. Here I think the Go language got the essence of the idea exactly right: Do not communicate by sharing memory; instead, share memory by communicating.

Even though Lua doesn't support multiple machine threads running concurrently, concurrency between fibers can still be fraught with bugs. Tony Hoare's Communicating Sequential Processes showed that we can avoid a class of these bugs by treating communication as a first-class concept.

Happily, the Concurrent ML project showed that it's possible to build these first-class communication facilities as a library, provided the language you are working in has threads of some kind, and fibers are enough. Last year I built a Concurrent ML library for Guile Scheme, and when in Snabb we had a similar need, I ported that code over to Lua. As it's a new take on the problem in a different language, I think I've been able to simplify things even more.

So let's take a crack at implementing Concurrent ML in Lua. In CML, the fundamental primitive for communication is the operation. An operation represents the potential for communication. For example, if you have a channel, it would have methods to return "get operations" and "put operations" on that channel. Actually receiving or sending a message on a channel occurs by performing those operations. One operation can be performed many times, or not at all.

Compared to a system like Go, for example, there are two main advantages of CML. The first is that CML allows non-deterministic choice between a number of potential operations in a generic way. For example, you can construct a operation that, when performed, will either get on one channel or wait for a condition variable to be signalled, whichever comes first. In Go, you can only select between operations on channels.

The other interesting part of CML is that operations are built from a uniform protocol, and so users can implement new kinds of operations. Compare again to Go where all you have are channels, and nothing else.

The CML operation protocol consists three related functions: try which attempts to directly complete an operation in a non-blocking way; block, which is called after a fiber has suspended, and which arranges to resume the fiber when the operation completes; and wrap, which is called on the result of a successfully performed operation.

In Lua, we can call this an implementation of an operation, and create it like this:

function new_op_impl(try, block, wrap)
   return { try=try, block=block, wrap=wrap }
end

Now let's go ahead and write the guts of CML: the operation implementation. We'll represent an operation as a Lua object with two methods. The perform method will attempt to perform the operation, and return the resulting value. If the operation can complete immediately, the call to perform will return directly. Otherwise, perform will suspend the current fiber and arrange to continue only when the operation completes.

The wrap method "decorates" an operation, returning a new operation that, if and when it completes, will "wrap" the result of the completed operation with a function, by applying the function to the result. It's useful to distinguish the sub-operations of a non-deterministic choice from each other.

Here our new_op function will take an array of operation implementations and return an operation that, when performed, will synchronize on the first available operation. As you can see, it already has the equivalent of Go's select built in.

function new_op(impls)
   local op = { impls=impls }
   
   function op.perform()
      for _,impl in ipairs(impls) do
         local success, val = impl.try()
         if success then return impl.wrap(val) end
      end
      local function block(fiber)
         local suspension = new_suspension(fiber)
         for _,impl in ipairs(impls) do
            impl.block(suspension, impl.wrap)
         end
      end
      local wrap, val = suspend_current_fiber(block)
      return wrap(val)
   end

   function op.wrap(f)
      local wrapped = {}
      for _, impl in ipairs(impls) do
         local function wrap(val)
            return f(impl.wrap(val))
         end
         local impl = new_op_impl(impl.try, impl.block, wrap)
         table.insert(wrapped, impl)
      end
      return new_op(wrapped)
   end

   return op
end

There's only one thing missing there, which is new_suspension. When you go to suspend a fiber because none of the operations that it's trying to do can complete directly (i.e. all of the try functions of its impls returned false), at that point the corresponding block functions will publish the fact that the fiber is waiting. However the fiber only waits until the first operation is ready; subsequent operations becoming ready should be ignored. The suspension is the object that manages this state.

function new_suspension(fiber)
   local waiting = true
   local suspension = {}
   function suspension.waiting() return waiting end
   function suspension.complete(wrap, val)
      assert(waiting)
      waiting = false
      local function resume()
         resume_fiber(fiber, wrap, val)
      end
      schedule_task(resume)
   end
   return suspension
end

As you can see, the suspension's complete method is also the bit that actually arranges to resume a suspended fiber.

Finally, just to round out the implementation, here's a function implementing non-deterministic choice from among a number of sub-operations:

function choice(...)
   local impls = {}
   for _, op in ipairs({...}) do
      for _, impl in ipairs(op.impls) do
         table.insert(impls, impl)
      end
   end
   return new_op(impls)
end

on cml

OK, I'm sure this seems a bit abstract at this point. Let's implement something concrete in terms of these primitives: channels.

Channels expose two similar but different kinds of operations: put operations, which try to send a value, and get operations, which try to receive a value. If there's a sender already waiting to send when we go to perform a get_op, the operation continues directly, and we resume the sender; otherwise the receiver publishes its suspension to a queue. The put_op case is similar.

Finally we add some synchronous put and get convenience methods, in terms of their corresponding CML operations.

function new_channel()
   local ch = {}
   -- Queues of suspended fibers waiting to get or put values
   -- via this channel.
   local getq, putq = {}, {}

   local function default_wrap(val) return val end
   local function is_empty(q) return #q == 0 end
   local function peek_front(q) return q[1] end
   local function pop_front(q) return table.remove(q, 1) end
   local function push_back(q, x) q[#q+1] = x end

   -- Since a suspension could complete in multiple ways
   -- because of non-deterministic choice, it could be that
   -- suspensions on a channel's putq or getq are already
   -- completed.  This helper removes already-completed
   -- suspensions.
   local function remove_stale_entries(q)
      local i = 1
      while i <= #q do
         if q[i].suspension.waiting() then
            i = i + 1
         else
            table.remove(q, i)
         end
      end
   end

   -- Make an operation that if and when it completes will
   -- rendezvous with a receiver fiber to send VAL over the
   -- channel.  Result of performing operation is nil.
   function ch.put_op(val)
      local function try()
         remove_stale_entries(getq)
         if is_empty(getq) then
            return false, nil
         else
            local remote = pop_front(getq)
            remote.suspension.complete(remote.wrap, val)
            return true, nil
         end
      end
      local function block(suspension, wrap)
         remove_stale_entries(putq)
         push_back(putq, {suspension=suspension, wrap=wrap, val=val})
      end
      return new_op({new_op_impl(try, block, default_wrap)})
   end

   -- Make an operation that if and when it completes will
   -- rendezvous with a sender fiber to receive one value from
   -- the channel.  Result is the value received.
   function ch.get_op()
      local function try()
         remove_stale_entries(putq)
         if is_empty(putq) then
            return false, nil
         else
            local remote = pop_front(putq)
            remote.suspension.complete(remote.wrap)
            return true, remote.val
         end
      end
      local function block(suspension, wrap)
         remove_stale_entries(getq)
         push_back(getq, {suspension=suspension, wrap=wrap})
      end
      return new_op({new_op_impl(try, block, default_wrap)})
   end

   function ch.put(val) return ch.put_op(val).perform() end
   function ch.get()    return ch.get_op().perform()    end

   return ch
end

a wee example

You might be wondering what it's like to program with channels in Lua, so here's a little example that shows a prime sieve based on channels. It's not a great example of concurrency in that it's not an inherently concurrent problem, but it's cute to show computations in terms of infinite streams.

function prime_sieve(count)
   local function sieve(p, rx)
      local tx = new_channel()
      spawn_fiber(function ()
         while true do
            local n = rx.get()
            if n % p ~= 0 then tx.put(n) end
         end
      end)
      return tx
   end

   local function integers_from(n)
      local tx = new_channel()
      spawn_fiber(function ()
         while true do
            tx.put(n)
            n = n + 1
         end
      end)
      return tx
   end

   local function primes()
      local tx = new_channel()
      spawn_fiber(function ()
         local rx = integers_from(2)
         while true do
            local p = rx.get()
            tx.put(p)
            rx = sieve(p, rx)
         end
      end)
      return tx
   end

   local done = false
   spawn_fiber(function()
      local rx = primes()
      for i=1,count do print(rx.get()) end
      done = true
   end)

   while not done do run_tasks() end
end

Here you also see an example of running the scheduler in the last line.

where next?

Let's put this into perspective: in a couple hundred lines of code, we've gone from minimal Lua to a language with lightweight multitasking, extensible CML-based operations, and CSP-style channels; truly a delight.

There are a number of possible ways to extend this code. One of them is to implement true multithreading, if the language you are working in supports that. In that case there are some small protocol modifications to take into account; see the notes on the Guile CML implementation and especially the Manticore Parallel CML project.

The implementation above is pleasantly small, but it could be faster with the choice of more specialized data structures. I think interested readers probably see a number of opportunities there.

In a library, you might want to avoid the global task_queue and implement nested or multiple independent schedulers, and of course in a parallel situation you'll want core-local schedulers as well.

The implementation above has no notion of time. What we did in the Snabb implementation of fibers was to implement a timer wheel, inspired by Juho Snellman's Ratas, and then add that timer wheel as a task source to Snabb's scheduler. In Snabb, every time the equivalent of run_tasks() is called, a scheduler asks its sources to schedule additional tasks. The timer wheel implementation schedules expired timers. It's straightforward to build CML timeout operations in terms of timers.

Additionally, your system probably has other external sources of communication, such as sockets. The trick to integrating sockets into fibers is to suspend the current fiber whenever an operation on a file descriptor would block, and arrange to resume it when the operation can proceed. Here's the implementation in Snabb.

The only difficult bit with getting nice nonblocking socket support is that you need to be able to suspend the calling thread when you see the EWOULDBLOCK condition, and for coroutines that is often only possible if you implemented the buffered I/O yourself. In Snabb that's what we did: we implemented a compatible replacement for Lua's built-in streams, in Lua. That lets us handle EWOULDBLOCK conditions in a flexible manner. Integrating epoll as a task source also lets us sleep when there are no runnable tasks.

Likewise in the Snabb context, we are also working on a TCP implementation. In that case you want to structure TCP endpoints as fibers, and arrange to suspend and resume them as appropriate, while also allowing timeouts. I think the scheduler and CML patterns are going to allow us to do that without much trouble. (Of course, the TCP implementation will give us lots of trouble!)

Additionally your system might want to communicate with fibers from other threads. It's entirely possible to implement CML on top of pthreads, and it's entirely possible as well to support communication between pthreads and fibers. If this is interesting to you, see Guile's implementation.

When I talked about fibers in an earlier article, I built them in terms of delimited continuations. Delimited continuations are fun and more expressive than coroutines, but it turns out that for fibers, all you need is the expressive power of coroutines -- multi-shot continuations aren't useful. Also I think the presentation might be more straightforward. So if all your language has is coroutines, that's still good enough.

There are many more kinds of standard CML operations; implementing those is also another next step. In particular, I have found semaphores and condition variables to be quite useful. Also, standard CML supports "guards", invoked when an operation is performed, and "nacks", invoked when an operation is definitively not performed because a choice selected some other operation. These can be layered on top; see the Parallel CML paper for notes on "primitive CML".

Also, the choice operator above is left-biased: it will prefer earlier impls over later ones. You might want to not always start with the first impl in the list.

The scheduler shown above is the simplest thing I could come up with. You may want to experiment with other scheduling algorithms, e.g. capability-based scheduling, or kill-safe abstractions. Do it!

Or, it could be you already have a scheduler, like some kind of main loop that's already there. Cool, you can use it directly -- all that fibers needs is some way to schedule functions to run.

godspeed

In summary, I think Concurrent ML should be better-known. Its simplicity and expressivity make it a valuable part of any concurrent system. Already in Snabb it helped us solve some longstanding gnarly issues by making the right solutions expressible.

As Adam Solove says, Concurrent ML is great, but it has a branding problem. Its ideas haven't penetrated the industrial concurrent programming world to the extent that they should. This article is another attempt to try to get the word out. Thanks to Adam for the observation that CML is really a protocol; I'm sure the concepts could be made even more clear, but at least this is a step forward.

All the code in this article is up on a gitlab snippet along with instructions for running the example program from the command line. Give it a go, and happy hacking with CML!

16 May 2018 3:17pm GMT

Alberto Ruiz: GNOME Performance Hackfest

We're about to finish the three days long first GNOME Performance Hackfest here in Cambridge.

We started covering a few topics, there are three major areas we've covered and in each one of those there has been a bunch of initiatives.

photo_2018-05-15_16-05-20

photo_2018-05-16_16-45-16

GNOME Shell performance

Jonas Adahl, Marco Trevisan, Eric Anholt, Emmanuele Bassi, Carlos Garnacho and Daniel Stone have been flocking together around Shell performance. There has been some high level discussions about the pipeline, Clutter, Cogl, cairo/gtk3 and gtk4.

The main effort has been around creating probes across the stack to help Christian Hergert with sysprof (in drm, mutter, gjs…) so that we can actually measure performance bottlenecks at different levels and pinpoint culprits.

We've been also looking at the story behind search providers and see if we can rearchitect things a bit to have less roundtrips and avoid persistent session daemons to achieve the same results. Discussions are still ongoing on that front.

GNOME Session resource consumption

Hans de Goede put together a summary of the resource consumed in a typical GNOME session in Fedora and tweaks to avoid those, you can check the list in the agenda.

There are some issues specific to Fedora there, but the biggest improvement that we can achieve is shutting down's GDM's own gnome-shell instance, for which Hans already has a working patch. This should reduce resource consumption by 280megs of RAM.

The second biggest target is GNOME Software, which we keep running primarily for the shell provider. Richard Hughes was here yesterday and is already working on a solution for this.

We are also looking into the different GNOME Settings Daemon processes and trying to figure out which ones we can shut down until needed.

Surely there's stuff I've missed, and hopefully we'll see blogposts and patches surfacing soon after we wrap up the event. Hopefully we can follow up during GUADEC and start showing the results.

On Tuesday we enjoyed some drinks out kindly hosted by Collabora.

collabora-logo-small
I'd like to thank the Eben Upton and the Raspberry Pi Foundation for sponsoring the venue and sending Eric Anholt over.

raspberry-pi-logo-8240ABBDFE-seeklogo.com

16 May 2018 3:06pm GMT

Eisha Chen-yen-su: Internationalization of Fractal (part 2)

A textual program using gettext

For my investigations, I first tried to write a textual program that works with gettext. I spent quite some time to figure out how all of this works but I finally was able to make it work. And that means that we should be able to implement i18n for Fractal using gettext!

I created a new Rust binary project with cargo and I added gettext-rs as a dependency in the Cargo.toml file by adding this line:

gettext-rs = { git = "https://github.com/Koka/gettext-rs", features = ["gettext-system"] }

I edited the main.rs source file to have a basic test to see if gettext-rs works correctly, here is its content:

extern crate gettextrs;
use gettextrs::*;

fn main() {
     setlocale(LocaleCategory::LcAll, "");
     bindtextdomain("test", "./po");
     textdomain("test");

    println!("Original: Open");
    println!("Translated: {}", gettext("Open"));

    println!("Original: Text Editor");
    println!("Translated singular: {}", ngettext("Text Editor", "Text Editors", 1));
    println!("Translated plural: {}", ngettext("Text Editor", "Text Editors", 2));
}

In the root directory of the project, I created a directory named "po", then a subdirectory named "fr" (as I want to do a little French translation) inside "po" and then again, a directory "LC_MESSAGES" inside "fr". So that I had the same folder hierarchy inside my "po" directory as in "/usr/share/locale", because gettext will look for the translations in MO files that are in directories like "[TARGET]/[LANGUAGE]/LC_MESSAGES/[PACKAGE].mo". [TARGET] and [PACKAGE] are defined by the call "bindtextdomain([PACKAGE], [TARGET]);" in the program. So here, the package name is "test" and the targeted directory is "./po".

Next, I used the utility xgettext to extract all the translatable strings from the source file with this command:

xgettext -o po/template.pot src/*

And I edited template.pot to replace "CHARSET" with "UTF-8". To get the file in which I would do the French translation, I used:

msginit --locale=fr --input=po/template.pot --output=po/fr/test.po

Finally, I added the translations of the strings on the lines with "msgid" in the lines with "msgstr" and my PO file had this content:

# French translations for PACKAGE package.
# Copyright (C) 2018 THE PACKAGE'S COPYRIGHT HOLDER
# This file is distributed under the same license as the PACKAGE package.
# Eisha <eisha@ernesto>, 2018.
#
msgid ""
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2018-05-15 12:33+0200\n"
"PO-Revision-Date: 2018-05-15 12:34+0200\n"
"Last-Translator: Eisha <eisha@ernesto>\n"
"Language-Team: French\n"
"Language: fr\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=(n > 1);\n"

#: ../src/main.rs:10
msgid "Open"
msgstr "Ouvrir"

#: ../src/main.rs:13 ../src/main.rs:14
msgid "Text Editor"
msgid_plural "Text Editors"
msgstr[0] "Un éditeur de texte"
msgstr[1] "Des éditeurs de texte"

I compiled the MO file like this:

mgsfmt po/fr/test.po  -o po/fr/LC_MESSAGES/test.mo

The the program was finally ready to test, so when launching:

LANG=en_US cargo run

I get this output:

Original: Open
Translated: Open
Original: Text Editor
Translated singular: Text Editor
Translated plural: Text Editors

And when I launch it like this:

LANG=fr_FR cargo run

I get the expected output:

Original: Open
Translated: Ouvrir
Original: Text Editor
Translated singular: Un éditeur de texte
Translated plural: Des éditeurs de texte

You can find the program I have written in this repository.

A GTK program using gettext

When I got this textual program working, I had to test it with a GTK program in order to know if gettext-rs could still translate the strings in the source file and most importantly, if/how it could also translate strings from Glade files.

So I wrote a GTK program (with the crate gtk-rs) for which I created an empty window with the following title "A window title from Glade" with Glade, I saved this minimal UI in the file ui/main_window.glade. And I wrote the following code in main.rs:

extern crate gtk;
extern crate gettextrs;

use gtk::prelude::*;

use gettextrs::*;

fn main() {
    setlocale(LocaleCategory::LcAll, "");
    bindtextdomain("test", "./po");
    textdomain("test");

    gtk::init().unwrap();

    let window: gtk::Window = gtk::Builder::new_from_file("./ui/main_window.glade")
        .get_object("main_window").expect("Failed to load the main window");

    // UI initialization
    let label = gtk::Label::new(gettext("A label from the source code").as_str());
    window.add(&label);
    window.show_all();

    window.connect_delete_event(|_, _| {
        gtk::main_quit();
        Inhibit(false)
    });

    gtk::main();
}

Then, I followed the same procedure as before to generate the POT, PO and MO files, except that I also indicated to xgettext to parse the Glade file like this:

xgettext -o po/template.pot src/* ui/*

So that the content of po/fr/test.po was (at the end of the procedure):

# French translations for PACKAGE package.
# Copyright (C) 2018 THE PACKAGE'S COPYRIGHT HOLDER
# This file is distributed under the same license as the PACKAGE package.
# Eisha <eisha@ernesto>, 2018.
#
msgid ""
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2018-05-15 19:16+0200\n"
"PO-Revision-Date: 2018-05-15 19:17+0200\n"
"Last-Translator: Eisha <eisha@ernesto>\n"
"Language-Team: French\n"
"Language: fr\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=(n > 1);\n"

#: ../src/main.rs:19
msgid "A label from the source code"
msgstr "Une étiquette du code source"

#: ../ui/main_window.glade:7
msgid "A window title from Glade"
msgstr "Un titre de fenêtre de Glade"

When running this program with:

LANG=en_US cargo run

The window opened is like this:

Capture du 2018-05-16 11-09-51

And when running it with:

LANG=fr_FR cargo run

The window opened is like this:

Capture du 2018-05-16 11-11-54

So it seems that GtkBuilder is calling gettext when parsing the UI file.

You can find this other program I have written in this repository.

So now I can start to implement i18n support within Fractal! Although I will also have to figure out how to properly integrate this with meson.

16 May 2018 9:37am GMT

Alexander Larsson: Introducing the 1.8 freedesktop runtime in the gnome nightly builds

All the current Flatpak runtimes in wide use are based on the 1.6 Freedesktop runtime. This is a two-layered beast where the lower layer is built using Yocto and the upper layer is built using flatpak-builder.

Yocto let us get a basic system bootstrapped, but it is not really a great match for the needs of a Flatpak runtime. We've long wanted a base that targeted sandboxed builds which is closer to upstream and without the embedded cross-compiled legacy of Yocto. This is part of the reason why Tristan started working on BuildStream, and why the 1.8 Freedesktop runtime was created from scratch, using it.

After a herculean effort from the people at Codethink (who sponsor this effort) we now have a working Yocto-free version of the Platform and Sdk. You can download the unstable version from here and start playing with it. It is not yet frozen or API/ABI stable, but its getting there.

The next step in this effort is to test it as widely as possible to catch any issues before the release is frozen. In order to do this I rebased the Gnome nightly runtime builds on top of the new Freedesktop version this week. This is a good match for a test release, because it has no real ABI requirements (being rebuilt with the apps daily), yet gets a fair amount of testing.

WARNING: During the initial phase it is likely that there will be problems. Please test your apps extra carefully and report all the issues you find.

In the future, the goal is to also convert the Gnome runtimes to BuildStream. Work on this has started, but for now we want to focus on getting the base runtime stable.

16 May 2018 8:05am GMT

Tobias Bernard: Fractal Hackfest in Strasbourg

Last week we had an intense 4-day hackfest in Strasbourg to map out the future of Fractal, a native GNOME Matrix messaging app. The event was held at Epitech in Strasbourg's old town, and organized by Alexandre Franke. Among the attendees were core Fractal contributors Daniel, Alexandre, Eisha, and Julian, as well as Dorota, Adrien, and Francois from Purism. Special thanks go to Matthew from the Matrix core team for joining us on the first two days.

Our main priorities for the hackfest were to plan the roadmap for the next months, decide on the tasks for our GSoC students (Eisha and Julian), and work on the design of some important missing features, like the room settings.

I personally attended the hackfest in both my role as designer on the Fractal project and as a Purism employee currently working on the apps for the Librem 5. One of the reasons why several members of the Librem 5 team attended the hackfest was that we will need a Matrix messaging app on the phone and wanted to explore a potential collaboration.

The hackfest was extremely productive, so much so that I'll need multiple blog posts to report on all the things we worked on. Here's a quick outline of some of the most important things that happened:

Matthew explaining device verification for E2E chats

Thanks everyone for attending, Epitech and Alexandre for the venue, and Purism for sponsoring the hackfest! It was amazingly productive to have everyone in one place, and I look forward to seeing you all at GUADEC in July :)

16 May 2018 7:07am GMT