fidzu : kde

A few months ago, we received a phone call from a bioinformatics group at a European university. The problem they were having appeared very simple. They wanted to know how to usemmap() to be able to load a large data set into RAM at once. OK I thought, no problem, I can handle that one. Turns out this has grown into a complex and interesting exercise in profiling and threading.

The background is that they are performing Markov-Chain Monte Carlo simulations by sampling at random from data sets containing SNP (pronounced "snips") genetic markers for a selection of people. It boils down to a large 2D matrix of floats where each column corresponds to an SNP and each row to a person. They provided some small and medium sized data sets for me to test with, but their full data set consists of 500,000 people with 38 million SNP genetic markers!

The analysis involves selecting a column (SNP) at random in the data set and then performing some computations on the data for all of the individuals and collecting some summary statistics. Do that for all of the columns in the data set, and then repeat for a large number of iterations. This allows you to approximate the underlying true distribution from the discreet data that has been collected.

That's the 10,000 ft view of the problem, so what was actually involved? Well we undertook a bit of an adventure and learned some interesting stuff along the way, hence this blog series.

The stages we went through were:

1. Preprocessing
2. Loading the Data
3. Fine-grained Threading
4. Preprocessing Reprise
5. Coarse Threading

In this blog, I'll detail stages 1 and 2. The rest of the process will be revealed as the blog series unfolds, and I'll include a final summary at the end.

1. Preprocessing

The first thing we noticed when looking at the code they already had is that there is quite some work being done when reading in the data for each column. They do some summary statistics on the column, then scale and bias all the data points in that column such that the mean is zero. Bearing in mind that each column will be processed many times, (typically 10k - 1 million), this is wasteful to repeat every time the column is used.

So, reusing some general advice from 3D graphics, we moved this work further up the pipeline to a preprocessing step. The SNP data is actually stored in a compressed form which takes the form of quantizing 4 SNP values into a few bytes which we then decompress when loading. So the preprocessing step does the decompression of SNP data, calculates the summary statistics, adjusts the data and then writes the floats out to disk in the form of a ppbed file (preprocessed bed where bed is a standard format used for this kind of data).

The upside is that we avoid all of this work on every iteration of the Monte Carlo simulation at runtime. The downside is that 1 float per SNP per person adds up to a hell of a lot of data for the larger data sets! In fact, for the full data set it's just shy of 69 TB of floating point data! But to get things going, we were just worrying about smaller subsets. We will return to this later.

2. Loading the data

Even on moderately sized data sets, loading the entirety of the data set into physical RAM at once is a no-go as it will soon exhaust even the beefiest of machines. They have some 40 core, many-many-GB-of-RAM machine which was still being exhausted. This is where the original enquiry was aimed - how to use mmap(). Turns out it's pretty easy as you'd expect. It's just a case of setting the correct flags so that the kernel doesn't actually take a copy of the data in the file. Namely, PROT_READ and MAP_SHARED:

void Data::mapPreprocessBedFile(const string &preprocessedBedFile)
{
    // Calculate the expected file sizes - cast to size_t so that we don't overflow the unsigned int's
    // that we would otherwise get as intermediate variables!
    const size_t ppBedSize = size_t(numInds) * size_t(numIncdSnps) * sizeof(float);
 
    // Open and mmap the preprocessed bed file
    ppBedFd = open(preprocessedBedFile.c_str(), O_RDONLY);
    if (ppBedFd == -1)
        throw("Error: Failed to open preprocessed bed file [" + preprocessedBedFile + "]");
 
    ppBedMap = reinterpret_cast<float *>(mmap(nullptr, ppBedSize, PROT_READ, MAP_SHARED, ppBedFd, 0));
    if (ppBedMap == MAP_FAILED)
        throw("Error: Failed to mmap preprocessed bed file");
 
    ...
}

When dealing with such large amounts of data, be careful of overflows in temporaries! We had a bug where ppBedSize was overflowing and later causing a segfault.

So, at this point we have a float *ppBed pointing at the start of the huge 2D matrix of floats. That's all well and good but not very convenient for working with. The code base already made use of Eigen for vector and matrix operations so it would be nice if we could interface with the underlying data using that.

Turns out we can (otherwise I wouldn't have mentioned it). Eigen provides VectorXf and MatrixXf types for vectors and matrices but these own the underlying data. Luckily Eigen also provides a wrapper around these in the form of Map. Given our pointer to the raw float data which is mmap()'d, we can use the placement new operator to wrap it up for Eigen like so:

class Data
{
public:
    Data();
 
    // mmap related data
    int ppBedFd;
    float *ppBedMap;
    Map<MatrixXf> mappedZ;
}
 
 
void Data::mapPreprocessBedFile(const string &preprocessedBedFile)
{
    ...
 
    ppBedMap = reinterpret_cast<float *>(mmap(nullptr, ppBedSize, PROT_READ, MAP_SHARED, ppBedFd, 0));
    if (ppBedMap == MAP_FAILED)
        throw("Error: Failed to mmap preprocessed bed file");
 
    new (&mappedZ) Map<MatrixXf>(ppBedMap, numRows, numCols);
}

At this point we can now do operations on the mappedZ matrix and they will operate on the huge data file which will be paged in by the kernel as needed. We never need to write back to this data so we didn't need the PROT_WRITE flag for mmap.

Yay! Original problem solved and we've saved a bunch of work at runtime by preprocessing. But there's a catch! It's still slow. See the next blog in the series for how we solved this.

The post Little Trouble in Big Data - Part 1 appeared first on KDAB.

23 May 2019 1:05pm GMT

Elisa is a music player developed by the KDE community that strives to be simple and nice to use. We also recognize that we need a flexible product to account for the different workflows and use-cases of our users.

We focus on a very good integration with the Plasma desktop of the KDE community without compromising the support for other platforms (other Linux desktop environments, Windows and Android).

We are creating a reliable product that is a joy to use and respects our users privacy. As such, we will prefer to support online services where users are in control of their data.

I am happy to announce the release of 0.4.0 version of the Elisa music player.

The new features are explained in the following posts New features in Elisa, New Features in Elisa: part 2 and Elisa 0.4 Beta Release and More New Features.

There have been a couple more changes not yet covered.

Improved Grid Views Elements

Nate Graham has reworked the grid elements (especially visible with the albums view).

I must confess that I was a bit uneasy with this change (it was a part mostly unchanged since the early versions). I am now very happy about this change.

Before After

Getting Involved

I would like to thank everyone who contributed to the development of Elisa, including code contributions, testing, and bug reporting and triaging. Without all of you, I would have stopped working on this project.

New features and fixes are already being worked on. If you enjoy using Elisa, please consider becoming a contributor yourself. We are happy to get any kind of contributions!

We have some tasks that would be perfect junior jobs. They are a perfect way to start contributing to Elisa. There are more not yet reported here but reported in bugs.kde.org.

The flathub Elisa package allows an easy way to test this new release.

Elisa source code tarball is available here. There is no Windows setup. There is currently a blocking problem with it (no icons) that is being investigated. I hope to be able to provide installers for later bugfix versions.

The phone/tablet port project could easily use some help to build an optimized interface on top of Kirigami. It remains to be seen how to handle this related to the current desktop UI.

22 May 2019 8:28pm GMT

Thank you all for the large amount of feedback I have received after my previous Wayland Itches blog post. I've received over 40 mails, below is an attempt at summarizing all the mails.

Highlights

1. Middle click on title / header bar to lower the Window does not work for native apps. Multiple people have reported this issue to me. A similar issue was fixed for not being able to raise Windows. It should be easy to apply a similar fix for the lowering problem. There are bugs open for this here, here and here.

2. Running graphical apps via sudo or pxexec does not work. There are numerous examples of apps breaking because of this, such as lshw-gui and usbivew. At least for X11 apps this is not that hard to fix. But sofar this has deliberately not been fixed. The reasoning behind this is described in this bug. I agree with the reasoning behind this, but I think it is not pragmatic to immediately disallow all GUI apps to connect when run as root starting today.

We need some sort of transition period. So when I find some time for this, I plan to submit a merge-requests which optionally makes gnome-shell/mutter start Xwayland with an xauth file, like how it is done when running in GNOME on Xorg mode. This will be controlled by a gsettings option, which will probably default to off upstream and then distros can choice to override this for now, giving us a transition period

Requests for features implemented as external programs on X11

There are various features which can be implemented as external programs
on X11, but because of the tighter security need to be integrated into the
compositor with Wayland:

• 
  
• Hiding of the mouse-cursor when not used à la unclutter-xfixes, xbanish.
• 
  
• Rotating screen 90 / 270 degrees à la "xrandr -o [left|right]" mostly used through custom hotkeys, possible fix is defining bindable actions for this in gsd-media-keys.
• 
  
• Mapping actions to mouse buttons à la easystroke
• 
  
• Some touchscreen's, e.g. so called smart-screens for education, need manual calibration. Under X11 there are some tools to get the callibration matrix for the touchscreen, after which this can be manually applied through xinput. Even under X11 this currently is far from ideal but at least it is possible there.
• 
  
• Keys Indicator gnome-shell extension. This still works when using Wayland, but only works for apps using Xwayland, it does not work for native apps.
• 
  
• Some sort of xkill and xdotool utility equivalents would be nice
• 
  
• The GNOME on screen keyboard is not really suitable for use with apps which are not touch-enabled, as it lacks a way to send ctrl + key, etc. Because of this some users have reported that it is impossible to use alternative on screen keyboards with Wayland. Not being able to use alternative on screen keyboards is by design and IMHO the proper fix here is to improve GNOME's on screen keyboard.
• 
  

App specific problems

• 
  
• Citrix ICA Client does not work well with Xwayland
• 
  
• Eclipse does not work well with Xwayland
• 
  
• Teamviewer does not work with Wayland. It needs to be updated to use pipewire for screencapturing and the RemoteDesktop portal to inject keyboard and mouse events.
• 
  
• Various apps lack screenrecording / capture support due to the app not having support for pipewire: gImageReader, green-recorder, OBS studio, peek, screenrecorder, slack
• 
  
• For apps which do support pipewire, there is not an option to share the contents of a window, other then the window making the request. On Xorg it is possible to share a random window and since pipewire allows sharing the whole desktop I see no security reason why we would not allow sharing another window.
• 
  
• guake window has incorrect size when using HiDPI scaling, see this issue
• 
  

Miscellaneous problems

• 
  
• Mouse cursor is slow / lags
• 
  
• Drag and drop sometimes does not work, e.g. dragging files into file roller to compress or out of file roller to extract.
• 
  
• Per keyboard layouts. On X11 after plugging in a keyboard, the layout/keymap for just that one keyboard can be updated manually using xinput, allowing different keyboard layouts for different keyboards when multiple keyboards are connected
• 
  
• No-title-bar shell extension, X button can be hit unintentionally, see this issue
• 
  
• Various issues with keyboard layout switching
• 
  

Hard to fix issues

• 
  
• Alt-F2, r equivalent (restart the gnome-shell)
• 
  
• X11 apps running on top of Xwayland do not work well on HiDPI screens
• 
  
• Push-to-talk (passive key grab on space) does not work in Mumble when using native Wayland apps, see this issue
• 
  

Problems with other compositors then GNOME3 / mutter

I've also received several reports about issues when using another Wayland compositor as GNOME / mutter (Weston, KDE, Sway). I'm sorry but I have not looked very closely into these reports. I believe that it is great that Linux users have multiple Desktop Environments to choose from and I wish for the other DEs to thrive. But there are only so many hours in a day so I've chosen to mainly focus on GNOME.

22 May 2019 7:01pm GMT

First of all I do not want people to get their hopes up about $subject of this blogpost. Improving gaming support is a subjects which holds my personal interest and it is an issue I plan to spend time on trying to improve. But this will take a lot of time (think months for simple things, years for more complex things).

As I see it there are currently 2 big issues when running games under Wayland:

1. Many games show as a smal centered image with a black border (letterbox) around the image when running fullscreen.

For 2D games this is fixed by switching to SDL2 which will transparently scale the pixmap the game renders to the desktop resolution. This assumes that 2D games in general do not demand a lot of performance and thus will not run into performance issues when introducing an extra scaling step. A problem here is that many games still use SDL1.2 (and some games do not use SDL at all).

I plan to look into the recently announced SDL1.2 compatibility wrapper around SDL2. If this works well this should fix this issue for all SDL1.2 2D games, by making them use SDL2 under the hood.

For 3D games this can be fixed by rendering at the desktop resolution, but this might be slow and rendering at a lower resolution leads to the letterbox issue.

Recently mutter has has grown support for the WPviewport extension, which allows Wayland apps to tell the compositor to scale the pixmap the app gives to the compositor before presenting it. If we add support to SDL2's Wayland backend for this then, this can be used to allow rendering 3D apps at a lower resolution and still have them fill the entire screen.

Unfortunately there are 2 problems with this plan:

1. 
   
2. SDL2 by default uses its x11 backend, not its wayland backend. I'm not sure what fixes need to be done to change this, at a minimum we need a fix at either the SDL or mutter side for this issue, which is going to be tricky.
   
3. This only helps for SDL2 apps, again hopefully the SDL1.2 compatibility wrapper for SDL2 can help here, at least for games using SDL.
   

2. Fullscreen performance is bad with many games.

Since under Wayland games cannot change the monitor resolution, they need to either render at the full desktop resolution, which can be very slow; or they render at a lower resolution and then need to do an extra scaling step each frame.

If we manage to make SDL2's Wayland backend the default and then add WPviewport support to it then this should help by reducing an extra memcpy/blit of a desktop-sized pixmap. Currently what apps which use scaling do is:

1. 
   
2. render lower-res-pixmap;
   
3. scale lower-res-pixmap to desktop-res-pixmap
   
4. give desktop-res-pixmap to the compositor;
   
5. compositor does a hardware blit of the desktop-res-pixmap to the framebuffer.
   

With viewport support this becomes:

1. 
   
2. render lower-res-pixmap;
   
3. give low-res-pixmap to the compositor;
   
4. compositor uses hardware to do a scaling blit from the low-res-pixmap to the desktop-res framebuffer
   

Also with viewport support, the compositor could in the case of there only being the one fullscreen app even keep the framebuffer in lowres and use a hardware scaling drm-plane to send the low-res framebuffer scaled to desktop-res to the output while only reading the low-res framebuffer from memory saving a ton of memory bandwidth. But this optimization is going to be a challenge to pull off.

22 May 2019 4:44pm GMT

KDAB has released a new version of KDSoap. This is version 1.8.0 and comes more than one year since the last release (1.7.0).

KDSoap is a tool for creating client applications for web services without the need for any further component such as a dedicated web server.

KDSoap lets you interact with applications which have APIs that can be exported as SOAP objects. The web service then provides a machine-accessible interface to its functionality via HTTP. Find out more...

Version 1.8.0 has a large number of improvements and fixes:

General

• Fixed internally-created faults lacking an XML element name (so e.g. toXml() would abort)
• KDSoapMessage::messageAddressingProperties() is now correctly filled in when receiving a message with WS-Addressing in the header

Client-side

• Added support for timing out requests (default 30 minutes, configurable with KDSoapClientInterface::setTimeout())
• Added support for soap 1.2 faults in faultAsString()
• Improved detection of soap 1.2 faults in HTTP response
• Stricter namespace check for Fault elements being received
• Report client-generated faults as SOAP 1.2 if selected
• Fixed error code when authentication failed
• Autodeletion of jobs is now configurable (github pull #125)
• Added error details in faultAsString() - and the generated lastError() - coming from the SOAP 1.2 detail element.
• Fixed memory leak in KDSoapClientInterface::callNoReply
• Added support for WS-UsernameToken, see KDSoapAuthentication
• Extended KDSOAP_DEBUG functionality (e.g. "KDSOAP_DEBUG=http,reformat" will now print http-headers and pretty-print the xml)
• Added support for specifying requestHeaders as part of KDSoapJob via KDSoapJob::setRequestHeaders()
• Renamed the missing KDSoapJob::returnHeaders() to KDSoapJob::replyHeaders(), and provide an implementation
• Made KDSoapClientInterface::soapVersion() const
• Added lastFaultCode() for error handling after sync calls. Same as lastErrorCode() but it returns a QString rather than an int.
• Added conversion operator from KDDateTime to QVariant to void implicit conversion to base QDateTime (github issue #123).

Server-side

• New method KDSoapServerObjectInterface::additionalHttpResponseHeaderItems to let server objects return additional http headers. This can be used to implement support for CORS, using KDSoapServerCustomVerbRequestInterface to implement OPTIONS response, with "Access-Control-Allow-Origin" in the headers of the response (github issue #117).
• Stopped generation of two job classes with the same name, when two bindings have the same operation name. Prefixed one of them with the binding name (github issue #139 part 1)
• Prepended this-> in method class to avoid compilation error when the variable and the method have the same name (github issue #139 part 2)

WSDL parser / code generator changes, applying to both client and server side

• Source incompatible change: all deserialize() functions now require a KDSoapValue instead of a QVariant. If you use a deserialize(QVariant) function, you need to port your code to use KDSoapValue::setValue(QVariant) before deserialize()
• Source incompatible change: all serialize() functions now return a KDSoapValue instead of a QVariant. If you use a QVariant serialize() function, you need to port your code to use QVariant KDSoapValue::value() after serialize()
• Source incompatible change: xs:QName is now represented by KDQName instead of QString, which allows the namespace to be extracted. The old behaviour is available via KDQName::qname().
• Fixed double-handling of empty elements
• Fixed fault elements being generated in the wrong namespace, must be SOAP-ENV:Fault (github issue #81).
• Added import-path argument for setting the local path to get (otherwise downloaded) files from.
• Added -help-on-missing option to kdwsdl2cpp to display extra help on missing types.
• Added C++17 std::optional as possible return value for optional elements.
• Added -both to create both header(.h) and implementation(.cpp) files in one run
• Added -namespaceMapping @mapping.txt to import url=code mappings, affects C++ class name generation
• Added functionality to prevent downloading the same WSDL/XSD file twice in one run
• Added "hasValueFor{MemberName}()" accessor function, for optional elements
• Generated services now include soapVersion() and endpoint() accessors to match the setSoapVersion(…) and setEndpoint(…) mutators
• Added support for generating messages for WSDL files without services or bindings
• Fixed erroneous QT_BEGIN_NAMESPACE around forward-declarations like Q17__DialogType.
• KDSoapValue now stores the namespace declarations during parsing of a message and writes
• namespace declarations during sending of a message
• Avoid serialize crash with required polymorphic types, if the required variable wasn't actually provided
• Fixed generated code for restriction to base class (it wouldn't compile)
• Prepended "undef daylight" and "undef timezone" to all generated files, to fix compilation errors in wsdl files that use those names, due to nasty Windows macros
• Added generation for default attribute values.

Get KDSoap…

KDSoap on github…

The post KDSoap 1.8.0 released appeared first on KDAB.

22 May 2019 9:32am GMT

The just released 5.2-rc1 kernel includes improved support for Logitech wireless keyboards and mice. Until now we were relying on the generic HID keyboard and mouse emulation for 27 MHz and non-unifying 2.4 GHz wireless receivers.

Starting with the 5.2 kernel instead we actually look at the devices behind the receiver. This allows us to provide battery monitoring support and to have per device quirks, like device specific HID-code to evdev-code mappings where necessary. Until now device specific quirks where not possible because the receivers have a generic product-id which is the same independent of the device behind the receiver.

The per device key-mapping is especially important for 27MHz wireless devices, these use the same HID-code for Fn + F1 to Fn + F12 for all devices, but the markings on the keys differ per model. Sofar it was impossible for Linux to get the mapping for this right, but now that we have per device product-ids for the devices behind the receiver we can finally fix this. As is the case with other devices with vendor specific mappings, the actual mapping is done in userspace through hwdb.

If you have a 27 MHz device (often using this receiver, keyboard marked as canada 210 or canada 310 at the bottom). Please give 5.2 a try. Download the latest 60-keyboard.hwdb file and place it in /lib/udev/hwdb.d (replacing the existing file) and then run "sudo udevadm hwdb --update", before booting into the 5.2 kernel. Then run "sudo evemu-record" select your keyboard and try Fn + F1 to Fn + F12 and any other special keys. If any keys do not work, edit 60-keyboard.hwdb, search for Logitech and add an entry for your keyboard, see the existing Logitech entries. After editing you need to re-run "sudo udevadm hwdb --update", followed by "sudo udevadm trigger" for the changes to take effect. Once you have a working entry, submit a pull-req to systemd to get the changes upstream. If you need any help drop me an email.

We still have some old code for the generic HID emulation for 27 MHz receivers with a product-id of c50c, these should work fine with the new code, but we've been unable to test this. I would really like to move the c50c id over to the new code and remove all the old code. If you've a 27 MHz Logitech device, please run lsusb, if your device has a product-id of c50c and you are willing to test, please drop me an email.

Likewise I suspect that 2.4GHz receivers with a product-id of c531 should work fine with the new support for non-unifying 2.4 GHz receivers, if you have one of those also please drop me an email.

21 May 2019 8:26am GMT