24 Mar 2017

feedPlanet Maemo

Making something that is ‘undoable editable’ with Qt

Among the problems we'll face is that we want asynchronous APIs that are undoable and that we want to switch to read only, undoable editing, non-undoable editing and that QML doesn't really work well with QFuture. At least not yet. We want an interface that is easy to talk with from QML. Yet we want to switch between complicated behaviors.

We will also want synchronous mode and asynchronous mode. Because I just invented that requirement out of thin air.

Ok, first the "design". We see a lot of behaviors, for something that can do something. The behaviors will perform for that something, the actions it can do. That is the strategy design pattern, then. It's the one about ducks and wing fly behavior and rocket propelled fly behavior and the ostrich that has a can't fly behavior. For undo and redo, we have the command pattern. We have this neat thing in Qt for that. We'll use it. We don't reinvent the wheel. Reinventing the wheel is stupid.

Let's create the duck. I mean, the thing-editor as I will use "Thing" for the thing that is being edited. We want copy (sync is sufficient), paste (must be aysnc), and edit (must be async). We could also have insert and delete, but those APIs would be just like edit. Paste is usually similar to insert, of course. Except that it can be a combined delete and insert when overwriting content. The command pattern allows you to make such combinations. Not the purpose of this example, though.

Enough explanation. Let's start! The ThingEditor, is like the flying Duck in strategy. This is going to be more or less the API that we will present to the QML world. It could be your ViewModel, for example (ie. you could let your ThingViewModel subclass ThingEditor).

class ThingEditor : public QObject
{
    Q_OBJECT

    Q_PROPERTY ( ThingEditingBehavior* editingBehavior READ editingBehavior
                 WRITE setEditingBehavior NOTIFY editingBehaviorChanged )
    Q_PROPERTY ( Thing* thing READ thing WRITE setThing NOTIFY thingChanged )

public:
    explicit ThingEditor( QSharedPointer<Thing> &a_thing,
            ThingEditingBehavior *a_editBehavior,
            QObject *a_parent = nullptr );

    explicit ThingEditor( QObject *a_parent = nullptr );

    Thing* thing() const { return m_thing.data(); }
    virtual void setThing( QSharedPointer<Thing> &a_thing );
    virtual void setThing( Thing *a_thing );

    ThingEditingBehavior* editingBehavior() const { return m_editingBehavior.data(); }
    virtual void setEditingBehavior ( ThingEditingBehavior *a_editingBehavior );

    Q_INVOKABLE virtual void copyCurrentToClipboard ( );
    Q_INVOKABLE virtual void editCurrentAsync( const QString &a_value );
    Q_INVOKABLE virtual void pasteCurrentFromClipboardAsync( );

signals:
    void editingBehaviorChanged ();
    void thingChanged();
    void editCurrentFinished( EditCurrentCommand *a_command );
    void pasteCurrentFromClipboardFinished( EditCurrentCommand *a_command );

private slots:
    void onEditCurrentFinished();
    void onPasteCurrentFromClipboardFinished();

private:
    QScopedPointer<ThingEditingBehavior> m_editingBehavior;
    QSharedPointer<Thing> m_thing;
    QList<QFutureWatcher<EditCurrentCommand*> *> m_editCurrentFutureWatchers;
    QList<QFutureWatcher<EditCurrentCommand*> *> m_pasteCurrentFromClipboardFutureWatchers;
};

For the implementation of this class, I'll only provide the non-obvious pieces. I'm sure you can do that setThing, setEditingBehavior and the constructor yourself. I'm also providing it only once, and also only for the EditCurrentCommand. The one about paste is going to be exactly the same.

void ThingEditor::copyCurrentToClipboard ( )
{
    m_editingBehavior->copyCurrentToClipboard( );
}

void ThingEditor::onEditCurrentFinished( )
{
    QFutureWatcher<EditCurrentCommand*> *resultWatcher
            = static_cast<QFutureWatcher<EditCurrentCommand*>*> ( sender() );
    emit editCurrentFinished ( resultWatcher->result() );
    if (m_editCurrentFutureWatchers.contains( resultWatcher )) {
        m_editCurrentFutureWatchers.removeAll( resultWatcher );
    }
    delete resultWatcher;
}

void ThingEditor::editCurrentAsync( const QString &a_value )
{
    QFutureWatcher<EditCurrentCommand*> *resultWatcher
            = new QFutureWatcher<EditCurrentCommand*>();
    connect ( resultWatcher, &QFutureWatcher<EditCurrentCommand*>::finished,
              this, &ThingEditor::onEditCurrentFinished, Qt::QueuedConnection );
    resultWatcher->setFuture ( m_editingBehavior->editCurrentAsync( a_value ) );
    m_editCurrentFutureWatchers.append ( resultWatcher );
}

For QUndo we'll need a QUndoCommand. For each undoable action we indeed need to make such a command. You could add more state and pass it to the constructor. It's common, for example, to pass Thing, or the ThingEditor or the behavior (this is why I used QSharedPointer for those: as long as your command lives in the stack, you'll need it to hold a reference to that state).

class EditCurrentCommand: public QUndoCommand
{
public:
    explicit EditCurrentCommand( const QString &a_value,
                                 QUndoCommand *a_parent = nullptr )
        : QUndoCommand ( a_parent )
        , m_value ( a_value ) { }
    void redo() Q_DECL_OVERRIDE {
       // Perform action goes here
    }
    void undo() Q_DECL_OVERRIDE {
      // Undo what got performed goes here
    }
private:
    const QString &m_value;
};

You can (and probably should) also make this one abstract (and/or a so called pure interface), as you'll usually want many implementations of this one (one for every kind of editing behavior). Note that it leaks the QUndoCommand instances unless you handle them (ie. storing them in a QUndoStack). That in itself is a good reason to keep it abstract.

class ThingEditingBehavior : public QObject
{
    Q_OBJECT

    Q_PROPERTY ( ThingEditor* editor READ editor WRITE setEditor NOTIFY editorChanged )
    Q_PROPERTY ( Thing* thing READ thing NOTIFY thingChanged )

public:
    explicit ThingEditingBehavior( ThingEditor *a_editor,
                                   QObject *a_parent = nullptr )
        : QObject ( a_parent )
        , m_editor ( a_editor ) { }

    explicit ThingEditingBehavior( QObject *a_parent = nullptr )
        : QObject ( a_parent ) { }

    ThingEditor* editor() const { return m_editor.data(); }
    virtual void setEditor( ThingEditor *a_editor );
    Thing* thing() const;

    virtual void copyCurrentToClipboard ( );
    virtual QFuture<EditCurrentCommand*> editCurrentAsync( const QString &a_value, bool a_exec = true );
    virtual QFuture<EditCurrentCommand*> pasteCurrentFromClipboardAsync( bool a_exec = true );

protected:
    virtual EditCurrentCommand* editCurrentSync( const QString &a_value, bool a_exec = true );
    virtual EditCurrentCommand* pasteCurrentFromClipboardSync( bool a_exec = true );

signals:
    void editorChanged();
    void thingChanged();

private:
    QPointer<ThingEditor> m_editor;
    bool m_synchronous = true;
};

That setEditor, the constructor, etc: these are too obvious to write here. Here are the non-obvious ones:

void ThingEditingBehavior::copyToClipboard ( )
{
}

EditCurrentCommand* ThingEditingBehavior::editCurrentSync( const QString &a_value, bool a_exec )
{
    EditCurrentCommand *ret = new EditCurrentCommand ( a_value );
    if ( a_exec )
        ret->redo();
    return ret;
}

QFuture<EditCurrentCommand*> ThingEditingBehavior::editCurrentAsync( const QString &a_value, bool a_exec )
{
    QFuture<EditCurrentCommand*> resultFuture =
            QtConcurrent::run( QThreadPool::globalInstance(), this,
                               &ThingEditingBehavior::editCurrentSync,
                               a_value, a_exec );
    if (m_synchronous)
        resultFuture.waitForFinished();
    return resultFuture;
}

And now we can make the whole thing undoable by making a undoable editing behavior. I'll leave a non-undoable editing behavior as an exercise to the reader (ie. just perform redo() on the QUndoCommand, don't store it in the QUndoStack and immediately delete or cmd->deleteLater() the instance).

Note that if m_synchronous is false, that (all access to) m_undoStack, and the undo and redo methods of your QUndoCommands, must be (made) thread-safe. The thread-safety is not the purpose of this example, though.

class UndoableThingEditingBehavior : public ThingEditingBehavior
{
    Q_OBJECT
public:
    explicit UndoableThingEditingBehavior( ThingEditor *a_editor,
                                           QObject *a_parent = nullptr );
protected:
    EditCellCommand* editCurrentSync( const QString &a_value, bool a_exec = true ) Q_DECL_OVERRIDE;
    EditCurrentCommand* pasteCurrentFromClipboardSync( bool a_exec = true ) Q_DECL_OVERRIDE;
private:
    QScopedPointer<QUndoStack> m_undoStack;
};

EditCellCommand* UndoableThingEditingBehavior::editCurrentSync( const QString &a_value, bool a_exec )
{
    Q_UNUSED(a_exec)
    EditCellCommand *undoable = ThingEditingBehavior::editCurrentSync(  a_value, false );
    m_undoStack->push( undoable );
    return undoable;
}

EditCellCommand* UndoableThingEditingBehavior::pasteCurrentFromClipboardSync( bool a_exec )
{
    Q_UNUSED(a_exec)
    EditCellCommand *undoable = ThingEditingBehavior::pasteCurrentFromClipboardSync( false );
    m_undoStack->push( undoable );
    return undoable;
}

0 Add to favourites0 Bury

24 Mar 2017 9:42am GMT

23 Mar 2017

feedPlanet Maemo

Perfection

Perfection has been reached not when there is nothing left to add, but when there is nothing left to take away.

0 Add to favourites0 Bury

23 Mar 2017 12:17am GMT

20 Mar 2017

feedPlanet Maemo

Media Source Extensions upstreaming, from WPE to WebKitGTK+

A lot of good things have happened to the Media Source Extensions support since my last post, almost a year ago.

The most important piece of news is that the code upstreaming has kept going forward at a slow, but steady pace. The amount of code Igalia had to port was pretty big. Calvaris (my favourite reviewer) and I considered that the regular review tools in WebKit bugzilla were not going to be enough for a good exhaustive review. Instead, we did a pre-review in GitHub using a pull request on my own repository. It was an interesting experience, because the change set was so large that it had to be (artificially) divided in smaller commits just to avoid reaching GitHub diff display limits.

394 GitHub comments later, the patches were mature enough to be submitted to bugzilla as child bugs of Bug 157314 - [GStreamer][MSE] Complete backend rework. After some comments more in bugzilla, they were finally committed during Web Engines Hackfest 2016:

Some unforeseen regressions in the layout tests appeared, but after a couple of commits more, all the mediasource WebKit tests were passing. There are also some other tests imported from W3C, but I kept them still skipped because webm support was needed for many of them. I'll focus again on that set of tests at its due time.

Igalia is proud of having brought the MSE support up to date to WebKitGTK+. Eventually, this will improve the browser video experience for a lot of users using Epiphany and other web browsers based on that library. Here's how it enables the usage of YouTube TV at 1080p@30fps on desktop Linux:

Our future roadmap includes bugfixing and webm/vp9+opus support. This support is important for users from countries enforcing patents on H.264. The current implementation can't be included in distros such as Fedora for that reason.

As mentioned before, part of this upstreaming work happened during Web Engines Hackfest 2016. I'd like to thank our sponsors for having made this hackfest possible, as well as Metrological for giving upstreaming the importance it deserves.

Thank you for reading.

0 Add to favourites0 Bury

20 Mar 2017 11:55am GMT

17 Mar 2017

feedPlanet Maemo

Duck typing

Imagine you have a duck. Imagine you have a wall. Now imagine you throw the duck with a lot of force against a wall. Duck typing means that the duck hitting the wall quacks like a duck would.

ps. Replace wall with API and duck with ugly stupid script written by an idiot. You can leave quacks.

0 Add to favourites0 Bury

17 Mar 2017 10:34am GMT

10 Feb 2017

feedPlanet Maemo

Migrating from owncloud 9.1 to nextcloud 11

First one should ask though: why? My main motivation was that many of the apps I use were easily available in the nextcloud store, while with owncloud I had to manually pull them from github.
Additionally some of the app authors migrated to nextcloud and did not provide further updates for owncloud.

Another reason is this:

the graphs above show the number of commits for owncloud and nextcloud. Owncloud has taken a very noticeable hit here after the fork - even though they deny it.

From the user perspective the lack of contribution is visible for instance in the admin interface where with nextcloud you get a nice log browser and system stats while with owncloud you do not. Furthermore the nextcloud android app handles Auto-Upload much better and generally seems more polished - I think one can expect nextcloud to advance faster in general.

Migrating

For migrating you can follow the excellent instructions of Jos Poortvliet.

In my case owncloud 9.1 was installed on Ubuntu in /var/www/owncloud and I put nextcloud 11 to /var/www/nextcloud. Then the following steps had to be applied:

  1. put owncloud in maintenance mode
    sudo -u www-data php occ maintenance:mode --on
    
  2. copy over the config.php
    cp /var/www/owncloud/config/config.php /var/www/nextcloud/config/
    
  3. adapt the path in config.php
    # from 
    'path' => '/var/www/owncloud/apps',
    # to
    'path' => '/var/www/nextcloud/apps',
    
  4. adapt the path in crontab
    sudo crontab -u www-data -e
    
  5. adapt the paths in the apache config
  6. run the upgrade script which takes care of the actual migration. Then disable the maintanance mode.
    sudo -u www-data php occ upgrade
    sudo -u www-data php occ maintenance:mode --off
    

and thats it.

0 Add to favourites0 Bury

10 Feb 2017 11:33pm GMT

Working on an Android tablet, 2017 edition

Back in 2013 I was working exclusively on an Android tablet. Then with the NoFlo Kickstarter I needed a device with a desktop browser. What followed were brief periods working on a Chromebook, on a 12" MacBook, and even an iPad Pro.

But from April 2016 onwards I've been again working with an Android device. Some people have asked me about my setup, and so here is an update.

Information technology

Why work on a tablet?

When I started on this path in 2013, using a tablet for "real work" was considered crazy. While every story on tablet productivity still brings out the people claiming it is not a real computer for real work, using tablets for real work is becoming more and more common.

A big contributor to this has been the plethora of work-oriented tablets and convertibles released since then. Microsoft's popular Surface Pro line brought the PC to tablet form factor, and Apple's iPad Pro devices gave the iPad a keyboard.

Here are couple of great posts talking about how it feels to work on an iPad:

With all the activity going on, one could claim using a tablet for work has been normalized. But why work on a tablet instead of a "real computer"? Here are some reasons, at least for me:

Free of legacy cruft

Desktop operating systems have become clunky. Window management. File management. Multiple ways to discover, install, and uninstall applications. Broken notification mechanisms.

With a tablet you can bypass pretty much all of that, and jump into a simpler, cleaner interface designed for the modern connected world.

I think this is also the reason driving some developers back to Linux and tiling window managers - cutting manual tweaking and staying focused.

Amazing endurance

Admittedly, laptop battery life has increased a lot since 2013. But with some manufacturers using this an excuse to ship thinner devices, tablets still win the endurance game.

With my current work tablet, I'm customarily getting 12 or more hours of usage. This means I can power through the typical long days of a startup founder without having to plug in. And when traveling, I really don't have to care where power sockets are located on trains, airplanes, and conference centers.

Low power usage also means that I can really get a lot of more runtime by utilizing the mobile battery pack I originally bought to use with my phone. While I've never actually had to try this, back-of-the-envelope math claims I should be able to get a full workweek from the combo without plugging in.

Work and play

The other aspect of using a tablet is that it becomes a very nice content consumption device after I'm done working. Simply disconnect the keyboard and lean back, and the same device you used for writing software becomes a great e-reader, video player, or a gaming machine.

Livestreaming a SpaceX launch

This combined with the battery life has meant that I've actually stopped carrying a Kindle with me. While an e-ink screen is still nicer to read, not needing an extra device has its benefits, especially for a frequent one-bag traveller.

The setup

I'm writing this on a Pixel C, a 10.2" Android tablet made by Google. I got the device last spring when there were developer discounts available at ramp-up to the Android 7 release, and have been using it full-time since.

Software

My Android homescreen

Surprisingly little has changed in my software use since 2013 - I still spend the most of the time writing software in either Flowhub or terminal. Here are the apps I use on daily basis:

Looking back to the situation in early 2013, the biggest change is that Slack has pretty much killed work email.

Termux is a new app that has done a lot to improve the local development situation. By starting the app you get a very nice Linux chroot environment where a lot of software is only a quick apt install away.

Since much of my non-Flowhub work is done in tmux and vim, I get the exactly same working environment on both local chroot and cloud machines by simply installing my dotfiles on each of them.

Keyboard

Laptop tablet

When I'm on the road I'm using the Pixel C keyboard. This doubles as a screen protector, and provides a reasonable laptop-like typing environment. It attaches to the tablet with very strong magnets and allows a good amount of flexibility on the screen angles.

However, when stationary, no laptop keyboard compares to a real mechanical keyboard. When I'm in the office I use a Filco MiniLa Air, a bluetooth keyboard with quiet-ish Cherry MX brown switches.

Desktop tablet

This tenkeyless (60%) keyboard is extremely comfortable to type on. However, the sturdy metal case means that it is a little too big and heavy to carry on a daily basis.

In practice I've only taken with me when there has been a longer trip where I know that I'll be doing a lot of typing. To solve this, I'm actually looking to build a more compact custom mechanical keyboard so I could always have it with me.

Comparison with iOS

So, why work on Android instead of getting an iPad Pro? I've actually worked on both, and here are my reasons:

Of course, iOS has its own benefits. Apple has a stronger stance on privacy than Google. And there is more well-made tablet software available for iPads than Android. But when almost everything I use is available on the web, this doesn't matter that much.

The future

Hacking on the c-base patio

As a software developer working on Android tablets, the weakest point of the platform is still that there are no browser developer tools available. This was a problem in 2013, and it is still a problem now.

From my conversations with some Chrome developers, it seems Google has very little interest in addressing this. However, there is a bright spot: the new breed of convertible Chromebooks being released now. And they run Android apps:

Chrome OS is another clean, legacy free, modern computing interface. With these new devices you get the combination of a full desktop browser and the ability to run all Android tablet software.

The Samsung Chromebook Pro/Plus mentioned above is definitely interesting. A high-res 12" screen and a digital pen which I see as something very promising for visual programming purposes.

However, given that I already have a great mechanical keyboard, I'd love a device that shipped without an attached keyboard. We'll see what kind of devices get out later this year.

0 Add to favourites0 Bury

10 Feb 2017 12:00am GMT

09 Feb 2017

feedPlanet Maemo

How to expose a QList in a ViewModel to QML

MyPlugin/MyPlugin.cpp:

#include <ViewModels/MyListClass.h>
#include <ViewModels/DisplayViewModel.h>

qmlRegisterUncreatableType<MyListClass>( a_uri, 1, 0, "MyListClass",
         "Use access via DisplayViewModel instead");
qmlRegisterType<DisplayViewModel>( a_uri, 1, 0, "DisplayViewModel");

Utils/MyQMLListUtils.h

#define MY_DECLARE_QML_LIST(type, name, owner, prop) \
QQmlListProperty<type> name(){ \
   return QQmlListProperty<type>( \
               this, 0,&owner::count ## type ## For ## name ## List, \
               &owner::at ## type ## For ## name ## List); \
} \
static int count ## type ## For ## name ## List(QQmlListProperty<type>*property){ \
   owner *m = qobject_cast<owner *>(property->object); \
   return m->prop.size(); \
} \
static type *at ## type ## For ## name ## List( \
        QQmlListProperty<type>*property, int index){ \
   owner *m = qobject_cast<owner *>(property->object); \
   return m->prop[index]; \
}

ViewModels/DisplayViewModel.h

#ifndef DISPLAYVIEWMODEL_H
#define DISPLAYVIEWMODEL_H

#include <QObject>
#include <QtQml>
#include <ViewModels/MyListClass.h>
#include <Utils/MyQMLListUtils.h>

class DisplayViewModel : public QObject
{
    Q_OBJECT

    Q_PROPERTY(constQString title READ title WRITE setTitle NOTIFY titleChanged )
    Q_PROPERTY(constQList<MyListClass*> objects READ objects
                                          NOTIFY objectsChanged ) 
    Q_PROPERTY( QQmlListProperty<MyListClass> objectList READ objectList
                                              NOTIFY objectsChanged )
public:
    explicit DisplayViewModel( QObject *a_parent = nullptr );
    explicit DisplayViewModel( const QString &a_title,
                               QList<MyListClass*> a_objects,
                               QObject *a_parent = nullptr );
    const QString title()
        { return m_title; }
    void setTitle( const QString &a_title ); 
    const QList<MyListClass*> objects ()
        { return m_objects; } 
    Q_INVOKABLE void appendObject( MyListClass *a_object);
    Q_INVOKABLE void deleteObject( MyListClass *a_object);
    Q_INVOKABLE void reset( );

protected:
    MY_DECLARE_QML_LIST(MyListClass, objectList, DisplayViewModel, m_objects)

signals:
    void titleChanged();
    void objectsChanged();

private:
    QString m_title;
    QList<MyListObject*> m_objects;
};

#endif// DISPLAYVIEWMODEL_H

DisplayViewModel.cpp

#include "DisplayViewModel.h"

DisplayViewModel::DisplayViewModel( const QString &a_title,
                                    QList<MyListClass*> a_objects,
                                    QObject *a_parent )
    : QObject ( a_parent )
    , m_title ( a_title )
    , m_objects ( a_objects )
{
    foreach (MyListClass* mobject, m_objects) {
        mobject->setParent (this);
    }
}

void DisplayViewModel::setTitle (const QString &a_title )
{
    if ( m_title != a_title ) {
        m_title = a_title;
        emit titleChanged();
    }
}

void DisplayViewModel::reset( )
{
    foreach ( MyListClass *mobject, m_objects ) {
        mobject->deleteLater();
    }
    m_objects.clear();
    emit objectsChanged();
}

void DisplayViewModel::appendObject( MyListClass *a_object )
{
    a_object->setParent( this );
    m_objects.append( a_object );
    emit objectsChanged();
}

void DisplayViewModel::deleteObject( MyListClass *a_object )
{
    if (m_objects.contains( a_object )) {
        m_objects.removeOne( a_object );
        a_object->deleteLater();
        emit objectsChanged();
    }
}

Tester.cpp

#include <ViewModels/DisplayViewModel.h>
#include <ViewModels/MyListClass.h>

QList<MyListClass*> objectList;
for( int i = 0; i < 100 ; ++i ) {
    objectList.append ( new MyListClass (i) );
}
DisplayViewModel *viewModel = new DisplayViewModel (objectList);
viewModel->appendObject ( new MyListClass (101) );

Display.qml

import QtQuick 2.5
import MyPlugin 1.0

Repeater { 
    property DisplayViewModel viewModel: DisplayViewModel { } 
    model: viewModel.objectList
    delegate: Item {
        property MyListClass object: modelData
        Text {
            text: object.property
        }
    }
}

0 Add to favourites0 Bury

09 Feb 2017 8:07pm GMT

08 Feb 2017

feedPlanet Maemo

QEMU and the qcow2 metadata checks

When choosing a disk image format for your virtual machine one of the factors to take into considerations is its I/O performance. In this post I'll talk a bit about the internals of qcow2 and about one of the aspects that can affect its performance under QEMU: its consistency checks.

As you probably know, qcow2 is QEMU's native file format. The first thing that I'd like to highlight is that this format is perfectly fine in most cases and its I/O performance is comparable to that of a raw file. When it isn't, chances are that this is due to an insufficiently large L2 cache. In one of my previous blog posts I wrote about the qcow2 L2 cache and how to tune it, so if your virtual disk is too slow, you should go there first.

I also recommend Max Reitz and Kevin Wolf's qcow2: why (not)? talk from KVM Forum 2015, where they talk about a lot of internal details and show some performance tests.

qcow2 clusters: data and metadata

A qcow2 file is organized into units of constant size called clusters. The cluster size defaults to 64KB, but a different value can be set when creating a new image:

qemu-img create -f qcow2 -o cluster_size=128K hd.qcow2 4G

Clusters can contain either data or metadata. A qcow2 file grows dynamically and only allocates space when it is actually needed, so apart from the header there's no fixed location for any of the data and metadata clusters: they can appear mixed anywhere in the file.

Here's an example of what it looks like internally:

In this example we can see the most important types of clusters that a qcow2 file can have:

Metadata overlap checks

In order to detect corruption when writing to qcow2 images QEMU (since v1.7) performs several sanity checks. They verify that QEMU does not try to overwrite sections of the file that are already being used for metadata. If this happens, the image is marked as corrupted and further access is prevented.

Although in most cases these checks are innocuous, under certain scenarios they can have a negative impact on disk write performance. This depends a lot on the case, and I want to insist that in most scenarios it doesn't have any effect. When it does, the general rule is that you'll have more chances of noticing it if the storage backend is very fast or if the qcow2 image is very large.

In these cases, and if I/O performance is critical for you, you might want to consider tweaking the images a bit or disabling some of these checks, so let's take a look at them. There are currently eight different checks. They're named after the metadata sections that they check, and can be divided into the following categories:

  1. Checks that run in constant time. These are equally fast for all kinds of images and I don't think they're worth disabling.
    • main-header
    • active-l1
    • refcount-table
    • snapshot-table
  2. Checks that run in variable time but don't need to read anything from disk.
    • refcount-block
    • active-l2
    • inactive-l1
  3. Checks that need to read data from disk. There is just one check here and it's only needed if there are internal snapshots.
    • inactive-l2

By default all tests are enabled except for the last one (inactive-l2), because it needs to read data from disk.

Disabling the overlap checks

Tests can be disabled or enabled from the command line using the following syntax:

-drive file=hd.qcow2,overlap-check.inactive-l2=on
-drive file=hd.qcow2,overlap-check.snapshot-table=off

It's also possible to select the group of checks that you want to enable using the following syntax:

-drive file=hd.qcow2,overlap-check.template=none
-drive file=hd.qcow2,overlap-check.template=constant
-drive file=hd.qcow2,overlap-check.template=cached
-drive file=hd.qcow2,overlap-check.template=all

Here, none means that no tests are enabled, constant enables all tests from group 1, cached enables all tests from groups 1 and 2, and all enables all of them.

As I explained in the previous section, if you're worried about I/O performance then the checks that are probably worth evaluating are refcount-block, active-l2 and inactive-l1. I'm not counting inactive-l2 because it's off by default. Let's look at the other three:

Conclusion

The qcow2 consistency checks are useful to detect data corruption, but they can affect write performance.

If you're unsure and you want to check it quickly, open an image with overlap-check.template=none and see for yourself, but remember again that this will only affect write operations. To obtain more reliable results you should also open the image with cache=none in order to perform direct I/O and bypass the page cache. I've seen performance increases of 50% and more, but whether you'll see them depends a lot on your setup. In many cases you won't notice any difference.

I hope this post was useful to learn a bit more about the qcow2 format. There are other things that can help QEMU perform better, and I'll probably come back to them in future posts, so stay tuned!

Acknowledgments

My work in QEMU is sponsored by Outscale and has been made possible by Igalia and the help of the rest of the QEMU development team.

0 Add to favourites0 Bury

08 Feb 2017 8:52am GMT

05 Jan 2017

feedPlanet Maemo

Process API for NoFlo components

It has been a while that I've written about flow-based programming - but now that I'm putting most of my time to Flowhub things are moving really quickly.

One example is the new component API in NoFlo that has been emerging over the last year or so.

Most of the work described here was done by Vladimir Sibirov from The Grid team.

Introducing the Process API

NoFlo programs consist of graphs where different nodes are connected together. These nodes can themselves be graphs, or they can be components written in JavaScript.

A NoFlo component is simply a JavaScript module that provides a certain interface that allows NoFlo to run it. In the early days there was little convention on how to write components, but over time some conventions emerged, and with them helpers to build well-behaved components more easily.

Now with the upcoming NoFlo 0.8 release we've taken the best ideas from those helpers and rolled them back into the noflo.Component base class.

So, how does a component written using the Process API look like?

// Load the NoFlo interface
var noflo = require('noflo');
// Also load any other dependencies you have
var fs = require('fs');

// Implement the getComponent function that NoFlo's component loader
// uses to instantiate components to the program
exports.getComponent = function () {
  // Start by instantiating a component
  var c = new noflo.Component();

  // Provide some metadata, including icon for visual editors
  c.description = 'Reads a file from the filesystem';
  c.icon = 'file';

  // Declare the ports you want your component to have, including
  // their data types
  c.inPorts.add('in', {
    datatype: 'string'
  });
  c.outPorts.add('out', {
    datatype: 'string'
  });
  c.outPorts.add('error', {
    datatype: 'object'
  });

  // Implement the processing function that gets called when the
  // inport buffers have packets available
  c.process(function (input, output) {
    // Precondition: check that the "in" port has a data packet.
    // Not necessary for single-inport components but added here
    // for the sake of demonstration
    if (!input.hasData('in')) {
      return;
    }

    // Since the preconditions matched, we can read from the inport
    // buffer and start processing
    var filePath = input.getData('in');
    fs.readFile(filePath, 'utf-8', (err, contents) {
      // In case of errors we can just pass the error to the "error"
      // outport
      if (err) {
        output.done(err);
        return;
      }

      // Send the file contents to the "out" port
      output.send({
        out: contents
      });
      // Tell NoFlo we've finished processing
      output.done();
    });
  });

  // Finally return to component to the loader
  return c;
}

Most of this is still the same component API we've had for quite a while: instantiation, component metadata, port declarations. What is new is the process function and that is what we'll focus on.

When is process called?

NoFlo components call their processing function whenever they've received packets to any of their regular inports.

In general any new information packets received by the component cause the process function to trigger. However, there are some exceptions:

Handling preconditions

When the processing function is called, the first job is to determine if the component has received enough data to act. These "firing rules" can be used for checking things like:

The NoFlo component input handler provides methods for checking the contents of the input buffer. Each of these return a boolean if the conditions are matched:

For convenience, has and hasData can be used to check multiple ports at the same time. For example:

// Fail precondition check unless both inports have a data packet
if (!input.hasData('in1', 'in2')) return;

For more complex checking it is also possible to pass a validation function to the has method. This function will get called for each information packet in the port(s) buffer:

// We want to process only when color is green
var validator = function (packet) {
  if (packet.data.color === 'green') {
    return true;
  }
  return false;
}
// Run all packets in in1 and in2 through the validator to
// check that our firing conditions are met
if (!input.has('in1', 'in2', validator)) return;

The firing rules should be checked in the beginning of the processing function before we start actually reading packets from the buffer. At that stage you can simply finish the run with a return.

Processing packets

Once your preconditions have been met, it is time to read packets from the buffers and start doing work with them.

For reading packets there are equivalent get functions to the has functions used above:

For get and getStream you receive whole IP objects. For convenience, getData returns just the data payload of the data packet.

When you have read the packets you want to work with, the next step is to do whatever your component is supposed to do. Do some simple data processing, call some remote API function, or whatever. NoFlo doesn't really care whether this is done synchronously or asynchronously.

Note: once you read packets from an inport, the component activates. After this it is necessary to finish the process by calling output.done() when you're done.

Sending packets

While the component is active, it can send packets to any number of outports using the output.send method. This method accepts a map of port names and information packets.

output.send({
  out1: new noflo.IP('data', "some data"),
  out2: new noflo.IP('data', [1, 2, 3])
});

For data packets you can also just send the data as-is, and NoFlo will wrap it to an information packet.

Once you've finished processing, simply call output.done() to deactivate the component. There is also a convenience method that is a combination of send and done. This is useful for simple components:

c.process(function (input, output) {
  var data = input.getData('in');
  // We just add one to the number we received and send it out
  output.sendDone({
    out: data + 1
  });
});

In normal situations there packets are transmitted immediately. However, when working on individual packets that are part of a stream, NoFlo components keep an output buffer to ensure that packets from the stream are transmitted in original order.

Component lifecycle

In addition to making input processing easier, the other big aspect of the Process API is to help formalize NoFlo's component and program lifecycle.

NoFlo program lifecycle

The component lifecycle is quite similar to the program lifecycle shown above. There are three states:

Once all components in a NoFlo network have deactivated, the whole program is finished.

Components are only allowed to do work and send packets when they're activated. They shouldn't do any work before receiving input packets, and should not send anything after deactivating.

Generator components

Regular NoFlo components only send data associated with input packets they've received. One exception is generators, a class of components that can send packets whenever something happens.

Some examples of generators include:

The same rules of "only send when activated" apply also to generators. However, they can utilize the processing context to self-activate as needed:

exports.getComponent = function () {
 var c = new noflo.Component();
 c.inPorts.add('start', { datatype: 'bang' });
 c.inPorts.add('stop', { datatype: 'bang' });
 c.outPorts.add('out', { datatype: 'bang' });
 // Generators generally want to send data immediately and
 // not buffer
 c.autoOrdering = false;

 // Helper function for clearing a running timer loop
 var cleanup = function () {
   // Clear the timer
   clearInterval(c.timer.interval);
   // Then deactivate the long-running context
   c.timer.deactivate();
   c.timer = null;
 }

 // Receive the context together with input and output
 c.process(function (input, output, context) {
   if (input.hasData('start')) {
     // We've received a packet to the "start" port
     // Stop the previous interval and deactivate it, if any
     if (c.timer) {
       cleanup();
     }
     // Activate the context by reading the packet
     input.getData('start');
     // Set the activated context to component so it can
     // be deactivated from the outside
     c.timer = context
     // Start generating packets
     c.timer.interval = setInterval(function () {
       // Send a packet
       output.send({
         out: true
       });
     }, 100);
     // Since we keep the generator running we don't
     // call done here
   }

   if (input.hasData('stop')) {
     // We've received a packet to the "stop" port
     input.getData('stop');
     if (!c.timer) {
       // No timers running, we can just finish here
       output.done();
       return;
     }
     // Stop the interval and deactivate
     cleanup();
     // Also call done for this one
     output.done();
   }
 });

 // We also may need to clear the timer at network shutdown
 c.tearDown = function (callback) {
   if (c.timer) {
     // Stop the interval and deactivate
     cleanup();
   }
   c.emit('end');
   c.started = false;
   callback();
 }

 return c;
}

Time to prepare

NoFlo 0.7 included a preview version of the Process API. However, last week during the 33C3 conference we finished some tricky bits related to process lifecycle and automatic bracket forwarding that make it more useful for real-life NoFlo applications.

These improvements will land in NoFlo 0.8, due out soon.

So, if you're maintaining a NoFlo application, now is a good time to give the git version a spin and look at porting your components to the new API. Make sure to report any issues you encounter!

We're currently migrating all the hundred-plus NoFlo open source modules to latest build and testing process so that they can be easily updated to the new APIs when they land.

1 Add to favourites0 Bury

05 Jan 2017 12:00am GMT

22 Dec 2016

feedPlanet Maemo

OGRECave 1.10 release

The 1.10.0 release of the OGRECave fork was just created. This means that the code is considered stable enough for general usage and the current interfaces will be supported in subsequent patch releases (i.e. 1.10.1, 1.10.2 …).

SampleBrowser running GLES2 on desktop

This release represents more than 3 years of work from various contributors when compared to the previous 1.9 release. At the time of writing it contains all commits from the bitbucket version as well as many fork specific features and fixes.

If you are reading about the fork for the first time and wonder why it was created, see this blog post. For a comparison between the github and bitbucket version see this log.

For a general overview of the 1.10 features when compared to 1.9, see the OGRECave 1.10 release notes.

The highlights probably are:

For further information see the github page of the fork.

0 Add to favourites0 Bury

22 Dec 2016 1:26am GMT

28 Nov 2016

feedPlanet Maemo

Nokia and their standard batteries

Nokia. A company everyone knows and most of us probably even used one of their phones in past. They were better or worse but one thing was good - most of them shared batteries…

My daughter (8.5y old) uses Nokia E50 as her daily phone. Sim card is covered by duct tape to not fall out when phone hit a floor (previous one went missing in such situation). Mira records how she and her friends sing, does some photo sessions to her dolls etc.

But during weekend phone stopped charging. Hm… Is it charger? Nope, it was original Nokia one. Tried some crappy Chinese one with same result. So let's check the battery.

Opened drawer, took Nokia 101. Inside was BL-5CB battery. Inserted into E50 got phone back online. But I like my 101 and keep it as a spare just in case.

Digged in a drawer with old devices. The one where I keep Sharp Zaurus c760, Sony Ericsson k750i, Openmoko FIC-GTA01bv3 and few other pieces of junk with some sentimental value. What I found there was Nokia 6230i which I got from Ross Burton during GUADEC 2007. Last time I used it about 5 years ago. But it had original Nokia BL-5C inside!

So I put that battery inside of E50, plugged charger and guess what… It started charging and phone booted! With over 11 years old battery!

During next few days I will buy BL-5C clone somewhere (they are 3-8€ now) and put it in my daughter's phone.

Related posts:

  1. Sending files over Bluetooth to S60 devices
  2. Defining good Contacts application
  3. Nokia N900 discount

0 Add to favourites0 Bury

28 Nov 2016 9:57pm GMT

28 Oct 2016

feedPlanet Maemo

The 1st Maemo Developer Regatta - The starting line

The Maemo Community e.V. proudly presents

The 1st Maemo Developer Regatta - The starting line

sponsored by Jolla


The wait is over!

The first Maemo Coding Competition for all Maemo derived systems, including but not limited to Maemo and SailfishOS.

This year's competition has the following three categories: Something new, Fixing/Updating and Beginner. Whether you are an experienced developer, porter, hacker or just a beginner on your very first hacks/codes, we have a category that suits you and your coding skills! If you are new to programming or want to learn, here's the excuse to finally make something. Anything. Entries for Maemo & Mer-based devices are eligible.

The timetable for the competition is the following: The competitions begins on Tuesday, 1st of November, and will be running for 3 months until Tuesday 31st of January 2016. You can enroll into the competition at any time, however, all the entries for the competition must be submitted no later than by 23:59 on Tuesday, 31st of January(UTC time). Testing/reviewing will take two weeks, from 01.02.2017 to 15.02.2017 and voting will run from 16th of February until 28th of February 2017, 23:59 UTC.

Follow the links to the wiki below for more information.

Devices & Platforms
Categories
How to Submit an Entry
Prizes
Participants
Rules
Voting
Developer Resources

If you would like to have an App, a Port, a Fix, a Hack or etc. but you can't code it yourself, don't worry, just write your ideas in here! Many developers would be happy to get a new idea input.

We are relying on your donations towards the Community Prize Fund.

The entry is free of charge.

Thanks for your interest in taking part, we can't wait to see what you will develop! :)

You still have questions? Contact us!

Regards

Maemo Community e.V.

0 Add to favourites0 Bury

28 Oct 2016 11:41am GMT

11 Sep 2016

feedPlanet Maemo

Maemo Community e.V. - new Invitation to the General Assembly 2016

Maemo Community e.V. - Einladung zur Mitgliederversammlung 2016

Sehr geehrtes Mitglied,

aufgrund zu geringer Teilnehmerzahl und daraus resultierender Beschlussunfähigkeit, musste die letzte ordentliche Mitgliederversammlung am 04.09.2016 vorzeitig beendet werden.
Die Mitgliederversammlung findet nun am 09.10.2016 14:00 UTC im IRC Freenode Channel #maemo-meeting statt.
*Bitte sorgt dafür, dass ihr diesmal anwesend seid.*

Auf der Tagesordnung stehen folgende Themen:
1. Begrüßung durch die Vorsitzende des Vorstands
2. Feststellung der ordnungsgemäßen Einberufung und der Beschlussfähigkeit der Mitgliederversammlung
3. Entgegennahme des Jahresberichts für das abgelaufene Geschäftsjahr und Entlastung des Vorstands
4. Neuwahl des Vorstandes
5. Änderung der Satzung §7 und §8, und der Vereinsordnung §1.1, §3.2 und §4.1. Erläuterung im Anhang [1].
6. Verschiedenes

Anträge auf Ergänzungen der Tagesordnung müssen bis eine Woche vor der Versammlung schriftlich beim Vorstand eingereicht werden (§ 9 Abs. 2 der Satzung).

Mit freundlichen Grüßen
Win7Mac/Gido Griese


########################## English ##########################

Maemo Community e.V. - Invitation to the General Assembly 2016

Dear Member,

due to too low number of participants and the resulting lack of a quorum, our last regular General Assembly on 2016-09-04 had to be ceased prematurely.
The General Meeting will now be held on Sunday, 2016-10-09 at 14:00 UTC on IRC Freenode channel #maemo-meeting.
*Please make sure to join the meeting this time.*

Unless any further issues are raised, the agenda includes the following topics:
1. Welcome by the Chairman of the Board
2. Determination of the proper convocation and the quorum of the General Assembly
3. Acceptance of the annual report for the fiscal year and actions of the Executive
4. Election of the Board Directors
5. Amendment of Articles of Association §7 and §8, Association Rules §1.1, §3.2 and §4.1 - please see appendix for details [1].
6. Any other business

Requests for additions to the agenda must be submitted to the Board in writing one week prior to the meeting (§ 9.2 of the Statutes).

Yours sincerely
Win7Mac/Gido Griese


[1] http://maemo.org/community/council/referendum_2015

Anhang / APPENDIX

** Änderungen der Satzung / Amendments to the Articles of Association **

§7 Vorstand (Board of directors)
(5) Der Vorstand führt die Beschlüsse der aktiven Mitgliederversammlung und der des Vorstandsrats aus, sofern dem Vorstandsrat diese Entscheidungshoheit laut Satzung oder Vereinsordnung eingeräumt wird.

(5) The Board of Directors carries out the resolutions of the General Assembly and of the Council in so far as this decision's jurisdiction is delegated to Council according to the statutes.

§8 Vorstandsrat (Council)
(4) Aufgaben des Vorstandsrats sowie Regelungen zu dessen Arbeit, inklusive der Einberufung von Sitzungen, deren Ablauf und die Durchführung von Abstimmungen, sind in der Vereinsordnung festgelegt.

(4) The council's duties, rules to announce meetings, their proceedings and executions of votes are regulated by the Association Rules.


** Änderungen der Vereinsordnung / Amendments to the Association Rules **

§ 1.1
(2) § 4.1 (Vorstandsratswahlen) dieser Vereinsordnung kann durch ein Referendum der passiven Mitgliederversammlung geändert oder aufgehoben werden. Änderungen dieser Wahlordnung durch die aktive Mitgliederversammlung müssen durch Beschluss der passiven Mitgliederversammlung bestätigt werden (Referendum). Sollte die Änderung von der passiven Mitgliederversammlung abgelehnt werden, kann der Änderungsvorschlag überarbeitet und erneut zur Abstimmung gebracht werden.

(2) § 4.1 (Council/election rules) of these association rules can be changed or dismissed with a referendum by the Passive Members' Meeting. Changes to these election rules by the General Assembly must be confirmed by a resolution of the passive members meeting (referendum). Should the amendment be rejected by the passive members meeting, the proposed amendment can be revised and brought again to the vote.


§3.2 Vorstandsrat /(council)

(7) Aufgaben und Zuständigkeiten des Vorstandsrats sind wie folgt:
a) Vertretung der assoziierten Mitglieder und ihrer Interessen
b) Beratung des Vorstands in allgemeinen Belangen
c) Der Vorstandsrat ist für die Durchführung aller Wahlen zuständig

(7) Tasks and competences of the Council are as follows:
a) Representation of the associated members (garage account) and their interests.
b) Counseling of the Board of directors in general matters
c) Council is responsible to execute all elections


§ 4.1 Vorstandsratswahlen / Council Elections

(6) Kandidaten mit einem professionellen bzw. witschaftlichen Interesse in Maemo, z.B. Mitarbeiter von Firmen, die in Bezug zu Maemo Hard- oder Software stehen, müssen ihr Interesse bei der Nominierung erklären. Bleibt dies aus, kann der Vorstandsrat die Kandidatur des Mitglieds für ungültig erklären und somit ausschliessen.

(6) Nominees with a professional interest in Maemo, such as working for a company involved in Maemo-related software development - must declare their interest when advertising their nomination. Failure to do so may result in the Board of Directors, or the outgoing Council, declaring their nomination invalid and so bar them from standing in the current election.

(9) Das Datum der nächsten Wahl soll mindestens 4 Wochen vorher auf maemo.org bekannt gegeben werden.

(9) The election date must be publicised at least 4 weeks in advance of the election.

(11) Sollten nach Ende der Nominierungsphase weniger als 4 Kandidaten zur Wahl stehen, kann die Wahl nicht abgehalten werden.
a) Die Nominierungsphase wird dann um 4 Wochen verlängert und die Wahl entsprechend verschoben.
b) Sollten nach der verlängerten Nominierungsphase weniger als 4 Kandidaten zur Wahl stehen, bestimmt der scheidende Vorstandsrat zusammen mit dem Vorstand das weitere Vorgehen. Eine Lösung, die zu weiteren Kandidaten und einer ordentlichen Wahl führt, soll dabei bevorzugt werden, andere Möglichkeiten werden nicht ausgeschlossen.

(11) If there are less than 4 candidates when the nominations close, the election can't be held.
a) The nomination period will be extended by 4 weeks and the election postponed similarly.
b) If, after 4 weeks's delay, there are still fewer than 4 candidates; the outgoing council will decide - in conjunction with Board of Directors - about how to proceed. The preferred solution is to encourage more members of the maemo.org community to participate and re-run the election, but all options are open.

(12) Änderungen der Vorstandsratswahlen müssen durch Beschluss durch die aktive Mitgliederversammlung oder die passive Mitgliederversammlung bestätigt werden.
a) Die Teilnahme an Referenda unterliegt den gleichen Bestimmungen, die für Vorstandsratswahlen gelten.
b) Bevor ein Referendum zur Abstimmung gelangen kann, muss ein entsprechender Entwurf für mindestens 4 Wochen zur Diskussion stehen.
c) Der Zeitraum, in welchem über das Referendum abgestimmt werden kann, soll genauso lang sein, wie bei regulären Vorstandsratswahlen.

(12)Changes to any of the above rules must be approved by a resolution of the Passive Members' Meeting (community referendum) or General Assembly.
a) Voting in such, will be open to anyone eligible to vote in the respective meeting.
b) The referendum options must be debated for a minimum of 4 weeks prior to the referendum.
c) Referendum voting will be open for the same length of time as the council elections.

1 Add to favourites0 Bury

11 Sep 2016 5:35pm GMT

30 Aug 2016

feedPlanet Maemo

2016-08-09 Meeting Minutes

Meeting held 2016-08-09 on FreeNode, channel #maemo-meeting (logs)

Attending: Win7Mac, eekkelund, juiceme, chem|st

Partial:

Absent: pichlo, reinob

Summary of topics (ordered by discussion):


(Topic Coding Competition):


Action Items:
  • old items:
    • Coding competition planning (eekkelund)
    • Set up ftp/sftp site for CC (reinob)
    • Create twitter account for maemo community(eekkelund)
  • new items:


Solved Action Items:
  • The next GA meeting should be announced soon.

0 Add to favourites0 Bury

30 Aug 2016 8:41am GMT

2016-08-02 Meeting Minutes

Meeting held 2016-08-02 on FreeNode, channel #maemo-meeting (logs)

Attending: pichlo, Win7Mac, eekkelund, reinob, juiceme

Partial:

Absent:

Summary of topics (ordered by discussion):


(Topic warfare replaced all SSL certificates):

(Topic MC e.V.):


Action Items:
  • old items:
    • Coding competition planning (eekkelund)
    • Set up ftp/sftp site for CC (reinob)
    • Create twitter account for maemo community(eekkelund)
  • new items:


Solved Action Items:
  • The next GA meeting should be announced soon.

0 Add to favourites0 Bury

30 Aug 2016 8:39am GMT

18 Aug 2016

feedPlanet Maemo

2016-07-26 Meeting Minutes

Meeting held 2016-07-26 on FreeNode, channel #maemo-meeting (logs)

Attending: pichlo, Win7Mac, eekkelund, reinob, juiceme

Partial: chem|st

Absent:

Summary of topics (ordered by discussion):


(Topic Approve pending project request on https://garage.maemo.org/admin/approve-pending.php):

(Topic Coding Competition):

(Topic Twitter account):

(Topic MC e.V.):


Action Items:
  • old items:
    • Coding competition planning (eekkelund)
    • The next GA meeting should be announced soon.
  • new items:
    • Set up ftp/sftp site for CC (reinob)
    • Create twitter account for maemo community(eekkelund)


Solved Action Items:

Find out if https is doable 0 Add to favourites0 Bury

18 Aug 2016 6:39am GMT