26 Jul 2016

feedPlanet Maemo

Loading truly truly huge text files with a QAbstractListModel

Sometimes people want to do crazy stuff like loading a gigabyte sized plain text file into a Qt view that can handle QAbstractListModel. Like for example a QML ListView. You know, the kind of files you generate with this commando:

base64 /dev/urandom | head -c 100000000 > /tmp/file.txt

But, how do they do it?

FileModel.h

So we will make a custom QAbstractListModel. Its private member fields I will explain later:

#ifndef FILEMODEL_H
#define FILEMODEL_H

#include <QObject>
#include <QVariant>
#include <QAbstractListModel>
#include <QFile>

class FileModel: public QAbstractListModel {
    Q_OBJECT

    Q_PROPERTY(QString fileName READ fileName WRITE setFileName NOTIFY fileNameChanged )
public:
    explicit FileModel( QObject* a_parent = nullptr );
    virtual ~FileModel();

    int columnCount(const QModelIndex &parent) const;
    int rowCount( const QModelIndex& parent =  QModelIndex() ) const Q_DECL_OVERRIDE;
    QVariant data( const QModelIndex& index, int role = Qt::DisplayRole ) const  Q_DECL_OVERRIDE;
    QVariant headerData( int section, Qt::Orientation orientation,
                         int role = Qt::DisplayRole ) const  Q_DECL_OVERRIDE;
    void setFileName(const QString &fileName);
    QString fileName () const
        { return m_file->fileName(); }
signals:
    void fileNameChanged();
private:
    QFile *m_file, *m_index;
    uchar *map_file;
    uchar *map_index;
    int m_rowCount;
    void clear();
};

#endif// FILEMODEL_H

FileModel.cpp

We will basically scan the very big source text file for newline characters. We'll write the offsets of those to a file suffixed with ".mmap". We'll use that new file as a sort of "partition table" for the very big source text file, in the data() function of QAbstractListModel. But instead of sectors and files, it points to newlines.

The reason why the scanner itself isn't using the mmap's address space is because apparently reading blocks of 4kb is faster than reading each and every byte from the mmap in search of \n characters. Or at least on my hardware it was.

You should probably do the scanning in small qEventLoop iterations (make sure to use nonblocking reads, then) or in a thread, as your very big source text file can be on a unreliable or slow I/O device. Plus it's very big, else you wouldn't be doing this (please promise me to just read the entire text file in memory unless it's hundreds of megabytes in size: don't micro optimize your silly homework notepad.exe clone).

Note that this is demo code with a lot of bugs like not checking for \r and god knows what memory leaks and stuff was remaining when it suddenly worked. I leave it to the reader to improve this. An example is that you should check for validity of the ".mmap" file: your very big source text file might have changed since the newline partition table was made.

Knowing that I'll soon find this all over the place without any of its bugs fixed, here it comes ..

#include "FileModel.h"

#include <QDebug>

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <pthread.h>
#include <unistd.h>

FileModel::FileModel( QObject* a_parent )
    : QAbstractListModel( a_parent )
    , m_file (nullptr)
    , m_index(nullptr)
    , m_rowCount ( 0 ) { }

FileModel::~FileModel() { clear(); }

void FileModel::clear()
{
    if (m_file) {
        if (m_file->isOpen() && map_file != nullptr)
            m_file->unmap(map_file);
        delete m_file;
    }
    if (m_index) {
        if (m_index->isOpen() && map_index != nullptr)
            m_index->unmap(map_index);
        delete m_index;
    }
}

void FileModel::setFileName(const QString &fileName)
{
   clear();
   m_rowCount = 0;
   m_file = new QFile(fileName);
   int cur = 0;
   m_index = new QFile(m_file->fileName() + ".mmap");
   if (m_file->open(QIODevice::ReadOnly)) {
       if (!m_index->exists()) {
           char rbuffer[4096];
           m_index->open(QIODevice::WriteOnly);
           char nulbuffer[4];
           int idxnul = 0;
           memset( nulbuffer +0, idxnul >> 24 & 0xff, 1 );
           memset( nulbuffer +1, idxnul >> 16 & 0xff, 1 );
           memset( nulbuffer +2, idxnul >>  8 & 0xff, 1 );
           memset( nulbuffer +3, idxnul >>  0 & 0xff, 1 );
           m_index->write( nulbuffer, sizeof(quint32));
           qDebug() << "Indexing to" << m_index->fileName();
           while (!m_file->atEnd()) {
               int in = m_file->read(rbuffer, 4096);
               if (in == -1)
                   break;
               char *newline = (char*) 1;
               char *last = rbuffer;
               while (newline != 0) {
                   newline = strchr ( last, '\n');
                   if (newline != 0) {
                     char buffer[4];
                     int idx = cur + (newline - rbuffer);
                     memset( buffer +0, idx >> 24 & 0xff, 1 );
                     memset( buffer +1, idx >> 16 & 0xff, 1 );
                     memset( buffer +2, idx >>  8 & 0xff, 1 );
                     memset( buffer +3, idx >>  0 & 0xff, 1 );
                     m_index->write( buffer, sizeof(quint32));
                     m_rowCount++;
                     last = newline + 1;
                  }
               }
               cur += in;
           }
           m_index->close();
           m_index->open(QFile::ReadOnly);
           qDebug() << "done";
       } else {
           m_index->open(QFile::ReadOnly);
           m_rowCount = m_index->size() / 4;
       }
       map_file= m_file->map(0, m_file->size(), QFileDevice::NoOptions);
       qDebug() << "Done loading " << m_rowCount << " lines";
       map_index = m_index->map(0, m_index->size(), QFileDevice::NoOptions);
   }
   beginResetModel();
   endResetModel();
   emit fileNameChanged();
}

static quint32
read_uint32 (const quint8 *data)
{
    return data[0] << 24 |
           data[1] << 16 |
           data[2] << 8 |
           data[3];
}

int FileModel::rowCount( const QModelIndex& parent ) const
{
    Q_UNUSED( parent );
    return m_rowCount;
}

int FileModel::columnCount(const QModelIndex &parent) const
{
    Q_UNUSED( parent );
    return 1;
}

QVariant FileModel::data( const QModelIndex& index, int role ) const
{
    if( !index.isValid() )
        return QVariant();
    if (role == Qt::DisplayRole) {
        QVariant ret;
        quint32 pos_i = read_uint32(map_index + ( 4 * index.row() ) );
        quint32 end_i;
        if ( index.row() == m_rowCount-1 )
            end_i = m_file->size();
        else
            end_i = read_uint32(map_index + ( 4 * (index.row()+1) ) );
        uchar *position;
        position = map_file +  pos_i;
        uchar *end = map_file + end_i;
        int length = end - position;
        char *buffer = (char*) alloca(length +1);
        memset (buffer, 0, length+1);
        strncpy (buffer, (char*) position, length);
        ret = QVariant(QString(buffer));
        return ret;
    }
    return QVariant();
}

QVariant FileModel::headerData( int section, Qt::Orientation orientation, int role ) const
{
    Q_UNUSED(section);
    Q_UNUSED(orientation);
    if (role != Qt::DisplayRole)
           return QVariant();
    return QString("header");
}

main.cpp

#include <QGuiApplication>
#include <QQmlApplicationEngine>
#include <QtQml>// qmlRegisterType

#include "FileModel.h"

int main(int argc, char *argv[])
{
    QGuiApplication app(argc, argv);
    qmlRegisterType<FileModel>( "FileModel", 1, 0, "FileModel" );
    QQmlApplicationEngine engine;
    engine.load(QUrl(QStringLiteral("qrc:/main.qml")));
    return app.exec();
}

main.qml

import QtQuick 2.3
import QtQuick.Window 2.2
import FileModel 1.0

Window {
    visible: true

    FileModel { id: fileModel }
    ListView {
        id: list
        anchors.fill: parent
        delegate: Text { text: display }
        MouseArea {
            anchors.fill: parent
            onClicked: {
                list.model = fileModel
                fileModel.fileName = "/tmp/file.txt"
            }
        }
    }
}

profile.pro

TEMPLATE = app
QT += qml quick
CONFIG += c++11
SOURCES += main.cpp \
    FileModel.cpp
RESOURCES += qml.qrc
HEADERS += \
    FileModel.h

qml.qrc

<RCC>
    <qresource prefix="/">
        <file>main.qml</file>
    </qresource>
</RCC>

0 Add to favourites0 Bury

26 Jul 2016 7:15pm GMT

09 Jul 2016

feedPlanet Maemo

Creating PyGTK app snaps with snapcraft

Snap is a new packaging format introduced by Ubuntu as an successor to dpkg aka debian package. It offers sandboxing and transactional updates and thus is a competitor to the flatpak format and resembles docker images.

As with every new technology the weakest point of working with snaps is the documentation. Your best bet so far is the snappy-playpen repository.

There are also some rough edges regarding desktop integration and python interoperability, so this is what the post will be about.

I will introduce some quircks that were needed to get teatime running, which is written in Python3 and uses Unity and GTK3 via GObject introspection.

The most important thing to be aware of is that snaps are similar to containers in that each snap has its own rootfs and only restricted access outside of it. This is basically what the sandboxing is about.
However a typical desktop application needs to know quite a lot about the outside world:

To declare that we want to write to home, play back sound and use unity features we use the plugs keyword like

apps:
    teatime:
         # ...
         plugs: [unity7, home, pulseaudio]

However we must also tell our app to look for the according libraries inside its snap instead of the system paths. For this one must change quite a few environment variables manually. Fortunately Ubuntu provides wrapper scripts that take care of this for us. They are called desktop-launchers.

To use the launcher the configures the GTK3 environment we have to extend the teatime part like this:

parts:
    teatime:
         # ...
         after: [desktop/gtk3]

However the desktop-launch script currently misses the environment varibles for GI repository files. To fix this we must use this rather lengthy startup command

env GI_TYPELIB_PATH=$SNAP/usr/lib/girepository-1.0:$SNAP/usr/lib/x86_64-linux-gnu/girepository-1.0 desktop-launch $SNAP/usr/share/teatime/teatime.py

the bad part here is that we hard-code the architecture. The proper solution is to just fix the desktop-launch script, which I did here.

Update

The change was merged, so the startup command simpifies to

apps:
    teatime:
        command: desktop-launch $SNAP/usr/share/teatime/teatime.py
        # ...

You can see the full snapcraft.yml here.

After this teatime will start, but the paths still have to be fixed. Inside a snap "/" still refers to the system root, so all absolute paths must be prefixed with $SNAP.

Actually I think the design of flatpak is more elegant in this regard where "/" points to the local rootfs and one does not have to change absolute paths. To bring in the system parts flatpak uses bind mounts.

Conclusions

Once you get the hang of how snaps work, packaging becomes quite straightforward, however currently there are still some drawbacks

0 Add to favourites0 Bury

09 Jul 2016 1:45pm GMT

01 Jul 2016

feedPlanet Maemo

Composition and aggregation with QObject

Consider these rather simple relationships between classes

Continuing on this subject, here are some code examples.

Class1 & Class2: Composition
An instance of Class1 can not exist without an instance of Class2.

Example of composition is typically a Bicycle and its Wheels, Saddle and a HandleBar: without these the Bicycle is no longer a Bicycle but just a Frame.

It can no longer function as a Bicycle. Example of when you need to stop thinking about composition versus aggregation is whenever you say: without the other thing can't in our software the first thing work.

Note that you must consider this in the context of Class1. You use aggregation or composition based on how Class2 exists in relation to Class1.

Class1 with QScopedPointer:

#ifndef CLASS1_H
#define CLASS1_H

#include <QObject>
#include <QScopedPointer>
#include <Class2.h>

class Class1: public QObject
{
    Q_PROPERTY( Class2* class2 READ class2 WRITE setClass2 NOTIFY class2Changed)
public:
    Class1( QObject *a_parent = nullptr )
        : QObject ( a_parent) {
        // Don't use QObject parenting on top here
        m_class2.reset (new Class2() );
    }
    Class2* class2() {
        return m_class2.data();
    }
    void setClass2 ( Class2 *a_class2 ) {
        Q_ASSERT (a_class2 != nullptr); // Composition can't set a nullptr!
        if ( m_class2.data() != a_class2 ) {
            m_class2.reset( a_class2 );
            emit class2Changed()
        }
    }
signals:
    void class2Changed();
private:
    QScopedPointer<Class2> m_class2;
};

#endif// CLASS1_H

Class1 with QObject parenting:

#ifndef CLASS1_H
#define CLASS1_H

#include <QObject>
#include <Class2.h>

class Class1: public QObject
{
    Q_PROPERTY( Class2* class2 READ class2 WRITE setClass2 NOTIFY class2Changed)
public:
    Class1( QObject *a_parent = nullptr )
        : QObject ( a_parent )
        , m_class2 ( nullptr ) {
        // Make sure to use QObject parenting here
        m_class2 = new Class2( this );
    }
    Class2* class2() {
        return m_class2;
    }
    void setClass2 ( Class2 *a_class2 ) {
         Q_ASSERT (a_class2 != nullptr); // Composition can't set a nullptr!
         if ( m_class2 != a_class2 ) {
             // Make sure to use QObject parenting here
             a_class2->setParent ( this );
             delete m_class2; // Composition can never be nullptr
             m_class2 = a_class2;
             emit class2Changed();
         }
    }
signals:
    void class2Changed();
private:
    Class2 *m_class2;
};

#endif// CLASS1_H

Class1 with RAII:

#ifndef CLASS1_H
#define CLASS1_H

#include <QObject>
#include <QScopedPointer>

#include <Class2.h>

class Class1: public QObject
{
    Q_PROPERTY( Class2* class2 READ class2 CONSTANT)
public:
    Class1( QObject *a_parent = nullptr )
        : QObject ( a_parent ) { }
    Class2* class2()
        { return &m_class2; }
private:
    Class2 m_class2;
};
#endif// CLASS1_H

Class3 & Class4: Aggregation

An instance of Class3 can exist without an instance of Class4. Example of composition is typically a Bicycle and its driver or passenger: without the Driver or Passenger it is still a Bicycle. It can function as a Bicycle.

Example of when you need to stop thinking about composition versus aggregation is whenever you say: without the other thing can in our software the first thing work.

Class3:

#ifndef CLASS3_H
#define CLASS3_H

#include <QObject>

#include <QPointer>
#include <Class4.h>

class Class3: public QObject
{
    Q_PROPERTY( Class4* class4 READ class4 WRITE setClass4 NOTIFY class4Changed)
public:
    Class3( QObject *a_parent = nullptr );
    Class4* class4() {
        return m_class4.data();
    }
    void setClass4 (Class4 *a_class4) {
         if ( m_class4 != a_class4 ) {
             m_class4 = a_class4;
             emit class4Changed();
         }
    }
signals:
    void class4Changed();
private:
    QPointer<Class4> m_class4;
};
#endif// CLASS3_H

Class5, Class6 & Class7: Shared composition
An instance of Class5 and-or an instance of Class6 can not exist without a instance of Class7 shared by Class5 and Class6. When one of Class5 or Class6 can and one can not exist without the shared instance, use QWeakPointer at that place.

Class5:

#ifndef CLASS5_H
#define CLASS5_H

#include <QObject>
#include <QSharedPointer>

#include <Class7.h>

class Class5: public QObject
{
    Q_PROPERTY( Class7* class7 READ class7 CONSTANT)
public:
    Class5( QObject *a_parent = nullptr, Class7 *a_class7 );
        : QObject ( a_parent )
        , m_class7 ( a_class7 ) { }
    Class7* class7()
        { return m_class7.data(); }
private:
    QSharedPointer<Class7> m_class7;
};

Class6:

#ifndef CLASS6_H
#define CLASS6_H

#include <QObject>
#include <QSharedPointer>

#include <Class7.h>

class Class6: public QObject
{
    Q_PROPERTY( Class7* class7 READ class7 CONSTANT)
public:
    Class6( QObject *a_parent = nullptr, Class7 *a_class7 )
        : QObject ( a_parent )
        , m_class7 ( a_class7 ) { }
    Class7* class7()
        { return m_class7.data(); }
private:
    QSharedPointer<Class7> m_class7;
};
#endif// CLASS6_H

Interfaces with QObject

FlyBehavior:

#ifndef FLYBEHAVIOR_H
#define FLYBEHAVIOR_H
#include <QObject>
// Don't inherit QObject here (you'll break multiple-implements)
class FlyBehavior {
    public:
        Q_INVOKABLE virtual void fly() = 0;
};
Q_DECLARE_INTERFACE(FlyBehavior , "be.codeminded.Flying.FlyBehavior /1.0") 
#endif// FLYBEHAVIOR_H

FlyWithWings:

#ifndef FLY_WITH_WINGS_H
#define FLY_WITH_WINGS_H
#include <QObject>  
#include <Flying/FlyBehavior.h>
// Do inherit QObject here (this is a concrete class)
class FlyWithWings: public QObject, public FlyBehavior
{
    Q_OBJECT
    Q_INTERFACES( FlyBehavior )
public:
    explicit FlyWithWings( QObject *a_parent = nullptr ): QObject ( *a_parent ) {}
    ~FlyWithWings() {}

    virtual void fly() Q_DECL_OVERRIDE;
}
#endif// FLY_WITH_WINGS_H

0 Add to favourites0 Bury

01 Jul 2016 11:38am GMT

22 Jun 2016

feedPlanet Maemo

2016-06-14 Meeting Minutes

Meeting held 2016-06-14 on FreeNode, channel #maemo-meeting (logs)

Attending:reinob, Win7Mac, M4rtinK, eekkelund, pichlo

Partial: chem|st

Absent: juiceme

Summary of topics (ordered by discussion):


(Topic Coding Competetion):

(Topic Bitcoins, HTTPS):


Action Items:
  • old items:
    • The next GA meeting should be announced soon.
    • Tax-exempt status
    • Could we make coding competetion happen
  • new items:
    • Find legal disclaimer for Bitcoin donations


Solved Action Items:
  • Ask about tor and https from techstaff

0 Add to favourites0 Bury

22 Jun 2016 7:27pm GMT

2016-06-07 Meeting Minutes

Meeting held 2016-06-07 on FreeNode, channel #maemo-meeting (logs)

Attending:reinob, eekkelund, Win7Mac, juiceme, pichlo

Partial:

Absent:

Summary of topics (ordered by discussion):


(Topic Tax-exempt):

(Topic More Coding Competetion discussion):


Action Items:
  • old items:
    • The next GA meeting should be announced soon.
    • Ask about tor and https from techstaff
    • Could we make coding competetion happen
  • new items:
    • Tax-exempt status


Solved Action Items:
  • Find out if https is doable (pichlo)
  • Check missing priviledges to the Council blog (juiceme)

0 Add to favourites0 Bury

22 Jun 2016 7:26pm GMT

07 Jun 2016

feedPlanet Maemo

2016-05-31 Meeting Minutes

Meeting held 2016-05-31 on FreeNode, channel #maemo-meeting (logs)

Attending: eekkelund, reinob, Win7Mac, juiceme

Partial: pichlo

Absent:

Summary of topics (ordered by discussion):


(Topic Coding competition):


Action Items:
  • old items:
    • The next GA meeting should be announced soon.
    • Ask about tor and https from techstaff (reinob)
  • new items:
    • Could we make coding competetion happen


Solved Action Items:
  • Check missing priviledges to the Council blog (juiceme)
  • Find out if https is doable (pichlo)
  • Arrange Council permissions on TMO to new Councillors. (juiceme)

0 Add to favourites0 Bury

07 Jun 2016 5:50pm GMT

2016-05-24 Meeting Minutes

Meeting held 2016-05-24 on FreeNode, channel #maemo-meeting (logs)

Attending: eekkelund, pichlo, Win7Mac, juiceme, reinob

Partial:

Absent:

Summary of topics (ordered by discussion):


(Topic https and tor support for maemo.org):

(Topic Missing priviligies for some councillors):


Action Items:
  • old items:
    • The next GA meeting should be announced soon.
  • new items:
    • Ask about tor and https from techstaff (reinob)
    • Check missing priviledges to the Council blog (juiceme)


Solved Action Items:
  • Find out if https is doable (pichlo)
  • Arrange Council permissions on TMO to new Councillors. (juiceme)
  • Add new Councillors to the mailing list and remove the old. (juiceme)
  • Get cloaks for new Councillors. (juiceme)

0 Add to favourites0 Bury

07 Jun 2016 5:48pm GMT

31 May 2016

feedPlanet Maemo

2016-05-17 Meeting Minutes

Meeting held 2016-05-17 on FreeNode, channel #maemo-meeting (logs)

Attending: eekkelund, pichlo, Win7Mac, juiceme, M4rtinK

Partial: chem|st

Absent: reinob

Summary of topics (ordered by discussion):


(Topic Tor users being blacklisted, read-only option for blacklisted IPs):

(Full httpS support for maemo.org):

(Spam to garage email):

(Bitcoin):


Action Items:
  • old items:
    • The next GA meeting should be announced soon.
    • Arrange Council permissions on TMO to new Councillors. (juiceme)
  • new items:
    • Find out if https is doable (pichlo)


Solved Action Items:
  • Add new Councillors to the mailing list and remove the old. (juiceme)
  • Get cloaks for new Councillors. (juiceme)

0 Add to favourites0 Bury

31 May 2016 7:04pm GMT

24 May 2016

feedPlanet Maemo

I/O bursts with QEMU 2.6

QEMU 2.6 was released a few days ago. One new feature that I have been working on is the new way to configure I/O limits in disk drives to allow bursts and increase the responsiveness of the virtual machine. In this post I'll try to explain how it works.

The basic settings

First I will summarize the basic settings that were already available in earlier versions of QEMU.

Two aspects of the disk I/O can be limited: the number of bytes per second and the number of operations per second (IOPS). For each one of them the user can set a global limit or separate limits for read and write operations. This gives us a total of six different parameters.

I/O limits can be set using the throttling.* parameters of -drive, or using the QMP block_set_io_throttle command. These are the names of the parameters for both cases:

-drive block_set_io_throttle
throttling.iops-total iops
throttling.iops-read iops_rd
throttling.iops-write iops_wr
throttling.bps-total bps
throttling.bps-read bps_rd
throttling.bps-write bps_wr

It is possible to set limits for both IOPS and bps at the same time, and for each case we can decide whether to have separate read and write limits or not, but if iops-total is set then neither iops-read nor iops-write can be set. The same applies to bps-total and bps-read/write.

The default value of these parameters is 0, and it means unlimited.

In its most basic usage, the user can add a drive to QEMU with a limit of, say, 100 IOPS with the following -drive line:

-drive file=hd0.qcow2,throttling.iops-total=100

We can do the same using QMP. In this case all these parameters are mandatory, so we must set to 0 the ones that we don't want to limit:

   { "execute": "block_set_io_throttle",
     "arguments": {
        "device": "virtio0",
        "iops": 100,
        "iops_rd": 0,
        "iops_wr": 0,
        "bps": 0,
        "bps_rd": 0,
        "bps_wr": 0
     }
   }

I/O bursts

While the settings that we have just seen are enough to prevent the virtual machine from performing too much I/O, it can be useful to allow the user to exceed those limits occasionally. This way we can have a more responsive VM that is able to cope better with peaks of activity while keeping the average limits lower the rest of the time.

Starting from QEMU 2.6, it is possible to allow the user to do bursts of I/O for a configurable amount of time. A burst is an amount of I/O that can exceed the basic limit, and there are two parameters that control them: their length and the maximum amount of I/O they allow. These two can be configured separately for each one of the six basic parameters described in the previous section, but here we'll use 'iops-total' as an example.

The I/O limit during bursts is set using 'iops-total-max', and the maximum length (in seconds) is set with 'iops-total-max-length'. So if we want to configure a drive with a basic limit of 100 IOPS and allow bursts of 2000 IOPS for 60 seconds, we would do it like this (the line is split for clarity):

   -drive file=hd0.qcow2,
          throttling.iops-total=100,
          throttling.iops-total-max=2000,
          throttling.iops-total-max-length=60

Or with QMP:

   { "execute": "block_set_io_throttle",
     "arguments": {
        "device": "virtio0",
        "iops": 100,
        "iops_rd": 0,
        "iops_wr": 0,
        "bps": 0,
        "bps_rd": 0,
        "bps_wr": 0,
        "iops_max": 2000,
        "iops_max_length": 60,
     }
   }

With this, the user can perform I/O on hd0.qcow2 at a rate of 2000 IOPS for 1 minute before it's throttled down to 100 IOPS.

The user will be able to do bursts again if there's a sufficiently long period of time with unused I/O (see below for details).

The default value for 'iops-total-max' is 0 and it means that bursts are not allowed. 'iops-total-max-length' can only be set if 'iops-total-max' is set as well, and its default value is 1 second.

Controlling the size of I/O operations

When applying IOPS limits all I/O operations are treated equally regardless of their size. This means that the user can take advantage of this in order to circumvent the limits and submit one huge I/O request instead of several smaller ones.

QEMU provides a setting called throttling.iops-size to prevent this from happening. This setting specifies the size (in bytes) of an I/O request for accounting purposes. Larger requests will be counted proportionally to this size.

For example, if iops-size is set to 4096 then an 8KB request will be counted as two, and a 6KB request will be counted as one and a half. This only applies to requests larger than iops-size: smaller requests will be always counted as one, no matter their size.

The default value of iops-size is 0 and it means that the size of the requests is never taken into account when applying IOPS limits.

Applying I/O limits to groups of disks

In all the examples so far we have seen how to apply limits to the I/O performed on individual drives, but QEMU allows grouping drives so they all share the same limits.

This feature is available since QEMU 2.4. Please refer to the post I wrote when it was published for more details.

The Leaky Bucket algorithm

I/O limits in QEMU are implemented using the leaky bucket algorithm (specifically the "Leaky bucket as a meter" variant).

This algorithm uses the analogy of a bucket that leaks water constantly. The water that gets into the bucket represents the I/O that has been performed, and no more I/O is allowed once the bucket is full.

To see the way this corresponds to the throttling parameters in QEMU, consider the following values:

  iops-total=100
  iops-total-max=2000
  iops-total-max-length=60

bucket

The bucket is initially empty, therefore water can be added until it's full at a rate of 2000 IOPS (the burst rate). Once the bucket is full we can only add as much water as it leaks, therefore the I/O rate is reduced to 100 IOPS. If we add less water than it leaks then the bucket will start to empty, allowing for bursts again.

Note that since water is leaking from the bucket even during bursts, it will take a bit more than 60 seconds at 2000 IOPS to fill it up. After those 60 seconds the bucket will have leaked 60 x 100 = 6000, allowing for 3 more seconds of I/O at 2000 IOPS.

Also, due to the way the algorithm works, longer burst can be done at a lower I/O rate, e.g. 1000 IOPS during 120 seconds.

Acknowledgments

As usual, my work in QEMU is sponsored by Outscale and has been made possible by Igalia and the help of the QEMU development team.

igalia-outscale

Enjoy QEMU 2.6!

0 Add to favourites0 Bury

24 May 2016 11:47am GMT

08 May 2016

feedPlanet Maemo

2016-05-03 Meeting Minutes

Meeting held 2016-05-03 on FreeNode, channel #maemo-meeting (logs)

Attending: eekkelund, reinob, pichlo, Win7Mac, juiceme, Oksana

Partial:

Absent: peterleinchen, HtheB

Summary of topics (ordered by discussion):


(Topic Handover meeting for the new Council):


Action Items:
  • old items:
    • The next GA meeting should be announced soon.
  • new items:
    • Add new Councillors to the mailing list and remove the old. (juiceme)
    • Arrange Council permissions on TMO to new Councillors. (juiceme)
    • Get cloaks for new Councillors. (juiceme)

0 Add to favourites0 Bury

08 May 2016 1:44pm GMT

02 May 2016

feedPlanet Maemo

Introducing the OGRE fork on GitHub

in this post I want to introduce the OGRE fork on github. The goal of the fork is to provide a stable and reliable OGRE 1.x series while at the same time modernizing parts under the hood updates.

The idea behind this is that there are many existing 1.x codebases, actually a whole 1.x ecosystem, that can be modernized that way.
The last release of the 1.x series was over 2 years ago, so using the current 1.10 branch already gives a lot of improvements.

However the 1.10 branch contains some unnecessary changes that make it incompatible to the 1.9 release. These were reverted, so old code should compile again.

Additionally there are modernizing changes when compared with the upstream at bitbucket, which are discussed in more detail in the following.

This will be still a high-level overview though. If you need even more details, take a look on the git-ChangeLog.

Replace legacy renderers

With 1.9 the only stable renderers were GL(2.0) and DX9 which by today are outdated and produce different results than the GLES2 renderer that is used for Web and Mobile, which makes porting applications harder.

Therefore the goal is to get the GL3Plus renderer into a shape where it can act as a drop-in replacement for the old GL render and then drop the latter.

You can see the current status here - while it says that only 41/86 tests pass, most of the failed tests actually only differ in only 1 or 2 pixels which is pretty good when considering we are comparing fixed function vs. dynamically generated shaders (RTSS).

But the fork does not stop here; the GL renderers now also share the context creation code. This allows using the GLES2 renderer on the desktop or creating a GL3+ context using EGL - which is a prerequisite for headless rendering and for running on Wayland/MIR. Overall this makes your applications much more portable.

Improved regression testing

Changing the renderer requires being able to continuously monitor whether the rendering is still correct and to immediately detect regressions.

For this the Testing and VisualTesting frameworks were fixed and now correctly run on Linux and OSX. They run on each pull request to catch errors before the code even touches master.

Furthermore the tests can now be built without input handling (OIS) which hopefully will lead to wider adoption.

The unit tests still use cppunit which is a pain to use when compared to gtest, but changing that would require a large rewrite.

Batteries Included

The build now automatically handles the dependencies by downloading and building them as needed. On all platforms. So you do not have to care about mismatching compile settings any more.

Furthermore the SampleBrowser can now be build without input handling which eases quick testing. And in case you want input, now SDL2 is used instead of the esoteric and outdated OIS.

Next bringing your applications to the web is now easier; OGRE was able to run in the browser for a long time already - but the process was only badly documented. The fork tracks the Emscripten sample code inside the repository and also has documentation on how to use it.

Bindings for Scripting Languages

For using scripting languages, there is traditionally MOGRE (C#), Ogre4j, Python-Ogre which are by now basically all outdated and unmaintained.

To put all these efforts on a solid foundation I took some inspiration from ruby-ogre and added SWIG support.

At the time of writing there are already working Python bindings (see sample here) and a proof-of-concept for C#.

By using SWIG it is now easily to add further bindings for Ruby, Java, JavaScript (Node.js) as well. But as I care mostly for Python this will be an exercise for the reader.

A word on OGRE 2.1

You might wonder why one should care about the 1.x and not go for 2.1 directly. The obvious reason is that you have an existing codebase and OGRE 2.1 drastically changes API and even the material file format.

Then, while faster than 1.x, OGRE 2.x is still far from feature completeness with 1.x - it still completely lacks Web and Mobile support.
Faster also actually depends on your scene/ material usage; For instance, if you only render a single object (product showcase) 2.x will not offer you any advantages.

Finally the main functionality advance in 2.1, namely the new material system (HLMS) and with it physically based shading were backported to 1.x.

Yet if you only care about desktop and want to render large immersive worlds (lots of nodes, lots of materials), 2.1 is the way to go.

0 Add to favourites0 Bury

02 May 2016 2:38pm GMT

23 Mar 2016

feedPlanet Maemo

Q2 2016 Community Council Election Announcement

Dear friends and Maemoans, six months and even a bit more has passed and again it is time to elect a new Community Council for us.

The last election finished on Saturday 13th of September 2015.

The schedule of the voting is as follows:

To get our community to continue strong, we need fresh people with fresh viewpoints to carry on the torch, so please think about volunteering for Maemo Council.

On behalf of the outgoing community council,

Jussi Ohenoja

0 Add to favourites0 Bury

23 Mar 2016 6:17pm GMT

07 Jan 2016

feedPlanet Maemo

2015-12-01 Meeting Minutes

Meeting held 2015-12-01 on FreeNode, channel #maemo-meeting (logs)

Attending: Peter Leinchen (peterleinchen), Halit Bal (HtheB), Gido Griese (Win7Mac), Jussi Ohenoja (juiceme)

Partial: Bernardo Reino (reinob)

Absent:

Summary of topics (ordered by discussion):


(Topic Discussing handling of ex-members accounts):

(Topic Automatic calculation of election results):

(Topic Finalizing the rules for the coding competition):


Action Items:
  • old items:
    • Looking into automatic calculation of election results ...
    • Discuss spam handling with techstaff.
  • new items:
    • Finalize the coding competition rules.
    • Post to TMO announcing the competition and requesting for donations.
    • Discuss competition entries storing with OpenRepos people.

0 Add to favourites0 Bury

07 Jan 2016 11:18am GMT

29 Dec 2015

feedPlanet Maemo

Learning Modern 3D Graphics Programming

one of the best resources to learn modern OpenGL and the one which helped me quite a lot is the Book at www.arcsynthesis.org/gltut/ - or lets better say was. Unfortunately the domain expired so the content is no longer reachable.

Luckily the Book was designed as an open source project and the code to generate the website is still available at Bitbucket. Unfortunately this repository does not seem to be actively maintained any more.

Therefore I set out to make the Book to be available again using Github Pages. You can find the results here:

https://paroj.github.io/gltut/

However I did not simply mirror the pages, but also improved it at several places. So what has changed so far?

hopefully these changes will generate some momentum so this great Book gets extended again. As there were also non-cosmetical changes like the new Chapter I also tagged a 0.3.9 release.

I the process of the above work I found out that there is also a mirror of the original Book at http://alfonse.bitbucket.org/oldtut/. This one is however at the state of the 0.3.8 release, meaning it does not only misses the above changes but also some adjustment happened post 0.3.8 at bitbucket.

1 Add to favourites0 Bury

29 Dec 2015 10:26pm GMT

18 Dec 2015

feedPlanet Maemo

Coding Competition 2016

As you may have already noticed in the recent Meeting Minutes, current Council is about to prepare a Coding Competition (CC) for 2016. In times like these, without Nokia or any other corporate entity sponsoring prizes/devices, there are of course limited resources for big giveaways. Nevertheless we believe the approach is worth it. :) We are relying on your donations.

Looking at last CC's, this time it will probably be more of the supportive kind than rival in the first place. Basically we want to support the ongoing AND future software development in a way that ideally, the winners will get the "right" devices to carry on their efforts or even to the next level (e.g. cross-platform).

Now since no current Councilor has ever arranged a CC, we would like to hear your thoughts on how to get it "right", satisfying community demands (SW) as well as individual developer demands (HW). Please share your ideas! And if you've been involved in organizing recent CC's and are willing to help, please holler. And we sure need help in having an uptodate wiki page.

Furthermore, we are calling all owners of now unused devices to consider donating* them for this CC, respectively to MC e.V., who will be responsible for collecting donations and funds. Especially we are looking at those that got rewarded in recent CC's or device programs (N950 anyone? ;)).

* or selling them for a reasonable price.

The voting webinterface will be the same as for the Maemo Council elections, so jury will be you. We also need you to scrutinize the submitted code (ideally installable/executable + binary). It is undecided yet wether or not to accept closed source apps. As the Council is organizing this CC, it will only intervene if a need occurs. Please help make sure conditions and rules are distinct.

Ideas for categories:

- Old NIT's

- Fremantle

- Harmattan

- Sailfish OS

- "Port it/cross-platform"

- "Maintain it" (packages, CSSU)

- "Update/Fix it"

- "Bugfix it" (known, yet unresolved bugs)

- Core development or "Free it" (backport, clone, replace or supersede closed packages)

- Recent contributions (recent major releases)

- Wishlist (rewarded if/when completed)

- Beginner (first release)

A variant could be to have category "Update/Fix it" for each of the 4 device categories if it can be expected to have enough entries (and prizes).

Ideas and suggestions welcome. :)

1 Add to favourites0 Bury

18 Dec 2015 8:55pm GMT

01 Dec 2015

feedPlanet Maemo

2015-11-24 Meeting Minutes

Meeting held 2015-11-24 on FreeNode, channel #maemo-meeting (logs)

Attending: Peter Leinchen (peterleinchen), Bernardo Reino (reinob), Halit Bal (HtheB), Gido Griese (Win7Mac), Jussi Ohenoja (juiceme)

Partial: Rudiger Schiller (chem|st)

Absent:

Summary of topics (ordered by discussion):


(Topic Council roles reorganization):

(Topic Spam mail through Maemo accounts):

(Topic How to handle user requests to remove projects):

(Topic Code of Conduct):

(Topic Maemo Coding Competition 2015):

(Topic Elections handling and management):


Action Items:
  • old items:
    • The selected Code of Conduct (KDE) still needs to be published on TMO.
    • Looking into automatic calculation of election results ...
    • Arrange Council status on TMO for new Council members
  • new items:
    • Discuss spam handling with techstaff.
    • Finalize the coding competition rules.

1 Add to favourites0 Bury

01 Dec 2015 9:20am GMT