24 May 2016

feedPlanet Maemo

I/O bursts with QEMU 2.6

QEMU 2.6 was released a few days ago. One new feature that I have been working on is the new way to configure I/O limits in disk drives to allow bursts and increase the responsiveness of the virtual machine. In this post I'll try to explain how it works.

The basic settings

First I will summarize the basic settings that were already available in earlier versions of QEMU.

Two aspects of the disk I/O can be limited: the number of bytes per second and the number of operations per second (IOPS). For each one of them the user can set a global limit or separate limits for read and write operations. This gives us a total of six different parameters.

I/O limits can be set using the throttling.* parameters of -drive, or using the QMP block_set_io_throttle command. These are the names of the parameters for both cases:

-drive block_set_io_throttle
throttling.iops-total iops
throttling.iops-read iops_rd
throttling.iops-write iops_wr
throttling.bps-total bps
throttling.bps-read bps_rd
throttling.bps-write bps_wr

It is possible to set limits for both IOPS and bps at the same time, and for each case we can decide whether to have separate read and write limits or not, but if iops-total is set then neither iops-read nor iops-write can be set. The same applies to bps-total and bps-read/write.

The default value of these parameters is 0, and it means unlimited.

In its most basic usage, the user can add a drive to QEMU with a limit of, say, 100 IOPS with the following -drive line:

-drive file=hd0.qcow2,throttling.iops-total=100

We can do the same using QMP. In this case all these parameters are mandatory, so we must set to 0 the ones that we don't want to limit:

   { "execute": "block_set_io_throttle",
     "arguments": {
        "device": "virtio0",
        "iops": 100,
        "iops_rd": 0,
        "iops_wr": 0,
        "bps": 0,
        "bps_rd": 0,
        "bps_wr": 0
     }
   }

I/O bursts

While the settings that we have just seen are enough to prevent the virtual machine from performing too much I/O, it can be useful to allow the user to exceed those limits occasionally. This way we can have a more responsive VM that is able to cope better with peaks of activity while keeping the average limits lower the rest of the time.

Starting from QEMU 2.6, it is possible to allow the user to do bursts of I/O for a configurable amount of time. A burst is an amount of I/O that can exceed the basic limit, and there are two parameters that control them: their length and the maximum amount of I/O they allow. These two can be configured separately for each one of the six basic parameters described in the previous section, but here we'll use 'iops-total' as an example.

The I/O limit during bursts is set using 'iops-total-max', and the maximum length (in seconds) is set with 'iops-total-max-length'. So if we want to configure a drive with a basic limit of 100 IOPS and allow bursts of 2000 IOPS for 60 seconds, we would do it like this (the line is split for clarity):

   -drive file=hd0.qcow2,
          throttling.iops-total=100,
          throttling.iops-total-max=2000,
          throttling.iops-total-max-length=60

Or with QMP:

   { "execute": "block_set_io_throttle",
     "arguments": {
        "device": "virtio0",
        "iops": 100,
        "iops_rd": 0,
        "iops_wr": 0,
        "bps": 0,
        "bps_rd": 0,
        "bps_wr": 0,
        "iops_max": 2000,
        "iops_max_length": 60,
     }
   }

With this, the user can perform I/O on hd0.qcow2 at a rate of 2000 IOPS for 1 minute before it's throttled down to 100 IOPS.

The user will be able to do bursts again if there's a sufficiently long period of time with unused I/O (see below for details).

The default value for 'iops-total-max' is 0 and it means that bursts are not allowed. 'iops-total-max-length' can only be set if 'iops-total-max' is set as well, and its default value is 1 second.

Controlling the size of I/O operations

When applying IOPS limits all I/O operations are treated equally regardless of their size. This means that the user can take advantage of this in order to circumvent the limits and submit one huge I/O request instead of several smaller ones.

QEMU provides a setting called throttling.iops-size to prevent this from happening. This setting specifies the size (in bytes) of an I/O request for accounting purposes. Larger requests will be counted proportionally to this size.

For example, if iops-size is set to 4096 then an 8KB request will be counted as two, and a 6KB request will be counted as one and a half. This only applies to requests larger than iops-size: smaller requests will be always counted as one, no matter their size.

The default value of iops-size is 0 and it means that the size of the requests is never taken into account when applying IOPS limits.

Applying I/O limits to groups of disks

In all the examples so far we have seen how to apply limits to the I/O performed on individual drives, but QEMU allows grouping drives so they all share the same limits.

This feature is available since QEMU 2.4. Please refer to the post I wrote when it was published for more details.

The Leaky Bucket algorithm

I/O limits in QEMU are implemented using the leaky bucket algorithm (specifically the "Leaky bucket as a meter" variant).

This algorithm uses the analogy of a bucket that leaks water constantly. The water that gets into the bucket represents the I/O that has been performed, and no more I/O is allowed once the bucket is full.

To see the way this corresponds to the throttling parameters in QEMU, consider the following values:

  iops-total=100
  iops-total-max=2000
  iops-total-max-length=60

bucket

The bucket is initially empty, therefore water can be added until it's full at a rate of 2000 IOPS (the burst rate). Once the bucket is full we can only add as much water as it leaks, therefore the I/O rate is reduced to 100 IOPS. If we add less water than it leaks then the bucket will start to empty, allowing for bursts again.

Note that since water is leaking from the bucket even during bursts, it will take a bit more than 60 seconds at 2000 IOPS to fill it up. After those 60 seconds the bucket will have leaked 60 x 100 = 6000, allowing for 3 more seconds of I/O at 2000 IOPS.

Also, due to the way the algorithm works, longer burst can be done at a lower I/O rate, e.g. 1000 IOPS during 120 seconds.

Acknowledgments

As usual, my work in QEMU is sponsored by Outscale and has been made possible by Igalia and the help of the QEMU development team.

igalia-outscale

Enjoy QEMU 2.6!

0 Add to favourites0 Bury

24 May 2016 11:47am GMT

19 May 2016

feedPlanet Maemo

QML coding conventions checker that uses QML parser’s own abstract syntax tree

My colleague Henk Van Der Laak made a interesting tool that checks your code against the QML coding conventions. It uses the internal parser's abstract syntax tree of Qt 5.6 and a visitor design.

It has a command line, but being developers ourselves we want an API too of course. Then we can integrate it in our development environments without having to use popen!

So this is how to use that API:

// Parse the code
QQmlJS::Engine engine;
QQmlJS::Lexer lexer(&engine);
QQmlJS::Parser parser(&engine);

QFileInfo info(a_filename);
bool isJavaScript = info.suffix().toLower() == QLatin1String("js");
lexer.setCode(code,  1, !isJavaScript);
bool success = isJavaScript ? parser.parseProgram() : parser.parse();
if (success) {
    // Check the code
    QQmlJS::AST::UiProgram *program = parser.ast();
    CheckingVisitor checkingVisitor(a_filename);
    program->accept(&checkingVisitor);
    foreach (const QString &warning, checkingVisitor.getWarnings()) {
        qWarning() << qPrintable(warning);
    }
}

0 Add to favourites0 Bury

19 May 2016 1:31pm GMT

18 May 2016

feedPlanet Maemo

Item isChild of another Item in QML

Damned, QML is inconsistent! Things have a content, data or children. And apparently they can all mean the same thing. So how do we know if something is a child of something else?

After a failed stackoverflow search I gave up on copy-paste coding and invented the damn thing myself.

function isChild( a_child, a_parent ) {
        if ( a_parent === null ) {
                return false
        }

        var tmp = ( a_parent.hasOwnProperty("content") ? a_parent.content
                : ( a_parent.hasOwnProperty("children") ? a_parent.children : a_parent.data ) )

        if ( tmp === null || tmp === undefined ) {
                return false
        }

        for (var i = 0; i < tmp.length; ++i) {

                if ( tmp[i] === a_child ) {
                        return true
                } else {
                        if ( isChild ( a_child, tmp[i] ) ) {
                                return true
                        }
                }
        }
        return false
}

0 Add to favourites0 Bury

18 May 2016 7:30am GMT

13 May 2016

feedPlanet Maemo

Composition and aggregation to choose memory types in Qt

As we all know has Qt types like QPointer, QSharedPointer and we know about its object trees. So when do we use what?

Let's first go back to school, and remember the difference between composition and aggregation. Most of you probably remember drawings like this?

It thought us when to use composition, and when to use aggregation:

This model in the picture will for example tell us that a car's passenger must have ten fingers.

But what does this have to do with QPointer, QSharedPointer and Qt's object trees?

First situation is a shared composition. Both Owner1 and Owner2 can't survive without Shared (composition, filled up diamonds). For this situation you would typically use a QSharedPointer<Shared> at Owner1 and Owner2:

If there is no other owner, then it's probably better to just use Qt's object trees and setParent() instead. Note that for example QML's GC is not very well aware of QSharedPointer, but does seem to understand Qt's object trees.

Second situation are shared users. User1 and User2 can stay alive when Shared goes away (aggregation, empty diamonds). In this situation you typically use a QPointer<Shared> at User1 and at User2. You want to be aware when Shared goes away. QPointer<Shared>'s isNull() will become true after that happened.

Third situation is a mixed one. In this case you could at Owner use a QSharedPointer<Shared> or a parented raw QObject pointer (using setParent()), but a QPointer<Shared> at User. When Owner goes away and its destructor (due to the parenting) deletes Shared, User can check for it using the previously mentioned isNull check.

Finally if you have a typical object tree, then use QObject's infrastructure for this.

0 Add to favourites0 Bury

13 May 2016 4:53pm GMT

08 May 2016

feedPlanet Maemo

2016-05-03 Meeting Minutes

Meeting held 2016-05-03 on FreeNode, channel #maemo-meeting (logs)

Attending: eekkelund, reinob, pichlo, Win7Mac, juiceme, Oksana

Partial:

Absent: peterleinchen, HtheB

Summary of topics (ordered by discussion):


(Topic Handover meeting for the new Council):


Action Items:
  • old items:
    • The next GA meeting should be announced soon.
  • new items:
    • Add new Councillors to the mailing list and remove the old. (juiceme)
    • Arrange Council permissions on TMO to new Councillors. (juiceme)
    • Get cloaks for new Councillors. (juiceme)

0 Add to favourites0 Bury

08 May 2016 1:44pm GMT

02 May 2016

feedPlanet Maemo

Visitor for Klartext

Felt good about explaining my work last time. For no reason. I guess I'm happy, or I no longer feel PGO's pressure or something. Having to be politically correct all the times, sucks. Making technically and architecturally good solutions is what drives me.

Today I explained the visitor pattern. We want to parse Klartext in such a way that we can present its structure in a editing component. It's the same component for which I utilized a LRU last week. We want to visualize significant lines like tool changes, but also make cycles foldable like SciTe does with source code and a whole lot of other stuff that I can't tell you because of teh secretz. Meanwile these files are, especially when generated using cad-cam software, amazingly huge.

Today I had some success with explaining visitor using the Louvre as that what is "visitable" (the AST) and a Japanese guy who wants to collect state (photos) as a visitor of fine arts. Hoping my good-taste solutions (not my words, it's how Matthias Hasselmann describes my work at Nokia) will once again yield a certain amount of success.

ps. I made sure that all the politically correcting categories are added to this post. So if you'd have filtered away the condescending and controversial posts from my blog, you could have protected yourself from being in total shock now (because I used the sexually tinted word "sucks", earlier). Guess you didn't. Those categories have been in place on my blog's infrastructure since many years. They are like the Körperwelten (Bodyworlds) exhibitions; you don't have to visit them.

0 Add to favourites0 Bury

02 May 2016 8:06pm GMT

Introducing the OGRE fork on GitHub

in this post I want to introduce the OGRE fork on github. The goal of the fork is to provide a stable and reliable OGRE 1.x series while at the same time modernizing parts under the hood updates.

The idea behind this is that there are many existing 1.x codebases, actually a whole 1.x ecosystem, that can be modernized that way.
The last release of the 1.x series was over 2 years ago, so using the current 1.10 branch already gives a lot of improvements.

However the 1.10 branch contains some unnecessary changes that make it incompatible to the 1.9 release. These were reverted, so old code should compile again.

Additionally there are modernizing changes when compared with the upstream at bitbucket, which are discussed in more detail in the following.

This will be still a high-level overview though. If you need even more details, take a look on the git-ChangeLog.

Replace legacy renderers

With 1.9 the only stable renderers were GL(2.0) and DX9 which by today are outdated and produce different results than the GLES2 renderer that is used for Web and Mobile, which makes porting applications harder.

Therefore the goal is to get the GL3Plus renderer into a shape where it can act as a drop-in replacement for the old GL render and then drop the latter.

You can see the current status here - while it says that only 41/86 tests pass, most of the failed tests actually only differ in only 1 or 2 pixels which is pretty good when considering we are comparing fixed function vs. dynamically generated shaders (RTSS).

But the fork does not stop here; the GL renderers now also share the context creation code. This allows using the GLES2 renderer on the desktop or creating a GL3+ context using EGL - which is a prerequisite for headless rendering and for running on Wayland/MIR. Overall this makes your applications much more portable.

Improved regression testing

Changing the renderer requires being able to continuously monitor whether the rendering is still correct and to immediately detect regressions.

For this the Testing and VisualTesting frameworks were fixed and now correctly run on Linux and OSX. They run on each pull request to catch errors before the code even touches master.

Furthermore the tests can now be built without input handling (OIS) which hopefully will lead to wider adoption.

The unit tests still use cppunit which is a pain to use when compared to gtest, but changing that would require a large rewrite.

Batteries Included

The build now automatically handles the dependencies by downloading and building them as needed. On all platforms. So you do not have to care about mismatching compile settings any more.

Furthermore the SampleBrowser can now be build without input handling which eases quick testing. And in case you want input, now SDL2 is used instead of the esoteric and outdated OIS.

Next bringing your applications to the web is now easier; OGRE was able to run in the browser for a long time already - but the process was only badly documented. The fork tracks the Emscripten sample code inside the repository and also has documentation on how to use it.

Bindings for Scripting Languages

For using scripting languages, there is traditionally MOGRE (C#), Ogre4j, Python-Ogre which are by now basically all outdated and unmaintained.

To put all these efforts on a solid foundation I took some inspiration from ruby-ogre and added SWIG support.

At the time of writing there are already working Python bindings (see sample here) and a proof-of-concept for C#.

By using SWIG it is now easily to add further bindings for Ruby, Java, JavaScript (Node.js) as well. But as I care mostly for Python this will be an exercise for the reader.

A word on OGRE 2.1

You might wonder why one should care about the 1.x and not go for 2.1 directly. The obvious reason is that you have an existing codebase and OGRE 2.1 drastically changes API and even the material file format.

Then, while faster than 1.x, OGRE 2.x is still far from feature completeness with 1.x - it still completely lacks Web and Mobile support.
Faster also actually depends on your scene/ material usage; For instance, if you only render a single object (product showcase) 2.x will not offer you any advantages.

Finally the main functionality advance in 2.1, namely the new material system (HLMS) and with it physically based shading were backported to 1.x.

Yet if you only care about desktop and want to render large immersive worlds (lots of nodes, lots of materials), 2.1 is the way to go.

0 Add to favourites0 Bury

02 May 2016 2:38pm GMT

29 Apr 2016

feedPlanet Maemo

Putting an LRU in your code

For the ones who didn't find the LRU in Tracker's code (and for the ones who where lazy).

Let's say we will have instances of something called a Statement. Each of those instances is relatively big. But we can recreate them relatively cheap. We will have a huge amount of them. But at any time we only need a handful of them.

The ones that are most recently used are most likely to be needed again soon.

First you make a structure that will hold some administration of the LRU:

typedef struct {
        Statement *head;
        Statement *tail;
        unsigned int size;
        unsigned int max;
} StatementLru;

Then we make the user of a Statement (a view or a model). I'll be using a Map here. You can in Qt for example use QMap for this. Usually I want relatively fast access based on a key. You could also each time loop the stmt_lru to find the instance you want in useStatement based on something in Statement itself. That would rid yourself of the overhead of a map.

class StatementUser
{
        StatementUser();
        ~StatementUser();
        void useStatement(KeyType key);
private:
        StatementLru stmt_lru;
        Map<KeyType, Statement*> stmts;
        StatementFactory stmt_factory;
}

Then we will add to the private fields of the Statement class the members prev and next: We'll make a circular doubly linked list.

class Statement: QObject {
        Q_OBJECT
    ...
private:
        Statement *next;
        Statement *prev;
};

Next we initialize the LRU:

StatementUser::StatementUser() 
{
        stmt_lru.max = 500;
        stmt_lru.size = 0;            
}

Then we implement using the statements

void StatementUser::useStatement(KeyType key)
{
        Statement *stmt;

        if (!stmts.get (key, &stmt)) {

                stmt = stmt_factory.createStatement(key);

                stmts.insert (key, stmt);

                /* So the ring looks a bit like this: *
           *                                    *
           *    .--tail  .--head                *
           *    |        |                      *
           *  [p-n] -> [p-n] -> [p-n] -> [p-n]  *
           *    ^                          |    *
           *    `- [n-p] <- [n-p] <--------'    */

                if (stmt_lru.size >= stmt_lru.max) {
                   Statement *new_head;

          /* We reached max-size of the LRU stmt cache. Destroy current
           * least recently used (stmt_lru.head) and fix the ring. For
           * that we take out the current head, and close the ring.
           * Then we assign head->next as new head. */

                        new_head = stmt_lru.head->next;
                        auto to_del = stmts.find (stmt_lru.head);
                        stmts.remove (to_del);
                        delete stmt_lru.head;
                        stmt_lru.size--;
                        stmt_lru.head = new_head;
                } else {
                        if (stmt_lru.size == 0) {
                                stmt_lru.head = stmt;
                                stmt_lru.tail = stmt;
                        }
                }

        /* Set the current stmt (which is always new here) as the new tail
   * (new most recent used). We insert current stmt between head and
   * current tail, and we set tail to current stmt. */

                stmt_lru.size++;
                stmt->next = stmt_lru.head;
                stmt_lru.head->prev = stmt;

                stmt_lru.tail->next = stmt;
                stmt->prev = stmt_lru.tail;
                stmt_lru.tail = stmt;

        } else {
                if (stmt == stmt_lru.head) {

                /* Current stmt is least recently used, shift head and tail
           * of the ring to efficiently make it most recently used. */

                        stmt_lru.head = stmt_lru.head->next;
                        stmt_lru.tail = stmt_lru.tail->next;
                } else if (stmt != stmt_lru.tail) {

                /* Current statement isn't most recently used, make it most
           * recently used now (less efficient way than above). */

                /* Take stmt out of the list and close the ring */
                        stmt->prev->next = stmt->next;
                        stmt->next->prev = stmt->prev;

                /* Put stmt as tail (most recent used) */
                        stmt->next = stmt_lru.head;
                        stmt_lru.head->prev = stmt;
                        stmt->prev = stmt_lru.tail;
                        stmt_lru.tail->next = stmt;
                        stmt_lru.tail = stmt;
                }

        /* if (stmt == tail), it's already the most recently used in the
   * ring, so in this case we do nothing of course */
        }

        /* Use stmt */

        return;
}

In case StatementUser and Statement form a composition (StatementUser owns Statement, which is what makes most sense), don't forget to delete the instances in the destructor of StatementUser. In the example's case we used heap objects. You can loop the stmt_lru or the map here.

StatementUser::~StatementUser()
{
        Map<KeyType, Statement*>::iterator i;
        for (i = stmts.begin(); i != stmts.end(); ++i) {
                delete i.value();
        }
}

0 Add to favourites0 Bury

29 Apr 2016 9:30pm GMT

27 Apr 2016

feedPlanet Maemo

Secretly reusing my own LRU code

Last week, I secretly reused my own LRU code in the model of the editor of a CNC machine (has truly huge files, needs a statement editor). I rewrote my own code, of course. It's Qt based, not GLib. Wouldn't work in original form anyway. But the same principle. Don't tell Jürg who helped me write that, back then.

Extra points and free beer for people who can find it in Tracker's code.

0 Add to favourites0 Bury

27 Apr 2016 10:55am GMT

23 Mar 2016

feedPlanet Maemo

Q2 2016 Community Council Election Announcement

Dear friends and Maemoans, six months and even a bit more has passed and again it is time to elect a new Community Council for us.

The last election finished on Saturday 13th of September 2015.

The schedule of the voting is as follows:

To get our community to continue strong, we need fresh people with fresh viewpoints to carry on the torch, so please think about volunteering for Maemo Council.

On behalf of the outgoing community council,

Jussi Ohenoja

0 Add to favourites0 Bury

23 Mar 2016 6:17pm GMT

07 Jan 2016

feedPlanet Maemo

2015-12-01 Meeting Minutes

Meeting held 2015-12-01 on FreeNode, channel #maemo-meeting (logs)

Attending: Peter Leinchen (peterleinchen), Halit Bal (HtheB), Gido Griese (Win7Mac), Jussi Ohenoja (juiceme)

Partial: Bernardo Reino (reinob)

Absent:

Summary of topics (ordered by discussion):


(Topic Discussing handling of ex-members accounts):

(Topic Automatic calculation of election results):

(Topic Finalizing the rules for the coding competition):


Action Items:
  • old items:
    • Looking into automatic calculation of election results ...
    • Discuss spam handling with techstaff.
  • new items:
    • Finalize the coding competition rules.
    • Post to TMO announcing the competition and requesting for donations.
    • Discuss competition entries storing with OpenRepos people.

0 Add to favourites0 Bury

07 Jan 2016 11:18am GMT

29 Dec 2015

feedPlanet Maemo

Learning Modern 3D Graphics Programming

one of the best resources to learn modern OpenGL and the one which helped me quite a lot is the Book at www.arcsynthesis.org/gltut/ - or lets better say was. Unfortunately the domain expired so the content is no longer reachable.

Luckily the Book was designed as an open source project and the code to generate the website is still available at Bitbucket. Unfortunately this repository does not seem to be actively maintained any more.

Therefore I set out to make the Book to be available again using Github Pages. You can find the results here:

https://paroj.github.io/gltut/

However I did not simply mirror the pages, but also improved it at several places. So what has changed so far?

hopefully these changes will generate some momentum so this great Book gets extended again. As there were also non-cosmetical changes like the new Chapter I also tagged a 0.3.9 release.

I the process of the above work I found out that there is also a mirror of the original Book at http://alfonse.bitbucket.org/oldtut/. This one is however at the state of the 0.3.8 release, meaning it does not only misses the above changes but also some adjustment happened post 0.3.8 at bitbucket.

1 Add to favourites0 Bury

29 Dec 2015 10:26pm GMT

18 Dec 2015

feedPlanet Maemo

Coding Competition 2016

As you may have already noticed in the recent Meeting Minutes, current Council is about to prepare a Coding Competition (CC) for 2016. In times like these, without Nokia or any other corporate entity sponsoring prizes/devices, there are of course limited resources for big giveaways. Nevertheless we believe the approach is worth it. :) We are relying on your donations.

Looking at last CC's, this time it will probably be more of the supportive kind than rival in the first place. Basically we want to support the ongoing AND future software development in a way that ideally, the winners will get the "right" devices to carry on their efforts or even to the next level (e.g. cross-platform).

Now since no current Councilor has ever arranged a CC, we would like to hear your thoughts on how to get it "right", satisfying community demands (SW) as well as individual developer demands (HW). Please share your ideas! And if you've been involved in organizing recent CC's and are willing to help, please holler. And we sure need help in having an uptodate wiki page.

Furthermore, we are calling all owners of now unused devices to consider donating* them for this CC, respectively to MC e.V., who will be responsible for collecting donations and funds. Especially we are looking at those that got rewarded in recent CC's or device programs (N950 anyone? ;)).

* or selling them for a reasonable price.

The voting webinterface will be the same as for the Maemo Council elections, so jury will be you. We also need you to scrutinize the submitted code (ideally installable/executable + binary). It is undecided yet wether or not to accept closed source apps. As the Council is organizing this CC, it will only intervene if a need occurs. Please help make sure conditions and rules are distinct.

Ideas for categories:

- Old NIT's

- Fremantle

- Harmattan

- Sailfish OS

- "Port it/cross-platform"

- "Maintain it" (packages, CSSU)

- "Update/Fix it"

- "Bugfix it" (known, yet unresolved bugs)

- Core development or "Free it" (backport, clone, replace or supersede closed packages)

- Recent contributions (recent major releases)

- Wishlist (rewarded if/when completed)

- Beginner (first release)

A variant could be to have category "Update/Fix it" for each of the 4 device categories if it can be expected to have enough entries (and prizes).

Ideas and suggestions welcome. :)

1 Add to favourites0 Bury

18 Dec 2015 8:55pm GMT

01 Dec 2015

feedPlanet Maemo

2015-11-24 Meeting Minutes

Meeting held 2015-11-24 on FreeNode, channel #maemo-meeting (logs)

Attending: Peter Leinchen (peterleinchen), Bernardo Reino (reinob), Halit Bal (HtheB), Gido Griese (Win7Mac), Jussi Ohenoja (juiceme)

Partial: Rudiger Schiller (chem|st)

Absent:

Summary of topics (ordered by discussion):


(Topic Council roles reorganization):

(Topic Spam mail through Maemo accounts):

(Topic How to handle user requests to remove projects):

(Topic Code of Conduct):

(Topic Maemo Coding Competition 2015):

(Topic Elections handling and management):


Action Items:
  • old items:
    • The selected Code of Conduct (KDE) still needs to be published on TMO.
    • Looking into automatic calculation of election results ...
    • Arrange Council status on TMO for new Council members
  • new items:
    • Discuss spam handling with techstaff.
    • Finalize the coding competition rules.

1 Add to favourites0 Bury

01 Dec 2015 9:20am GMT

23 Nov 2015

feedPlanet Maemo

IOCTL Android

IOCTL is a very useful system call: it is simple and multiplexes the different commands to the appropriate kernel space function. In this post, I want to describe how you can implement a module with IOCTL support for Android. There are a lot of good articles about it (links below), and I just describe the differences regarding to the Android platform.

So, let's create a very simple misc device and let's play with it, doing some reads and writes. Initially, let's define a very simple kernel module (most of the code was taken from here).

#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/fs.h>
#include <linux/miscdevice.h>
#include <asm/uaccess.h>


#define MY_MACIG 'G'
#define READ_IOCTL _IOR(MY_MACIG, 0, int)
#define WRITE_IOCTL _IOW(MY_MACIG, 1, int)

static int used;
static char msg[200];

static ssize_t device_read(struct file *filp, char __user *buffer, size_t length, loff_t *offset)
{
return simple_read_from_buffer(buffer, length, offset, msg, 200);
}

static ssize_t device_write(struct file *filp, const char __user *buff, size_t len, loff_t *off)
{
if (len > 199)
return -EINVAL;
copy_from_user(msg, buff, len);
msg[len] = '\0';
return len;
}

static int device_open(struct inode *inode, struct file *file)
{
used++;
return 0;
}

static int device_release(struct inode *inode, struct file *file)
{
used--;
return 0;
}

char buf[200];
int device_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
{
int len = 200;
switch(cmd) {
case READ_IOCTL:
copy_to_user((char *)arg, buf, 200);
break;

case WRITE_IOCTL:
copy_from_user(buf, (char *)arg, len);
break;

default:
return -ENOTTY;
}
return len;
}

static struct file_operations fops = {
.owner = THIS_MODULE,
.read = device_read,
.write = device_write,
.open = device_open,
.release = device_release,
.unlocked_ioctl = device_ioctl
};

static struct miscdevice my_miscdev = {
.name = "my_device",
.mode = S_IRWXUGO,
.fops = &fops,
};

static int __init cdevexample_module_init(void)
{
int ret = misc_register(&my_miscdev);
if (ret < 0) {
printk ("Registering the character device failed\n");
return ret;
}
printk("create node with mknod /dev/my_device\n");
return 0;
}

static void __exit cdevexample_module_exit(void)
{
misc_deregister(&my_miscdev);
}

module_init(cdevexample_module_init);
module_exit(cdevexample_module_exit);
MODULE_LICENSE("GPL");


Compile it and insert into Android kernel.

shell@flounder:/ $ su
shell@flounder:/ $ insmod my_module.ko

Now, we can check the new device in /dev:

shell@flounder:/ $ su
shell@flounder:/ $ ls -l /dev
...
crw-rw---- root mtp 10, 23 2015-11-23 18:04 mtp_usb
crw------- root root 10, 0 2015-11-23 18:11 my_device
crw------- root root 10, 36 2015-11-23 18:04 network_latency
crw------- root root 10, 35 2015-11-23 18:04 network_throughput
crw-rw-rw- root root 1, 3 2015-11-23 18:04 null

...

See that the permissions are limited. Don't forget to set it to:

shell@flounder:/ $ chmod 666 /dev/my_device
shell@flounder:/ $ ls -l /dev
...
crw-rw---- root mtp 10, 23 2015-11-23 18:04 mtp_usb
crw-rw-rw- root root 10, 0 2015-11-23 18:11 my_device
crw------- root root 10, 36 2015-11-23 18:04 network_latency
crw------- root root 10, 35 2015-11-23 18:04 network_throughput
crw-rw-rw- root root 1, 3 2015-11-23 18:04 null

...

Now, let's try to do some operations with our device driver:

shell@flounder:/ $ echo "Hello world" > /dev/my_device
shell@flounder:/ $ cat /dev/my_device

You will see the following error on the logcat:

avc: denied { read write } for name="my_device" dev="tmpfs" scontext=u:r:system_app:s0 tcontext

This means that SELinux (yes, Android makes heavy usage of it) also controls the access to device drivers and you cannot read/write from/to your new drive. You have two options: i) disable SELinux in Android (you need to change some kernel options and rebuild it) or ii) add some new rules into SELinux. Let's do the last to learn a bit more :-)

So, we change the following files and give access (read, write, getattr, ioctl, open and create) to our new device /dev/my_device. If you need to restrict the access, you can adapt the policies according to your needs. For more information about SELinux and Android, take a look in this doc (specially the section "Implementation").

external/sepolicy/device.te
type fscklogs, dev_type;
type full_device, dev_type;
type my_device, dev_type;


external/sepolicy/file_contexts
/dev/rproc_user u:object_r:rpmsg_device:s0
/dev/my_device u:object_r:my_device:s0
/dev/snd(/.*)? u:object_r:audio_device:s0


external/sepolicy/app.te
allow appdomain usb_device:chr_file { read write getattr ioctl };
allow appdomain usbaccessory_device:chr_file { read write getattr };
allow appdomain my_device:chr_file { read write getattr ioctl open create };


Now, let's build the Android framework again and flash the device. Everything should work fine.

shell@flounder:/ $ echo "Hello world" > /dev/my_device
shell@flounder:/ $ cat /dev/my_device
Hello world

That's it!! You can also check the following links


0 Add to favourites0 Bury

23 Nov 2015 6:33pm GMT

17 Nov 2015

feedPlanet Maemo

2015-11-03 Meeting Minutes

Meeting held 2015-11-03 on FreeNode, channel #maemo-meeting (logs)

Attending: Bernardo Reino (reinob), Jussi Ohenoja (juiceme), Peter Leinchen (peterleinchen)

Partial: Rudiger Schiller (chem|st)

Absent: Halit Bal (HtheB), Gido Griese (Win7Mac)

Summary of topics (ordered by discussion):


(Topic TMO forum swearword filter):

(Topic Maemo Coding Competition 2015):


Action Items:
  • old items:
    • The selected Code of Conduct (KDE) still needs to be published on TMO.
    • Looking into automatic calculation of election results ...
  • new items:
    • Arrange Council status on TMO for new Council members

1 Add to favourites0 Bury

17 Nov 2015 6:05pm GMT