05 Jan 2017

feedPlanet Maemo

Process API for NoFlo components

It has been a while that I've written about flow-based programming - but now that I'm putting most of my time to Flowhub things are moving really quickly.

One example is the new component API in NoFlo that has been emerging over the last year or so.

Most of the work described here was done by Vladimir Sibirov from The Grid team.

Introducing the Process API

NoFlo programs consist of graphs where different nodes are connected together. These nodes can themselves be graphs, or they can be components written in JavaScript.

A NoFlo component is simply a JavaScript module that provides a certain interface that allows NoFlo to run it. In the early days there was little convention on how to write components, but over time some conventions emerged, and with them helpers to build well-behaved components more easily.

Now with the upcoming NoFlo 0.8 release we've taken the best ideas from those helpers and rolled them back into the noflo.Component base class.

So, how does a component written using the Process API look like?

// Load the NoFlo interface
var noflo = require('noflo');
// Also load any other dependencies you have
var fs = require('fs');

// Implement the getComponent function that NoFlo's component loader
// uses to instantiate components to the program
exports.getComponent = function () {
  // Start by instantiating a component
  var c = new noflo.Component();

  // Provide some metadata, including icon for visual editors
  c.description = 'Reads a file from the filesystem';
  c.icon = 'file';

  // Declare the ports you want your component to have, including
  // their data types
  c.inPorts.add('in', {
    datatype: 'string'
  });
  c.outPorts.add('out', {
    datatype: 'string'
  });
  c.outPorts.add('error', {
    datatype: 'object'
  });

  // Implement the processing function that gets called when the
  // inport buffers have packets available
  c.process(function (input, output) {
    // Precondition: check that the "in" port has a data packet.
    // Not necessary for single-inport components but added here
    // for the sake of demonstration
    if (!input.hasData('in')) {
      return;
    }

    // Since the preconditions matched, we can read from the inport
    // buffer and start processing
    var filePath = input.getData('in');
    fs.readFile(filePath, 'utf-8', (err, contents) {
      // In case of errors we can just pass the error to the "error"
      // outport
      if (err) {
        output.done(err);
        return;
      }

      // Send the file contents to the "out" port
      output.send({
        out: contents
      });
      // Tell NoFlo we've finished processing
      output.done();
    });
  });

  // Finally return to component to the loader
  return c;
}

Most of this is still the same component API we've had for quite a while: instantiation, component metadata, port declarations. What is new is the process function and that is what we'll focus on.

When is process called?

NoFlo components call their processing function whenever they've received packets to any of their regular inports.

In general any new information packets received by the component cause the process function to trigger. However, there are some exceptions:

Handling preconditions

When the processing function is called, the first job is to determine if the component has received enough data to act. These "firing rules" can be used for checking things like:

The NoFlo component input handler provides methods for checking the contents of the input buffer. Each of these return a boolean if the conditions are matched:

For convenience, has and hasData can be used to check multiple ports at the same time. For example:

// Fail precondition check unless both inports have a data packet
if (!input.hasData('in1', 'in2')) return;

For more complex checking it is also possible to pass a validation function to the has method. This function will get called for each information packet in the port(s) buffer:

// We want to process only when color is green
var validator = function (packet) {
  if (packet.data.color === 'green') {
    return true;
  }
  return false;
}
// Run all packets in in1 and in2 through the validator to
// check that our firing conditions are met
if (!input.has('in1', 'in2', validator)) return;

The firing rules should be checked in the beginning of the processing function before we start actually reading packets from the buffer. At that stage you can simply finish the run with a return.

Processing packets

Once your preconditions have been met, it is time to read packets from the buffers and start doing work with them.

For reading packets there are equivalent get functions to the has functions used above:

For get and getStream you receive whole IP objects. For convenience, getData returns just the data payload of the data packet.

When you have read the packets you want to work with, the next step is to do whatever your component is supposed to do. Do some simple data processing, call some remote API function, or whatever. NoFlo doesn't really care whether this is done synchronously or asynchronously.

Note: once you read packets from an inport, the component activates. After this it is necessary to finish the process by calling output.done() when you're done.

Sending packets

While the component is active, it can send packets to any number of outports using the output.send method. This method accepts a map of port names and information packets.

output.send({
  out1: new noflo.IP('data', "some data"),
  out2: new noflo.IP('data', [1, 2, 3])
});

For data packets you can also just send the data as-is, and NoFlo will wrap it to an information packet.

Once you've finished processing, simply call output.done() to deactivate the component. There is also a convenience method that is a combination of send and done. This is useful for simple components:

c.process(function (input, output) {
  var data = input.getData('in');
  // We just add one to the number we received and send it out
  output.sendDone({
    out: data + 1
  });
});

In normal situations there packets are transmitted immediately. However, when working on individual packets that are part of a stream, NoFlo components keep an output buffer to ensure that packets from the stream are transmitted in original order.

Component lifecycle

In addition to making input processing easier, the other big aspect of the Process API is to help formalize NoFlo's component and program lifecycle.

NoFlo program lifecycle

The component lifecycle is quite similar to the program lifecycle shown above. There are three states:

Once all components in a NoFlo network have deactivated, the whole program is finished.

Components are only allowed to do work and send packets when they're activated. They shouldn't do any work before receiving input packets, and should not send anything after deactivating.

Generator components

Regular NoFlo components only send data associated with input packets they've received. One exception is generators, a class of components that can send packets whenever something happens.

Some examples of generators include:

The same rules of "only send when activated" apply also to generators. However, they can utilize the processing context to self-activate as needed:

exports.getComponent = function () {
 var c = new noflo.Component();
 c.inPorts.add('start', { datatype: 'bang' });
 c.inPorts.add('stop', { datatype: 'bang' });
 c.outPorts.add('out', { datatype: 'bang' });
 // Generators generally want to send data immediately and
 // not buffer
 c.autoOrdering = false;

 // Helper function for clearing a running timer loop
 var cleanup = function () {
   // Clear the timer
   clearInterval(c.timer.interval);
   // Then deactivate the long-running context
   c.timer.deactivate();
   c.timer = null;
 }

 // Receive the context together with input and output
 c.process(function (input, output, context) {
   if (input.hasData('start')) {
     // We've received a packet to the "start" port
     // Stop the previous interval and deactivate it, if any
     if (c.timer) {
       cleanup();
     }
     // Activate the context by reading the packet
     input.getData('start');
     // Set the activated context to component so it can
     // be deactivated from the outside
     c.timer = context
     // Start generating packets
     c.timer.interval = setInterval(function () {
       // Send a packet
       output.send({
         out: true
       });
     }, 100);
     // Since we keep the generator running we don't
     // call done here
   }

   if (input.hasData('stop')) {
     // We've received a packet to the "stop" port
     input.getData('stop');
     if (!c.timer) {
       // No timers running, we can just finish here
       output.done();
       return;
     }
     // Stop the interval and deactivate
     cleanup();
     // Also call done for this one
     output.done();
   }
 });

 // We also may need to clear the timer at network shutdown
 c.shutdown = function () {
   if (c.timer) {
     // Stop the interval and deactivate
     cleanup();
   }
   c.emit('end');
   c.started = false;
 }
}

Time to prepare

NoFlo 0.7 included a preview version of the Process API. However, last week during the 33C3 conference we finished some tricky bits related to process lifecycle and automatic bracket forwarding that make it more useful for real-life NoFlo applications.

These improvements will land in NoFlo 0.8, due out soon.

So, if you're maintaining a NoFlo application, now is a good time to give the git version a spin and look at porting your components to the new API. Make sure to report any issues you encounter!

We're currently migrating all the hundred-plus NoFlo open source modules to latest build and testing process so that they can be easily updated to the new APIs when they land.

0 Add to favourites0 Bury

05 Jan 2017 12:00am GMT

22 Dec 2016

feedPlanet Maemo

OGRECave 1.10 release

The 1.10.0 release of the OGRECave fork was just created. This means that the code is considered stable enough for general usage and the current interfaces will be supported in subsequent patch releases (i.e. 1.10.1, 1.10.2 …).

SampleBrowser running GLES2 on desktop

This release represents more than 3 years of work from various contributors when compared to the previous 1.9 release. At the time of writing it contains all commits from the bitbucket version as well as many fork specific features and fixes.

If you are reading about the fork for the first time and wonder why it was created, see this blog post. For a comparison between the github and bitbucket version see this log.

For a general overview of the 1.10 features when compared to 1.9, see the OGRECave 1.10 release notes.

The highlights probably are:

For further information see the github page of the fork.

0 Add to favourites0 Bury

22 Dec 2016 1:26am GMT

28 Nov 2016

feedPlanet Maemo

Nokia and their standard batteries

Nokia. A company everyone knows and most of us probably even used one of their phones in past. They were better or worse but one thing was good - most of them shared batteries…

My daughter (8.5y old) uses Nokia E50 as her daily phone. Sim card is covered by duct tape to not fall out when phone hit a floor (previous one went missing in such situation). Mira records how she and her friends sing, does some photo sessions to her dolls etc.

But during weekend phone stopped charging. Hm… Is it charger? Nope, it was original Nokia one. Tried some crappy Chinese one with same result. So let's check the battery.

Opened drawer, took Nokia 101. Inside was BL-5CB battery. Inserted into E50 got phone back online. But I like my 101 and keep it as a spare just in case.

Digged in a drawer with old devices. The one where I keep Sharp Zaurus c760, Sony Ericsson k750i, Openmoko FIC-GTA01bv3 and few other pieces of junk with some sentimental value. What I found there was Nokia 6230i which I got from Ross Burton during GUADEC 2007. Last time I used it about 5 years ago. But it had original Nokia BL-5C inside!

So I put that battery inside of E50, plugged charger and guess what… It started charging and phone booted! With over 11 years old battery!

During next few days I will buy BL-5C clone somewhere (they are 3-8€ now) and put it in my daughter's phone.

Related posts:

  1. Nokia N810 and fscked charging subsystem
  2. 3rdparty chargers for Neo1973
  3. Things to check with Nokia N900

0 Add to favourites0 Bury

28 Nov 2016 9:57pm GMT

28 Oct 2016

feedPlanet Maemo

The 1st Maemo Developer Regatta - The starting line

The Maemo Community e.V. proudly presents

The 1st Maemo Developer Regatta - The starting line

sponsored by Jolla


The wait is over!

The first Maemo Coding Competition for all Maemo derived systems, including but not limited to Maemo and SailfishOS.

This year's competition has the following three categories: Something new, Fixing/Updating and Beginner. Whether you are an experienced developer, porter, hacker or just a beginner on your very first hacks/codes, we have a category that suits you and your coding skills! If you are new to programming or want to learn, here's the excuse to finally make something. Anything. Entries for Maemo & Mer-based devices are eligible.

The timetable for the competition is the following: The competitions begins on Tuesday, 1st of November, and will be running for 3 months until Tuesday 31st of January 2016. You can enroll into the competition at any time, however, all the entries for the competition must be submitted no later than by 23:59 on Tuesday, 31st of January(UTC time). Testing/reviewing will take two weeks, from 01.02.2017 to 15.02.2017 and voting will run from 16th of February until 28th of February 2017, 23:59 UTC.

Follow the links to the wiki below for more information.

Devices & Platforms
Categories
How to Submit an Entry
Prizes
Participants
Rules
Voting
Developer Resources

If you would like to have an App, a Port, a Fix, a Hack or etc. but you can't code it yourself, don't worry, just write your ideas in here! Many developers would be happy to get a new idea input.

We are relying on your donations towards the Community Prize Fund.

The entry is free of charge.

Thanks for your interest in taking part, we can't wait to see what you will develop! :)

You still have questions? Contact us!

Regards

Maemo Community e.V.

0 Add to favourites0 Bury

28 Oct 2016 11:41am GMT

11 Sep 2016

feedPlanet Maemo

Maemo Community e.V. - new Invitation to the General Assembly 2016

Maemo Community e.V. - Einladung zur Mitgliederversammlung 2016

Sehr geehrtes Mitglied,

aufgrund zu geringer Teilnehmerzahl und daraus resultierender Beschlussunfähigkeit, musste die letzte ordentliche Mitgliederversammlung am 04.09.2016 vorzeitig beendet werden.
Die Mitgliederversammlung findet nun am 09.10.2016 14:00 UTC im IRC Freenode Channel #maemo-meeting statt.
*Bitte sorgt dafür, dass ihr diesmal anwesend seid.*

Auf der Tagesordnung stehen folgende Themen:
1. Begrüßung durch die Vorsitzende des Vorstands
2. Feststellung der ordnungsgemäßen Einberufung und der Beschlussfähigkeit der Mitgliederversammlung
3. Entgegennahme des Jahresberichts für das abgelaufene Geschäftsjahr und Entlastung des Vorstands
4. Neuwahl des Vorstandes
5. Änderung der Satzung §7 und §8, und der Vereinsordnung §1.1, §3.2 und §4.1. Erläuterung im Anhang [1].
6. Verschiedenes

Anträge auf Ergänzungen der Tagesordnung müssen bis eine Woche vor der Versammlung schriftlich beim Vorstand eingereicht werden (§ 9 Abs. 2 der Satzung).

Mit freundlichen Grüßen
Win7Mac/Gido Griese


########################## English ##########################

Maemo Community e.V. - Invitation to the General Assembly 2016

Dear Member,

due to too low number of participants and the resulting lack of a quorum, our last regular General Assembly on 2016-09-04 had to be ceased prematurely.
The General Meeting will now be held on Sunday, 2016-10-09 at 14:00 UTC on IRC Freenode channel #maemo-meeting.
*Please make sure to join the meeting this time.*

Unless any further issues are raised, the agenda includes the following topics:
1. Welcome by the Chairman of the Board
2. Determination of the proper convocation and the quorum of the General Assembly
3. Acceptance of the annual report for the fiscal year and actions of the Executive
4. Election of the Board Directors
5. Amendment of Articles of Association §7 and §8, Association Rules §1.1, §3.2 and §4.1 - please see appendix for details [1].
6. Any other business

Requests for additions to the agenda must be submitted to the Board in writing one week prior to the meeting (§ 9.2 of the Statutes).

Yours sincerely
Win7Mac/Gido Griese


[1] http://maemo.org/community/council/referendum_2015

Anhang / APPENDIX

** Änderungen der Satzung / Amendments to the Articles of Association **

§7 Vorstand (Board of directors)
(5) Der Vorstand führt die Beschlüsse der aktiven Mitgliederversammlung und der des Vorstandsrats aus, sofern dem Vorstandsrat diese Entscheidungshoheit laut Satzung oder Vereinsordnung eingeräumt wird.

(5) The Board of Directors carries out the resolutions of the General Assembly and of the Council in so far as this decision's jurisdiction is delegated to Council according to the statutes.

§8 Vorstandsrat (Council)
(4) Aufgaben des Vorstandsrats sowie Regelungen zu dessen Arbeit, inklusive der Einberufung von Sitzungen, deren Ablauf und die Durchführung von Abstimmungen, sind in der Vereinsordnung festgelegt.

(4) The council's duties, rules to announce meetings, their proceedings and executions of votes are regulated by the Association Rules.


** Änderungen der Vereinsordnung / Amendments to the Association Rules **

§ 1.1
(2) § 4.1 (Vorstandsratswahlen) dieser Vereinsordnung kann durch ein Referendum der passiven Mitgliederversammlung geändert oder aufgehoben werden. Änderungen dieser Wahlordnung durch die aktive Mitgliederversammlung müssen durch Beschluss der passiven Mitgliederversammlung bestätigt werden (Referendum). Sollte die Änderung von der passiven Mitgliederversammlung abgelehnt werden, kann der Änderungsvorschlag überarbeitet und erneut zur Abstimmung gebracht werden.

(2) § 4.1 (Council/election rules) of these association rules can be changed or dismissed with a referendum by the Passive Members' Meeting. Changes to these election rules by the General Assembly must be confirmed by a resolution of the passive members meeting (referendum). Should the amendment be rejected by the passive members meeting, the proposed amendment can be revised and brought again to the vote.


§3.2 Vorstandsrat /(council)

(7) Aufgaben und Zuständigkeiten des Vorstandsrats sind wie folgt:
a) Vertretung der assoziierten Mitglieder und ihrer Interessen
b) Beratung des Vorstands in allgemeinen Belangen
c) Der Vorstandsrat ist für die Durchführung aller Wahlen zuständig

(7) Tasks and competences of the Council are as follows:
a) Representation of the associated members (garage account) and their interests.
b) Counseling of the Board of directors in general matters
c) Council is responsible to execute all elections


§ 4.1 Vorstandsratswahlen / Council Elections

(6) Kandidaten mit einem professionellen bzw. witschaftlichen Interesse in Maemo, z.B. Mitarbeiter von Firmen, die in Bezug zu Maemo Hard- oder Software stehen, müssen ihr Interesse bei der Nominierung erklären. Bleibt dies aus, kann der Vorstandsrat die Kandidatur des Mitglieds für ungültig erklären und somit ausschliessen.

(6) Nominees with a professional interest in Maemo, such as working for a company involved in Maemo-related software development - must declare their interest when advertising their nomination. Failure to do so may result in the Board of Directors, or the outgoing Council, declaring their nomination invalid and so bar them from standing in the current election.

(9) Das Datum der nächsten Wahl soll mindestens 4 Wochen vorher auf maemo.org bekannt gegeben werden.

(9) The election date must be publicised at least 4 weeks in advance of the election.

(11) Sollten nach Ende der Nominierungsphase weniger als 4 Kandidaten zur Wahl stehen, kann die Wahl nicht abgehalten werden.
a) Die Nominierungsphase wird dann um 4 Wochen verlängert und die Wahl entsprechend verschoben.
b) Sollten nach der verlängerten Nominierungsphase weniger als 4 Kandidaten zur Wahl stehen, bestimmt der scheidende Vorstandsrat zusammen mit dem Vorstand das weitere Vorgehen. Eine Lösung, die zu weiteren Kandidaten und einer ordentlichen Wahl führt, soll dabei bevorzugt werden, andere Möglichkeiten werden nicht ausgeschlossen.

(11) If there are less than 4 candidates when the nominations close, the election can't be held.
a) The nomination period will be extended by 4 weeks and the election postponed similarly.
b) If, after 4 weeks's delay, there are still fewer than 4 candidates; the outgoing council will decide - in conjunction with Board of Directors - about how to proceed. The preferred solution is to encourage more members of the maemo.org community to participate and re-run the election, but all options are open.

(12) Änderungen der Vorstandsratswahlen müssen durch Beschluss durch die aktive Mitgliederversammlung oder die passive Mitgliederversammlung bestätigt werden.
a) Die Teilnahme an Referenda unterliegt den gleichen Bestimmungen, die für Vorstandsratswahlen gelten.
b) Bevor ein Referendum zur Abstimmung gelangen kann, muss ein entsprechender Entwurf für mindestens 4 Wochen zur Diskussion stehen.
c) Der Zeitraum, in welchem über das Referendum abgestimmt werden kann, soll genauso lang sein, wie bei regulären Vorstandsratswahlen.

(12)Changes to any of the above rules must be approved by a resolution of the Passive Members' Meeting (community referendum) or General Assembly.
a) Voting in such, will be open to anyone eligible to vote in the respective meeting.
b) The referendum options must be debated for a minimum of 4 weeks prior to the referendum.
c) Referendum voting will be open for the same length of time as the council elections.

1 Add to favourites0 Bury

11 Sep 2016 5:35pm GMT

30 Aug 2016

feedPlanet Maemo

2016-08-09 Meeting Minutes

Meeting held 2016-08-09 on FreeNode, channel #maemo-meeting (logs)

Attending: Win7Mac, eekkelund, juiceme, chem|st

Partial:

Absent: pichlo, reinob

Summary of topics (ordered by discussion):


(Topic Coding Competition):


Action Items:
  • old items:
    • Coding competition planning (eekkelund)
    • Set up ftp/sftp site for CC (reinob)
    • Create twitter account for maemo community(eekkelund)
  • new items:


Solved Action Items:
  • The next GA meeting should be announced soon.

0 Add to favourites0 Bury

30 Aug 2016 8:41am GMT

2016-08-02 Meeting Minutes

Meeting held 2016-08-02 on FreeNode, channel #maemo-meeting (logs)

Attending: pichlo, Win7Mac, eekkelund, reinob, juiceme

Partial:

Absent:

Summary of topics (ordered by discussion):


(Topic warfare replaced all SSL certificates):

(Topic MC e.V.):


Action Items:
  • old items:
    • Coding competition planning (eekkelund)
    • Set up ftp/sftp site for CC (reinob)
    • Create twitter account for maemo community(eekkelund)
  • new items:


Solved Action Items:
  • The next GA meeting should be announced soon.

0 Add to favourites0 Bury

30 Aug 2016 8:39am GMT

18 Aug 2016

feedPlanet Maemo

2016-07-26 Meeting Minutes

Meeting held 2016-07-26 on FreeNode, channel #maemo-meeting (logs)

Attending: pichlo, Win7Mac, eekkelund, reinob, juiceme

Partial: chem|st

Absent:

Summary of topics (ordered by discussion):


(Topic Approve pending project request on https://garage.maemo.org/admin/approve-pending.php):

(Topic Coding Competition):

(Topic Twitter account):

(Topic MC e.V.):


Action Items:
  • old items:
    • Coding competition planning (eekkelund)
    • The next GA meeting should be announced soon.
  • new items:
    • Set up ftp/sftp site for CC (reinob)
    • Create twitter account for maemo community(eekkelund)


Solved Action Items:

Find out if https is doable 0 Add to favourites0 Bury

18 Aug 2016 6:39am GMT

2016-07-19 Meeting Minutes

Meeting held 2016-07-19 on FreeNode, channel #maemo-meeting (logs)

Attending: Win7Mac, eekkelund, reinob

Partial:

Absent: pichlo, juiceme

Summary of topics (ordered by discussion):


(Topic Coding Competition):


Action Items:
  • old items:
    • Coding competition planning (eekkelund)
    • The next GA meeting should be announced soon.
  • new items:


Solved Action Items:

Find out if https is doable 0 Add to favourites0 Bury

18 Aug 2016 6:38am GMT

2016-07-12 Meeting Minutes

Meeting held 2016-07-12 on FreeNode, channel #maemo-meeting (logs)

Attending: eekkelund, pichlo, reinob

Partial:

Absent: Win7Mac, juiceme

Summary of topics (ordered by discussion):


(Topic Coding Competition):


Action Items:
  • old items:
    • Coding competition planning (eekkelund)
    • The next GA meeting should be announced soon.
  • new items:


Solved Action Items:

Find out if https is doable 0 Add to favourites0 Bury

18 Aug 2016 6:36am GMT

2016-07-05 Meeting Minutes

Meeting held 2016-07-05 on FreeNode, channel #maemo-meeting (logs)

Attending: juiceme, eekkelund, pichlo, Win7Mac, reinob

Partial:

Absent:

Summary of topics (ordered by discussion):


(Topic read-only option for TOR endpoints):

(Topic Coding Competition):


Action Items:
  • old items:
    • The next GA meeting should be announced soon.
  • new items:
    • Coding competition planning (eekkelund)


Solved Action Items:

Find out if https is doable 0 Add to favourites0 Bury

18 Aug 2016 6:35am GMT

05 Aug 2016

feedPlanet Maemo

On OGRE versions

Currently one can choose between the following OGRE versions
1.9, 1.10, 2.0 and 2.1

However the versioning scheme has become completely arbitrary while still resembling semantic versioning.
As a consequence somebody even had to put a "What version to choose?" guide on the OGRE homepage.

Unfortunately the guide confuses more than it helps:

You might wonder why 2.1 while having the latest features, lacks some bugfixes of older releases. Read on..

Lack of branching concept

Usually projects have a default master or trunk branch where all development eventually ends up and optionally some maintenance and bleeding edge branches. See this for a small overview of the according workflows.

Now OGRE has the following branches: v1-9, v1-10, v2-0, v2-1 and default. By common sense default would contain the most recent code while the other branches would be maintenance branches of the respective releases. But not today! Actually all of these branches are under active development with 2.1 containing the latest features and default representing the current stable version. (actually default and 1.10 are pretty much the same)

As there is no clear code flow between the branches this means that branches are diverging. Fixes for issues in one branch do not distribute across the others. Examples:

As you can see we got all possible directions here. The situation can not be easily resolved by successive merging branches into each other.

With the github fork, I therefore manually compared the different branches and cherry-picked all fixed into master (v1-10).

This is why I initially called it the stable fork. Even though it contains new features as discussed in my last post, the API is backwards compatible to existing code/ add-ons and it aggregates all bugfixes in one branch.

0 Add to favourites0 Bury

05 Aug 2016 12:24pm GMT

30 Jul 2016

feedPlanet Maemo

Maemo Community e.V. - Invitation to the General Assembly 2016

Maemo Community e.V. - Einladung zur Mitgliederversammlung 2016

Sehr geehrtes Mitglied, unsere diesjährige ordentliche Mitgliederversammlung findet am 04.09.2016 16:00 UTC im IRC Freenode Channel #maemo-meeting statt.

Auf der Tagesordnung stehen folgende Themen:
1. Begrüßung durch die Vorsitzende des Vorstands
2. Feststellung der ordnungsgemäßen Einberufung und der Beschlussfähigkeit der Mitgliederversammlung
3. Entgegennahme des Jahresberichts für das abgelaufene Geschäftsjahr und Entlastung des Vorstands
4. Neuwahl des Vorstandes
5. Änderung der Satzung §7 und §8, und der Vereinsordnung §1.1, §3.2 und §4.1. Erläuterung im Anhang.
6. Verschiedenes

Anträge auf Ergänzungen der Tagesordnung müssen bis eine Woche vor der Versammlung schriftlich beim Vorstand eingereicht werden (§ 9 Abs. 2 der Satzung).

Mit freundlichen Grüßen
Win7Mac/Gido Griese

############################ English##########################

Maemo Community e.V. - Invitation to the General Assembly 2016

Dear Member, our Annual General Meeting will be held on Sunday, 2016-09-04 at 16:00 UTC on IRC Freenode channel #maemo-meeting.

Unless any further issues are raised, the agenda includes the following topics:
1. Welcome by the Chairman of the Board
2. Determination of the proper convocation and the quorum of the General Assembly
3. Acceptance of the annual report for the fiscal year and actions of the Executive
4. Election of the Board Directors
5. Amendment of Articles of Association §7 and §8, Association Rules §1.1, §3.2 and $4.1 - please see appendix for details [1].
6. Any other business

Requests for additions to the agenda must be submitted to the Board in writing one week prior to the meeting (§ 9 para. 2 of the Statutes).

Yours sincerely
Win7Mac/Gido Griese


[1]http://maemo.org/community/council/referendum_2015/


Anhang / APPENDIX

* Änderungen der Satzung/Amendments to the Articles of Association:
§7 Vorstand (Board of directors)
(5) Der Vorstand führt die Beschlüsse der aktiven Mitgliederversammlung und der des Vorstandsrats aus, sofern dem Vorstandsrat diese Entscheidungshoheit laut Satzung oder Vereinsordnung eingeräumt wird.
(5) The Board of Directors carries out the resolutions of the General Assembly and of the Council in so far as this decision's jurisdiction is delegated to Council according to the statutes.

§8 Vorstandsrat (Council)
(4) Aufgaben des Vorstandsrats sowie Regelungen zu dessen Arbeit, inklusive der Einberufung von Sitzungen, deren Ablauf und die Durchführung von Abstimmungen, sind in der Vereinsordnung festgelegt.
(4) The council's duties, rules to announce meetings, their proceedings and executions of votes are regulated by the Association Rules.

* Änderungen der Vereinsordnung/Amendments to the Association Rules
§ 1.1
(2) § 4.1 (Vorstandsratswahlen) dieser Vereinsordnung kann durch ein Referendum der passiven Mitgliederversammlung geändert oder aufgehoben werden. Änderungen dieser Wahlordnung durch die aktive Mitgliederversammlung müssen durch Beschluss der passiven Mitgliederversammlung bestätigt werden (Referendum). Sollte die Änderung von der passiven Mitgliederversammlung abgelehnt werden, kann der Änderungsvorschlag überarbeitet und erneut zur Abstimmung gebracht werden.

(2) § 4.1 (Council election rules) of these association rules can be changed or dismissed with a referendum by the Passive Members' Meeting. Changes to these election rules by the General Assembly must be confirmed by a resolution of the passive members meeting (referendum). Should the amendment be rejected by the passive members meeting, the proposed amendment can be revised and brought again to the vote.

§3.2 Vorstandsrat (council)

(7) Aufgaben und Zuständigkeiten des Vorstandsrats sind wie folgt:
a) Vertretung der assoziierten Mitglieder und ihrer Interessen
b) Beratung des Vorstands in allgemeinen Belangen
c) Der Vorstandsrat ist für die Durchführung aller Wahlen zuständig

(7) Tasks and competences of the Council are as follows:
a) Representation of the associated members (garage account) and their interests.
b) Counseling of the Board of directors in general matters
c) Council is responsible to execute all elections

§ 4.1 Vorstandsratswahlen (Council Elections)

(6) Kandidaten mit einem professionellen bzw. witschaftlichen Interesse in Maemo, z.B. Mitarbeiter von Firmen, die in Bezug zu Maemo Hard- oder Software stehen, müssen ihr Interesse bei der Nominierung erklären. Bleibt dies aus, kann der Vorstandsrat die Kandidatur des Mitglieds für ungültig erklären und somit ausschliessen.

(6) Nominees with a professional interest in Maemo, such as working for a company involved in Maemo-related software development - must declare their interest when advertising their nomination. Failure to do so may result in the Board of Directors, or the outgoing Council, declaring their nomination invalid and so bar them from standing in the current election.

(9) Das Datum der nächsten Wahl soll mindestens 4 Wochen vorher auf maemo.org bekannt gegeben werden.

(9) The election date must be publicised at least 4 weeks in advance of the election.

(11) Sollten nach Ende der Nominierungsphase weniger als 4 Kandidaten zur Wahl stehen, kann die Wahl nicht abgehalten werden.
a) Die Nominierungsphase wird dann um 4 Wochen verlängert und die Wahl entsprechend verschoben.
b) Sollten nach der verlängerten Nominierungsphase weniger als 4 Kandidaten zur Wahl stehen, bestimmt der scheidende Vorstandsrat zusammen mit dem Vorstand das weitere Vorgehen. Eine Lösung, die zu weiteren Kandidaten und einer ordentlichen Wahl führt, soll dabei bevorzugt werden, andere Möglichkeiten werden nicht ausgeschlossen.

(11) If there are less than 4 candidates when the nominations close, the election can't be held.
a) The nomination period will be extended by 4 weeks and the election postponed similarly.
b) If, after 4 weeks's delay, there are still fewer than 4 candidates; the outgoing council will decide - in conjunction with Board of Directors - about how to proceed. The preferred solution is to encourage more members of the maemo.org community to participate and re-run the election, but all options are open.

(12) Änderungen der Vorstandsratswahlen müssen durch Beschluss durch die aktive Mitgliederversammlung oder die passive Mitgliederversammlung bestätigt werden.
a) Die Teilnahme an Referenda unterliegt den gleichen Bestimmungen, die für Vorstandsratswahlen gelten.
b) Bevor ein Referendum zur Abstimmung gelangen kann, muss ein entsprechender Entwurf für mindestens 4 Wochen zur Diskussion stehen.
c) Der Zeitraum, in welchem über das Referendum abgestimmt werden kann, soll genauso lang sein, wie bei regulären Vorstandsratswahlen.

(12)Changes to any of the above rules must be approved by a resolution of the Passive Members' Meeting (community referendum) or General Assembly.
a) Voting in such, will be open to anyone eligible to vote in the respective meeting.
b) The referendum options must be debated for a minimum of 4 weeks prior to the referendum.
c) Referendum voting will be open for the same length of time as the council elections.1 Add to favourites0 Bury

30 Jul 2016 4:11pm GMT

28 Jul 2016

feedPlanet Maemo

Truly huge files and the problem of continuous virtual address space

As we all know does mmap, or even worse on Windows CreateFileMapping, need contiguous virtual address space for a given mapping size. That can become a problem when you want to load a file of a gigabyte with mmap.

The solution is of course to mmap the big file using multiple mappings. For example like adapting yesterday's demo this way:

void FileModel::setFileName(const QString &fileName)
{
    ...
    if (m_file->open(QIODevice::ReadOnly)) {
        if (m_file->size() > MAX_MAP_SIZE) {
            m_mapSize = MAX_MAP_SIZE;
            m_file_maps.resize(1 + m_file->size() / MAX_MAP_SIZE, nullptr);
        } else {
            m_mapSize = static_cast(m_file->size());
            m_file_maps.resize(1, nullptr);
        }
        ...
    } else {
        m_index->open(QFile::ReadOnly);
        m_rowCount = m_index->size() / 4;
    }
    m_file_maps[0] = m_file->map(0, m_mapSize, QFileDevice::NoOptions);
    qDebug() << "Done loading " << m_rowCount << " lines";
    map_index = m_index->map(0, m_index->size(), QFileDevice::NoOptions);

    beginResetModel();
    endResetModel();
    emit fileNameChanged();
}

And in the data() function:

QVariant FileModel::data( const QModelIndex& index, int role ) const
{
    QVariant ret;
    ...
    quint32 mapIndex = pos_i / MAX_MAP_SIZE;
    quint32 map_pos_i = pos_i % MAX_MAP_SIZE;
    quint32 map_end_i = end_i % MAX_MAP_SIZE;
    uchar* map_file = m_file_maps[mapIndex];
    if (map_file == nullptr)
        map_file = m_file_maps[mapIndex] = m_file->map(mapIndex * m_mapSize, m_mapSize, QFileDevice::NoOptions);
    position = m_file_maps[mapIndex] + map_pos_i;
    if (position) {
            const int length = static_cast(end_i - pos_i);
            char *buffer = (char*) alloca(length+1);
            if (map_end_i >= map_pos_i)
                strncpy (buffer, (char*) position, length);
            else {
                const uchar *position2 = m_file_maps[mapIndex+1];
                if (position2 == nullptr) {
                    position2 = m_file_maps[mapIndex+1] = m_file->map((mapIndex+1) *
                         m_mapSize, m_mapSize, QFileDevice::NoOptions);
                }
                strncpy (buffer, (char*) position, MAX_MAP_SIZE - map_pos_i);
                strncpy (buffer + (MAX_MAP_SIZE - map_pos_i), (char*) position2, map_end_i);
            }
            buffer[length] = 0;
            ret = QVariant(QString(buffer));
        }
    }
    return ret;
}

You could also not use mmap for the very big source text file and use m_file.seek(map_pos_i) and m_file.read(buffer, length). The most important mapping is of course the index one, as the reading of the individual lines can also be done fast enough with normal read() calls (as long as you don't have to do it for each and every line of the very big file and as long as you know in a O(1) way where the QAbstractListModel's index.row()'s data is).

But you already knew that. Right?

0 Add to favourites0 Bury

28 Jul 2016 12:42pm GMT

26 Jul 2016

feedPlanet Maemo

Loading truly truly huge text files with a QAbstractListModel

Sometimes people want to do crazy stuff like loading a gigabyte sized plain text file into a Qt view that can handle QAbstractListModel. Like for example a QML ListView. You know, the kind of files you generate with this commando:

base64 /dev/urandom | head -c 100000000 > /tmp/file.txt

But, how do they do it?

FileModel.h

So we will make a custom QAbstractListModel. Its private member fields I will explain later:

#ifndef FILEMODEL_H
#define FILEMODEL_H

#include <QObject>
#include <QVariant>
#include <QAbstractListModel>
#include <QFile>

class FileModel: public QAbstractListModel {
    Q_OBJECT

    Q_PROPERTY(QString fileName READ fileName WRITE setFileName NOTIFY fileNameChanged )
public:
    explicit FileModel( QObject* a_parent = nullptr );
    virtual ~FileModel();

    int columnCount(const QModelIndex &parent) const;
    int rowCount( const QModelIndex& parent =  QModelIndex() ) const Q_DECL_OVERRIDE;
    QVariant data( const QModelIndex& index, int role = Qt::DisplayRole ) const  Q_DECL_OVERRIDE;
    QVariant headerData( int section, Qt::Orientation orientation,
                         int role = Qt::DisplayRole ) const  Q_DECL_OVERRIDE;
    void setFileName(const QString &fileName);
    QString fileName () const
        { return m_file->fileName(); }
signals:
    void fileNameChanged();
private:
    QFile *m_file, *m_index;
    uchar *map_file;
    uchar *map_index;
    int m_rowCount;
    void clear();
};

#endif// FILEMODEL_H

FileModel.cpp

We will basically scan the very big source text file for newline characters. We'll write the offsets of those to a file suffixed with ".mmap". We'll use that new file as a sort of "partition table" for the very big source text file, in the data() function of QAbstractListModel. But instead of sectors and files, it points to newlines.

The reason why the scanner itself isn't using the mmap's address space is because apparently reading blocks of 4kb is faster than reading each and every byte from the mmap in search of \n characters. Or at least on my hardware it was.

You should probably do the scanning in small qEventLoop iterations (make sure to use nonblocking reads, then) or in a thread, as your very big source text file can be on a unreliable or slow I/O device. Plus it's very big, else you wouldn't be doing this (please promise me to just read the entire text file in memory unless it's hundreds of megabytes in size: don't micro optimize your silly homework notepad.exe clone).

Note that this is demo code with a lot of bugs like not checking for \r and god knows what memory leaks and stuff was remaining when it suddenly worked. I leave it to the reader to improve this. An example is that you should check for validity of the ".mmap" file: your very big source text file might have changed since the newline partition table was made.

Knowing that I'll soon find this all over the place without any of its bugs fixed, here it comes ..

#include "FileModel.h"

#include <QDebug>

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <pthread.h>
#include <unistd.h>

FileModel::FileModel( QObject* a_parent )
    : QAbstractListModel( a_parent )
    , m_file (nullptr)
    , m_index(nullptr)
    , m_rowCount ( 0 ) { }

FileModel::~FileModel() { clear(); }

void FileModel::clear()
{
    if (m_file) {
        if (m_file->isOpen() && map_file != nullptr)
            m_file->unmap(map_file);
        delete m_file;
    }
    if (m_index) {
        if (m_index->isOpen() && map_index != nullptr)
            m_index->unmap(map_index);
        delete m_index;
    }
}

void FileModel::setFileName(const QString &fileName)
{
   clear();
   m_rowCount = 0;
   m_file = new QFile(fileName);
   int cur = 0;
   m_index = new QFile(m_file->fileName() + ".mmap");
   if (m_file->open(QIODevice::ReadOnly)) {
       if (!m_index->exists()) {
           char rbuffer[4096];
           m_index->open(QIODevice::WriteOnly);
           char nulbuffer[4];
           int idxnul = 0;
           memset( nulbuffer +0, idxnul >> 24 & 0xff, 1 );
           memset( nulbuffer +1, idxnul >> 16 & 0xff, 1 );
           memset( nulbuffer +2, idxnul >>  8 & 0xff, 1 );
           memset( nulbuffer +3, idxnul >>  0 & 0xff, 1 );
           m_index->write( nulbuffer, sizeof(quint32));
           qDebug() << "Indexing to" << m_index->fileName();
           while (!m_file->atEnd()) {
               int in = m_file->read(rbuffer, 4096);
               if (in == -1)
                   break;
               char *newline = (char*) 1;
               char *last = rbuffer;
               while (newline != 0) {
                   newline = strchr ( last, '\n');
                   if (newline != 0) {
                     char buffer[4];
                     int idx = cur + (newline - rbuffer);
                     memset( buffer +0, idx >> 24 & 0xff, 1 );
                     memset( buffer +1, idx >> 16 & 0xff, 1 );
                     memset( buffer +2, idx >>  8 & 0xff, 1 );
                     memset( buffer +3, idx >>  0 & 0xff, 1 );
                     m_index->write( buffer, sizeof(quint32));
                     m_rowCount++;
                     last = newline + 1;
                  }
               }
               cur += in;
           }
           m_index->close();
           m_index->open(QFile::ReadOnly);
           qDebug() << "done";
       } else {
           m_index->open(QFile::ReadOnly);
           m_rowCount = m_index->size() / 4;
       }
       map_file= m_file->map(0, m_file->size(), QFileDevice::NoOptions);
       qDebug() << "Done loading " << m_rowCount << " lines";
       map_index = m_index->map(0, m_index->size(), QFileDevice::NoOptions);
   }
   beginResetModel();
   endResetModel();
   emit fileNameChanged();
}

static quint32
read_uint32 (const quint8 *data)
{
    return data[0] << 24 |
           data[1] << 16 |
           data[2] << 8 |
           data[3];
}

int FileModel::rowCount( const QModelIndex& parent ) const
{
    Q_UNUSED( parent );
    return m_rowCount;
}

int FileModel::columnCount(const QModelIndex &parent) const
{
    Q_UNUSED( parent );
    return 1;
}

QVariant FileModel::data( const QModelIndex& index, int role ) const
{
    if( !index.isValid() )
        return QVariant();
    if (role == Qt::DisplayRole) {
        QVariant ret;
        quint32 pos_i = read_uint32(map_index + ( 4 * index.row() ) );
        quint32 end_i;
        if ( index.row() == m_rowCount-1 )
            end_i = m_file->size();
        else
            end_i = read_uint32(map_index + ( 4 * (index.row()+1) ) );
        uchar *position;
        position = map_file +  pos_i;
        uchar *end = map_file + end_i;
        int length = end - position;
        char *buffer = (char*) alloca(length +1);
        memset (buffer, 0, length+1);
        strncpy (buffer, (char*) position, length);
        ret = QVariant(QString(buffer));
        return ret;
    }
    return QVariant();
}

QVariant FileModel::headerData( int section, Qt::Orientation orientation, int role ) const
{
    Q_UNUSED(section);
    Q_UNUSED(orientation);
    if (role != Qt::DisplayRole)
           return QVariant();
    return QString("header");
}

main.cpp

#include <QGuiApplication>
#include <QQmlApplicationEngine>
#include <QtQml>// qmlRegisterType

#include "FileModel.h"

int main(int argc, char *argv[])
{
    QGuiApplication app(argc, argv);
    qmlRegisterType<FileModel>( "FileModel", 1, 0, "FileModel" );
    QQmlApplicationEngine engine;
    engine.load(QUrl(QStringLiteral("qrc:/main.qml")));
    return app.exec();
}

main.qml

import QtQuick 2.3
import QtQuick.Window 2.2
import FileModel 1.0

Window {
    visible: true

    FileModel { id: fileModel }
    ListView {
        id: list
        anchors.fill: parent
        delegate: Text { text: display }
        MouseArea {
            anchors.fill: parent
            onClicked: {
                list.model = fileModel
                fileModel.fileName = "/tmp/file.txt"
            }
        }
    }
}

profile.pro

TEMPLATE = app
QT += qml quick
CONFIG += c++11
SOURCES += main.cpp \
    FileModel.cpp
RESOURCES += qml.qrc
HEADERS += \
    FileModel.h

qml.qrc

<RCC>
    <qresource prefix="/">
        <file>main.qml</file>
    </qresource>
</RCC>

0 Add to favourites0 Bury

26 Jul 2016 7:15pm GMT

09 Jul 2016

feedPlanet Maemo

Creating PyGTK app snaps with snapcraft

Snap is a new packaging format introduced by Ubuntu as an successor to dpkg aka debian package. It offers sandboxing and transactional updates and thus is a competitor to the flatpak format and resembles docker images.

As with every new technology the weakest point of working with snaps is the documentation. Your best bet so far is the snappy-playpen repository.

There are also some rough edges regarding desktop integration and python interoperability, so this is what the post will be about.

I will introduce some quircks that were needed to get teatime running, which is written in Python3 and uses Unity and GTK3 via GObject introspection.

The most important thing to be aware of is that snaps are similar to containers in that each snap has its own rootfs and only restricted access outside of it. This is basically what the sandboxing is about.
However a typical desktop application needs to know quite a lot about the outside world:

To declare that we want to write to home, play back sound and use unity features we use the plugs keyword like

apps:
    teatime:
         # ...
         plugs: [unity7, home, pulseaudio]

However we must also tell our app to look for the according libraries inside its snap instead of the system paths. For this one must change quite a few environment variables manually. Fortunately Ubuntu provides wrapper scripts that take care of this for us. They are called desktop-launchers.

To use the launcher the configures the GTK3 environment we have to extend the teatime part like this:

apps:
    teatime:
        command: desktop-launch $SNAP/usr/share/teatime/teatime.py
        # ...
parts:
    teatime:
         # ...
         after: [desktop/gtk3]

The desktop-launch script takes care of telling PyGTK where the GI repository files are.

You can see the full snapcraft.yml here.

Update:

Before my fix, one had to use this rather lengthy startup command

env GI_TYPELIB_PATH=$SNAP/usr/lib/girepository-1.0:$SNAP/usr/lib/x86_64-linux-gnu/girepository-1.0 desktop-launch $SNAP/usr/share/teatime/teatime.py

which hard-coded the architecture.

/Update

After this teatime will start, but the paths still have to be fixed. Inside a snap "/" still refers to the system root, so all absolute paths must be prefixed with $SNAP.

Actually I think the design of flatpak is more elegant in this regard where "/" points to the local rootfs and one does not have to change absolute paths. To bring in the system parts flatpak uses bind mounts.

Conclusions

Once you get the hang of how snaps work, packaging becomes quite straightforward, however currently there are still some drawbacks

0 Add to favourites0 Bury

09 Jul 2016 1:45pm GMT