26 Jun 2019

feedAndroid Community

OneDrive increases storage, introduces secure Personal Vault

If you think your OneDrive storage isn't enough anymore to store all your files, we have good news for you. Microsoft is making some changes in how much storage you get for your subscription and it's a pretty good change since they're doubling it. Also, they're introducing a new OneDrive Personal Vault where there's an […]

26 Jun 2019 10:00pm GMT

MIUI beta lets you wave your phone around with “3D Air Gestures”

Opening apps on your phone is not that hard but of course some people want it to be either more convenient or more unusual. It looks like Xiaomi wants to do a little of both as their MIUI beta is trying out something new for the Mi 9 device. They are bringing "3D Air Gestures" […]

26 Jun 2019 8:00pm GMT

Pixel ambient display feature is getting palm rejection

Not many people may realize it but the Pixel series comes with an ambient display feature. It's more popular as the Always-on display feature on some phones. It's actually now a standard on most phones that have OLED displays. LCD screens can still have it but it looks better on OLED. The Pixel line comes […]

26 Jun 2019 7:00pm GMT

feedAndroid Developers Blog

Advanced in-app billing: handling alternative purchase flows

Posted by Oscar Rodriguez, Developer Advocate

When designing and developing an app or game, at some point you may ask yourself if you want to monetize it.

If you choose to do so by selling products via Google Play, you will most likely have a store screen that shows available items for sale, and use the Google Play Billing Library to display dialogs that allow your users to complete their purchase.

While there is a more detailed explanation in the documentation and in the Billing Library TrivialDrive samples, the general flow is as follows:

  1. Call the launchBillingFlow() method from the UI thread to launch the Google Play purchase dialog.
  2. If the purchase was successful, Google Play calls the onPurchasesUpdated() method to deliver the result of the purchase operation.
  3. If your app has a server, we strongly recommend that you verify the purchase from your server by using the Subscriptions and In-App Purchases API.
  4. Acknowledge the purchase either with consumeAsync() for consumable items or with acknowledgePurchase() for non-consumable items.
  5. Finally, grant entitlement to the purchased item inside the app.

If your app is still using the Google Play Billing AIDL API, it is also possible to perform the same task. Keep in mind that the AIDL API is now deprecated, so we strongly recommend you migrate to the Google Play Billing Library as soon as possible.

If you are using the AIDL API, the flow is very similar:

  1. Send a getBuyIntent() or getBuyIntentExtraParams() request to specify the item to purchase, and then call startIntentSenderForResult() to launch the Google Play purchase dialog.
  2. When the purchase dialog finishes, Google Play sends a response Intent to your onActivityResult() method, where you can verify if the purchase was successful.
  3. If your app has a server, we strongly recommend that you verify the purchase from your server by using the Subscriptions and In-App Purchases API.
  4. If the purchase was successful, call the getPurchases() method to retrieve a list of owned items that are still not consumed. For consumable items, call the consumePurchase() method to make the item available for purchase again.
  5. Finally, grant entitlement to the purchased item inside the app.

Nevertheless, just implementing the above mentioned flow is not enough to correctly handle all types of purchases. There are two main cases in which purchases will not be correctly handled by this flow.

The first case happens when the purchase flow is interrupted before it finishes. The app may have crashed, the user may have killed the app, or the user's Internet connection may have been lost. In any case, it is possible for the app not to have delivered the item to the user even though Google Play has already processed the payment. In this case, the item is in limbo, because Google Play will not allow an item to be re-purchased until it is consumed, but the app or game won't consume the item outside of the flow mentioned above.

The second case happens during alternative purchase flows, such as in-app promotions, the recently announced out-of-app subscription surfaces, promo codes for subscriptions, or other promotions in collaboration with Google. In these cases, a user gets an item directly on the Play Store app, while the target app or game may be paused, not running, or even not installed.

For these cases, the Google Play Billing Library and the Google Play Billing AIDL API offer a mechanism to detect purchases that are not acknowledged or consumed.

When using the Google Play Billing API, do the following:

  1. In your app's onResume() callback, call the queryPurchases() method to retrieve a list of items, so you can determine which ones are unacknowledged.
  2. If your app has a server, we strongly recommend that you verify the purchase from your server by using the Subscriptions and In-App Purchases API.
  3. If there are owned but unacknowledged items, acknowledge the purchase either with consumeAsync() for consumable items or with acknowledgePurchase() for non-consumable items.
  4. Grant entitlement to the purchased item inside the app.

For the Google Play Billing AIDL API, do the following:

  1. In your app's onResume() callback, call the getPurchases() method to retrieve a list of owned items that are still not consumed.
  2. If your app has a server, we strongly recommend that you verify the purchase from your server by using the Subscriptions and In-App Purchases API.
  3. For consumable items, call the consumePurchase() method to make the item available for purchase again.
  4. Finally, grant entitlement to the purchased item inside the app.

In either case, when you detect and process an unconsumed item in this manner, users will expect the app or game to communicate about it. We suggest that you display a dialog, message box, or notification that tells the user that they have successfully received their item.

Keep in mind that your app's onResume() callback will be called when its process is started, as well as when it is brought to the foreground, regardless of which screen the app or game was in before it was paused. For example, a game with a home screen, a store screen, and a game screen might get its onResume() called from any of those screens. For an optimal user experience, we suggest you make it so your app or game handles unacknowledged or unconsumed items regardless of the screen you display when onResume() gets called. Thorough testing of this process in each screen is crucial to deliver a great user experience.

Finally, there is one more case your app must handle: when a user acquires an item from the Play Store app, and both the Play Store app and your app are visible at the same time with multi-window mode.

To support this scenario with the Google Play Billing Library, do the following:

  1. Google Play calls the onPurchasesUpdated() method to notify your app that there is a new pending item.
  2. If your app has a server, we strongly recommend that you verify the purchase from your server by using the Subscriptions and In-App Purchases API.
  3. Acknowledge the purchase either with consumeAsync() for consumable items or with acknowledgePurchase() for non-consumable items.
  4. Finally, grant entitlement to the purchased item inside the app.

For the Google Play Billing AIDL API, do the following:

  1. In your app's onResume() callback, register a PurchasesUpdatedListener to receive the com.android.vending.billing.PURCHASES_UPDATED intent. Also, in your app's onPause() callback, unregister the listener.
  2. If your app has a server, we strongly recommend that you verify the purchase from your server by using the Subscriptions and In-App Purchases API.
  3. Google Play calls your listener to notify your app that there is a new pending item. Inside it, call the getPurchases() method to retrieve a list of owned items that are still not consumed. For consumable items, call the consumePurchase() method to make the item available for purchase again.
  4. Finally, grant entitlement to the purchased item inside the app.

Just as before, you should display a dialog, message box, or notification that tells the user that they have successfully received their item.

If you follow these steps, your app or game will be better prepared to robustly handle purchase flow interruptions and alternative purchase flows.

26 Jun 2019 5:31pm GMT

feedTalkAndroid

AT&T facing class action lawsuit over hidden fees

A new lawsuit filed in California alleges AT&T is charging customers more than their advertised rates. The additional amounts charged are in the form of an "Administrative Fee" that AT&T tacks on to customer bills after they sign up for service. Currently the fee is $1.99 per month. According to the complaint that was filed, […]


Come comment on this article: AT&T facing class action lawsuit over hidden fees

Visit TalkAndroid

26 Jun 2019 5:16pm GMT

OnePlus 6 and OnePlus 6T get new open beta builds with small performance tweaks

OnePlus has released a few open betas for the OnePlus 6 and OnePlus 6T, bringing those phones to beta version 21 and 13, respectively. It's pretty much the same software for both phones, but like all betas, it may be a little less stable than a regular OTA update, so keep that in mind. The […]


Come comment on this article: OnePlus 6 and OnePlus 6T get new open beta builds with small performance tweaks

Visit TalkAndroid

26 Jun 2019 5:15pm GMT

Google says not yet to claims that Android Auto update is available

After a user on reddit posted an image claiming they had received a major update to their Android Auto app in their vehicle, Google has responded to indicate that no such update is rolling out to users. Google said earlier this year that an overhaul of their vehicle interface system was in the works with […]


Come comment on this article: Google says not yet to claims that Android Auto update is available

Visit TalkAndroid

26 Jun 2019 4:52pm GMT

13 Jun 2019

feedPlanet Maemo

WIP: changing the backend for contacts in Ubports

More than one year has passed since the initial announcement of my plan to investigate using a different backend for contact storage. If you want to get a better understanding of the plan, that mail is still a good read -- not much has changed since them, planning wise.

The reason for this blog post is to give a small update on what has happened since then, and as a start nothing can be better than a couple of screenshots:

Adding CardDAV accounts in the Addressbook application Aggregated contact details from multiple sources

In other words, that means that contact synchonisation works, both with the new CardDAV protocol (for which we'll have preconfigured setups for NextCloud and OwnCloud accounts) and with Google Contacts, for which we are now using a different engine. What you see in the second screenshot (although indeed it's not obvious at all) is that the new qtcontacts-sqlite backend performs automatic contact merging based on some simple heuristics, meaning that when synchonising the same contact from multiple sources you should not happen to find a multitude of semi-identical copies of the contact, but a single one having all the aggregated details.

Before you get too excited, I have to say that this code is pre-alpha quality and that it's not even available for testing yet. The next step is indeed to setup CI so that the packaggqes get automatically built and published to a public repository, at which point I'll probably issue another update here in my blog.

The boring stuff

And now some detail for those who might wonder why this feature is not ready yet, or would like to get an idea on the time-frame for its completion.

Apart from a chronical lack of time from my part, the feature complexity is due to the large number of components involved:

  • qtcontacts-sqlite: the QtContacts backend we are migrating to. This is a backend for the QtContacts API (used by our Addressbook application) which uses a SQLite database as storage for your contacts.
  • buteo-sync-plugin-carddav: the CardDAV plugin for Buteo (our synchronisation manager). This plugin is loaded by Buteo and synchronises the contacts between a CardDAV remote source and the qtcontacts-sqlite database.
  • buteo-sync-plugins-social: a Buteo plugin which can synchronise contacts from a multitude of sources, including Google, Facebook and Vk. At the moment we only care about Google, but once this feature has landed we can easily extend it to work with the other two as well.
  • address-book-app: this is our well-known Contacts application. It needs some minor changes to adapt to the qtcontacts-sqlite backend and to support the creation of new CardDAV, NextCloud and OwnCloud accounts.
  • QtPim: the contacts and calendar API developed by the Qt project. Our Contacts application is using the front-end side of this API, and the qtcontacts-sqlite component implements the backend side. There are some improvements proposed by Jolla, which we need to include in order to support grouping contacts by their initials.

The other tricky aspect is that the first three projects are maintained by Jolla as part of the Sailfish OS, and while on one side this means that we can share the development and maintenance burden with Jolla, on the other side of the coin it means that we need to apply extra care when submitting changes, in order not to step on each other's shoes. Specifically, Sailfish OS is using a much older version of QtPim than Ubports is, and the APIs between the two versions have changes in an incompatible version, so that it's nearly impossible to have a single code base working with both versions of QtPim. Luckily git supports branches, and Chris from Jolla was kind enough to create a branch for us in their upstream repository where I've proposed our changes (and they are a lot!).

However, this is not as bad as it sounds, and the fact that I have a roughly working version on my development device is a good sign that things are moving forwards.

0 Add to favourites0 Bury

13 Jun 2019 2:59pm GMT

11 Jun 2019

feedAndroid Developers Blog

Moving Android Studio and Android Emulator to 64-bit versions

Posted by Sam Lin, Product Manager, Android

With Project Marble, the Android Studio team focused our efforts on making the fundamental features and flows of the Integrated Development Environment (IDE) rock-solid. Performance is an underlying tenant to delivering a high quality IDE. To this end, we are sharpening our product focus and we will only support 64-bit operating systems going forward. Using Android Studio with an 64-bit operating systems enables efficient access to memory for both the IDE and the Android Emulator, and overall leads to a better development experience. While this change will not affect most Android Studio users, this change does have an impact if you use 32-bit versions of Microsoft® Windows®. To aid in this transition for those developers using 32-bit versions of Microsoft Windows, we want to give you details on the upcoming depreciation timeline plus steps to take to be ready for this upcoming change.

Timeline

To minimize the impact of this change towards exclusively supporting 64-bit operating systems, we will first deprecate the 32-bit version. During the depreciation phase, both Android Studio and the Android Emulator will continue to work but the products will not receive new feature updates. During this transition period you can still download the product from the Android Studio web site. After one year, we will officially end product support and will remove the 32-bit product version download links. Note, if you have the 32-bit version of Android Studio previously installed during this period then the product should continue to work, but we will not provide a link for you to re-download the product. The exact dates for the depreciation and end-of-support period are in the table below:

Supported 32-bit Product Version Deprecation from End of Support on
Android Studio IDE 3.6 December 31, 2019 December 31, 2020
Android Emulator 28.0.25 Jun 30, 2019 December 31, 2020

Advantages of a 64-bit development environment

There are a few advantages to using a 64-bit version of Android Studio, which include:

Next steps

To recap, before ending support for the 32-bit version of Android Studio, we want to inform you in advance, provide guidance, and allow for a one-year lead time to help you migrate to a 64-bit operating system. You can still use 32-bit versions of Android Studio, but be mindful that these version will not receive future updates. Therefore, if you want to migrate we suggest you start planning early so that you can continue to get the latest product updates and take advantage of the performance improvements of a 64-bit development environment.

11 Jun 2019 6:12pm GMT

05 Jun 2019

feedAndroid Developers Blog

Android Q Beta 4 and Final APIs!

Posted by Dave Burke, VP of Engineering

AndroidQ logo

Last month at Google I/O we talked about what's new for Android developers, from new features in Android Q to the latest in Kotlin and Jetpack.

With Android Q, we highlighted three themes: innovation, security and privacy, and digital wellbeing. We want to help you take advantage of the latest new technology -- 5G, foldables, edge-to-edge screens, on-device machine learning, and more -- while making sure users' security, privacy, and wellbeing are always a top priority.

We also talked about how we're going increasingly Kotlin-first, and continuing to expand Jetpack with new libraries like CameraX, Jetpack Security and Jetpack Compose -- a modern reactive-style UI toolkit for Android that takes advantage of Kotlin. If you missed the livestream for the keynotes or tech sessions, make sure to check out the full playlist of Android and Play sessions.

Today we're releasing Beta 4 with the final Android Q APIs and official SDK -- the time is now to get your apps ready for the final release later in the summer!

You can get Beta 4 today on Pixel devices by enrolling here. If you're already enrolled and received the Beta 3 on your Pixel device, you'll automatically get the update to Beta 4. Partners participating in the Android Q Beta program will also be updating their devices to Beta 4 over the coming weeks.

To get started with Android Q Beta, visit developer.android.com/preview.

What's in Beta 4?

The Beta 4 update includes the latest Android Q system images for Pixel and Android Emulator, along with the final Android Q developer APIs (API level 29), the official API 29 SDK, and updated build tools for Android Studio. Together, these give you everything you need to test your apps for compatibility with Android Q and build with Android Q features and APIs.

To get started, download the official API 29 SDK and tools into the stable release of Android Studio 3.4, or for the latest Android Q support update to Android Studio 3.5 Beta. Then follow these instructions to configure your environment, and see the release notes for known issues.

Make your apps compatible with Android Q!

With the developer APIs finalized and release candidate builds coming soon, it's critical for all Android developers to test their current apps for compatibility with Android Q. We recommend getting started as soon as possible.

Just install your current app from Google Play onto an Android Q Beta device or emulator, then test. As you work through the flows, your app should run and look great and handle all of the Android Q behavior changes properly. Watch for impacts from privacy changes, gestural navigation, changes to dynamic linker paths for Bionic libraries, and others.

Make sure that you test with the Android Q privacy features, such as the new location permissions, restrictions on background activity starts, changes to data and identifiers, and other key privacy features. See the privacy checklist to get started, and review the behavior changes doc for more areas to test.

Android Developers YouTube channel UI on landscape mode.

You can use the updated Android Emulator to test your apps for compatibility.

If you plan to update your platform targeting to API 29, also make sure to test with scoped storage, location permission for wireless scans, and permission for fullscreen intents. You can read about other changes that could affect apps here.

It's also important to test for uses of restricted non-SDK interfaces and move to public SDK or NDK equivalents instead. Watch for logcat warnings that highlight these accesses and use the StrictMode method detectNonSdkApiUsage() to catch them programmatically.

Last, make sure to fully test the libraries and SDKs in your app to make sure they work as expected on Android Q and follow best practices for privacy, performance, UX, data handling, and permissions. If you find an issue, try updating to the latest version of the SDK, or reach out to the SDK developer for help. You can also report SDK compatibility issues here.

When you've finished your testing and made any updates, we recommend publishing your compatible app right away. This lets Android Beta users test the app now, and helps you deliver a smooth transition to users as they update to Android Q.

We realize that supporting these changes is an investment for you too, and we're working to minimize the impact on your apps and be responsive to your input as we move toward the final release in the coming months.

Enhance your app with Android Q features and APIs

When you're ready, dive into Android Q and learn about the new features and APIs that you can use in your apps. Android Q features can help you engage users, give them more control and security, and even improve your app's performance.

mobile device notification window

Android Q provides system-suggested replies and actions in notifications.

For example, you can deliver seamless, edge-to-edge experiences on today's innovative devices by optimizing for foldables and supporting gestural navigation in your app. To engage more users, try supporting Dark Theme, suggested replies and actions in notifications, sharing shortcuts, and settings panels.

Google Maps app closes to display home screen with ocean aerial image

Gestural navigation lets you offer an edge-to-edge experience in your apps.

If your app manages IoT devices over Wi-Fi, try the new network connection APIs for functions like configuring, downloading, or printing. If your app manages Wi-Fi internet connections, try the network suggestion APIs as an easier way to surface preferred Wi-Fi networks, without needing to request location permission.

If you use the camera, learn about dynamic depth format. For media, you can use AV1 for video streaming and HDR10+ for high dynamic range video. For speech and music streaming, you can use Opus encoding, and for musicians, a native MIDI API is available.

Dynamic Depth lets you offer specialized blurs and bokeh options in your app.

To support captioning or gameplay recording, enable audio playback capture -- it's a great way to reach more users and get your app noticed. If your app uses power intensively, try using the new thermal API to optimize app performance based on device temperature.

BiometricPrompt is now the preferred way to support fingerprint auth on modern devices, so all developers using fingerprint or other biometric auth should move to using this API as soon as possible. To make the transition easy, use the backwards-compatible BiometricPrompt API that we're providing in the AndroidX library. Android Q supports both standard and passive (no confirmation, for face and other passive modes) auth flows.

These are just a few of the many new features and APIs in Android Q -- to see them all, visit the Android Q Beta site for developers.

Publish your app updates to Google Play

Today with Android Q Beta 4 we're also opening up publishing on Google Play to apps that are compiled against, or optionally targeting, API 29. This means you can now push your updates to users now through Google Play to test your app's compatibility, including on devices running Android Q Beta 4.

How do I get Beta 4?

It's easy! Just enroll any supported Pixel device here to get the update over-the-air. If you're already enrolled, you'll receive the update soon and no action is needed on your part. Downloadable system images are also available here. Partners that are participating in the Android Q Beta program will be updating their devices over the coming weeks. See android.com/beta for details.

For even broader testing on supported devices, you can also get Android GSI images, and if you don't have a device you can test on the Android Emulator.

As always, your input is critical, so please continue to let us know what you think. You can use our hotlists for filing platform issues (including privacy and behavior changes), app compatibility issues, and third-party SDK issues. You've shared great feedback with us so far and we're working to integrate as much of it as possible in the next Beta release.

We're looking forward to seeing your apps on Android Q!

05 Jun 2019 5:08pm GMT

13 May 2019

feedPlanet Maemo

Ubuntu on the Lenovo D330

The Lenovo D330 2-in-1 convertible (or netbook as we used to say) is a quite interesting device. It is based on Intels current low-power core platform, Gemini Lake (GLK), and thus offers great battery-life and a fan-less design.

This similar to what you would from an ARM based tablet. However being x86 based and Windows focused we can expect to get Ubuntu Linux running - without requiring any out-of-tree drivers or custom kernels that never get updated as we are used-to from the ARM world.
This post will be about my experiences on doing so.

For this I will use the most recent Ubuntu 19.04 release as it contains fractional scaling support, which is essential for a 10″ 1920x1200px device. Also the orientation sensor (mostly) works out of the box, when compared to the 18.04 LTS release.

Getting to the desktop

After booting the live USB, you will notice that the screen stays black. This is caused by the i915 driver not correctly setting up the internal (DSI) screen (see FDO#109267).

A quick workaround for this is to rotate the device, causes the i915 driver to re-initialise the screen, which will work at this point. Alternatively you can also suspend/ resume the device.

Wrong screen orientation

Once on the desktop, you will notice that it is rotated by 90° which is caused by a missing mount matrix entry in udev shipped with 19.04.

First we manually rotate the screen to landscape mode by opening a terminal via Ctrl+Alt+T and typing

xrandr -o right

Now we can continue to add the missing accelerometer mount matrix. Create the file /etc/udev/hwdb.d/61-sensor-local.hwdb with the following content:

# IdeaPad D330
sensor:modalias:acpi:BOSC0200*:dmi:*:svnLENOVO:pn81H3:*
    ACCEL_MOUNT_MATRIX=0, 1, 0; -1, 0, 0; 0, 0, 1

To immediately apply the changes do

sudo systemd-hwdb update
sudo udevadm trigger -v -p DEVNAME=/dev/iio:device0
sudo service iio-sensor-proxy restart

Touch input rotation

Now you can rotate the device and the screen content will be correctly oriented. However, if you try to use the touchscreen, you will notice that it only behaves correctly in portrait mode.

This is caused by a bug in GNOME/ mutter introduced in the 3.32 release. It will be fixed in the 3.32.2 point-release. In the meantime you can use the updated mutter packages, where I have backported the fix.

Enabling fractional scaling

Finally we need to magnify the UI by 50% in order not to damage our eyes. Unfortunately the fractional scaling in Ubuntu 19.04 is hidden by default. To enable it enter the following in a console

gsettings set org.gnome.mutter experimental-features "['scale-monitor-framebuffer', 'x11-randr-fractional-scaling']"

This will enable fractional scaling for both Wayland and Xorg sessions. Here, you will get a noticeable performance hit when using Xorg. However you should choose it regardless due to the following on Wayland

This probably also helps understanding why Xorg is still default on the majority of Linux distributions.

What does not work

Surprisingly little, actually. The only remaining issue is that you will lose sound after a suspend/ resume cycle due to some bug in snd-hda-intel-realtek.

What does work

State of touch UI on Linux

However there is more than getting the hardware to start when using the device on a daily basis - you will also have to fight the linux apps. Currently only GTK3 apps have some understanding of touch events - most other toolkits need updating. To estimate the time until this will happen keep in mind that touch on Linux is a niche inside a niche.

Firefox for instance does not understand touch events and you will have to manually enable them as described here.

But even then you will be able to drag the window around - thanks to client side decoration (CSD) window placement is handled by Firefox itself, but it does not react to touch events #1530070 (same applies to Chrome btw).

Next all apps continue to launch in windowed mode. On device with limited screen space, you will soon notice why everything is full-screen on Android (and Windows Tablet Mode).

Finally the GNOME on-screen keyboard feels quite sluggish and lacks visual feedback. Therefore you will often end up with missing letters.

So in summary it is not a very pleasant experience right now.

Fortunately GNOME/ Ubuntu are supposedly on it for the next release.

Until then I would recommend sticking with Windows 10; fractional scaling just works there and you will really value its on-screen keyboard and tablet mode when having the physical keyboard detached.

0 Add to favourites0 Bury

13 May 2019 5:03pm GMT

24 Apr 2019

feedPlanet Maemo

A critical view on the blockchain

At the beginning of this month I participated to the foss-north conference, in Gothenburg, and took the stage to give a short presentation of the blockchain technology. Given that my talk was somehow critical of the blockchain (or rather, of the projects using it without due reason) I was prepared to receive a wave of negative remarks, assuming that all the hype surrounding this technology would have infected a good part of my audience as well. I was therefore positively surprised when several people came to me afterwords to express their appreciation for my speech, appreciation that now makes me confident enough to share the video of the presentation here too:

I want to publicly thank Johan Thelin and all the other foss-north staff and volunteers who organized such a successful conference. They also managed to get the video recordings out in a surprisingly short time. Indeed, the above video is taken from the foss-north YouTube channel, which I recommend you to visit as there were a lot of good talks at the conference; the topics were so varied, that I'm sure you'll find at least a couple of talks of your interest.

1 Add to favourites0 Bury

24 Apr 2019 7:59pm GMT

25 Sep 2017

feedPlanet Openmoko

Holger "zecke" Freyther: Brain dump what fascinates me

A small brain dump of topics that currently fascinate me. These are mostly pointers and maybe it is interesting to follow it.

Books/Reading:

My kobo ebook reader has the Site Reliability Engineering book and I am now mostly done. It is kind of a revelation and explains my interest to write code but also to operate infrastructure (like struggling with ruby, rmagick, nginx…). I am interested in backends since… well ever. The first time I noticed it when we talked about Kolab at LinuxTag and I was more interested in the backend than the KDE client. At sysmocom we built an IoT product and the backend was quite some fun, especially the scale of one instance and many devices/users, capacity planning and disk commissioning, lossless upgrades.

It can be seen in my non FOSS SS7 map work on traffic masquerading and transparent rewriting. It is also clear to see which part of engineering is needed for scale (instead of just installing and restarting servers).

Lang VM design

One technology that made Java fast (Hotspot) and has seen its way into JavaScript is dynamic optimization. Most Just in Time Compilers start with generating native code per method, either directly or after the first couple of calls when the methods size is significant enough. The VM records which call paths are hot, which types are used and then can generate optimized code (e.g. specialized for integers, remove type checks). A technique pioneered at Sun for the "Self" language (and then implemented for Strongtalk and then brought to Java) was "adaptive optimization and deoptimization" and was the Phd topic of Urs Hoelzle (Google's VP of Engineering). One of the key aspects is inlining across method boundaries as this removes method look-up, call stack handling and opens the way for code optimization across method boundaries (at the cost of RAM usage).

In OpenJDK, V8 and JavaScriptCore this adaptive optimization is typically implemented in C++ and requires quite some code. The code is complicated as it needs to optimize but also need to return to a basic function (deoptimize, e.g. if a method changed or the types passed don't match anymore), e.g. in the middle of a for loop with tons of inlined code (think of Array.map being inlined but then need to be de-inlined). A nice and long blog post of JSC can be found here describing the On Stack Replacement (OSR).

Long introduction and now to the new thing. In the OpensmalltalkVM a new approach called Sista has been picked and I find it is genius. Like with many problems the point of view and approach really matters. Instead of writing a lot of code in the VM the optimizer runs next to the application code. The key parts seem to be:

The revelation is the last part. By just optimizing from bytecode to bytecode the VM remains in charge of creating and managing machine code. The next part is that tooling in the higher language is better or at least the roundtrip is more quick (edit code and just compile the new method instead of running make, c++, ld). The productivity thanks to the abstraction and tooling is likely higher.

As last part the OSR is easier as well. In Smalltalk thisContext (the current stack frame, activation record) is an object as well. At the right point (when the JIT has either written back variables from register to the stack or at least knows where the value is) one can just manipulate thisContext, create and link news ones and then resume execution without all the magic in other VMs.

Go, Go and escape analysis

Ken Thompson and Robert Pike are well known persons and their Go programming language is a very interesting system programming language. Like with all new languages I try to get real world experience with the language, the tooling and which kind of problems can be solved with it. I have debugged and patched some bigger projects and written two small applications with it.

There is plenty I like. The escape analysis of the compiler is fun (especially now that I know it was translated from the Plan9 C compiler from C to Go), the concurrency model is good (though allowing shared state), the module system makes sense (but makes forking harder than necessary), being able to cross compile to any target from any system.

Knowing a bit of Erlang (and continuing to read the Phd Thesis of Joe Armstrong) and being a heavy Smalltalk user there are plenty of things missing. It starts with vague runtime error messages (e.g. panicslice not having parameters) and goes to runtime and post-runtime inspection. In Smalltalk thanks to the abstraction a lot of hard things are easy and I would have wished for some of them to be in Go. Serialize all unrecovered panics? Debugging someone else's code seems like pre 1980…

So for many developers Go is a big improvement but for some people with a wider view it might look like a lost opportunity. But that can only be felt by developers that have experienced higher abstraction and productivity.

Unsupervised machine learning

but that is for another dump…

25 Sep 2017 10:11am GMT

02 Sep 2017

feedPlanet Openmoko

Harald "LaF0rge" Welte: Purism Librem 5 campaign

There's a new project currently undergoing crowd funding that might be of interest to the former Openmoko community: The Purism Librem 5 campaign.

Similar to Openmoko a decade ago, they are aiming to build a FOSS based smartphone built on GNU/Linux without any proprietary drivers/blobs on the application processor, from bootloader to userspace.

Furthermore (just like Openmoko) the baseband processor is fully isolated, with no shared memory and with the Linux-running application processor being in full control.

They go beyond what we wanted to do at Openmoko in offering hardware kill switches for camera/phone/baseband/bluetooth. During Openmoko days we assumed it is sufficient to simply control all those bits from the trusted Linux domain, but of course once that might be compromised, a physical kill switch provides a completely different level of security.

I wish them all the best, and hope they can leave a better track record than Openmoko. Sure, we sold some thousands of phones, but the company quickly died, and the state of software was far from end-user-ready. I think the primary obstacles/complexities are verification of the hardware design as well as the software stack all the way up to the UI.

The budget of ~ 1.5 million seems extremely tight from my point of view, but then I have no information about how much Puri.sm is able to invest from other sources outside of the campaign.

If you're a FOSS developer with a strong interest in a Free/Open privacy-first smartphone, please note that they have several job openings, from Kernel Developer to OS Developer to UI Developer. I'd love to see some talents at work in that area.

It's a bit of a pity that almost all of the actual technical details are unspecified at this point (except RAM/flash/main-cpu). No details on the cellular modem/chipset used, no details on the camera, neither on the bluetooth chipset, wifi chipset, etc. This might be an indication of the early stage of their plannings. I would have expected that one has ironed out those questions before looking for funding - but then, it's their campaign and they can run it as they see it fit!

I for my part have just put in a pledge for one phone. Let's see what will come of it. In case you feel motivated by this post to join in: Please keep in mind that any crowdfunding campaign bears significant financial risks. So please make sure you made up your mind and don't blame my blog post for luring you into spending money :)

02 Sep 2017 10:00pm GMT

01 Sep 2017

feedPlanet Openmoko

Harald "LaF0rge" Welte: First actual XMOS / XCORE project

For many years I've been fascinated by the XMOS XCore architecture. It offers a surprisingly refreshing alternative virtually any other classic microcontroller architectures out there. However, despite reading a lot about it years ago, being fascinated by it, and even giving a short informal presentation about it once, I've so far never used it. Too much "real" work imposes a high barrier to spending time learning about new architectures, languages, toolchains and the like.

Introduction into XCore

Rather than having lots of fixed-purpose built-in "hard core" peripherals for interfaces such as SPI, I2C, I2S, etc. the XCore controllers have a combination of

  • I/O ports for 1/4/8/16/32 bit wide signals, with SERDES, FIFO, hardware strobe generation, etc
  • Clock blocks for using/dividing internal or external clocks
  • hardware multi-threading that presents 8 logical threads on each core
  • xCONNECT links that can be used to connect multiple processors over 2 or 5 wires per direction
  • channels as a means of communication (similar to sockets) between threads, whether on the same xCORE or a remote core via xCONNECT
  • an extended C (xC) programming language to make use of parallelism, channels and the I/O ports

In spirit, it is like a 21st century implementation of some of the concepts established first with Transputers.

My main interest in xMOS has been the flexibility that you get in implementing not-so-standard electronics interfaces. For regular I2C, UART, SPI, etc. there is of course no such need. But every so often one encounters some interface that's very rately found (like the output of an E1/T1 Line Interface Unit).

Also, quite often I run into use cases where it's simply impossible to find a microcontroller with a sufficient number of the related peripherals built-in. Try finding a microcontroller with 8 UARTs, for example. Or one with four different PCM/I2S interfaces, which all can run in different clock domains.

The existing options of solving such problems basically boil down to either implementing it in hard-wired logic (unrealistic, complex, expensive) or going to programmable logic with CPLD or FPGAs. While the latter is certainly also quite interesting, the learning curve is steep, the tools anything but easy to use and the synthesising time (and thus development cycles) long. Furthermore, your board design will be more complex as you have that FPGA/CPLD and a microcontroller, need to interface the two, etc (yes, in high-end use cases there's the Zynq, but I'm thinking of several orders of magnitude less complex designs).

Of course one can also take a "pure software" approach and go for high-speed bit-banging. There are some ARM SoCs that can toggle their pins. People have reported rates like 14 MHz being possible on a Raspberry Pi. However, when running a general-purpose OS in parallel, this kind of speed is hard to do reliably over long term, and the related software implementations are going to be anything but nice to write.

So the XCore is looking like a nice alternative for a lot of those use cases. Where you want a microcontroller with more programmability in terms of its I/O capabilities, but not go as far as to go full-on with FPGA/CPLD development in Verilog or VHDL.

My current use case

My current use case is to implement a board that can accept four independent PCM inputs (all in slave mode, i.e. clock provided by external master) and present them via USB to a host PC. The final goal is to have a board that can be combined with the sysmoQMOD and which can interface the PCM audio of four cellular modems concurrently.

While XMOS is quite strong in the Audio field and you can find existing examples and app notes for I2S and S/PDIF, I couldn't find any existing code for a PCM slave of the given requirements (short frame sync, 8kHz sample rate, 16bit samples, 2.048 MHz bit clock, MSB first).

I wanted to get a feeling how well one can implement the related PCM slave. In order to test the slave, I decided to develop the matching PCM master and run the two against each other. Despite having never written any code for XMOS before, nor having used any of the toolchain, I was able to implement the PCM master and PCM slave within something like ~6 hours, including simulation and verification. Sure, one can certainly do that in much less time, but only once you're familiar with the tools, programming environment, language, etc. I think it's not bad.

The biggest problem was that the clock phase for a clocked output port cannot be configured, i.e. the XCore insists on always clocking out a new bit at the falling edge, while my use case of course required the opposite: Clocking oout new signals at the rising edge. I had to use a second clock block to generate the inverted clock in order to achieve that goal.

Beyond that 4xPCM use case, I also have other ideas like finally putting the osmo-e1-xcvr to use by combining it with an XMOS device to build a portable E1-to-USB adapter. I have no clue if and when I'll find time for that, but if somebody wants to join in: Let me know!

The good parts

Documentation excellent

I found the various pieces of documentation extremely useful and very well written.

Fast progress

I was able to make fast progress in solving the first task using the XMOS / Xcore approach.

Soft Cores developed in public, with commit log

You can find plenty of soft cores that XMOS has been developing on github at https://github.com/xcore, including the full commit history.

This type of development is a big improvement over what most vendors of smaller microcontrollers like Atmel are doing (infrequent tar-ball code-drops without commit history). And in the case of the classic uC vendors, we're talking about drivers only. In the XMOS case it's about the entire logic of the peripheral!

You can for example see that for their I2C core, the very active commit history goes back to January 2011.

xSIM simulation extremely helpful

The xTIMEcomposer IDE (based on Eclipse) contains extensive tracing support and an extensible near cycle accurate simulator (xSIM). I've implemented a PCM mater and PCM slave in xC and was able to simulate the program while looking at the waveforms of the logic signals between those two.

The bad parts

Unfortunately, my extremely enthusiastic reception of XMOS has suffered quite a bit over time. Let me explain why:

Hard to get XCore chips

While the product portfolio on on the xMOS website looks extremely comprehensive, the vast majority of the parts is not available from stock at distributors. You won't even get samples, and lead times are 12 weeks (!). If you check at digikey, they have listed a total of 302 different XMOS controllers, but only 35 of them are in stock. USB capable are 15. With other distributors like Farnell it's even worse.

I've seen this with other semiconductor vendors before, but never to such a large extent. Sure, some packages/configurations are not standard products, but having only 11% of the portfolio actually available is pretty bad.

In such situations, where it's difficult to convince distributors to stock parts, it would be a good idea for XMOS to stock parts themselves and provide samples / low quantities directly. Not everyone is able to order large trays and/or capable to wait 12 weeks, especially during the R&D phase of a board.

Extremely limited number of single-bit ports

In the smaller / lower pin-count parts, like the XU[F]-208 series in QFN/LQFP-64, the number of usable, exposed single-bit ports is ridiculously low. Out of the total 33 I/O lines available, only 7 can be used as single-bit I/O ports. All other lines can only be used for 4-, 8-, or 16-bit ports. If you're dealing primarily with serial interfaces like I2C, SPI, I2S, UART/USART and the like, those parallel ports are of no use, and you have to go for a mechanically much larger part (like XU[F]-216 in TQFP-128) in order to have a decent number of single-bit ports exposed. Those parts also come with twice the number of cores, memory, etc- which you don't need for slow-speed serial interfaces...

Insufficient number of exposed xLINKs

The smaller parts like XU[F]-208 only have one xLINK exposed. Of what use is that? If you don't have at least two links available, you cannot daisy-chain them to each other, let alone build more complex structures like cubes (at least 3 xLINKs).

So once again you have to go to much larger packages, where you will not use 80% of the pins or resources, just to get the required number of xLINKs for interconnection.

Change to a non-FOSS License

XMOS deserved a lot of praise for releasing all their soft IP cores as Free / Open Source Software on github at https://github.com/xcore. The License has basically been a 3-clause BSD license. This was a good move, as it meant that anyone could create derivative versions, whether proprietary or FOSS, and there would be virtually no license incompatibilities with whatever code people wanted to write.

However, to my very big disappointment, more recently XMOS seems to have changed their policy on this. New soft cores (released at https://github.com/xmos as opposed to the old https://github.com/xcore) are made available under a non-free license. This license is nothing like BSD 3-clause license or any other Free Software or Open Source license. It restricts the license to use the code together with an XMOS product, requires the user to contribute fixes back to XMOS and contains references to importand export control. This license is incopatible with probably any FOSS license in existance, making it impossible to write FOSS code on XMOS while using any of the new soft cores released by XMOS.

But even beyond that license change, not even all code is provided in source code format anymore. The new USB library (lib_usb) is provided as binary-only library, for example.

If you know anyone at XMOS management or XMOS legal with whom I could raise this topic of license change when transitioning from older sc_* software to later lib_* code, I would appreciate this a lot.

Proprietary Compiler

While a lot of the toolchain and IDE is based on open source (Eclipse, LLVM, ...), the actual xC compiler is proprietary.

Further Reading

01 Sep 2017 10:00pm GMT

12 Nov 2011

feedPlanet Linux-to-go

Paul 'pfalcon' Sokolovsky: Shopping for 3D TV...

Shopping for 3D TV (again), few findings:

12 Nov 2011 6:55pm GMT

Paul 'pfalcon' Sokolovsky: Hacking Luxeon SP-1

I finally going to get Arduino, and while I'm choosing flavor and waiting for it, I can't help but disassembling all devices I have at home, each time speaking: "This must have Arduino inside!" (meaning of course that I expect it to be based on general-purpose MCU). Gosh, I usually get "blob chip" (uncased chip with blob of epoxy on top).

Well, I finally had my expectations fulfilled - Luxeon SP-1 voltage stabilizer/cutter features ATMEGA48V-10PU (Flash: 4k, EEPROM: 256, RAM:512). Not only that, it is installed in DIP socket! Buy from Luxeon, they're hacker-friendly ;-).

I bought the device actually for a wattmeter it features (which fact is hard to figure out from common specs found in the shops, I accidentally read somebody mentioning it on a forum). The wattmeter is of course not bright - for a lamp rated 100W it shows 88W, and for more powerful equipment (like perforator) understates wattage even more (maybe it's difference between real and apparent power factor).

Still, for $17 you get Arudino-alike with voltage/current sensor and hacking possibility. Woot!

BOM:
High-power board:

MCU board:


12 Nov 2011 5:58pm GMT

10 Nov 2011

feedPlanet Linux-to-go

Paul 'pfalcon' Sokolovsky: Links for November 2011

Kindle:


Linux kernel module tricks:

10 Nov 2011 3:21pm GMT

19 Oct 2011

feedPlanet OpenEZX

Antonio Ospite: Gnome 3: go to Shell? Not just yet, thanks.

In Debian Unstable the transition to Gnome 3 is taking place; when Gnome 3.0 firstly came out some unnamed geeky users complained loudly about the design decisions of the development team to push strongly towards gnome-shell as a new default UI; gnome-shell was designed focusing on usability (usability is a metric relative to a certain target audience BTW) and simplicity, hiding a lot of details from the users. Obviously you can never make everyone happy so some of us simply happened to be "out of target": you know us computer people (*cough cough*), we like to be in charge and control The Machine... I must admit I still don't have a definitive opinion about the gnome-shell concept, for now I just know that it does not suit me; I am going to try it eventually, maybe I'll get used to it, but in the mean time I need my desktop back just like I shaped it through the years; can this be done without loosing all the good Gnome technologies (Empathy over all of them)?

To be completely fair I have to say that there is little to complain about with Gnome developers, we can still get our good old GNOME desktop fully back by using the fall-back mode based on gnome-panel and live happily ever after, let's take a look at how this can be accomplished.

NOTE: GNOME people state that the fall-back mode is meant for systems with older graphic cards which cannot run gnome-shell, however it can very well be seen as a good opportunity for those who do not want to run gnome-shell just yet.

Getting back to the topic: some minor touches are needed to make the panel look more like what we are used to, maybe some of these settings could even become default for fall-back mode, we'll see.

First, enable fall-back mode (on Debian there is a dedicated session you can choose from the Log-in Manager for that) and change some desktop settings, in a terminal type:

$ gsettings set org.gnome.desktop.session session-name 'gnome-fallback'
$ gsettings set org.gnome.desktop.interface 'menus-have-icons' true
$ gsettings set org.gnome.desktop.interface 'buttons-have-icons' true
$ gsettings set org.gnome.desktop.background 'show-desktop-icons' true

gnome-tweak-tool can be used for some of these settings like shown in the attached images.

Then rearrange the applets on the panel as you please (use Alt-RightClick to access the panel properties), and fix the theming using this patch to have a light panel again (against gnome-themes-standard=3.0.2-1):

$ mkdir $HOME/.themes
$ cd $HOME/.themes
$ cp -r /usr/share/themes/Adwaita Adwaita-fallback
$ cd Adwaita-fallback
$ patch -p1 < $HOME/adwaita-fallback-panel-theme.patch
$ gsettings set org.gnome.desktop.interface 'gtk-theme' 'Adwaita-fallback'

Some final touches for the Metacity window manager and to the clock applet, and we are all set:

$ gconftool-2 --type string --set /apps/metacity/general/focus_mode mouse
$ gconftool-2 --type boolean --set /apps/metacity/general/compositing_manager true
$ gconftool-2 --type string --set /apps/panel3-applets/clock/custom_format '<span color="#333">%a %d %b</span> <b>%H:%M</b>'
$ gconftool-2 --type string --set /apps/panel3-applets/clock/format custom

Ah, in the new gnome-panel based on Gtk3 there are still some details to take care of, I hope issues like that will be addressed and that the panel will be supported for quite some time.

Attached images:
Gnome Shell default look on Debian
gnome-tweak-tool show desktop icons
Gnome 3 fall-back mode default look on Debian
Gnome 3 fall-back mode applets rearranged
Gnome 3 fall-back mode rethemed to have a light panel
Attached files:
text/x-diff iconAdwaita theme patch for fall-back mode

19 Oct 2011 9:37pm GMT

09 Jun 2011

feedPlanet OpenEZX

Michael Lauer: The Eagle Has Landed!

After letting us wait for a bit longer than scheduled (13 days), the hospital initiated the contractions. For the first couple of hours, everything went just perfect, but then the little one got stuck on the way and we had to resort to a cesarean section. Lara Marie Lauer was born 8th of June at 04:41 (AM) with 3460 gramms and 49 cm.

Mummy was still on intensive care and so they gave her to me. I can't express the feelings I had in this very moment. I'm still kind of overwhelmed every time I see her. Thanks for all of you who waited anxiously with me and those who prayed for us. The most important tasks for the near future is getting Mummy to recover and Lara Marie to become accustomed to us and the rest of the outside world.

Please bear with me if in the next time I'm not as responsive as usually :)

Lara Marie Lauer

09 Jun 2011 4:06pm GMT

30 May 2011

feedPlanet OpenEZX

Michael Lauer: German Post on time!

And now for something completely different… while we are all waiting for my baby to arrive (who was scheduled for 25th of May), she just received her first greeting card - together with a personalized bib and a towel (with integrated hood - pretty fancy!) from my good friends at #openmoko-cdevel.

Guys, seeing this card was very heartwarming - it means a lot to me that you share my anticipation, thanks a lot! And I'm 100% sure she will appreciate her gifts… now let's cross fingers it won't take much longer… waiting is the hardest part of it :)

Yours,

Mickey.

30 May 2011 8:54am GMT