18 Nov 2025
Android Developers Blog
How Uber is reducing manual logins by 4 million per year with the Restore Credentials API
Posted by Niharika Arora - Senior Developer Relations Engineer at Google, Thomás Oliveira Horta - Android Engineer at Uber
How Uber is reducing manual logins by 4 million per year with the Restore Credentials API
Uber is the world's largest ridesharing company, getting millions of people from here to there while also supporting food delivery, healthcare transportation, and freight logistics. Simplicity of access is crucial to its success; when users switch to a new device, they expect a seamless transition without needing to log back into the Uber app or go through SMS-based one-time password authentication. This frequent device turnover presents a challenge, as well as an opportunity for strong user retention.
To maintain user continuity, Uber's engineers turned to the Restore Credentials feature, an essential tool for a time when 40% of people in the United States replace their smartphone every year. Following an assessment of user demand and code prototyping, they introduced Restore Credentials support in the Uber rider app. To validate that restoring credentials helps remove friction for re-logins, the Uber team ran a successful A/B experiment for a five-week period. The integration led to a reduction in manual logins that, when projected across Uber's massive user base, is estimated to eliminate 4 million manual logins annually.
Eliminating login friction with Restore Credentials

The Restore Credentials API eliminates the multi-step manual sign in process on new devices.
There were past attempts at account restoration on new devices using solutions like regular data backup and BlockStore, though both solutions required sharing authentication tokens directly, from source device to destination device. Since token information is highly sensitive, these solutions are only used to some extent, to pre-fill login fields on the destination device and reduce some friction during the sign-in flows. Passkeys are also used to provide a secure and fast login method, but their user-initiated nature limits their impact on seamless device transitions.
"Some users don't use the Uber app on a daily basis, but they expect it will just work when they need it," said Thomás Oliveira Horta, an Android engineer at Uber. "Finding out you're logged out just as you open the app to request a ride on your new Android phone can be an unpleasant, off-putting experience."
With Restore Credentials, the engineers were able to bridge this gap. The API generates a unique token on the old device, which is seamlessly and silently moved to the new device when the user restores their app data during the standard onboarding process. This process leverages Android OS's native backup and restore mechanism, ensuring the safe transfer of the restore key along with the app's data. The streamlined approach guarantees a simple and safe account transfer, meeting Uber's security requirements without any additional user input or development overhead.
|
Note: Restore keys and passkeys use the same underlying server implementation. However, when you save them in your database, you must differentiate between them. This distinction is crucial because user-created passkeys can be managed directly by the user, while restore keys are system-managed and hidden from the user interface. |
"With the adoption of Restore Credentials on Uber's rider app, we started seeing consistent usage," Thomás said. "An average of 10,000 unique daily users have signed in with Restore Credentials in the current rollout stage, and they've enjoyed a seamless experience when opening the app for the first time on a new device. We expect that number to double once we expand the rollout to our whole userbase."
Implementation Considerations
"Integration was pretty easy with minor adjustments on the Android side by following the sample code and documentation," Thomás said. "Our app already used Credential Manager for passkeys, and the backend required just a couple of small tweaks. Therefore, we simply needed to update the Credential Manager dependency to its latest version to get access to the new Restore Credentials API. We created a restore key via the same passkey creation flow and when our app is launched on a new device, the app proactively checks for this key by attempting a silent passkey retrieval. If the restore key is found, it is immediately utilized to automatically sign the user in, bypassing any manual login."
Throughout the development process, Uber's engineers navigated a few challenges during implementation-from choosing the right entry point to managing the credential lifecycle on the backend.
Choosing the Restore Credentials entry point
The engineers carefully weighed the tradeoffs between a perfectly seamless user experience and implementation simplicity when selecting which Restore Credentials entry point to use for recovery. Ultimately, they prioritized a solution that offered an ideal balance.
"This can take place during App Launch or in the background during device restoration and setup, using BackupAgent," Thomás said. "The background login entry point is more seamless for the user, but it presented challenges with background operations and required usage of the BackupAgent API, which would have led to increased complexity in a codebase as large as Uber's." They decided to implement the feature during the first app launch, which was significantly faster than the manual login.
Addressing server-side challenges
A few server-side challenges arose during integration with the backend WebAuthn APIs, as their design assumed user verification would always be required, and that all credentials would be listed in a user's account settings; neither of these assumptions worked for the non-user-managed Restore Credential keys.
The Uber team resolved this by making minor changes to the WebAuthn services, creating new credential types to distinguish passkeys from Restore Credentials and process them appropriately.
Managing the Restore Credentials lifecycle
Uber's engineers faced several challenges in managing the credential keys on the backend, with specialized support from backend engineer Ryan O'Laughlin:
-
Preventing orphaned keys: A significant challenge was defining a strategy for deleting registered Public Keys to prevent them from becoming "orphaned." For example, uninstalling the app deletes the local credential, but because this action doesn't signal the backend, it leaves an unused key on the server.
-
Balancing key lifespan: Keys needed a "time to live" that was long enough to handle edge cases. For example, if a user goes through a backup and restore, then manually logs out from the old device, the key is deleted from that old device. However, the key must remain valid on the server so the new device can still use it.
-
Supporting multiple devices: Since a user might have multiple devices (and could initiate a backup and restore from any of them), the backend needed to support multiple Restore Credentials per user (one for each device).
Uber's engineers addressed these challenges by establishing rules for server-side key deletion based on new credential registration and credential usage.
The feature went from design to delivery in a rapid two-month development and testing process. Afterward, a five-week A/B experiment (time to validate the feature with users) went smoothly and yielded undeniable results.
Preventing user drop-off with Restore Credentials
By eliminating manual logins on new devices, Uber retained users who might have otherwise abandoned the sign-in flow on a new device. This boost in customer ease was reflected in a wide array of improvements, and though they may seem slight at a glance, the impact is massive at the scale of Uber's user base:
-
3.4% decrease in manual logins (SMS OTP, passwords, social login).
-
1.2% reduction in expenses for logins requiring SMS OTP.
-
0.575% increase in Uber's access rate (% of devices that successfully reached the app home screen).
-
0.614% rise in devices with completed trips.
Today, Restore Credentials is well on its way to becoming a standard part of Uber's rider app, with over 95% of users in the trial group registered.
[UI flow]
During new device setup, users can restore app data and credentials from a backup. After selecting Uber for restoration and the background process finishes, the app will automatically sign the user in on the new device's first launch.
The invisible yet massive impact of Restore Credentials
In the coming months, Uber plans to expand the integration of Restore Credentials. Projecting from the trial's results, they estimate the change will eliminate 4 million manual logins annually. By simplifying app access and removing a key pain point, they are actively building a more satisfied and loyal customer base, one ride at a time.
"Integrating Google's RestoreCredentials allowed us to deliver the seamless 'it just works' experience our users expect on a new device," said Matt Mueller, Lead Project Manager for Core Identity at Uber. "This directly translated to a measurable increase in revenue, proving that reducing login friction is key to user engagement and retention."
Ready to enhance your app's login experience?
Learn how to facilitate a seamless login experience when switching devices with Restore Credentials and read more in the blog post. In the latest canary of the Android Studio Otter you can validate your integration, as new features help mock the backup and restoring mechanisms.
If you are new to Credential Manager, you can refer to our official documentation, codelab and samples for help with integration.
18 Nov 2025 10:00pm GMT
TalkAndroid
Boba Story Lid Recipes – 2025
Look no further for all the latest Boba Story Lid Recipes. They are all right here!
18 Nov 2025 5:11pm GMT
Dice Dreams Free Rolls – Updated Daily
Get the latest Dice Dreams free rolls links, updated daily! Complete with a guide on how to redeem the links.
18 Nov 2025 5:10pm GMT
Android Developers Blog
Configure and troubleshoot R8 Keep Rules

Posted by Ajesh R Pai - Developer Relations Engineer & Ben Weiss - Senior Developer Relations Engineer

In modern Android development, shipping a small, fast, and secure application is a fundamental user expectation. The Android build system's primary tool for achieving this is the R8 optimizer, the compiler that handles dead code and resource removal for shrinking, code renaming or minification, and app optimization.
Enabling R8 is a critical step in preparing an app for release, but it requires developers to provide guidance in the form of "Keep Rules."
After reading this article, check out the Performance Spotlight Week video on enabling, debugging and troubleshooting the R8 optimizer on YouTube.
Why Keep Rules are needed
The need to write Keep Rules stems from a core conflict: R8 is a static analysis tool, but Android apps often rely on dynamic execution patterns like reflection or calls in and out of native code using the JNI (Java Native Interface).
R8 builds a graph of used code by analyzing direct calls. When code is accessed in a dynamic way, R8's static analysis cannot predict that and it will identify that code as unused and remove it, leading to runtime crashes.
A keep rule is an explicit instruction to the R8 compiler, stating: "This specific class, method, or field is an entry point that will be accessed dynamically at runtime. You must keep it, even if you cannot find a direct reference to it."
See the official guide for more details on Keep Rules.
Where to write Keep Rules
Custom Keep Rules for an application are written in text file. By convention, this file is named proguard-rules.pro and is located in the root of the app or library module. This file is then specified in your module's build.gradle.kts file's release build type.
release {
isShrinkResources = true
isMinifyEnabled = true
proguardFiles(
getDefaultProguardFile("proguard-android-optimize.txt"),
"proguard-rules.pro",
)
}
Use the correct default file
The getDefaultProguardFile method imports a default set of rules provided by the Android SDK. When using the wrong file your app might not be optimized. Make sure to use proguard-android-optimize.txt. This file provides the default Keep Rules for standard Android components and enables R8's code optimizations. The outdated proguard-android.txt only provides the Keep Rules but does not enable R8's optimizations.
Since this is a serious performance problem, we are starting to warn developers about using the wrong file, starting in Android Studio Narwhal 3 Feature Drop. And starting with the Android Gradle Plugin Version 9.0 we're no longer supporting the outdated proguard-android.txt file. So make sure you upgrade to the optimized version.
How to write Keep Rules
A keep rule consists of three main parts:
-
An option like -keep or -keepclassmembers
-
Optional modifiers like allowshrinking
-
A class specification that defines the code to match
For the complete syntax and examples, refer to the guidance to add Keep Rules.
Keep Rule anti-patterns
It's important to know about best practices, but also about anti-patterns. These anti-patterns often arise from misunderstandings or troubleshooting shortcuts and can be catastrophic for a production build's performance.
Global options
These flags are global toggles that should never be used in a release build. They are only for temporary debugging to isolate a problem.
Using -dontotptimize effectively disables R8's performance optimizations leading to a slower app.
When using -dontobfuscate you disable all renaming and using -dontshrink turns off dead code removal. Both of these global rules increase app size.
Avoid using these global flags in a production environment wherever possible for a more performant app user experience.
Overly broad keep rules
The easiest way to nullify R8's benefits is to write overly-broad Keep Rules. Keep rules like the one below instruct the R8 optimizer to not shrink, not obfuscate, and not optimize any class in this package or any of its sub-packages. This completely removes R8's benefits for that entire package. Try to write narrow and specific Keep Rules instead.
-keep class com.example.package.** { *;} // WIDE KEEP RULES CAUSE PROBLEMS
The inversion operator (!)
The inversion operator (!) seems like a powerful way to exclude a package from a rule. But it's not that simple. Take this example:
-keep class !com.example.my_package.** { *; } // USE WITH CAUTION
You might think that this rule means "do not keep classes in com.example.package." But it actually means "keep every class, method and property in the entire application that is not in com.example.package." If that came as a surprise to you, best check for any negations in your R8 configuration.
Redundant rules for Android components
Another common mistake is to manually add Keep Rules for your app's Activities, Services, or BroadcastReceivers. This is unnecessary. The default proguard-android-optimize.txt file already includes the relevant rules for these standard Android components to work out of the box.
Also many libraries bring their own Keep Rules. So you should not have to write your own rules for these. In case there is a problem with Keep Rules from a library you're using, it is best to reach out to the library author to see what the problem is.
Keep Rule best practices
Now that you know what not to do, let's talk about best practices.
Write narrow Keep Rules
Good Keep Rules should be as narrow and specific as possible. They should preserve only what is necessary, allowing R8 to optimize everything else.
|
Rule |
Quality |
|---|---|
-keep class com.example.** { ; } |
Low: Keeps an entire package and its subpackages |
-keep class com.example.MyClass { ; } |
Low: Keeps an entire class which is likely still too wide |
-keepclassmembers class com.example.MyClass { private java.lang.String secretMessage; public void onNativeEvent(java.lang.String); } |
High: Only relevant methods and properties from a specific class are kept |
Use common ancestors
Instead of writing separate Keep Rules for multiple different data models, write one rule that targets a common base class or interface. The below rule tells R8 to keep any members of classes that implement this interface and is highly scalable.
# Keep all fields of any class that implements SerializableModel
-keepclassmembers class * implements com.example.models.SerializableModel {
<fields>;
}
Use Annotations to target multiple classes
Create a custom annotation (e.g., @Serialize) and use it to "tag" classes that need their fields preserved. This is another clean, declarative, and highly scalable pattern. You can create Keep Rules for already existing annotations from frameworks you're using as well.
# Keep all fields of any class annotated with @Serialize
-keepclassmembers class * {
@com.example.annotations.Serialize <fields>;
}
Choose the right Keep Option
The Keep Option is the most critical part of the rule. Choosing the wrong one can needlessly disable optimization.
|
Keep Option |
What It Does |
|
-keep |
Prevents the class and members mentioned in the declaration from being removed or renamed. |
|
-keepclassmembers |
Prevents the specified members from being removed or renamed, but allows the class itself to be removed but only on classes which are not otherwise removed. |
|
-keepclasseswithmembers |
A combination: Keeps the class and its members, only if all the specified members are present. |
You can find more about the keep option in our documentation for Keep Options.
Allow optimization with Modifiers
Modifiers like allowshrinking and allowobfuscation relax a broad -keep rule, giving optimization power back to R8. For example, if a legacy library forces you to use -keep on an entire class, you might be able to reclaim some optimization by allowing shrinking and obfuscation:
# Keep this class, but allow R8 to remove it if it's unused and allow R8 to rename it.
-keep,allowshrinking,allowobfuscation class com.example.LegacyClass
Add global options for additional optimization
Beyond Keep Rules, you can add global flags to your R8 configuration file to encourage even more optimization.
-repackageclasses is a powerful option that instructs R8 to move all obfuscated classes into a single package. This saves significant space in the DEX file by removing redundant package name strings.
-allowaccessmodification allows R8 to widen access (e.g., private to public) to enable more aggressive inlining. This is now enabled by default when using proguard-android-optimize.txt.
Warning: Library authors must never add these global optimization flags to their consumer rules, as they would be forcibly applied to the entire app.
And to make it even more clear, in version 9.0 of the Android Gradle Plugin we're going to start ignoring global optimization flags from libraries altogether.
Best practices for libraries
Every Android app relies on libraries one way or another. So let's talk about best practices for libraries.
For library developers
If your library uses reflection or JNI, you have the responsibility to provide the necessary Keep Rules to its consumers. These rules are placed in a consumer-rules.pro file, which is then automatically bundled inside the library's AAR file.
android {
defaultConfig {
consumerProguardFiles("consumer-rules.pro")
}
...
}
For library consumers
Filter out problematic Keep Rules
If you must use a library that includes problematic Keep Rules, you can filter them out in your build.gradle.kts file starting with AGP 9.0 This tells R8 to ignore the rules coming from a specific dependency.
release {
optimization.keepRules {
// Ignore all consumer rules from this specific library
it.ignoreFrom("com.somelibrary:somelibrary")
}
}
The best Keep Rule is no Keep Rule
The ultimate R8 configuration strategy is to remove the need to write Keep Rules altogether. For many apps can be achieved by choosing modern libraries that favor code generation over reflection. With code generation, the optimizer can more easily determine what code is actually used at runtime and what code can be removed. Also not using any dynamic reflection means no "hidden" entry points, and therefore, no Keep Rules are needed. When choosing a new library, always prefer a solution that uses code generation over reflection.
For more information about how to choose libraries, check choose library wisely.
Debugging and troubleshooting your R8 configuration
When R8 removes code it should have kept, or your APK is larger than expected, use these tools to diagnose the problem.
Find duplicate and global Keep Rules
Because R8 merges rules from dozens of sources, it can be hard to know what the "final" ruleset is. Adding this flag to your proguard-rules.pro file generates a complete report:
# Outputs the final, merged set of rules to the specified file
-printconfiguration build/outputs/logs/configuration.txt
You can search this file to find redundant rules or trace a problematic rule (like -dontoptimize) back to the specific library that included it.
Ask R8: Why are you keeping this?
If a class you expected to be removed is still in your app, R8 can tell you why. Just add this rule:
# Asks R8 to explain why it's keeping a specific class
class com.example.MyUnusedClass
-whyareyoukeeping
During the build, R8 will print the exact chain of references that caused it to keep that class, allowing you to trace the reference and adjust your rules.
For a full guide, check out the troubleshoot R8 section.
Next steps
R8 is a powerful tool for enhancing Android app performance. Its effectiveness, depends on a correct understanding of its operation as a static analysis engine.
By writing specific, member-level rules, leveraging ancestors and annotations, and carefully choosing the right keep options, you can preserve exactly what is necessary. The most advanced practice is to eliminate the need for rules entirely by choosing modern, codegen-based libraries over their reflection-based predecessors.
As you're following along Performance Spotlight Week, make sure to check out today's Spotlight Week video on YouTube and continue with our R8 challenge. Use #optimizationEnabled for any questions on enabling or troubleshooting R8. We're here to help.
It's time to see the benefits for yourself.
We challenge you to enable R8 full mode for your app today.
-
Follow our developer guides to get started: Enable app optimization.
-
Check if you still use proguard-android.txt and replace it with proguard-android-optimize.txt.
-
Then, measure the impact. Don't just feel the difference, verify it. Measure your performance gains by adapting the code from our Macrobenchmark sample app on GitHub to measure your startup times before and after.
We're confident you'll see a meaningful improvement in your app's performance.
While you're at it, use the social tag #AskAndroid to bring your questions. Throughout the week our experts are monitoring and answering your questions.
Stay tuned for tomorrow, where we'll talk about Profile Guided Optimization with Baseline and Startup Profiles, share how Compose rendering performance improved over the past releases and share performance considerations for background work.
18 Nov 2025 5:00pm GMT
Gemini 3 is now available for AI assistance in Android Studio

Posted by Tor Norbye - Senior Director of Engineering

The Gemini 3 Pro model, released today and engineered for better coding and agentic experiences, is now available for AI assistance in the latest version of Android Studio Otter. Android Studio is the best place for professional Android developers to use Gemini 3 for superior performance in Agent Mode, streamlined development workflows, and advanced problem solving capabilities. With agentic AI assistance to help you with boilerplate and complex development tasks, Android Studio helps you focus on what you do best-creating high quality apps for your users.
To get started with Gemini 3 Pro for Android development, download or update to the latest version of Android Studio Otter. For developers using Gemini in Android Studio at no-cost (Default Model), we are rolling out limited access to Gemini 3 with a 1 million token size window. For higher usage rate limits and longer sessions with Agent Mode, use a Gemini API key to leverage Gemini 3 in Android Studio for the highest tier of AI capability.
Adding a Gemini API key in Android Studio
This week we're rolling out Gemini 3 access for organizations, starting with users who have Gemini Code Assist Enterprise licenses. Your IT administrator will need to enable access to preview models through the Google Cloud console, and you'll need to sign up for the waitlist.
Try Gemini 3 Pro in Android Studio, and let us and the Android developer community know what you think. You can follow us across LinkedIn, Blog, YouTube, and X. We can't wait to see what you build!
18 Nov 2025 4:06pm GMT
TalkAndroid
Why teens are ditching Instagram for WhatsApp channels
Instagram may have the filters, Reels, and influencers - but when it comes to the next generation, another…
18 Nov 2025 7:30am GMT
15 Oct 2025
Planet Maemo
Dzzee 1.9.0 for N800/N810/N900/N9/Leste
15 Oct 2025 11:31am GMT
05 Jun 2025
Planet Maemo
Mobile blogging, the past and the future
This blog has been running more or less continuously since mid-nineties. The site has existed in multiple forms, and with different ways to publish. But what's common is that at almost all points there was a mechanism to publish while on the move.
Psion, documents over FTP
In the early 2000s we were into adventure motorcycling. To be able to share our adventures, we implemented a way to publish blogs while on the go. The device that enabled this was the Psion Series 5, a handheld computer that was very much a device ahead of its time.

The Psion had a reasonably sized keyboard and a good native word processing app. And battery life good for weeks of usage. Writing while underway was easy. The Psion could use a mobile phone as a modem over an infrared connection, and with that we could upload the documents to a server over FTP.
Server-side, a cron job would grab the new documents, converting them to HTML and adding them to our CMS.
In the early days of GPRS, getting this to work while roaming was quite tricky. But the system served us well for years.
If we wanted to include photos to the stories, we'd have to find an Internet cafe.
- To the Alps is a post from these times. Lots more in the motorcycling category
SMS and MMS
For an even more mobile setup, I implemented an SMS-based blogging system. We had an old phone connected to a computer back in the office, and I could write to my blog by simply sending a text. These would automatically end up as a new paragraph in the latest post. If I started the text with NEWPOST, an empty blog post would be created with the rest of that message's text as the title.
- In the Caucasus is a good example of a post from this era
As I got into neogeography, I could also send a NEWPOSITION message. This would update my position on the map, connecting weather metadata to the posts.
As camera phones became available, we wanted to do pictures too. For the Death Monkey rally where we rode minimotorcycles from Helsinki to Gibraltar, we implemented an MMS-based system. With that the entries could include both text and pictures. But for that you needed a gateway, which was really only realistic for an event with sponsors.
- Mystery of the Missing Monkey is typical. Some more in Internet Archive
Photos over email
A much easier setup than MMS was to slightly come back to the old Psion setup, but instead of word documents, sending email with picture attachments. This was something that the new breed of (pre-iPhone) smartphones were capable of. And by now the roaming question was mostly sorted.
And so my blog included a new "moblog" section. This is where I could share my daily activities as poor-quality pictures. Sort of how people would use Instagram a few years later.

- Internet Archive has some of my old moblogs but nowadays, I post similar stuff on Pixelfed
Pause
Then there was sort of a long pause in mobile blogging advancements. Modern smartphones, data roaming, and WiFi hotspots had become ubiquitous.
In the meanwhile the blog also got migrated to a Jekyll-based system hosted on AWS. That means the old Midgard-based integrations were off the table.
And I traveled off-the-grid rarely enough that it didn't make sense to develop a system.
But now that we're sailing offshore, that has changed. Time for new systems and new ideas. Or maybe just a rehash of the old ones?
Starlink, Internet from Outer Space
Most cruising boats - ours included - now run the Starlink satellite broadband system. This enables full Internet, even in the middle of an ocean, even video calls! With this, we can use normal blogging tools. The usual one for us is GitJournal, which makes it easy to write Jekyll-style Markdown posts and push them to GitHub.
However, Starlink is a complicated, energy-hungry, and fragile system on an offshore boat. The policies might change at any time preventing our way of using it, and also the dishy itself, or the way we power it may fail.
But despite what you'd think, even on a nerdy boat like ours, loss of Internet connectivity is not an emergency. And this is where the old-style mobile blogging mechanisms come handy.
- Any of the 2025 Atlantic crossing posts is a good example of this setup in action
Inreach, texting with the cloud
Our backup system to Starlink is the Garmin Inreach. This is a tiny battery-powered device that connects to the Iridium satellite constellation. It allows tracking as well as basic text messaging.
When we head offshore we always enable tracking on the Inreach. This allows both our blog and our friends ashore to follow our progress.
I also made a simple integration where text updates sent to Garmin MapShare get fetched and published on our blog. Right now this is just plain text-based entries, but one could easily implement a command system similar to what I had over SMS back in the day.
One benefit of the Inreach is that we can also take it with us when we go on land adventures. And it'd even enable rudimentary communications if we found ourselves in a liferaft.
- There are various InReach integration hacks that could be used for more sophisticated data transfer
Sailmail and email over HF radio
The other potential backup for Starlink failures would be to go seriously old-school. It is possible to get email access via a SSB radio and a Pactor (or Vara) modem.
Our boat is already equipped with an isolated aft stay that can be used as an antenna. And with the popularity of Starlink, many cruisers are offloading their old HF radios.
Licensing-wise this system could be used either as a marine HF radio (requiring a Long Range Certificate), or amateur radio. So that part is something I need to work on. Thankfully post-COVID, radio amateur license exams can be done online.
With this setup we could send and receive text-based email. The Airmail application used for this can even do some automatic templating for position reports. We'd then need a mailbox that can receive these mails, and some automation to fetch and publish.
- Sailmail and No Foreign Land support structured data via email to update position. Their formats could be useful inspiration
05 Jun 2025 12:00am GMT
16 Oct 2024
Planet Maemo
Adding buffering hysteresis to the WebKit GStreamer video player
The <video> element implementation in WebKit does its job by using a multiplatform player that relies on a platform-specific implementation. In the specific case of glib platforms, which base their multimedia on GStreamer, that's MediaPlayerPrivateGStreamer.
The player private can have 3 buffering modes:
- On-disk buffering: This is the typical mode on desktop systems, but is frequently disabled on purpose on embedded devices to avoid wearing out their flash storage memories. All the video content is downloaded to disk, and the buffering percentage refers to the total size of the video. A GstDownloader element is present in the pipeline in this case. Buffering level monitoring is done by polling the pipeline every second, using the
fillTimerFired()method. - In-memory buffering: This is the typical mode on embedded systems and on desktop systems in case of streamed (live) content. The video is downloaded progressively and only the part of it ahead of the current playback time is buffered. A GstQueue2 element is present in the pipeline in this case. Buffering level monitoring is done by listening to GST_MESSAGE_BUFFERING bus messages and using the buffering level stored on them. This is the case that motivates the refactoring described in this blog post, what we actually wanted to correct in Broadcom platforms, and what motivated the addition of hysteresis working on all the platforms.
- Local files: Files, MediaStream sources and other special origins of video don't do buffering at all (no GstDownloadBuffering nor GstQueue2 element is present on the pipeline). They work like the on-disk buffering mode in the sense that
fillTimerFired()is used, but the reported level is relative, much like in the streaming case. In the initial version of the refactoring I was unaware of this third case, and only realized about it when tests triggered the assert that I added to ensure that the on-disk buffering method was working in GST_BUFFERING_DOWNLOAD mode.
The current implementation (actually, its wpe-2.38 version) was showing some buffering problems on some Broadcom platforms when doing in-memory buffering. The buffering levels monitored by MediaPlayerPrivateGStreamer weren't accurate because the Nexus multimedia subsystem used on Broadcom platforms was doing its own internal buffering. Data wasn't being accumulated in the GstQueue2 element of playbin, because BrcmAudFilter/BrcmVidFilter was accepting all the buffers that the queue could provide. Because of that, the player private buffering logic was erratic, leading to many transitions between "buffer completely empty" and "buffer completely full". This, it turn, caused many transitions between the HaveEnoughData, HaveFutureData and HaveCurrentData readyStates in the player, leading to frequent pauses and unpauses on Broadcom platforms.

So, one of the first thing I tried to solve this issue was to ask the Nexus PlayPump (the subsystem in charge of internal buffering in Nexus) about its internal levels, and add that to the levels reported by GstQueue2. There's also a GstMultiqueue in the pipeline that can hold a significant amount of buffers, so I also asked it for its level. Still, the buffering level unstability was too high, so I added a moving average implementation to try to smooth it.
All these tweaks only make sense on Broadcom platforms, so they were guarded by ifdefs in a first version of the patch. Later, I migrated those dirty ifdefs to the new quirks abstraction added by Phil. A challenge of this migration was that I needed to store some attributes that were considered part of MediaPlayerPrivateGStreamer before. They still had to be somehow linked to the player private but only accessible by the platform specific code of the quirks. A special HashMap attribute stores those quirks attributes in an opaque way, so that only the specific quirk they belong to knows how to interpret them (using downcasting). I tried to use move semantics when storing the data, but was bitten by object slicing when trying to move instances of the superclass. In the end, moving the responsibility of creating the unique_ptr that stored the concrete subclass to the caller did the trick.
Even with all those changes, undesirable swings in the buffering level kept happening, and when doing a careful analysis of the causes I noticed that the monitoring of the buffering level was being done from different places (in different moments) and sometimes the level was regarded as "enough" and the moment right after, as "insufficient". This was because the buffering level threshold was one single value. That's something that a hysteresis mechanism (with low and high watermarks) can solve. So, a logical level change to "full" would only happen when the level goes above the high watermark, and a logical level change to "low" when it goes under the low watermark level.
For the threshold change detection to work, we need to know the previous buffering level. There's a problem, though: the current code checked the levels from several scattered places, so only one of those places (the first one that detected the threshold crossing at a given moment) would properly react. The other places would miss the detection and operate improperly, because the "previous buffering level value" had been overwritten with the new one when the evaluation had been done before. To solve this, I centralized the detection in a single place "per cycle" (in updateBufferingStatus()), and then used the detection conclusions from updateStates().
So, with all this in mind, I refactored the buffering logic as https://commits.webkit.org/284072@main, so now WebKit GStreamer has a buffering code much more robust than before. The unstabilities observed in Broadcom devices were gone and I could, at last, close Issue 1309.
16 Oct 2024 6:12am GMT
18 Sep 2022
Planet Openmoko
Harald "LaF0rge" Welte: Deployment of future community TDMoIP hub
I've mentioned some of my various retronetworking projects in some past blog posts. One of those projects is Osmocom Community TDM over IP (OCTOI). During the past 5 or so months, we have been using a number of GPS-synchronized open source icE1usb interconnected by a new, efficient but strill transparent TDMoIP protocol in order to run a distributed TDM/PDH network. This network is currently only used to provide ISDN services to retronetworking enthusiasts, but other uses like frame relay have also been validated.
So far, the central hub of this OCTOI network has been operating in the basement of my home, behind a consumer-grade DOCSIS cable modem connection. Given that TDMoIP is relatively sensitive to packet loss, this has been sub-optimal.
Luckily some of my old friends at noris.net have agreed to host a new OCTOI hub free of charge in one of their ultra-reliable co-location data centres. I'm already hosting some other machines there for 20+ years, and noris.net is a good fit given that they were - in their early days as an ISP - the driving force in the early 90s behind one of the Linux kernel ISDN stracks called u-isdn. So after many decades, ISDN returns to them in a very different way.
Side note: In case you're curious, a reconstructed partial release history of the u-isdn code can be found on gitea.osmocom.org
But I digress. So today, there was the installation of this new OCTOI hub setup. It has been prepared for several weeks in advance, and the hub contains two circuit boards designed entirely only for this use case. The most difficult challenge was the fact that this data centre has no existing GPS RF distribution, and the roof is ~ 100m of CAT5 cable (no fiber!) away from the roof. So we faced the challenge of passing the 1PPS (1 pulse per second) signal reliably through several steps of lightning/over-voltage protection into the icE1usb whose internal GPS-DO serves as a grandmaster clock for the TDM network.
The equipment deployed in this installation currently contains:
-
a rather beefy Supermicro 2U server with EPYC 7113P CPU and 4x PCIe, two of which are populated with Digium TE820 cards resulting in a total of 16 E1 ports
-
an icE1usb with RS422 interface board connected via 100m RS422 to an Ericsson GPS03 receiver. There's two layers of of over-voltage protection on the RS422 (each with gas discharge tubes and TVS) and two stages of over-voltage protection in the coaxial cable between antenna and GPS receiver.
-
a Livingston Portmaster3 RAS server
-
a Cisco AS5400 RAS server
For more details, see this wiki page and this ticket
Now that the physical deployment has been made, the next steps will be to migrate all the TDMoIP links from the existing user base over to the new hub. We hope the reliability and performance will be much better than behind DOCSIS.
In any case, this new setup for sure has a lot of capacity to connect many more more users to this network. At this point we can still only offer E1 PRI interfaces. I expect that at some point during the coming winter the project for remote TDMoIP BRI (S/T, S0-Bus) connectivity will become available.
Acknowledgements
I'd like to thank anyone helping this effort, specifically * Sylvain "tnt" Munaut for his work on the RS422 interface board (+ gateware/firmware) * noris.net for sponsoring the co-location * sysmocom for sponsoring the EPYC server hardware
18 Sep 2022 10:00pm GMT
08 Sep 2022
Planet Openmoko
Harald "LaF0rge" Welte: Progress on the ITU-T V5 access network front
Almost one year after my post regarding first steps towards a V5 implementation, some friends and I were finally able to visit Wobcom, a small German city carrier and pick up a lot of decommissioned POTS/ISDN/PDH/SDH equipment, primarily V5 access networks.
This means that a number of retronetworking enthusiasts now have a chance to play with Siemens Fastlink, Nokia EKSOS and DeTeWe ALIAN access networks/multiplexers.
My primary interest is in Nokia EKSOS, which looks like an rather easy, low-complexity target. As one of the first steps, I took PCB photographs of the various modules/cards in the shelf, take note of the main chip designations and started to search for the related data sheets.
The results can be found in the Osmocom retronetworking wiki, with https://osmocom.org/projects/retronetworking/wiki/Nokia_EKSOS being the main entry page, and sub-pages about
In short: Unsurprisingly, a lot of Infineon analog and digital ICs for the POTS and ISDN ports, as well as a number of Motorola M68k based QUICC32 microprocessors and several unknown ASICs.
So with V5 hardware at my disposal, I've slowly re-started my efforts to implement the LE (local exchange) side of the V5 protocol stack, with the goal of eventually being able to interface those V5 AN with the Osmocom Community TDM over IP network. Once that is in place, we should also be able to offer real ISDN Uk0 (BRI) and POTS lines at retrocomputing events or hacker camps in the coming years.
08 Sep 2022 10:00pm GMT
Harald "LaF0rge" Welte: Clock sync trouble with Digium cards and timing cables
If you have ever worked with Digium (now part of Sangoma) digital telephony interface cards such as the TE110/410/420/820 (single to octal E1/T1/J1 PRI cards), you will probably have seen that they always have a timing connector, where the timing information can be passed from one card to another.
In PDH/ISDN (or even SDH) networks, it is very important to have a synchronized clock across the network. If the clocks are drifting, there will be underruns or overruns, with associated phase jumps that are particularly dangerous when analog modem calls are transported.
In traditional ISDN use cases, the clock is always provided by the network operator, and any customer/user side equipment is expected to synchronize to that clock.
So this Digium timing cable is needed in applications where you have more PRI lines than possible with one card, but only a subset of your lines (spans) are connected to the public operator. The timing cable should make sure that the clock received on one port from the public operator should be used as transmit bit-clock on all of the other ports, no matter on which card.
Unfortunately this decades-old Digium timing cable approach seems to suffer from some problems.
bursty bit clock changes until link is up
The first problem is that downstream port transmit bit clock was jumping around in bursts every two or so seconds. You can see an oscillogram of the E1 master signal (yellow) received by one TE820 card and the transmit of the slave ports on the other card at https://people.osmocom.org/laforge/photos/te820_timingcable_problem.mp4
As you can see, for some seconds the two clocks seem to be in perfect lock/sync, but in between there are periods of immense clock drift.
What I'd have expected is the behavior that can be seen at https://people.osmocom.org/laforge/photos/te820_notimingcable_loopback.mp4 - which shows a similar setup but without the use of a timing cable: Both the master clock input and the clock output were connected on the same TE820 card.
As I found out much later, this problem only occurs until any of the downstream/slave ports is fully OK/GREEN.
This is surprising, as any other E1 equipment I've seen always transmits at a constant bit clock irrespective whether there's any signal in the opposite direction, and irrespective of whether any other ports are up/aligned or not.
But ok, once you adjust your expectations to this Digium peculiarity, you can actually proceed.
clock drift between master and slave cards
Once any of the spans of a slave card on the timing bus are fully aligned, the transmit bit clocks of all of its ports appear to be in sync/lock - yay - but unfortunately only at the very first glance.
When looking at it for more than a few seconds, one can see a slow, continuous drift of the slave bit clocks compared to the master :(
Some initial measurements show that the clock of the slave card of the timing cable is drifting at about 12.5 ppb (parts per billion) when compared against the master clock reference.
This is rather disappointing, given that the whole point of a timing cable is to ensure you have one reference clock with all signals locked to it.
The work-around
If you are willing to sacrifice one port (span) of each card, you can work around that slow-clock-drift issue by connecting an external loopback cable. So the master card is configured to use the clock provided by the upstream provider. Its other ports (spans) will transmit at the exact recovered clock rate with no drift. You can use any of those ports to provide the clock reference to a port on the slave card using an external loopback cable.
In this setup, your slave card[s] will have perfect bit clock sync/lock.
Its just rather sad that you need to sacrifice ports just for achieving proper clock sync - something that the timing connectors and cables claim to do, but in reality don't achieve, at least not in my setup with the most modern and high-end octal-port PCIe cards (TE820).
08 Sep 2022 10:00pm GMT




