27 Aug 2026

feedPlanet Mozilla

Firefox Tooling Announcements: MozPhab 2.18.0 Released

Bugs resolved in Moz-Phab 2.18.0:

Discuss these changes in #engineering-workflow on Slack or #Conduit Matrix.

1 post - 1 participant

Read full topic

27 Aug 2026 6:43pm GMT

26 Aug 2026

feedPlanet Mozilla

Serge Guelton: Pros and Cons of Unified Build

Unified builds (also know as Jumbo Builds) is a build techniques that aims at improving build time through the concatenation of several sources as a single unified source before compilation.

The goal is obtained through implicit caching of header instantiation, although it implies a trade-off with parallelism.

Let's illustrate this behavior through a simple example, two codes that implement variation of the same approach:

/* algo0.cpp */
#include <iostream>
#include <string>
#include <vector>
void translate(std::vector<std::string>& w, void (&t)(std::string&));
void translate(std::vector<std::string>& w_out, std::vector<std::string> const & w_in, void (&t)(std::string&)) {
    std::cout << "[log] through transform\n";
    w_out = w_in;
    translate(w_out, t);
}

/* algo1.cpp */
#include <algorithm>
#include <iostream>
#include <string>
#include <vector>
void translate(std::vector<std::string>& w, void (&t)(std::string&)) {
    std::cout << "[log] through for_each\n";
    std::for_each(w.begin(), w.end(), [&t](std::string& s) { t(s); });
}

Compiling individual files take the following times:

% hyperfine --warmup 5 "/usr/bin/clang++ -O2 algo0.cpp -c"
Benchmark 1: /usr/bin/clang++ -O2 algo0.cpp -c
  Time (mean ± σ):     267.7 ms ±   7.4 ms    [User: 234.6 ms, System: 30.4 ms]
  Range (min … max):   259.7 ms … 277.3 ms    11 runs

% hyperfine --warmup 5 "/usr/bin/clang++ -O2 algo1.cpp -c"
Benchmark 1: /usr/bin/clang++ -O2 algo1.cpp -c
  Time (mean ± σ):     173.6 ms ±  46.3 ms    [User: 149.6 ms, System: 22.1 ms]
  Range (min … max):   130.4 ms … 231.6 ms    13 runs

Creation of the unified file is just a matter of invoking cat, let's benchmark the compilation of the unified source:

% cat algo{0,1}.cpp > unified_algo.cpp
% hyperfine --warmup 5 "/usr/bin/clang++ -O2 unified_algo.cpp -c"
Benchmark 1: /usr/bin/clang++ -O2 unified_algo.cpp -c
  Time (mean ± σ):     223.8 ms ±  64.5 ms    [User: 193.0 ms, System: 28.4 ms]
  Range (min … max):   160.8 ms … 301.8 ms    10 runs

In that simple case, the weight of headers with respect to actual user code is such that compilation of the unified file takes almost the same time as the max compilation time among each individual file. That's roughly a 1.97x speedup on compilation time.

That's the promise given by unified builds. And it's a promise held.

Now let's have a look at the consequences of that deal.

Beforehand, we still need to introduce another parameter tied to unified builds: the unification parameter, say P. That parameter bounds the number of files that are unified together. Let's imagine we have a hundred of individual source files compiled with exactly the same compilation flags. Setting P to 5 leads to the generation of 20 unified sources compiled independently.

Remember the parameter P.

Quality of the Generated Code

Let's create a shared object from algo{0,1}.o (this implies a recompilation with -fPIC of the sources):

% /usr/bin/clang++ -O2 algo0.cpp -c -fPIC
% /usr/bin/clang++ -O2 algo1.cpp -c -fPIC
% /usr/bin/clang++ -shared algo{0,1}.o -fPIC -o algo.so

And do the same from unified_algo.o:

% /usr/bin/clang++ -O2 unified_algo.cpp -c -fPIC
% /usr/bin/clang++ -shared unified_algo.o -fPIC -o unified_algo.so

After stripping, comparing the size of the binaries yield a difference of a few bytes. After disassembling, it turns out the compiler decides to inline the call to void translate(std::vector<std::string>& w, void (&t)(std::string&)) from algo0.cpp when compiling the unified source, something the compiler cannot do when doing split compilation, as it does not know anything about the implementation of that function.

Interestingly, compiling with -flto=thin still lead the compiler instantiation through different optimization path.

Falling back to -flto=full finally yields to the same shared object, which makes sense because Full LTO is very close to performing source unification at the bytecode level and our sources are very simple. It's not a given though because the actual optimisation pipeline is still different in the two scenario.

Why does it matter? Depending on the value of P, the compiler will see different sets of files per unified file, which will result in different binary code. It's actually even worse: depending on the way we fill those unification sets, event with the same parameter P, we end up with different binaries. Let's call that the reunifying problem.

Even if we have an algorithm that seems to guarantee reproducibility, for instance working on a sorted list of files with a fixed P, variation can arise: the introduction of a new source file can lead to changes in every unified file (e.g. if the split is done by chunks and the new file ends up at the beginning of the file list).

So unified builds tend to improve performance, but they do not interact in a gentle way with performance reproducibility.

Recompilation Times

Let's denote S as the number of sources and C as the number of CPUs.

Intuitively, setting P=1 yields to the faster recompilation time when a single file is touched---a usual scenario when developing a new feature.

On the opposite, setting P=S yields to the slower recompilation time (if S >> C!) under the same scenario as all sources are recompiled under that scenario.

The form of the curve between those two extreme varies depending on the nature of the files, and the amount of header sharing between individual sources.

Caching tools like sccache is impacted by the same mechanism: as P gets greater, more cache misses are hit and more recompilation are done.

Marginally, introducing a new source also pollutes the cache or triggers recompilation for the unified source it gets added to, and eventually for all the unified sources derived from the associated file list. The reunifying problem strikes again.

So unified build make compilation faster, but recompilation slower. Setting P to an acceptable value is important depending on the usage scenario.

Correctness

Unified build changing the compilation unit frontier, which in turns modifies the semantic of the program. This change can be straight-forward or complex to debug, and even remain silent. I've listed a few instances of the two first categories below, and a crafted one for the latter category.

Macro / Symbol Redefinition

This one is trivial to spot (a preprocessor-warning is issued for the macro, and a compiler error is issued for the symbol redefinition):

/* pi0.cpp */
#define PI 3.141593
constexpr double pi() { return 3.141593; }

/* pi1.cpp */
#define PI 3.14159265
constexpr double pi() { return 3.14159265; }

The solution usually lies in moving the definition in a shared header, moving the declaration in a shared header and the definition in a single file, or renaming identifiers to avoid the name conflict. Note that depending on the solution we may change the visibility of the symbols, or impact code readability (assuming the identifier name was perfectly chosen in the first place).

Overload Conflicts

This one is also trivial to spot and may hint toward debatable design. But it exists and may be more complex to understand than the above:

/* overload0.cpp */
static float doit(float f) { return f;}
const float f = doit(1);

/* overload1.cpp */
static double doit(double d) { return d;}
const double d = doit(1);

The fix is generally to provide a perfect match for the overload, change the call site to avoid the ambiguity, or rename the functions/change their namespace to make the call site explicit.

Using Namespace Confusion

This one tends to creep a lot in codebase where using namespace is used. It generates ambiguity among potential symbols.

A caricatured situation is exhibited with the following situation:

/* using.h */
#pragma once
namespace a {
    namespace a {}
}

/* using0.cpp */
#include "using.h"
using namespace a;

/* using1.cpp */
#include "using.h"
using namespace a;

Once using{0,1}.cpp unified, the second using namespace a; directive is ambiguous.

A more realistic (but similar in spirit) situation arises when the same symbol is defined in different namespaces:

/* namespace0.cpp */
namespace a0 {
    int var;
}
using namespace a0;
int foo = var;

/* namespace1.cpp */
namespace a1 {
    int var;
}
using namespace a1;
int bar = var;

The problem with that category is that the fix is quite unsatisfying: there is no way to limit the scope of a using directive, removing using directive can lead to very verbose codebase, renaming symbols to avoid conflicts goes against the very purpose of namespaces...

Delicatessen

I spent a lot of time nailing that one down, so I wrote a small reproducer to illustrate the problem.

% tail -n +1 *.h *.cpp
==> header0.h <==
#ifndef H0
#define H0
namespace mozilla::dom {

class Lock final {};

}
#endif

==> header1.h <==
#ifndef H1
#define H1

#include "header0.h"

class Lock {};

class AutoUnlock {
    Lock *lock_;
};
#endif

==> src0.cpp <==
#include "header1.h"

==> src1.cpp <==
#include "header0.h"

==> src2.cpp <==
namespace mozilla::dom {};
using namespace mozilla::dom;
using namespace mozilla;


==> src3.cpp <==
#include "header1.h"

Let me comment that layout a bit: We basically have two different classes named Lock: one lives in the mozilla::dom namespace, and one lives at top-level. In header1.h, although we include the definition of mozilla::dom::Lock, we also get the definition of ::Lock, so a straight reference to Lock is not ambiguous.

Concerning source files, src0.cpp, src1.cpp and src3.cpp just include headers while src2.cpp contains the infamous using namespace modilla::dom; statement.

Let's now consider various partition of the file list src0.cpp, src1.cpp, src2.cpp, src3.cpp:

% for perm in 0,1 2,3 0,1,2 1,2,3 0,1,2,3; do printf "unifying $perm... " ; cat `eval echo src{$perm}.cpp` | clang++ -xc++ - -fsyntax-only 2>/dev/null && echo ok || echo ko ; done
unifying 0,1... ok
unifying 2,3... ko
unifying 0,1,2... ok
unifying 1,2,3... ko
unifying 0,1,2,3... ok

Isn't that amazing? Some intermediate unification, namely 0,1;2,3 and 0;1,2,3 fail, but other unifications, namely 0,1,2,3 and 0;1,2,3 fail. Did you notice that both non-unified and full unified build succeeds, while some intermediate unification fail? What a disaster. This basically mean that given a set of sources, and without putting restriction on the language (like banning using statement), the only way to be sure that a unified build always succeeds whatever the chosen partition is to test every partition. Not very satisfying.

As a side effect, we can also deduce that adding a new source file to a set of files to be unified can break compilation in files that used to compile fine. That's another instance of the reunifying problem.

Changing Semantic

It is quite easy to derive from the above an example whose semantic change once unified. Let's slightly change the overload conflict example from above:

/* silent0.cpp */
#include <cstdio>
static int doit(int f) { putchar('0'); return f;}
const int f = doit(1);

/* silent1.cpp */
#include <cstdio>
static double doit(double d) { putchar('1'); return d;}
const double d = doit(1);

When compiled independently, this results in a binary that prints a 0 and a 1 on the screen. But when compiled as a unified source, we only get a pair of 0.

Concluding Words

Remember that discussion between Luke and Yoda?

LUKE Vader. Is the dark side stronger?

YODA No… no… no. Quicker, easier, more seductive.

That's exactly my thoughts on unified builds: they give you quick wins in term of cold build speed and give faster builds. That's very good properties, and you rip the benefit of them very quickly. Then you realize that you're tied to a monster in terms of maintainability and developer experience, but you're already addict to the speed it gave you.

26 Aug 2026 10:00pm GMT

Mozilla Performance Blog: The Road to Better Performance Profiles – Part 2

Following the symbolication work from Part 1, I undertook two more efforts to improve performance profiles for the Performance team.

Native Profiling (Bugs 2030161, 2030166, 2047451, and 2030423)

The first effort was to use platform-specific profilers to generate profiles in CI of browsers running the Speedometer 3 benchmark, the industry-leading benchmark for browser performance. These 'native' profiles provide rich, system-wide insight into how browsers handle Speedometer 3, helping developers analyze and diagnose behaviour. They also provide us with profiles we can use to generate comparison reports for patches. We needed Raptor to generate native profiles for all of our platforms, namely Windows, macOS, Linux, and Android.

At a high level, the pipeline to generate native profiles is as follows:

  1. Start the system profiler.
  2. Run Speedometer 3 on Firefox or Chromium as Release using Raptor and Browsertime.
  3. Stop the system profiler.
  4. Symbolicate the profiles using samply and the corresponding build symbols.
  5. Post-process the symbolicated profiles using profiler-edit (formerly symbolicator-cli) and create compact and labelled variants of the profiles.
  6. Upload the processed profiles as artifacts.

To run tests with native profiling, you can use ./mach try fuzzy --full or ./mach try perf --full and run any test that has the -native-profiling suffix.

On Windows, we produced profiles using xperf, a tracer that uses the Event Tracing for Windows framework (ETW) to capture system-level data on applications running on Windows. To use xperf with our CI machines, scheduled tasks to run the tracer were configured in Puppet (see our ronin_puppet repository).

After adding support to Raptor to trigger these xperf tasks, we can start a trace and run the Speedometer 3 benchmark on Firefox or Chromium as Release (our custom Chromium build configured with release flags) using Browsertime. The Speedometer 3 benchmark runs 20 times, with each run in a separate browser cycle, to ensure we collect enough samples before stopping the trace.

Once completed, xperf provides a user trace and a kernel trace, which are combined into a full trace. We can then use samply to convert and symbolicate these traces into Firefox Profiler profiles, complete with markers and JIT information (Figure 1 and Figure 2).

Native profile of Speedometer 3 running on Firefox on Windows

Figure 1: Native profile of Speedometer 3 running on Firefox on Windows

Native profile of Speedometer 3 running on Chromium as Release on Windows

Figure 2: Native profile of Speedometer 3 running on Chromium as Release on Windows

On macOS, we took a similar approach. We used samply to collect and symbolicate profiles. After implementing a workaround to allow samply to profile across multiple browser cycles and sorting out permission issues in CI, samply could profile Speedometer 3 on Firefox with minimal CI configuration and symbolicate the profiles afterwards.

On Linux, we ran perf with elevated privileges to collect profiling data system-wide before using samply to symbolicate and convert the data into a Firefox Profiler profile - no additional machine configuration or workarounds were needed.

For Android, we took a slightly different approach. We used simpleperf to collect profiles on our mobile devices in CI (Samsung A55, Google Pixel 6, and Samsung S24) and samply to symbolicate them.

Initially, we wanted to profile 20 Speedometer 3 runs in a single simpleperf session, matching our desktop profiling workflow, but this proved unreliable during testing, as the on-device profiling appeared to be resource-intensive and would intermittently fail or crash.

We decided to use our simpleperf support introduced in Browsertime in Part 1 to profile the 20 individual test suites that make up a single Speedometer 3 run separately, which proved to be more stable (Figure 3). In the long term, we plan to support profiling complete Speedometer 3 runs with simpleperf on Android, bringing the workflow closer to what we currently use for desktop profiling (Bug 2032007). We will introduce native profiles for Chromium as Release (mobile) on Android in Bug 2067157.

Native profile of the TodoMVC-Vue Speedometer 3 test running on Firefox for Android

Figure 3: Native profile of the TodoMVC-Vue Speedometer 3 test running on Fenix (Firefox for Android)

To make our Speedometer 3 profiles clearer and more helpful, we post-process them with profiler-edit, which adds labels (groupings of JS frames) to the profile (Figure 5) and also provides a more compact version (Figure 4) where all runs are placed on the same process track.

Compact Speedometer 3 native profile

Figure 4: Compact Speedometer 3 native profile

Labelled Speedometer 3 native profile

Figure 5: Labelled Speedometer 3 native profile

In CI, we routinely generate Firefox profiles on autoland (Figure 6) and Chromium as Release profiles on mozilla-central. These native profiles have already been used to produce Speedometer 3 comparison reports (Figure 7) and investigate incidents in CI. Currently, profiles for Chromium as Release on macOS and Linux are works in progress (see Bug 2050869 and Bug 2065970, respectively).

Native profiles running routinely on autoland

Figure 6: Native profiles running routinely on autoland

Comparison reports using native profiles (preview courtesy of Markus Stange)

Figure 7: Comparison reports using native profiles (preview courtesy of Markus Stange)

Modernizing mozgeckoprofiler (Bug 1992000)

The second effort was to fully modernize symbolication in mozgeckoprofiler, the module responsible for symbolication across four performance testing frameworks: Raptor, Talos, XPCShell, and Mochitest. This mainly involved phasing out Eliot, an older symbolication API service scheduled to sunset in the near future, and fully transitioning the module's symbolication workflow to use samply and profiler-edit.

In Part 1, we added support that allowed Raptor and Talos to symbolicate their profiles using our new approach. This left two main cases that were still handled by Eliot: XPCShell and Mochitest symbolication, and local profile symbolication.

Implementing XPCShell and Mochitest profile symbolication (Bug 1998767) was relatively straightforward. We followed the same approach used for Talos and Raptor by adding our new symbolication dependencies to XPCShell and Mochitest taskgraph configurations. Since these tests and their profiles (Figure 8) can be generated in parallel, we made sure to perform symbolication only after all profiles had been generated. You can try these jobs by running ./mach try fuzzy --profiler and selecting any XPCShell and/or Mochitest job (Figure 9).

Symbolicated Mochitest profile

Figure 8: Symbolicated Mochitest profile

Mochitest tests running in CI produce symbolicated profiles

Figure 9: Mochitest tests running in CI produce symbolicated profiles

Originally, when generating profiles locally from one of our performance frameworks, Eliot was used to symbolicate those profiles. To replace it with our new approach, we needed to bootstrap samply and profiler-edit onto local developer machines. Now, when you run ./mach bootstrap, the latest builds of samply and profiler-edit available in CI for your platform are automatically installed under the local .mozbuild/ directory. With this change, the use of Eliot in PerfTest has been completely replaced by our new symbolication approach, and Eliot can safely be retired from mozgeckoprofiler!

What's Next

These patches are part of an ongoing effort to make performance profiling easier, more standardized, and more useful for Firefox developers. Here are a few related bugs to keep an eye on:

26 Aug 2026 7:28pm GMT