fidzu : Maemo

Yes, this is yet another post in the internet talking about using exceptions versus error returns. The topic has been flaming up at my workplace for quite some time now, and I felt that writing a blog post about it during the week-end would help me focus my thoughts and give me time to explain my point with the due care. In case you didn't know, I'm against using exceptions for error handling (maybe having spent many years working with Qt has had an effect on this); that does not mean that I never write code using exceptions: I certainly do my good share of try ... catch when dealing with third-party code (including the STL), but you won't find a throw in my programs.

I'm not going to write here all the reasons why I refrain myself from implementing error handling using exceptions; I'd rather like to focus on the one I consider to be the major one, and which I rarely see being given the due weight in the debate.

And please note that this post is about C++ only; it may be that exception handling in other languages is designed in such a way that all my concerns are addressed (either by the language itself, or by common error handling policies).

Code safety

I was about to title this "Code readability", but this is more about code verifiability, that is making sure that the code is correct and, ultimately, safe. As we all know, code is written once but read many times, and even if it's code you've written yourself, chances are that in a few weeks time you'll have forgotten several details about it; error cases and error handling are one typical thing that doesn't stick in our memory for long.

When I look at a small piece of code, such as the one that can fit into my screen, or which I can read from a merge request diff, I want to be able to ascertain that the code I'm looking at is correct. Let's look at some examples.

A throw-free project

I just made this up, so please bear with me if it doesn't make any sense. What I want to show is that code like the above has very few fault risks, if found in a project which bans throwing errors as exceptions: if we exclude out-of-memory errors, that are generally not handled to let the application crash (though you can always catch them if you like), the reader can easily verify that this code is safe. Coding style policies and naming conventions can guarantee that setMaximumSpeed() and setName() won't have a return value that needs to be checked, and all other method calls either return an error that our code is properly handling, or return some value. Of course, by just looking at this piece of code we cannot know if the engineTemperature() method has some other overloaded sibling which accepts passing a reference to a boolean and which could be used to detect an error; so, it may be that our code could be improved in that respect, if we had a look at the header files for the Car class - but this does deny the fact that a simple glance at this snippet tells us exactly what errors are handled and what could be going wrong.

Let's look at this code instead:

If we continue on the assumption that we are working on a project which bans throwing exceptions, we can immediately say that this code is not safe: we don't know if the car successfully executed a lap on the track, and our process will crash if boundingRectArea() is invoked on a null object.

Enter the exception

In a project where exceptions are actively used, the code from the second snippet is not obviously wrong anymore: maybe executeLap() cannot throw any exceptions, or, if does, the caller of this snippet is catching the exception? In order to figure out whether this code is correct, I need to see the declaration of the executeLap() method, and hope that there's a nice noexcept in there; if there isn't, I have to look at its implementation, and recursively descend through all the methods it calls - at which point the safest attitude is just to assume that it can throw. But that's only half of the story, because once I accept the fact that executeLap() can throw, I need to check whether the exception is properly handled: I have to check the implementation of all the callers of my method, and if I don't find a catch there, I'll have to recursively walk up the tree of their callers.

And indeed even the first snippet, which looked so harmless when exception throwing was banned, suddenly becomes not obviously correct anymore: what if executeLap() or getPath() also throw an exception? You might say that it would be quite a silly thing to do, and I'd certainly agree; but it may be that indeed they don't throw any exceptions in their implementation, but some of the methods they call does.

A compromise: catch early, catch often

The obvious solution to the above issue is having a policy of handling exceptions right away, and explicitly rethrowing them (or even better, rethrow a different, more appropriate exception) up the stack:

What I can tell from the above snippet is that the code is handling errors, and this is somehow a relief. I'm sure some of you would suggest using a more specific catch clause, but for the sake of this example let's assume that this one is fine.

(Quick note: the above example does not catch std::exception, because that would also catch the std::bad_alloc exception which is typically thrown in out-of-memory situations; my advice is not handle it at all, unless you know what you are doing)

In real life, though, you might find that try-ing on a rather large block of operations is not enough: suppose that the Car methods all emit the same exception type, and that you need to handle them differently depending on when they occur. Then you'd need to split up the try into smaller blocks, and at that point your code won't look any cleaner than the equivalent code which uses ifs on return values. Of course if you own the Car class you could modify it to throw different exceptions, in order to keep more operations inside the try block and have specific catches at the end.

The big catch (pun intended)

Even once you've refactored your methods to get the best out of exceptions (where "best" is highly subjective, but let's assume that it just means that you are happy with your exception-throwing code), there's something that still bothers me, and that's exactly the same thing that proponents of exceptions use as a "pro" in their argumentations: the business logic of your code gets separated from the error handling. You get a nice block of pure logic, not cluttered with error checking, and a catch section (which I call "the big catch") where error cases are handled.

I really don't see how that makes the code any more readable or safe: sure, the logic is not intertwined with error handling and might help focus on the expected flow of the operations (though, really, I do not think that normal brains have a problem skipping over if blocks), but that's hardly what I'm interested in when I want to check that the code is correct. Most of program errors and bugs lie in handling the edge cases and the abnormal situations, the seldomly taken code paths, and that's where I need to focus my attention.

When I see code like this one, I need to mentally build a mapping of "operationX() → possible exceptions" (which, unless exception naming is making this obvious, requires me to look at the implementation of the operationX() functions), and then mentally reconstruct the possible code paths in case operationX() fails, for each line of the try block.

Not seeing the errors right there, right away makes the correctness verification harder, which in turns means that the code becomes less safe. It will make you focus on the best case scenario, while ignoring all those annoying edge cases - too bad that 90% of the bugs are there.

Reading through the ISO C++ propaganda FAQ

I've been given a link to the C++ FAQ about exceptions, and unfortunately I read it. While there isn't much to argue on the technical side of it, it also carries some misleading statements, which might be true in absolute terms but don't let you see the big picture by not mentioning all that you need to know (which is the fundamental technique behind propaganda). An example is when they mention that eliminating ifs makes for more robust code, without mentioning that the same applies to all code branches, including exceptions.

Another argument that bothered me when I read it is the one about error propagation; this is the example they make:

The claim is that this code is more readable than the one with explicit error handling, because all the f2(), f3, …, f9() functions don't have to handle the error occurring in f10(). It is indeed a convincing argument, when presented in these terms, but is this really how our code looks like? In real life, you'll hardly have a chain of 1-liner functions, all defined next to each other in the same file. The moment that you realize that each one of these fn() functions might be twenty or thirty lines long, and that they might be scattered over different files, and be called not just by fn-1() but by any other function in the codebase, the picture does not look so rosy anymore: we get back to my main point of pain, that is that looking at the code of, say, f5(), I will not be able to tell if the errors thrown by it, or by any of the methods invoked by it, are properly handled.

Exceptions in APIs

A side note about projects using exceptions. I'm not really bothered when a library I need to use is throwing exceptions: having to write

is not less readable or less safe than the code I'd write if Foo::fetch() returned an error code. I still do have a little complaint, because the library author has given himself the right to decide that a failure in his library should be considered a critical fault, whereas it may be that in my program it is an expected failure and using exceptions imposes a penalty which could have been avoided. But I digress.

As long as the library documents which exceptions are thrown, it is used by many people (which hopefully means that it has few bugs) and it is a library that I don't need to contribute to, wrapping some of its methods in try blocks is something I can live with.

One situation where I actually wish that libraries threw an exception is in out-of-memory situations; in that case, of course, I'd expect them to throw nothing else than std::bad_alloc, which is the exception emitted by the standard library in such situations. That allows the caller to decide whether to ignore the exception and have the process terminated (which is what I usually do, at least in desktop applications) or try their luck and handle the failure - the latter is not easy, but it can certainly be done.

This is one case where error returns can be problematic, because it's likely that your code would look something like

and in this case there's actually a risk that your code is going to trigger an out-of-memory error in logging the message; this shouldn't be a concern in most cases, but I can imagine some situations where one might want to know which was the exact operation that first incurred in the out-of-memory failure.

So, I'm actually fine with new throwing. As for my code, my throw statement is actually spelt as return.

Mappero Geotagger has now moved from its previous page from this site to a new, separate website built with the awesome Nikola static website generator.

The main reason for this change is that I didn't have an online space where to host the application binaries, and I wanted to experiment with a different selling method. Now, downloads are (poorly) hidden behind a payment page, whereas in multiple places of the website I also mention that I can provide the application for free to whomever asks for it. While it might seem weird at first, I do honestly believe that this will not stop people from buying it: first of all, many people just think it's fair to pay for a software applications, and secondly, for some people writing an e-mail and establishing a personal contact with a stranger is actually harder than paying a small amount of money. And in all sincerity, the majority of the income I've had so far for Mappero Geotagger came from donations, rather than purchases; so, not much to lose here.

QBS and MXE

Anyway, since this is primarily a technical blog, I want to share my experiences with cross-building from Linux to Windows. As you might remember, some time ago I switched the build system of Mappero from qmake to QBS, and I haven't regretted it at all. I've managed to build the application in Linux (of course), macOS, as a Debian package on the Ubuntu PPA builders, on Windows with AppVeyor and, last but not least, on Linux for Windows using the mingw setup provided by the MXE project.

QBS worked surprisingly well also in this case, though I had to fight with a small bug on the toolchain detection, which is hopefully going to be fixed soon. For the few of you who are interested in achieving something similar, here's the steps I ran to configure QBS for mingw:

Sorry for using that many environment variables ☺. After qbs is configured, it's just a matter of running

to build the application. You will get a nice window binary and, once you collect all the needed library dependencies, you'll be able to run it on Windows. Or WINE ☺.

As part of this effort, I also had to build libraw, so I didn't miss the occasion to contribute its recipe to MXE. I'm also trying to get a change accepted, that would make MXE support the dynamic OpenGL selection available since Qt 5.4.

Today I've released Imaginario 0.9. The big feature coming with this new release is a face tagging flow which I believe will be the fastest and simplest you've ever used, despite it being all manual. I even sat down and spent some quality time with Blender to prepare a video to show it off:

While some people might actually think that I spent more time for making the video than for implementing the face tagging feature itself, this couldn't be farther from the truth: the face tagging branch has been being worked on for at least three months (of course, that's my spare time - so it's actually less than one hour per day) and consisted of more than 40 commits (after squashing all the fixups), whereas for the video I spent no more than a couple of hours.

I would appreciate if the curious could go and try it out, and let me know about any issues you should find: there are built packages for Linux (AppImage), macOS and Windows. I do also have an Ubuntu PPA where nightly images are built, but I'm not sure if I can recommend that one, since I've not been using it myself and have no idea whether those packages actually even start. But you are welcome to try :-)

Your feedback will help me do better, so please don't be shy!

I just released PhotoTeleport 0.12, which includes the feature mentioned in the title of this blog post. Given that it took me some time to understand how this could work with Qt, I think it might be worth spending a couple of lines about how to implement it.

In the target application

The first step (and the easiest one) is about adding the proper information to your .plist file: this is needed to tell MacOS what file types are supported by your application. The official documentation is here, but given that an example is better than a thousand words, here's what I had to add to PhotoTeleport.plist in order to have it registered as a handler for TIFF files:

  <key>CFBundleDocumentTypes</key>
  <array>
    <dict>
      <key>CFBundleTypeExtensions</key>
      <array>
        <string>tiff</string>
        <string>TIFF</string>
        <string>tif</string>
        <string>TIF</string>
      </array>
      <key>CFBundleTypeMIMETypes</key>
      <array>
        <string>image/tiff</string>
      </array>
      <key>CFBundleTypeName</key>
      <string>NSTIFFPboardType</string>
      <key>CFBundleTypeOSTypes</key>
      <array>
        <string>TIFF</string>
        <string>****</string>
      </array>
      <key>CFBundleTypeRole</key>
      <string>Viewer</string>
      <key>LSHandlerRank</key>
      <string>Default</string>
      <key>LSItemContentTypes</key>
      <array>
        <string>public.tiff</string>
      </array>
      <key>NSDocumentClass</key>
      <string>PVDocument</string>
    </dict>
    …more dict entries for other supported file formats…
  </array>

This is enough to have your application appear in Finder's "Open with…" menu and be started when the user selects it from the context menu, but it's only half of the story: to my big surprise, the selected files are not passed to your application as command line parameters, but via some MacOS-specific event which needs to be handled.

By grepping into the Qt source code, I've found out that Qt already handles the event, which is then transformed into a QFileOpenEvent. The documentation here is quite helpful, so I won't waste your time to repeat it here; what has hard for me was to actually find that this functionality exists and is supported by Qt.

In the source application

The above is only half of the story: what if you are writing an application which wants to send some files to some other application? Because of the sandboxing, you cannot just start the desired application in a QProcess and pass the files as parameters: again, we need to use the Apple Launch Services so that the target application would receive the files through the mechanism described above.

Unfortunately, as far as I could find this is not something that Qt supports; sure, with QDesktopServices::openUrlExternally() you can start the default handler for the given url, but what if you need to open more than one file at once? And what if you want to open the files in a specific application, and not just in the default one? Well, you need to get your hands dirty and use some MacOS APIs:

#import <CoreFoundation/CoreFoundation.h>
#import <ApplicationServices/ApplicationServices.h>

void MacOS::runApp(const QString &app, const QList<QUrl> &files)
{
    CFURLRef appUrl = QUrl::fromLocalFile(app).toCFURL();

    CFMutableArrayRef cfaFiles =
        CFArrayCreateMutable(kCFAllocatorDefault,
                             files.count(),
                             &kCFTypeArrayCallBacks);
    for (const QUrl &url: files) {
        CFURLRef u = url.toCFURL();
        CFArrayAppendValue(cfaFiles, u);
        CFRelease(u);
    }

    LSLaunchURLSpec inspec;
    inspec.appURL = appUrl;
    inspec.itemURLs = cfaFiles;
    inspec.asyncRefCon = NULL;
    inspec.launchFlags = kLSLaunchDefaults + kLSLaunchAndDisplayErrors;
    inspec.passThruParams = NULL;

    OSStatus ret;
    ret = LSOpenFromURLSpec(&inspec, NULL);
    CFRelease(appUrl);
}

In Imaginario I've saved this into a macos.mm file, added it to the source files, and also added the native MacOS libraries to the build (qmake):

LIBS += -framework CoreServices

You can see the commit implementing all this, it really doesn't get more complex than this. The first parameter to the MacOS::runApp() function is the name of the application; I've verified that the form /Applications/YourAppName.app works, but it may be that more human-friendly variants work as well.

Recently I've grown an interest to the indieweb: as big corporations are trying to dictate the way we live our digital life, I'm feeling the need to take a break from at least some of them and getting somehow more control over the technologies I use.

Some projects have been born which are very helpful with that (one above all: NextCloud), but there are also many older technologies which enable us to live the internet as a free distributed network with no owners: I'm referring here to protocols such as HTTP, IMAP, RSS, which I perceive to be under threat of being pushed aside in favor of newer, more convenient, but also more oppressive solutions.

Anyway. The indieweb community is promoting the empowerment of users, by teaching them how to regain control of their online presence: this pivots arund having one's own domain and use self-hosted or federated solutions as much as possible.

One of the lesser known technologies (yet widely used in the indieweb community) is webmentions: in simple terms, it's a way to reply to other people's blog posts by writing a reply in your own blog, and have it shown also on the original article you are replying to. The protocol behind this feature is an recommendation approved by the W3C, and it's actually one of the simplest protocol to implement. So, why not give it a try?

I already added support for comments in my blog (statically generated with Nikola) by deploying Isso, a self-hosted commenting system which can even run as a FastCGI application (hence, it can be deployed in a shared hosting with no support for long-running processes) - so I was looking for a solution to somehow convert webmentions into comments, in order hot to have to deal with two different commenting systems.

As expected, there was no ready solution for this; so I sat down and hacked up Bussator, a WSGI application which implements a webmention receiver and publishes the reply posts as Isso comments. The project is extensible, and Isso is only one of the possible commenting systems; sure, at the moment it's indeed the only one available, but there's no reason why a plugin for Static Man, Commento, Remark or others couldn't be written. I'll happily accept merge requests, don't be shy - or I can write it myself, if you convince me to (a nice Lego box would make me do anything 0 0

We live in strange times. People are so filled with hatred and prejudices that their brain becomes unable to parse the simplest sentences. I take this issue to heart, because it could happen to anyone - it has happened to me before (luckily, only in private online conversations), where an acquaintance of mine accused me of saying things I never said. And it happens to famous people all the time. Guys, just because you hate person X, you should not skip over parts of their speech or suppress context in order to make it look like they said something terrible or stupid, when they didn't.

Now it happend to Richard Stallman, with a whole wave of hateful people accusing him of saying something that he didn't say. Let's start with the VICE article, titled "Famed Computer Scientist Richard Stallman Described Epstein Victims As 'Entirely Willing'", which insists in quoting only two words out of Stallman's sentence:

Early in the thread, Stallman insists that the "most plausible scenario" is that Epstein's underage victims were "entirely willing" while being trafficked.

Except that he didn't say that. Why not quote the whole sentence? It's not such a long sentence, really! Just follow the link to the source, which provides a complete excerpt of Stallman's words:

We can imagine many scenarios, but the most plausible scenario is that she presented herself to him as entirely willing. Assuming she was being coerced by Epstein, he would have had every reason to tell her to conceal that from most of his associates.

Now, English is not my native language, but I read it well enough to understand that "to present oneself as" and "to be" are different expressions having very different meanings (and, in most context, actually opposite ones!). You don't need to be Shakespeare to understand that. You only need to either stop hating or, if you really cannot help it, at least stop projecting your prejudices onto the people you hate. Hate makes you blind.

It's sad to see otherwise intelligent people take stupid decisions because of such misunderstandings.

I for one, stand in solidarity with Richard Stallman and with the English language.

(please note that this is not an endorsement of everything Stallman might have said in the past; I don't follow him that closely, and it may be that he also happened to say terrible things in this very thread. I'm only commenting this very specific issue, and I know that in this very specific issue he's being wrongly accused)

The wonders of improvised Mastodon instances: one node disappears after an outage caused by a summer heatwave, leaving its users no way to migrate their data or to notify their followers.

After about one month of waiting for the node to come up or give some signals of life, I've decided to create a new account on another instance. If you use Mastodon and you were following me, please forgive me for the annoyance and follow me again here.

I was looking at TripAdvisor, some days ago. It's a very useful site, filled with user-generated advice and reviews, which has become almost a must for travellers. But I never like it when a private entity gets so much power over our lives (even if it's - currently - exercised in total fairness).

I would like to have a federated TripAdvisor-like network. I duckduckwent for a while, but I didn't find anything similar (if you know of some project of that kind, please let me know in the comments).

But thinking more about the issue, I realised that I probably wouldn't even bother to type my reviews into that site; I have a blog, so ideally I would like to have the option to write my review here, and then have the site import it. Technically, it could work with a webhook, or even a periodic check (real-time updates would not be a requirement here) over a URL I've linked to in my profile on that site. Then, in order for the posts to be imported, they would have to be entered in a standard format: maybe some keywords (or invisible HTML elements) could be used as markers for the content that need to be extracted from an otherwise ordinary blog post, or the relevant content could be replicated in a different format in the HTML headers (this, though, would require some additional work).

And while we are at it, why not extend this to other social networks? I use Mastodon, for example, and occasionally I send out a toot with a link to my latest blog post in there. But it would be much nicer if I could somehow set a special mark into my posts while composing them, to have them automatically tooted out on my account (this could probably be implemented as a standalone service, authorized to act on my Mastodon account - similarly to how the Mastodon-Twitter crossposter app works).

But I'm rather confident that I'm not the only one having this kind of needs, and that's why I'm writing this blog post: maybe someone out there has already found a solution, or has some more concrete ideas? If so, I'm all eyes!

A due premise: this is not a guide! While at the end I managed to build an image for my phone, it doesn't even boot. So, this post should be taken just as a collection of notes for myself, written to be able to reproduce the build and keep track of those steps I might want to revisit later.

The reason why I'm publishing these notes in my blog is that I believe they can still be useful for other people who should stumble into the same errors, and also to give you an idea of how much fun (or lack thereof) porting is.

It's a very long post, but unless you are starting to port Halium or Ubports to a new device, you can safely stop reading now.

I hope to write some better porting news in some future post. :-)

Before we start: getting the info

The first step is finding out a few informations about our device, and whether a LineageOS port for our device exists (hadn't there been a LineageOS port, I would have been out of luck: I certainly wouldn't have the time to do one myself). From the Settings application we can find the device name, the model number, the Android version and the kernel version:

• Device name: "Samsung Galaxy J3 (2016)"
• Model: SM-J320F
• Android version: 5.1.1
• Kernel version: 3.10.65-[…]

With this information we can look for a LineageOS port, either via a search engine or directly in the XDA forums; in my case the thread is this one. The post announcing the port should contain links to the source code, besides some other information. We do especially care about:

• the device codename: j3xlte
• the CM/LineageOS version: 14.1
• links to Device Tree, Kernel, Local Manifest (keep the local manifest handy, as you are going to need it soon)

To be on the safe side, get a copy of everything. In case these are GitHub repositories, forking them is enough.

Getting root access on the device

Another precondition for a successful port is the ability to be able to flash a custom kernel and rootfs image. The existance of a LineageOS port is not a guarantee that we can flash it: some devices are sold with a locked bootloader in certain markets or when sold by certain mobile operators. So, before spending time on the port, we should make sure that we can hack on our device freely.

In order to get adb working, I had to enable "Developer mode", which is done by hitting the "Build numer" item under the "Software info" page for 10 times in a row. Then from the newly appeared "Developer options" menu I enabled "USB debugging".

Being a total novice to Android phones, I had to go through several attempts and internet searches, before being able to get root access on the device (which I needed in order to read the /etc/fstab file, for example): by default, adb only let me as normal user, and su didn't work.

In order to flash a rootkit on this phone, we need to use Heimdall, because on Samsung devices this is what is used instead of fastboot. I just followed the instructions in the Linux/README file and could easily build and install it.

We are going to flash TWRP and SuperSU; a link to the former can be found in the XDA post about the LineageOS port for our device (under the "Download" button) and the latter from its own homepage (in my case I had to download the "Recovery flashable" zip file, and for some weird reason I got the 2.76 version, but I bet that any newer version works as well). Once I got these files, I followed these steps:

• Extract TWRP in my host machine: cd /tmp && tar xvf ~/Downloads/sm-j320fgm_twrp_3.1.1_f2fs.tar.md5 (that leaves a recovery.img file in my /tmp)
• Upload SuperSU to the phone:

    adb push ~/Downloads/UPDATE-SuperSU-v2.76-20160630161323.zip /sdcard/supersu.zip

• Power off the phone
• Power it on by keeping the VOLUME DOWN and HOME keys pressed while pressing the power button. The phone will be brought up into Odin mode.
• Run

    sudo heimdall flash  --RECOVERY /tmp/recovery.img

• Before the phone reboots, start keeping the VOLUME UP (note: it's not the same as before!) and HOME keys, to make it reboot into recovery and not into the ordinary image. If the phone boots into the ordinary image it will overwrite the TWRP we just flashed, so, if that happens, you need to start from scratch.

• Find supersu (the zip file) and install it.

If you reboot the phone and use adb shell, you'll be able to run su and get root rights. At this point, I charged myself with hopes and got to the real work.

Let the Odyssey start

I just followed the "First steps" instructions step by step, with no major surprises.

Once done with that, I moved to the "Get sources" page. Here it was not just a matter of following the instructions: assembling the manifest file took me several iterations, so I had to come back to this step a few times. Anyway, I followed the instructions for the halium-7.1 version, given that my device has a LineageOS 14.1 port. Then:

• I created a git branch in the halium/devices/ tree, to keep track of my changes
• I added the manifest halium/devices/manifests/samsung_j3xlte.xml (you can see the final version in my repository, but initially I was missing some important lines)
• Run JOBS=2 ./halium/devices/setup j3xlte
• The makefile needed to be changed: the device/samsung/j3xlte/setup-makefiles.sh script (from the device tree) was missing a mkdir -p ../../../$OUTDIR near the beginning.

At this point I continued to the "Build sources" page. Here starts a long trial and error phase, and the longer part of this post. For the sake of documentation, I will show you all the commands I ran, including failed attempts (there are many of them), just to give you an idea of what problems you might run into with your own port, and hopefully help in finding a solution.

I first ran the breakfast command for my device:

$ breakfast j3xlte
including vendor/cm/vendorsetup.sh
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist.  Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist.  Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist.  Stop.
Device j3xlte not found. Attempting to retrieve device repository from LineageOS Github (http://github.com/LineageOS).
Repository for j3xlte not found in the LineageOS Github repository list. If this is in error, you may need to manually add it to your local_manifests/roomservice.xml.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist.  Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist.  Stop.

** Don't have a product spec for: 'lineage_j3xlte'
** Do you have the right repo manifest?

It looks like the device/samsung/sharkls-common was not checked out properly. After re-doing with force-sync, it failed with other errors:

$ breakfast j3xlte
including vendor/cm/vendorsetup.sh
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist.  Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist.  Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist.  Stop.
Device j3xlte not found. Attempting to retrieve device repository from LineageOS Github (http://github.com/LineageOS).
Repository for j3xlte not found in the LineageOS Github repository list. If this is in error, you may need to manually add it to your local_manifests/roomservice.xml.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist.  Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist.  Stop.

** Don't have a product spec for: 'lineage_j3xlte'
** Do you have the right repo manifest?

Indeed, I had nothing under device/common. So, I looked under halium/devices/manifests to see if any other manifest was populating this directory, and found that the bq_krillin.xml file had such a line. So, I copied that line (the one adding the lineageos/android_device_common project) into my manifest. After doing that, the breakfast command succeeded:

$ breakfast j3xlte
including vendor/cm/vendorsetup.sh
Trying dependencies-only mode on a non-existing device tree?

============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=7.1.1
LINEAGE_VERSION=14.1-20190619-UNOFFICIAL-j3xlte
TARGET_PRODUCT=lineage_j3xlte
TARGET_BUILD_VARIANT=userdebug
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a-neon
TARGET_CPU_VARIANT=cortex-a7
TARGET_2ND_ARCH=
TARGET_2ND_ARCH_VARIANT=
TARGET_2ND_CPU_VARIANT=
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-4.15.0-51-generic-x86_64-with-Ubuntu-16.04-xenial
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=NOF27B
OUT_DIR=/mnt/Lavoro/halium/out
RECOVERY_VARIANT=twrp
WITH_SU=true
============================================

Building the kernel

According to the livestream, the proper script to check for the kernel config is not the Mer one, but halium/halium-boot/check-kernel-config (this is also documented in the Ubports - Building Halium boot documentation page).

At this point I realized that I didn't have a kernel: an entry starting with kernel/ should always be in the manifest file. So I added the kernel line taken from the LineageOS local manifest file.

I then looked for the config file, which I eventually found in arch/arm/configs/j3xlte_defconfig. So I ran

./halium/halium-boot/check-kernel-config kernel/samsung/sharkls/arch/arm/configs/j3xlte_defconfig -w

which reported no errors. However, in the command output I noticed this:

 CONFIG_TMPFS_POSIX_ACL is already set
 CONFIG_DEFAULT_SECURITY is set, but to y "selinux" y not "apparmor".
 Setting CONFIG_DEFAULT_SECURITY="apparmor" correctly
sed: -e expression #1, char 60: unterminated `s' command
 Setting CONFIG_SECURITY_APPARMOR_BOOTPARAM_VALUE=1

When I checked the config file, I noticed that AppArmor was not really enabled. So I changed it manually.

Next, the fstab. I had the Ubports and Halium guides both opened in my browser, and I went for the "Fix mounts" step from the Ubports guide. I used find to locate the fstab, and found two:

• device/samsung/sharkls-common/recovery/root/fstab.sc8830
• device/samsung/sharkls-common/rootdir/etc/fstab.sc8830

The recovery file seemed already fine to me, while the rootdir one had most entry duplicated; ones with f2fs and xattrs, the other ones with ext4. I commented out the f2fs ones and ran mka halium-boot:

[...]
/mnt/Lavoro/halium/out/build-aosp_arm.ninja is missing, regenerating...
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=7.1.1
LINEAGE_VERSION=
TARGET_PRODUCT=aosp_arm
TARGET_BUILD_VARIANT=eng
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a
TARGET_CPU_VARIANT=generic
TARGET_2ND_ARCH=
TARGET_2ND_ARCH_VARIANT=
TARGET_2ND_CPU_VARIANT=
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-4.15.0-51-generic-x86_64-with-Ubuntu-16.04-xenial
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=NOF27B
OUT_DIR=/mnt/Lavoro/halium/out
============================================
Checking build tools versions...
find: 'device/*/generic': No such file or directory
find: 'device/unknown': No such file or directory
find: 'device/android': No such file or directory
find: 'device/*/generic': No such file or directory
find: 'device/unknown': No such file or directory
find: 'device/android': No such file or directory
halium/hybris-boot/Android.mk:69: ********************* /boot appears to live on ERROR: *fstab* not found
halium/hybris-boot/Android.mk:70: ********************* /data appears to live on ERROR: *fstab* not found
halium/hybris-boot/Android.mk:73: *** There should be a one and only one device entry for HYBRIS_BOOT_PART and HYBRIS_DATA_PART.
build/core/ninja.mk:166: recipe for target '/mnt/Lavoro/halium/out/build-aosp_arm.ninja' failed
make: *** [/mnt/Lavoro/halium/out/build-aosp_arm.ninja] Error 1
make: Leaving directory '/mnt/Lavoro/halium'

#### make failed to build some targets (11 seconds) ####

this seems to be documented in the "Common system build errors" page: I had to rerun source build/envsetup.sh and breakfast j3xlte. After that, the mka halium-boot command failed with

device/samsung/sharkls-common/ims/sec_samsung/Android.mk:25: build/core/java_library.mk: No such file or directory
build/core/ninja.mk:166: recipe for target '/mnt/Lavoro/halium/out/build-lineage_j3xlte.ninja' failed
make: *** [/mnt/Lavoro/halium/out/build-lineage_j3xlte.ninja] Error 1
make: Leaving directory '/mnt/Lavoro/halium'

which looks like a progress. The issue about the missing java_library.mk has been raised in the #halium channel; possible solutions include:

• symlink from static_java_library.mk
• edit the Android.mk file

I went for the former; this probably needs to be fixed in Halium.

Reran mka halium-boot, got some improvement: fstab is found:

halium/hybris-boot/Android.mk:70: ********************* /boot appears to live on 
halium/hybris-boot/Android.mk:71: ********************* /data appears to live on /dev/block/platform/sdio_emmc/by-name/userdata
halium/hybris-boot/Android.mk:74: *** There should be a one and only one device entry for HYBRIS_BOOT_PART and HYBRIS_DATA_PART.
build/core/ninja.mk:166: recipe for target '/mnt/Lavoro/halium/out/build-lineage_j3xlte.ninja' failed
make: *** [/mnt/Lavoro/halium/out/build-lineage_j3xlte.ninja] Error 1
make: Leaving directory '/mnt/Lavoro/halium'

It looks like my fstab was not complete, but missing the /boot partition. given that none of the fstab I located in my source tree had a line for the boot partition, I checked out the Android's fstab from the phone, via adb shell. There is no fstab in /etc, but two fstab files right in the root directory of the filesystem: fstab.sc8830 with exactly the same contents as the recovery fstab I found before in device/samsung/sharkls-common/recovery/root/fstab.sc8830, and a fstab.goldfish:

127|root@j3xlte:/ # cat /fstab.goldfish                                        
# Android fstab file.
#<src>                                                  <mnt_point>         <type>    <mnt_flags and options>                              <fs_mgr_flags>
# The filesystem that contains the filesystem checker binary (typically /system) cannot
# specify MF_CHECK, and must come before any filesystems that do specify MF_CHECK
/dev/block/mtdblock0                                    /system             ext4      ro,noatime,barrier=1                                         wait
/dev/block/mtdblock1                                    /data               ext4      noatime,nosuid,nodev,barrier=1,nomblk_io_submit      wait,check
/dev/block/mtdblock2                                    /cache              ext4      noatime,nosuid,nodev  wait,check
/devices/platform/goldfish_mmc.0                        auto                vfat      defaults                                             voldmanaged=sdcard:auto

While mount says:

~ # mount
rootfs on / type rootfs (rw,seclabel)
tmpfs on /dev type tmpfs (rw,seclabel,nosuid,relatime,mode=755)
devpts on /dev/pts type devpts (rw,seclabel,relatime,mode=600)
proc on /proc type proc (rw,relatime,gid=3009,hidepid=2)
sysfs on /sys type sysfs (rw,seclabel,relatime)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,seclabel,relatime)
pstore on /sys/fs/pstore type pstore (rw,seclabel,relatime)
adb on /dev/usb-ffs/adb type functionfs (rw,relatime)
adb on /dev/usb-ffs/adb type functionfs (rw,relatime)
/dev/block/mmcblk0p27 on /data type ext4 (rw,seclabel,relatime,data=ordered)
/dev/block/mmcblk0p27 on /sdcard type ext4 (rw,seclabel,relatime,data=ordered)
/dev/block/mmcblk0p24 on /cache type ext4 (rw,seclabel,relatime,data=ordered)

None of the above helps in locating the boot partition. I then tried to look under /dev/block:

root@j3xlte:/dev/block/platform/sdio_emmc/by-name # ls -l
lrwxrwxrwx root     root              2019-06-26 19:13 CACHE -> /dev/block/mmcblk0p24
lrwxrwxrwx root     root              2019-06-26 19:13 FOTA_SIG -> /dev/block/mmcblk0p11
lrwxrwxrwx root     root              2019-06-26 19:13 HIDDEN -> /dev/block/mmcblk0p26
lrwxrwxrwx root     root              2019-06-26 19:13 KERNEL -> /dev/block/mmcblk0p20
lrwxrwxrwx root     root              2019-06-26 19:13 PARAM -> /dev/block/mmcblk0p16
lrwxrwxrwx root     root              2019-06-26 19:13 PERSDATA -> /dev/block/mmcblk0p23
lrwxrwxrwx root     root              2019-06-26 19:13 PERSISTENT -> /dev/block/mmcblk0p22
lrwxrwxrwx root     root              2019-06-26 19:13 RECOVERY -> /dev/block/mmcblk0p21
lrwxrwxrwx root     root              2019-06-26 19:13 RESERVED2 -> /dev/block/mmcblk0p19
lrwxrwxrwx root     root              2019-06-26 19:13 SBOOT -> /dev/block/mmcblk0p1
lrwxrwxrwx root     root              2019-06-26 19:13 SBOOT2 -> /dev/block/mmcblk0p2
lrwxrwxrwx root     root              2019-06-26 19:13 SYSTEM -> /dev/block/mmcblk0p25
lrwxrwxrwx root     root              2019-06-26 19:13 efs -> /dev/block/mmcblk0p17
lrwxrwxrwx root     root              2019-06-26 19:13 l_fixnv1 -> /dev/block/mmcblk0p3
lrwxrwxrwx root     root              2019-06-26 19:13 l_fixnv2 -> /dev/block/mmcblk0p4
lrwxrwxrwx root     root              2019-06-26 19:13 l_gdsp -> /dev/block/mmcblk0p9
lrwxrwxrwx root     root              2019-06-26 19:13 l_ldsp -> /dev/block/mmcblk0p7
lrwxrwxrwx root     root              2019-06-26 19:13 l_modem -> /dev/block/mmcblk0p8
lrwxrwxrwx root     root              2019-06-26 19:13 l_runtimenv1 -> /dev/block/mmcblk0p12
lrwxrwxrwx root     root              2019-06-26 19:13 l_runtimenv2 -> /dev/block/mmcblk0p13
lrwxrwxrwx root     root              2019-06-26 19:13 l_warm -> /dev/block/mmcblk0p10
lrwxrwxrwx root     root              2019-06-26 19:13 pm_sys -> /dev/block/mmcblk0p5
lrwxrwxrwx root     root              2019-06-26 19:13 prodnv -> /dev/block/mmcblk0p18
lrwxrwxrwx root     root              2019-06-26 19:13 rsvdfixnv1 -> /dev/block/mmcblk0p6
lrwxrwxrwx root     root              2019-06-26 19:13 td_runtimenv1 -> /dev/block/mmcblk0p14
lrwxrwxrwx root     root              2019-06-26 19:13 td_runtimenv2 -> /dev/block/mmcblk0p15
lrwxrwxrwx root     root              2019-06-26 19:13 userdata -> /dev/block/mmcblk0p27

This, combined from the PIT file which we can extract with Heimdall from the "Odin mode", gives us a better understanding of the device partitions:

$ sudo heimdall print-pit --no-reboot
Heimdall v1.4.2

Copyright (c) 2010-2017 Benjamin Dobell, Glass Echidna
http://www.glassechidna.com.au/

This software is provided free of charge. Copying and redistribution is
encouraged.

If you appreciate this software and you would like to support future
development please consider donating:
http://www.glassechidna.com.au/donate/

Initialising connection...
Detecting device...
Claiming interface...
Setting up interface...

Initialising protocol...
Protocol initialisation successful.

Beginning session...

Some devices may take up to 2 minutes to respond.
Please be patient!

Session begun.

Downloading device's PIT file...
PIT file download successful.

Entry Count: 31
Unknown 1: 1598902083
Unknown 2: 844251476
Unknown 3: 20563
Unknown 4: 17490
Unknown 5: 14136
Unknown 6: 13619
Unknown 7: 0
Unknown 8: 0


--- Entry #0 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 80
Attributes: 2 (STL Read-Only)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 0
Partition Block Count: 1024
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: BOOT
Flash Filename: spl.img
FOTA Filename: 


--- Entry #1 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 90
Attributes: 2 (STL Read-Only)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 0
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: BOOT2
Flash Filename: spl2.img
FOTA Filename: 


--- Entry #2 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 70
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 1024
Partition Block Count: 1024
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PIT
Flash Filename: J3XLTE.pit
FOTA Filename: 


--- Entry #3 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 71
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 2048
Partition Block Count: 6144
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: MD5HDR
Flash Filename: md5.img
FOTA Filename: 


--- Entry #4 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 1
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 8192
Partition Block Count: 4096
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: SBOOT
Flash Filename: sboot.bin
FOTA Filename: 


--- Entry #5 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 2
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 12288
Partition Block Count: 4096
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: SBOOT2
Flash Filename: sboot2.bin
FOTA Filename: 


--- Entry #6 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 3
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 16384
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_fixnv1
Flash Filename: nvitem1.bin
FOTA Filename: 


--- Entry #7 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 4
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 18432
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_fixnv2
Flash Filename: nvitem.bin
FOTA Filename: 


--- Entry #8 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 5
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 20480
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: pm_sys
Flash Filename: PM_sharkl_arm7.bin
FOTA Filename: 


--- Entry #9 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 6
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 22528
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: rsvdfixnv1
Flash Filename: 
FOTA Filename: 


--- Entry #10 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 7
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 24576
Partition Block Count: 8192
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_ldsp
Flash Filename: SPRDLTEDSP.img
FOTA Filename: 


--- Entry #11 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 8
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 32768
Partition Block Count: 32768
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_modem
Flash Filename: SPRDCP.img
FOTA Filename: 


--- Entry #12 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 9
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 65536
Partition Block Count: 8192
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_gdsp
Flash Filename: SPRDGDSP.img
FOTA Filename: 


--- Entry #13 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 10
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 73728
Partition Block Count: 8192
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_warm
Flash Filename: SPRDWDSP.img
FOTA Filename: 


--- Entry #14 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 11
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 81920
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: FOTA_SIG
Flash Filename: 
FOTA Filename: 


--- Entry #15 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 12
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 83968
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_runtimenv1
Flash Filename: 
FOTA Filename: 


--- Entry #16 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 13
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 86016
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_runtimenv2
Flash Filename: 
FOTA Filename: 


--- Entry #17 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 14
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 88064
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: td_runtimenv1
Flash Filename: 
FOTA Filename: 


--- Entry #18 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 15
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 90112
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: td_runtimenv2
Flash Filename: 
FOTA Filename: 


--- Entry #19 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 16
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 92160
Partition Block Count: 4096
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PARAM
Flash Filename: param.lfs
FOTA Filename: 


--- Entry #20 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 17
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 96256
Partition Block Count: 40960
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: efs
Flash Filename: efs.img
FOTA Filename: 


--- Entry #21 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 18
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 137216
Partition Block Count: 10240
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: prodnv
Flash Filename: prodnv.img
FOTA Filename: 


--- Entry #22 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 19
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 147456
Partition Block Count: 12288
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: RESERVED2
Flash Filename: 
FOTA Filename: 


--- Entry #23 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 20
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 159744
Partition Block Count: 40960
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: KERNEL
Flash Filename: boot.img
FOTA Filename: 


--- Entry #24 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 21
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 200704
Partition Block Count: 40960
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: RECOVERY
Flash Filename: recovery.img
FOTA Filename: 


--- Entry #25 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 22
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 241664
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PERSISTENT
Flash Filename: 
FOTA Filename: 


--- Entry #26 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 23
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 243712
Partition Block Count: 18432
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PERSDATA
Flash Filename: persdata.img
FOTA Filename: 


--- Entry #27 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 24
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 262144
Partition Block Count: 409600
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: CACHE
Flash Filename: cache.img
FOTA Filename: 


--- Entry #28 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 25
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 671744
Partition Block Count: 4194304
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: SYSTEM
Flash Filename: system.img
FOTA Filename: 


--- Entry #29 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 26
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 4866048
Partition Block Count: 81920
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: HIDDEN
Flash Filename: hidden.img
FOTA Filename: 


--- Entry #30 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 27
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 4947968
Partition Block Count: 0
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: userdata
Flash Filename: userdata.img
FOTA Filename: remained

Ending session...
Releasing device interface...

Looking at the name, size and flash filename of the partitions, we can guess that the BOOT and BOOT2 are about splash screens, and what we need to use as our boot partition is in fact the KERNEL partition: /dev/block/mmcblk0p20 (note how the Identifier field from the PIT file matches what we got from block devices in the /dev filesystem). So, we edit our fstab file and add a line

/dev/block/platform/sdio_emmc/by-name/KERNEL   /boot    ext4 ro     wait

From the IRC #halium channel, I understood that the fixup mountpoints step is needed. The block I added to the halium/hybris-boot/fixup-mountpoints script is this:

    "j3xlte")
        sed -i \
            -e 's block/platform/sdio_emmc/by-name/SYSTEM mmcblk0p25 ' \
            -e 's block/platform/sdio_emmc/by-name/userdata mmcblk0p27 ' \
            -e 's block/platform/sdio_emmc/by-name/CACHE mmcblk0p24 ' \
            -e 's block/platform/sdio_emmc/by-name/efs mmcblk0p17 ' \
            -e 's block/platform/sdio_emmc/by-name/prodnv mmcblk0p18 ' \
            -e 's block/platform/sdio_emmc/by-name/KERNEL mmcblk0p20 ' \
            "$@"
        ;;

I then ran mka halium-boot again, and it finally started to build the kernel. But it soon stopped:

/mnt/Lavoro/halium/kernel/samsung/sharkls/kernel/cgroup.c: In function 'subsys_cgroup_allow_attach':
/mnt/Lavoro/halium/kernel/samsung/sharkls/kernel/cgroup.c:2138:37: error: invalid operands to binary != (have 'kuid_t' and 'kuid_t')
   if (current != task && cred->euid != tcred->uid &&
                                     ^
/mnt/Lavoro/halium/kernel/samsung/sharkls/kernel/cgroup.c:2139:18: error: invalid operands to binary != (have 'kuid_t' and 'kuid_t')
       cred->euid != tcred->suid)
                  ^

Luckily, this is one of the known build errors, so I didn't waste a lot of time on it: setting CONFIG_USER_NS=n in kernel/samsung/sharkls/arch/arm/configs/j3xlte_defconfig fixed it. Then the build failed with

  LD      init/built-in.o
mm/built-in.o: In function `set_AKSM_level':
/mnt/Lavoro/halium/kernel/samsung/sharkls/mm/ksm.c:1822: undefined reference to `get_minfree_high_value'
/mnt/Lavoro/halium/kernel/samsung/sharkls/Makefile:786: recipe for target 'vmlinux' failed
make[1]: *** [vmlinux] Error 1
Makefile:130: recipe for target 'sub-make' failed
make: *** [sub-make] Error 2

Looking at the code, I understood I could probably switch off this code path with CONFIG_ADAPTIVE_KSM=n. After doing that, got another error:

  LD      init/built-in.o
arch/arm/lib/lib.a(memcmpksm.o):(.ARM.exidx+0x0): undefined reference to `__aeabi_unwind_cpp_pr0'
/mnt/Lavoro/halium/kernel/samsung/sharkls/Makefile:786: recipe for target 'vmlinux' failed
make[1]: *** [vmlinux] Error 1
Makefile:130: recipe for target 'sub-make' failed
make: *** [sub-make] Error 2

Disabling also CONFIG_KSM_ASSEMBLY_MEMCMP did the trick:

  CC      /mnt/Lavoro/halium/vendor/sprd/wcn/wifi/sc2331/6.0/sprdwl.mod.o
  LD [M]  /mnt/Lavoro/halium/vendor/sprd/wcn/wifi/sc2331/6.0/sprdwl.ko
make[1]: Leaving directory '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/KERNEL_OBJ'
make: Leaving directory '/mnt/Lavoro/halium/vendor/sprd/wcn/wifi/sc2331/6.0'
[ 77% 7/9] Target dt image: /mnt/Lavoro/halium/out/target/product/j3xlte/dt.img
DTB combiner:
  Input directory: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/KERNEL_OBJ/arch/arm/boot/dts/'
  Output file: '/mnt/Lavoro/halium/out/target/product/j3xlte/dt.img'
Found file: sprd-scx35l_sharkls_j3xlte_rev03.dtb ... chipset: 9830, platform: 3, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev02.dtb ... chipset: 9830, platform: 2, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev06.dtb ... chipset: 9830, platform: 6, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev07.dtb ... chipset: 9830, platform: 7, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev04.dtb ... chipset: 9830, platform: 4, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev01.dtb ... chipset: 9830, platform: 1, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev05.dtb ... chipset: 9830, platform: 5, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev00.dtb ... chipset: 9830, platform: 0, rev: 131072
=> Found 8 unique DTB(s)

Generating master DTB... completed
Made DT image: /mnt/Lavoro/halium/out/target/product/j3xlte/dt.img
[100% 9/9] Install: /mnt/Lavoro/halium/out/target/product/j3xlte/halium-boot.img
make: Leaving directory '/mnt/Lavoro/halium'

#### make completed successfully (01:28 (mm:ss)) ####

Building the Halium system image

Continuing following the Halium documentation, the next step was building the system image with the mka systemimage command. Of course, it failed:

make: Entering directory '/mnt/Lavoro/halium'
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=7.1.1
LINEAGE_VERSION=14.1-20190702-UNOFFICIAL-j3xlte
TARGET_PRODUCT=lineage_j3xlte
TARGET_BUILD_VARIANT=userdebug
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a-neon
TARGET_CPU_VARIANT=cortex-a7
TARGET_2ND_ARCH=
TARGET_2ND_ARCH_VARIANT=
TARGET_2ND_CPU_VARIANT=
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-4.15.0-52-generic-x86_64-with-Ubuntu-16.04-xenial
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=NOF27B
OUT_DIR=/mnt/Lavoro/halium/out
RECOVERY_VARIANT=twrp
WITH_SU=true
============================================
Running kati to generate build-lineage_j3xlte.ninja...
No need to regenerate ninja file
Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libcurl_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstrongswan_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
make: Leaving directory '/mnt/Lavoro/halium'

#### make failed to build some targets (2 seconds) ####

I found similar (though not exactly the same) errors in #halium: the solution is to locate the .mk file causing the build of the package requiring the missing file, and remove it. I ran a repo grep libstrongswan, and found that it's mentioned in device/samsung/sharkls-common/sharkls.mk. I removed it from there, but I got the same build error. Removing the whole block of PRODUCT_PACKAGES led to some progress:

build/core/binary.mk:1253: vendor/sprd/modules/libcamera/Android.mk: libcamisp2.0: Unused source files: isp2.0/calibration/backup isp2.0/third_lib/alc_ip
build/core/binary.mk:1253: vendor/sprd/open-source/libs/vpu/mmf/openmax/libomxil-bellagio-0.9.3/Android.mk: omxregister-bellagio: Unused source files: src/omxregister.h
No private recovery resources for TARGET_DEVICE j3xlte
build/core/Makefile:34: warning: overriding commands for target `/mnt/Lavoro/halium/out/target/product/j3xlte/system/etc/wifi/wpa_supplicant.conf'
build/core/base_rules.mk:316: warning: ignoring old commands for target `/mnt/Lavoro/halium/out/target/product/j3xlte/system/etc/wifi/wpa_supplicant.conf'
Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/STATIC_LIBRARIES/libbt-utils_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/EXECUTABLES/engpc_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

I then removed engpc from the same sharkls.mk file:

Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/STATIC_LIBRARIES/libbootloader_message_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/EXECUTABLES/vold_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

The last comment from here suggests that vold might not be needed (though some people in #halium claimed that it could be needed, and indeed in the Meizu Pro 5 by Canonical I do have that program under /system/bin/). Anyway, I edited build/target/product/base.mk and removed vold from the PRODUCT_PACKAGES variable. I also read that it's recommended to run the make command as LANG=C mka systemimage, since the outcome can be affected by the user locale. Next:

Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libfmjni_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengfm_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

This is mentioned in vendor/sprd/wcn/platform/wcnd/Android.mk, so let's remove it from there (LOCAL_SHARED_LIBRARIES variable):

Starting build with ninja
ninja: Entering directory `.'
ninja: error: 'vendor/samsung/j3xlte/proprietary/bin/at_distributor', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/system/bin/at_distributor', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

This is brought in by device/samsung/j3xlte/proprietary-files.txt, so I commented it out from there. This didn't seem to make any difference, and a mka clean didn't help either. Turns out it's also mentioned in vendor/samsung/j3xlte/j3xlte-vendor-blobs.mk, so I removed it from there too (PRODUCT_COPY_FILES variable). Reading the comment block on top of this makefile we understand that it's generated by the setup-makefiles.sh script based on the contents of proprietary-files.txt. So, one option is to re-run the setup-makefiles.sh script, but I preferred to take a no risk approach and edit both files by hand. The next failure was about mfgloader, which is also mentioned in the same file:

Starting build with ninja
ninja: Entering directory `.'
ninja: error: 'vendor/samsung/j3xlte/proprietary/bin/mfgloader', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/system/bin/mfgloader', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

Let's try to remove it as well, from the same two files:

Starting build with ninja
ninja: Entering directory `.'
ninja: error: 'vendor/samsung/j3xlte/proprietary/bin/sprdSleepLog', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/system/bin/sprdSleepLog', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

Next is modemd (exactly same logs). At this point I started suspecting that this was not the right way to proceed, that that instead I was just missing all the proprietary blobs. The device tree repository has an extract-files.sh which seems to get these binaries from a device. After looking at its source, it looks like it must be run from the device/samsung/j3xlte directory with the device connected and reachable via adb. Before running it, I reverted my changes to the proprietary-files.txt file, in order to retrieve all the needed blobs. Running the extraction tool failed once:

./extract-files.sh 
846 KB/s (92260 bytes in 0.106s)
108 KB/s (9664 bytes in 0.086s)
106 KB/s (9444 bytes in 0.086s)
279 KB/s (34092 bytes in 0.119s)
44 KB/s (1992 bytes in 0.043s)
115 KB/s (5296 bytes in 0.044s)
2 KB/s (265 bytes in 0.087s)
40 KB/s (3770 bytes in 0.090s)
25 KB/s (2247 bytes in 0.087s)
4 KB/s (427 bytes in 0.087s)
0 KB/s (70 bytes in 0.087s)
remote object '/system/bin/IPSecService' does not exist

but after removing the missing file from proprietary-files.txt, the extraction completed successfully. However, one file was still not retrieved, as this line from the output showed:

[...]
4779 KB/s (1386252 bytes in 0.283s)
405 KB/s (37928 bytes in 0.091s)
failed to copy '/system/vendor/firmware/vbc_eq' to '../../../vendor/samsung/j3xlte/proprietary/vendor/firmware/vbc_eq': Permission denied
3015 KB/s (415875 bytes in 0.134s)
[...]

I checked the situation by connecting to the device with adb shell, and the file was there but readable by root only. After copying the file to /data/user and calling chmod 0666 vbc_eq, I could manually retrieve it with

adb pull /data/user/vbc_eq ../../../vendor/samsung/j3xlte/proprietary/vendor/firmware/vbc_eq

I preferred not to touch the permissions directly on the original file, not to risk messing up with the system; and after retrieving the file, I removed my copy from the device. The I ran

JOBS=2 ./halium/devices/setup j3xlte

once again. That completed fine, and the next mka systemimage failed with another error:

Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/STATIC_LIBRARIES/libbootloader_message_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/EXECUTABLES/init_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

I played with the android_system_core repository, which is checked out in system/core/, to remove references to bootloader_message from init/builtins.cpp, and I also removed it from the LOCAL_STATIC_LIBRARIES variable in the system/core/init/Android.mk file. Next:

Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/STATIC_LIBRARIES/libminui_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/EXECUTABLES/healthd_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

This is listed in build/target/product/core_base.mk and build/target/product/embedded.mk and needs to be removed from both (it was disabled in Halium's init, so it's definitely not needed). It then failed with:

Starting build with ninja
ninja: Entering directory `.'
ninja: error: 'bootable/recovery-twrp/etc/init.rc', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/ramdisk-recovery.cpio', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

The recovery-twrp project is listed in device/samsung/sharkls-common/lineage.dependencies, so let's try adding a line

<project path="bootable/recovery-twrp" name="omnirom/android_bootable_recovery" remote="github" revision="android-7.1" />

to our manifest file and run JOBS=2 ./halium/devices/setup j3xlte once more. After that, mka systemimage gives another error:

Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/host/linux-x86/bin/imgdiff', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/recovery_patch_intermediates/recovery_from_boot.p', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

Adding bootable/recovery-twrp to the subdirs variable in build/core/main.mk seems to get us further. But before showing you the next error, I need to stop for a second, because at this point I started suspecting that the previous error about the bootloader_message library could go away now that I started using the same recovery-twrp tree used by the LineageOS porter. So I reverted my changes about the bootloader_message library and indeed it looked like the previous error was gone forever. And it might be that some of the other stuff I disabled above might be restored back; but I decided to leave it for later (it's documented in this blog post, after all, so I can go back to it at any time) and proceed with the build:

Starting build with ninja
ninja: Entering directory `.'
[  0% 125/16364] host C++: libadb <= system/core/adb/usb_linux.cpp
FAILED: /bin/bash -c "(PWD=/proc/self/cwd  prebuilts/clang/host/linux-x86/clang-2690385/bin/clang++ -I device/samsung/sharkls-common/include -I system/core/adb -I /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates -I /mnt/Lavoro/halium/out/host/linux-x86/gen/STATIC_LIBRARIES/libadb_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem tools/include -isystem /mnt/Lavoro/halium/out/host/linux-x86/obj/include -c    -fno-exceptions -Wno-multichar -m64 -Wa,--noexecstack -fPIC -no-canonical-prefixes -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector -D__STDC_FORMAT_MACROS -D__STDC_CONSTANT_MACROS -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -O2 -g -fno-strict-aliasing -DNDEBUG -UDEBUG  -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics   --gcc-toolchain=prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8 -fstack-protector-strong    --gcc-toolchain=prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8 --sysroot prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/sysroot -Bprebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/bin -target x86_64-linux-gnu   -Wsign-promo  -Wno-inconsistent-missing-override   --gcc-toolchain=prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8 --sysroot prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/sysroot -isystem prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/include/c++/4.8 -isystem prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/include/c++/4.8/x86_64-linux -isystem prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/include/c++/4.8/backward -target x86_64-linux-gnu  -Wall -Wextra -Werror -Wno-unused-parameter -Wno-missing-field-initializers -Wvla -DADB_REVISION='\"80811505f1ff-android\"' -fvisibility=hidden -DADB_HOST=1 -fPIC -std=c++14 -Wexit-time-destructors -D_USING_LIBCXX -std=gnu++14 -nostdinc++  -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast  -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type    -MD -MF /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.d -o /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.o system/core/adb/usb_linux.cpp ) && (cp /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.d /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.d >> /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.P; rm -f /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.d )"
In file included from system/core/adb/usb_linux.cpp:25:
In file included from device/samsung/sharkls-common/include/linux/usb/ch9.h:35:
In file included from device/samsung/sharkls-common/include/uapi/linux/usb/ch9.h:36:
In file included from device/samsung/sharkls-common/include/linux/types.h:5:
In file included from device/samsung/sharkls-common/include/uapi/linux/types.h:13:
In file included from prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/sysroot/usr/include/linux/posix_types.h:4:
device/samsung/sharkls-common/include/linux/stddef.h:11:2: error: expected identifier
        false   = 0,
        ^
prebuilts/clang/host/linux-x86/clang-2690385/bin/../lib64/clang/3.8.256229/include/stdbool.h:38:15: note: expanded from macro 'false'
#define false false
              ^

(see here for the full log). After some resultless internet search and asking around, I tried to look into the issue myself, and noticed that in that failed command there's a mix of headers coming from the userspace and from the kernel (which usually are not used together). Since we are building userspace programs now, it seems a bit strange that there are also kernel headers being used (those under device/samsung/sharkls-common). Looking at system/core/adb/usb_linux.cpp, I saw that it contains a line #include <linux/usb/ch9.h> which causes the inclusion of device/samsung/sharkls-common/include/linux/usb/ch9.h; but maybe there are other ch9.h files in the build tree, provided by userspace libraries and not by the kernel? Indeed, a find showed that I had also a ./prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/sysroot/usr/include/linux/usb/ch9.h file, which should probably be used instead of the kernel's one. So I copied the failing command line (starting from what comes after FAILED:) into a terminal and ran it, and could reproduce the issue; but if after removing the first include directive (-I device/samsung/sharkls-common/include) the command succeeded.

The line is added in build/core/binary.mk:

my_c_includes := $(TOPDIR)$(TARGET_SPECIFIC_HEADER_PATH) $(my_c_includes)

(I found it out by modifying the file, and adding random text strings around the right-hand parts of these assignments, then running the build again and see which random strings were surrounding the device/samsung/sharkls-common/include text in the failing command line). So, we need to find why TARGET_SPECIFIC_HEADER_PATH is set. I ran a repo grep TARGET_SPECIFIC_HEADER_PATH and found that it's being set in device/samsung/sharkls-common/BoardConfigCommon.mk. Browsing the various directory trees, and double-checking the manifest file, I realized that the device/samsung/sharkls-common should not contain the kernel source tree, yet in my local disk it did (and all the files were untracked by git). It was probably the result of one of my previous attempts of compiling the manifest file, where some projects were checked out locally but I didn't clean them up after updating the manifest. I therefore ran a git clean -dfx in that directory, and the kernel source were gone. The next build continued for a while, then failed:

[ 76% 12454/16364] target thumb C: libengbt <= vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c
FAILED: /bin/bash -c "(PWD=/proc/self/cwd  prebuilts/clang/host/linux-x86/clang-2690385/bin/clang -I device/samsung/sharkls-common/include -I system/bt/stack/include -I system/bt/include -I vendor/sprd/open-source/apps/engmode/bt -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libengbt_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c    -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG  -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc  -target arm-linux-androideabi    -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin    -std=gnu99     -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing   -DHAS_NO_BDROID_BUILDCFG -DSPRD_WCNBT_MARLIN -fPIC -D_USING_LIBCXX   -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast  -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type  -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.o vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d )"
vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c:30:10: fatal error: 'bt_types.h' file not found
#include "bt_types.h"
         ^
1 error generated.

Indeed the bt_types.h was nowhere to be found in my source tree. A search in GitHub reveals that this file is provided by libncf-nci, so it's probably related to NFC. The local manifest from LineageOS indeed had a few lines with NFC-related projects (at least, judging from their name):

<project path="vendor/nxp-nfc/opensource/Nfc" name="LineageOS/android_vendor_nxp-nfc_opensource_Nfc" revision="cm-14.1" />
<project path="vendor/nxp-nfc/opensource/frameworks" name="LineageOS/android_vendor_nxp-nfc_opensource_frameworks" revision="cm-14.1" />
<project path="vendor/nxp-nfc/opensource/libnfc-nci" name="LineageOS/android_vendor_nxp-nfc_opensource_libnfc-nci" revision="cm-14.1" />

I initially excluded them from my manifest, given that my device doesn't have NFC, but at this point I thought I'd better include them, just for the sake of building the binaries (the alternative being removing the binaries required these libraries, or hacking on them to remove the dependency; this is a decision I might want to revisit later). So I added them to my manifest file (adding the proper remote attribute!), re-ran the setup script which fetched the newly-added repositories, but the build still failed with the same error, as the include lines were not updated. However, adding vendor/nxp-nfc/opensource/libnfc-nci/src/include to the TARGET_SPECIFIC_HEADER_PATH variable in device/samsung/sharkls-common/BoardConfigCommon.mk caused the build to fail even earlier. I then tried to add it to the local project being built (engmode), by changing the LOCAL_C_INCLUDES variable in vendor/sprd/open-source/apps/engmode/bt/Android.mk and that brought me just a little further:

[ 97% 1507/1552] target thumb C: libengbt <= vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c
FAILED: /bin/bash -c "(PWD=/proc/self/cwd  prebuilts/clang/host/linux-x86/clang-2690385/bin/clang -I device/samsung/sharkls-common/include -I system/bt/stack/include -I vendor/nxp-nfc/opensource/libnfc-nci/src/include -I system/bt/include -I vendor/sprd/open-source/apps/engmode/bt -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libengbt_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c    -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG  -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc  -target arm-linux-androideabi    -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin    -std=gnu99     -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing   -DHAS_NO_BDROID_BUILDCFG -DSPRD_WCNBT_MARLIN -fPIC -D_USING_LIBCXX   -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast  -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type  -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.o vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d )"
In file included from vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c:30:
vendor/nxp-nfc/opensource/libnfc-nci/src/include/bt_types.h:22:10: fatal error: 'data_types.h' file not found
#include "data_types.h"
         ^
1 error generated.

(note that in order to speed up the development I was no longer running mka systemimage, but mka libengbt, which is the library affected by the failure). I had three candidates:

vendor/nxp-nfc/opensource/libnfc-nci/p61-jcop-kit/inc/data_types.h
vendor/nxp-nfc/opensource/libnfc-nci/src/gki/ulinux/data_types.h
vendor/nxp-nfc/opensource/libnfc-nci/halimpl/bcm2079x/gki/ulinux/data_types.h

they didn't seem to be very different, so I picked the first one. I got then another failure:

[ 35% 12/34] target thumb C: libengbt <= vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c
FAILED: /bin/bash -c "(PWD=/proc/self/cwd  prebuilts/clang/host/linux-x86/clang-2690385/bin/clang -I device/samsung/sharkls-common/include -I system/bt/stack/include -I vendor/nxp-nfc/opensource/libnfc-nci/src/include -I vendor/nxp-nfc/opensource/libnfc-nci/p61-jcop-kit/inc/ -I system/bt/include -I vendor/sprd/open-source/apps/engmode/bt -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libengbt_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c    -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG  -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc  -target arm-linux-androideabi    -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin    -std=gnu99     -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing   -DHAS_NO_BDROID_BUILDCFG -DSPRD_WCNBT_MARLIN -fPIC -D_USING_LIBCXX   -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast  -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type  -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.o vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d )"
vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c:31:10: fatal error: 'btm_api.h' file not found
#include "btm_api.h"
         ^
1 error generated.

I search in the internet revealed that this is part of Bluedroid, which I assume is the Android bluetooth stack; so I looked in the Halium manifest directory for a repo line:

$ git grep blue
motorola_potter.xml:    <project path="external/bluetooth/bluedroid" name="android_external_bluetooth_bluedroid" remote="los" revision="cm-12.1" />

This one seemed promising, so I added it to my manifest. I ran the setup script again, and had a look at the newly downloaded external/bluetooth/bluedroid/stack/include/ directory: that not only included the btm_api.h file, but also the bt_types.h which we were needing before. So I went back and tried removing the two include lines related to the nxp-nfc projects and replaced them with this one from bluedroid. I got the same issue about the missing data_types.h file, so I added back the vendor/nxp-nfc/opensource/libnfc-nci/p61-jcop-kit/inc line. I then got an error about a missing tb_target.h file, which is found in external/bluetooth/bluedroid/include/, so I added this line as well. This let me build libengbt successfully. The systemimage target instead failed later:

[ 12% 502/3912] target thumb C++: audio.primary.sc8830 <= vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp
FAILED: /bin/bash -c "(PWD=/proc/self/cwd  prebuilts/clang/host/linux-x86/clang-2690385/bin/clang++ -I device/samsung/sharkls-common/include -I external/tinyalsa/include -I external/expat/lib -I system/media/audio_utils/include -I system/media/audio_effects/include -I vendor/sprd/open-source/apps/engmode -I vendor/sprd/modules/audio/normal/vb_pga -I vendor/sprd/modules/audio/normal/record_process -I vendor/sprd/modules/audio/normal/include -I vendor/sprd/modules/audio/normal/DumpData -I vendor/sprd/modules/audio/normal/audiotest -I vendor/sprd/modules/audio/normal/record_nr -I vendor/sprd/modules/audio/normal/skd -I vendor/sprd/modules/audio/normal/resample_api -I vendor/sprd/modules/audio/normal/custom_mmi -I vendor/sprd/modules/resampler -I vendor/sprd/modules/audio/normal/libaudioril -I vendor/sprd/open-source/libs/libatci -I vendor/sprd/modules/audio/normal/vb_effect/v2 -I vendor/sprd/modules/audio/normal -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/audio.primary.sc8830_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c    -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG  -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc  -target arm-linux-androideabi    -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin    -fvisibility-inlines-hidden -Wsign-promo  -Wno-inconsistent-missing-override -nostdlibinc  -target arm-linux-androideabi   -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing  -fno-rtti -D_POSIX_SOURCE -Wno-multichar -g -DAUDIO_SPIPE_TD -D_LPA_IRAM -DVOIP_DSP_PROCESS -DAUDIO_HAL_ANDROID_N_API -DFM_VERSION_IS_GOOGLE -fPIC -D_USING_LIBCXX -std=gnu++14  -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast  -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type    -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.o vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.d )"
vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp:21:9: warning: 'LOG_TAG' macro redefined [-Wmacro-redefined]
#define LOG_TAG "SPRD_AUDIOTRACK"
        ^
system/core/include/log/log.h:65:9: note: previous definition is here
#define LOG_TAG NULL
        ^
vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp:186:39: warning: unused parameter 'user' [-Wunused-parameter]
void playerCallback( int event, void* user, void *info )
                                      ^
vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp:245:9: error: use of undeclared identifier 'AUDIO_DEVICE_OUT_FM_HEADSET'; did you mean 'AUDIO_DEVICE_OUT_WIRED_HEADSET'?
        AUDIO_DEVICE_OUT_FM_HEADSET,
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~
        AUDIO_DEVICE_OUT_WIRED_HEADSET
system/media/audio/include/system/audio.h:711:5: note: 'AUDIO_DEVICE_OUT_WIRED_HEADSET' declared here
    AUDIO_DEVICE_OUT_WIRED_HEADSET             = 0x4,
    ^

This is an issue reported here, The link mentioned there is for CyanogenMod 13, but I found one for LineageOS 14.1 here. So I ran

cd device/samsung/sharkls-common/patches/
./apply_sprd-diff.sh

This failed on a couple of patches, for which I reported an issue here, though it's likely not important for us (as it mainly regards Java code). The build proceeded:

[ 10% 436/4011] target thumb C++: libstagefright <= frameworks/av/media/libstagefright/CameraSource.cpp
FAILED: /bin/bash -c "(PWD=/proc/self/cwd  prebuilts/clang/host/linux-x86/clang-2690385/bin/clang++ -I device/samsung/sharkls-common/include -I ./frameworks/av/include/media/ -I ./frameworks/av/media/libavextensions -I ./frameworks/av/media/libstagefright/mpeg2ts -I ./frameworks/av/include/media/stagefright/timedtext -I ./frameworks/native/include/media/hardware -I ./frameworks/native/include/media/openmax -I ./external/flac/include -I ./external/tremolo -I ./external/libvpx/libwebm -I ./system/netd/include -I system/media/audio_utils/include -I hardware/qcom/media/sc8830/mm-core/inc -I ./external/stagefright-plugins/include -I frameworks/av/media/libstagefright -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libstagefright_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c    -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG  -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc  -target arm-linux-androideabi    -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin    -fvisibility-inlines-hidden -Wsign-promo  -Wno-inconsistent-missing-override -nostdlibinc  -target arm-linux-androideabi   -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing  -fno-rtti -Wno-multichar -Werror -Wno-error=deprecated-declarations -Wall -DENABLE_STAGEFRIGHT_EXPERIMENTS -DUSE_SPRD_COLORFORMAT -fPIC -D_USING_LIBCXX -fsanitize=unsigned-integer-overflow,signed-integer-overflow -fsanitize-trap=all -ftrap-function=abort -std=gnu++14  -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast  -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type    -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.o frameworks/av/media/libstagefright/CameraSource.cpp ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.d )"
frameworks/av/media/libstagefright/CameraSource.cpp:134:16: error: use of undeclared identifier 'OMX_SPRD_COLOR_FormatYVU420SemiPlanar'; did you mean 'OMX_QCOM_COLOR_FormatYVU420SemiPlanar'?
        return OMX_SPRD_COLOR_FormatYVU420SemiPlanar;
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
               OMX_QCOM_COLOR_FormatYVU420SemiPlanar
./frameworks/native/include/media/openmax/OMX_IVCommon.h:169:5: note: 'OMX_QCOM_COLOR_FormatYVU420SemiPlanar' declared here
    OMX_QCOM_COLOR_FormatYVU420SemiPlanar = 0x7FA30C00,
    ^
1 error generated.

I realized that there were more patches to be applied, and that the script stopped as soon as it stumbled into a failing patch. So, I took all the failing patches out of the way:

mv sprd-diff/frameworks_base.diff{,.skip}
mv sprd-diff/frameworks_opt.diff{,.skip}
mv sprd-diff/packages_apps.diff{,.skip}
mv sprd-diff/packages_services.diff{,.skip}
mv sprd-diff/system_bt.diff{,.skip} # This looks potentially worth a revisit: should we have a system/bt dir?

The next build failed here:

[ 81% 3506/4283] target thumb C: libbt-vendor <= vendor/sprd/wcn/bt/libbt/src/bt_vendor_sprd.c
FAILED: /bin/bash -c "(PWD=/proc/self/cwd  prebuilts/clang/host/linux-x86/clang-2690385/bin/clang -I device/samsung/sharkls-common/include -I vendor/sprd/wcn/bt/libbt/include -I system/bt/hci/include -I vendor/sprd/wcn/bt/libbt/conf/sprd/marlin/include -I vendor/sprd/wcn/bt/libbt -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libbt-vendor_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c    -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG  -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc  -target arm-linux-androideabi    -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin    -std=gnu99     -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing   -fPIC -D_USING_LIBCXX   -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast  -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type  -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.o vendor/sprd/wcn/bt/libbt/src/bt_vendor_sprd.c ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.d )"
In file included from vendor/sprd/wcn/bt/libbt/src/bt_vendor_sprd.c:31:
vendor/sprd/wcn/bt/libbt/include/bt_vendor_sprd.h:33:10: fatal error: 'bt_vendor_lib.h' file not found
#include "bt_vendor_lib.h"
         ^
1 error generated.

I had this file in external/bluetooth/bluedroid/hci/include/bt_vendor_lib.h, and I was about to add this directory to the vendor/sprd/wcn/bt/libbt/Android.mk file when I realized that this file was already including $(BDROID_DIR)/hci/include and defined BDROID_DIR := $(TOP_DIR)system/bt. So I understood that I should have changed my manifest file to check out the bluedroid repo under system/bt (which also meant that I was wrong in excluding the sprd-diff/system_bt.diff patch before). I also had a quick look at other manifest files, and I saw that they were typically getting system/bt from the LineageOS android_system_bt repo. Therefore I went to modify my manifest file, changing the bluedroid line to

<project path="system/bt" name="android_system_bt" remote="los" revision="cm-14.1" />

and removed the three vendor/nxp-nfc repositories I added a while before. Then I manually applied the patch in device/samsung/sharkls-common/patches/sprd-diff/system_bt.diff and I revisited the vendor/sprd/open-source/apps/engmode/bt/Android.mk to remove all the include lines I added before. The next build succeeded:

[...]
[ 99% 2250/2255] build /mnt/Lavoro/halium/out/target/product/j3xlte/obj/NOTICE.html
Combining NOTICE files into HTML
Combining NOTICE files into text
[ 99% 2254/2255] Target system fs image: /mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img
BuildImage: in_dir = /mnt/Lavoro/halium/out/target/product/j3xlte/system, out_file = /mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img
fs type is not ext4
Running:  mkuserimg.sh -s /mnt/Lavoro/halium/out/target/product/j3xlte/system /mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img ext4 system 2147483648 -D /mnt/Lavoro/halium/out/target/product/j3xlte/system -L system /mnt/Lavoro/halium/out/target/product/j3xlte/root/file_contexts.bin
make_ext4fs -s -T -1 -S /mnt/Lavoro/halium/out/target/product/j3xlte/root/file_contexts.bin -L system -l 2147483648 -a system /mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img /mnt/Lavoro/halium/out/target/product/j3xlte/system /mnt/Lavoro/halium/out/target/product/j3xlte/system
Creating filesystem with parameters:
    Size: 2147483648
    Block size: 4096
    Blocks per group: 32768
    Inodes per group: 8192
    Inode size: 256
    Journal blocks: 8192
    Label: system
    Blocks: 524288
    Block groups: 16
    Reserved block group size: 127
Created filesystem with 1381/131072 inodes and 55467/524288 blocks
Running ['mkuserimg.sh', '-s', '/mnt/Lavoro/halium/out/target/product/j3xlte/system', '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img', 'ext4', 'system', '2147483648', '-D', '/mnt/Lavoro/halium/out/target/product/j3xlte/system', '-L', 'system', '/mnt/Lavoro/halium/out/target/product/j3xlte/root/file_contexts.bin'] command, exit code = 0
[ 99% 2254/2255] Construct recovery from boot
failed to reconstruct target deflate chunk 1 [(null)]; treating as normal
chunk 0: type 0 start 0 len 5582858
chunk 1: type 2 start 5582858 len 8611584
chunk 2: type 0 start 9205682 len 673886
Construct patches for 3 chunks...
patch   0 is 212 bytes (of 5582858)
patch   1 is 4311951 bytes (of 3622824)
patch   2 is 207 bytes (of 673886)
chunk   0: normal   (         0,    5582858)         212
chunk   1: deflate  (   5582858,    8574717)     4311951  (null)
chunk   2: normal   (  14157575,     674057)         207
[100% 2255/2255] Install system fs image: /mnt/Lavoro/halium/out/target/product/j3xlte/system.img
/mnt/Lavoro/halium/out/target/product/j3xlte/system.img+/mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/recovery_patch_intermediates/recovery_from_boot.p maxsize=2192424960 blocksize=135168 total=197470210 reserve=22167552
make: Leaving directory '/mnt/Lavoro/halium'

#### make completed successfully (01:22 (mm:ss)) ####

Just to be on the safe side (to make sure I didn't have stale files from previous build attempts), I ran

mka clean
mka halium-boot
LANG=C mka systemimage

again, without encountering any errors. I also ran mka hybris-boot as per the Halium documentation, as it might be a good idea to test the Halium reference image first. hybris-boot.img was built successfully at the first try.

Flashing and testing the Halium reference image

Following the Halium documentation, I downloaded the Halium rootfs image:

mkdir rootfs
cd rootfs
wget 'http://bshah.in/halium/halium-rootfs-20170630-151006.tar.gz'

A small section of the documentation is dedicated on how to install the boot image for Samsung devices; it closely matches the steps I took before to install TWRP, which is comforting.

adb reboot download    # much easier than the weird key combinations!
sudo heimdall flash --BOOT out/target/product/j3xlte/hybris-boot.img

The last command ended with this output:

Uploading BOOT
100%
BOOT upload successful

Ending session...
ERROR: Failed to receive session end confirmation!
Releasing device interface...

and the device was stuck on the same screen. I then thought of using TWRP to flash this device. I rebooted in download mode and installed TWRP as explained near the beginning of this blog post, then I uploaded my boot image to the SD card:

adb push out/target/product/j3xlte/hybris-boot.img /sdcard/

In TWRP, I did a wipe first: I wiped off the "System", "Data" and "cache" partitions (you can choose which partitions to wipe under the "Advanced wipe" option). Then I went back and pressed the "Install" button, then "Install Image" and picked my hybris-boot.img file (and chose "Boot" as target, of course). Make sure you don't reboot at this stage, or you'll lose adb access and the rest of the commands will fail.

I then installed the halium-install script:

git clone https://gitlab.com/JBBgameich/halium-install.git

I then downloaded the original Samsung ROM (see below for the link) and unzipped it in the rootfs directory I created before. That produced a file named J320FXXU0AQK2_J320FOXE0AQK2_J320FXXU0AQI1_HOME.tar.md5, from which I extracted the system.img and uploaded it to the device along with the halium rootfs, as explained in the "Halium guide":

tar xvf J320FXXU0AQK2_J320FOXE0AQK2_J320FXXU0AQI1_HOME.tar.md5 system.img
../halium-install/halium-install -p halium -v halium-rootfs-20170630-151006.tar.gz system.img

The last command complained about simg2img not being in $PATH, despite the fact that it was. A small change to the script fixed this. Once that was fixed it successfully pushed the files to the device, after asking me which password I wanted to set for the root user (on the phone): I chose a familiar "0000".

After rebooting, however, the device didn't go beyond the boot screen (with a red line warning about SELinux not being enabled). It just got stuck there, and was not even detected via USB.

Therefore, it's time to end this post with a "To be continued" message, and with a bonus section on how to restore the stock image below.

Restoring the Samsung stock ROM

The stock ROM can be downloaded from sammobile.com; you can restore the original image with this commands:

unzip J320*.zip
tar xvf J320*.tar.md5
sudo heimdall flash --KERNEL boot.img --CACHE cache.img \
    --HIDDEN hidden.img --l_fixnv2 nvitem.bin --PARAM param.lfs \
    --pm_sys PM_sharkl_arm7.bin --RECOVERY recovery.img \
    --SBOOT sboot.bin --SBOOT2 sboot2.bin --BOOT spl.img \
    --l_modem SPRDCP.img --l_gdsp SPRDGDSP.img --l_ldsp SPRDLTEDSP.img \
    --l_warm SPRDWDSP.img --SYSTEM system.img

If you were so crazy as to follow my steps, now running the command above is probably the wisest thing you could do. :-)

You know that I'm not an early adopter. That's why it was only a couple of weeks ago when I decided to give Qbs a try, by using the good old Mappero (and its spin-off, Mappero Geotagger) as a test bench. Yes, I know that the Qt company is not going to maintain Qbs anymore in the future, but the little I knew about Qbs was enough to convince me that it's a project worth supporting. So, better late than never -- and hopefully the community (me included) will do a good job in keeping Qbs thriving.

Having Mappero build with Qbs was the simplest thing ever. The only issue I met was in building the unit tests, because I'm used to set the rpath on test executables in order to make it easy to run them uninstalled, and with qmake I achieved that with this:

QMAKE_RPATHDIR = $${QMAKE_LIBDIR}

In turns out that with Qbs you can do it in almost the same way, but for some reason I couldn't figure it out and I even reported a bug to which I got some nice suggestions, before eventually settling on this:

import qbs 1.0

Test {
    name: "path-test"

    files: [
        "path-test.cpp",
        "path-test.h",
        "paths.qrc",
    ]

    Depends { name: "Mappero" }
    cpp.rpaths: cpp.libraryPaths    // <-- this does the trick!
}

It's surprisingly similar to how it's done in qmake, so it's not clear even to me why I didn't guess that immediately. Anyway, that was literally my only problem, and you can see the whole set of Qbs files I wrote by having a look at this commit.

Given how easy the migration was, I thought I should also try to add a code coverage report; that's not something I had in my qmake build either, but it's something I really want to have in all my newer projects.

Teaching Qbs to make a code coverage report

Unfortunately, my search for examples on how to have Qbs prepare a coverage report was mostly insuccessful, but thanks to some amazing help from Christian in the #qbs IRC channel, this was not hard to achieve. So, I hope to be of some help myself too, by sharing how this works.

First of all, it must be said that Qbs doesn't know anything about code coverage, at all. However, it's possible (and often easy) to extend Qbs by adding your own Product with its own set of build rules, so here's the CoverageReport item for Mappero (though, it should be general enough to be reusable in your own project):

import qbs

Product {
    name: "coverage"

    property string outputDirectory: "coverage-html"
    property stringList extractPatterns: []

    builtByDefault: false
    files: ["**"]
    type: ["coverage.html"]

    Depends { productTypes: ["autotest-result"] }

    Rule {
        multiplex: true
        explicitlyDependsOnFromDependencies: ["autotest-result"]
        outputFileTags: "coverage.html"
        requiresInputs: false
        prepare: {
            var commands = []
            var captureCmd = new Command("lcov", [
                "--directory", project.sourceDirectory,
                "--capture",
                "--output-file", "coverage.info",
                "--no-checksum",
                "--compat-libtool",
            ]);
            captureCmd.description = "Collecting coverage data";
            captureCmd.highlight = "coverage";
            captureCmd.silent = false;
            commands.push(captureCmd);

            var extractArgs = []
            for (var i = 0; i < product.extractPatterns.length; i++) {
                extractArgs.push("--extract");
                extractArgs.push("coverage.info");
                extractArgs.push(product.extractPatterns[i]);
            }
            if (product.extractPatterns.length > 0) {
                extractArgs.push("-o");
                extractArgs.push("coverage.info");
                var extractCmd = new Command("lcov", extractArgs);
                extractCmd.description = "Extracting coverage data";
                extractCmd.highlight = "coverage";
                extractCmd.silent = false;
                commands.push(extractCmd);
            }

            var filterCmd = new Command("lcov", [
                "--remove", "coverage.info", 'moc_*.cpp',
                "--remove", "coverage.info", 'qrc_*.cpp',
                "--remove", "coverage.info", '*/tests/*',
                "-o", "coverage.info",
            ]);
            filterCmd.description = "Filtering coverage data";
            filterCmd.highlight = "coverage";
            filterCmd.silent = false;
            commands.push(filterCmd);

            var genhtmlCmd = new Command("genhtml", [
                "--prefix", project.sourceDirectory,
                "--output-directory", product.outputDirectory,
                "--title", "Code coverage",
                "--legend",
                "--show-details",
                "coverage.info",
            ]);
            genhtmlCmd.description = "Generate HTML coverage report";
            genhtmlCmd.highlight = "coverage";
            genhtmlCmd.silent = false;
            commands.push(genhtmlCmd);

            return commands;
        }
    }
}

The most important thing here are the references to the autotest-result tag: this is the tag used by the AutotestRunner Qbs item, which is responsible for running the unit tests. Referencing its product's tag in the Depends item and in the explicitlyDependsOnFromDependencies properties ensures that "building" our product will cause the unit tests to run. Other needed bits are the requiresInputs: false property, which means that our rule doesn't have any required inputs, and the builtByDefault: false property, which says that our coverage report should not be generated when just typing qbs. Instead, to run the tests and get the code coverage report one will have to request it explicitly, by typing

qbs -p coverage

The prepare property of the Rule is where the commands to generate the code coverage report are defined. Here we can use the Command item to invoke external programs, and we return a list of such items, so that the commands will be executed in sequence. Note that here I'm using lcov and expecting to find the coverage data produced by gcov, so this is probably not portable outside of Linux/gcc.

Using the CoverageReport item is quite easy: you just need to declare it, and specify which paths contain the coverage data that you are interested in (otherwise, lcov will collect data from all object files that it find under the build directory, which might not be what you desire):

    CoverageReport {
        condition: project.enableCoverage
        extractPatterns: [ '*/src/*.cpp', '*/lib/*.cpp' ]
    }

There's little more than that to be done. Of course, you need to find a way to pass the --coverage option to gcc when building your products, and for this I created a small buildconfig module in qbs/modules/buildconfig/BuildConfig.qbs which I depend on in all products which I wish to build with coverage enabled:

import qbs

Module {
    cpp.cxxFlags: project.enableCoverage ? ["--coverage"] : undefined
    cpp.dynamicLibraries: project.enableCoverage ? ["gcov"] : undefined

    Depends { name: "cpp" }
}

If all this looks scary, you should probably have a look at the diff which shows how I added code coverage reporting to qbs: hopefully you'll find that it's not that complex, after all.

I hope that Qbs users will find this interesting, and possibly improving my setup. Ideally we should try to get something like this part of Qbs itself, but portability outside of Linux / gcc is going to be an issue.