fidzu : Maemo

The wonders of improvised Mastodon instances: one node disappears after an outage caused by a summer heatwave, leaving its users no way to migrate their data or to notify their followers.

After about one month of waiting for the node to come up or give some signals of life, I've decided to create a new account on another instance. If you use Mastodon and you were following me, please forgive me for the annoyance and follow me again here.

I was looking at TripAdvisor, some days ago. It's a very useful site, filled with user-generated advice and reviews, which has become almost a must for travellers. But I never like it when a private entity gets so much power over our lives (even if it's - currently - exercised in total fairness).

I would like to have a federated TripAdvisor-like network. I duckduckwent for a while, but I didn't find anything similar (if you know of some project of that kind, please let me know in the comments).

But thinking more about the issue, I realised that I probably wouldn't even bother to type my reviews into that site; I have a blog, so ideally I would like to have the option to write my review here, and then have the site import it. Technically, it could work with a webhook, or even a periodic check (real-time updates would not be a requirement here) over a URL I've linked to in my profile on that site. Then, in order for the posts to be imported, they would have to be entered in a standard format: maybe some keywords (or invisible HTML elements) could be used as markers for the content that need to be extracted from an otherwise ordinary blog post, or the relevant content could be replicated in a different format in the HTML headers (this, though, would require some additional work).

And while we are at it, why not extend this to other social networks? I use Mastodon, for example, and occasionally I send out a toot with a link to my latest blog post in there. But it would be much nicer if I could somehow set a special mark into my posts while composing them, to have them automatically tooted out on my account (this could probably be implemented as a standalone service, authorized to act on my Mastodon account - similarly to how the Mastodon-Twitter crossposter app works).

But I'm rather confident that I'm not the only one having this kind of needs, and that's why I'm writing this blog post: maybe someone out there has already found a solution, or has some more concrete ideas? If so, I'm all eyes!

A due premise: this is not a guide! While at the end I managed to build an image for my phone, it doesn't even boot. So, this post should be taken just as a collection of notes for myself, written to be able to reproduce the build and keep track of those steps I might want to revisit later.

The reason why I'm publishing these notes in my blog is that I believe they can still be useful for other people who should stumble into the same errors, and also to give you an idea of how much fun (or lack thereof) porting is.

It's a very long post, but unless you are starting to port Halium or Ubports to a new device, you can safely stop reading now.

I hope to write some better porting news in some future post. :-)

Before we start: getting the info

The first step is finding out a few informations about our device, and whether a LineageOS port for our device exists (hadn't there been a LineageOS port, I would have been out of luck: I certainly wouldn't have the time to do one myself). From the Settings application we can find the device name, the model number, the Android version and the kernel version:

• Device name: "Samsung Galaxy J3 (2016)"
• Model: SM-J320F
• Android version: 5.1.1
• Kernel version: 3.10.65-[…]

With this information we can look for a LineageOS port, either via a search engine or directly in the XDA forums; in my case the thread is this one. The post announcing the port should contain links to the source code, besides some other information. We do especially care about:

• the device codename: j3xlte
• the CM/LineageOS version: 14.1
• links to Device Tree, Kernel, Local Manifest (keep the local manifest handy, as you are going to need it soon)

To be on the safe side, get a copy of everything. In case these are GitHub repositories, forking them is enough.

Getting root access on the device

Another precondition for a successful port is the ability to be able to flash a custom kernel and rootfs image. The existance of a LineageOS port is not a guarantee that we can flash it: some devices are sold with a locked bootloader in certain markets or when sold by certain mobile operators. So, before spending time on the port, we should make sure that we can hack on our device freely.

In order to get adb working, I had to enable "Developer mode", which is done by hitting the "Build numer" item under the "Software info" page for 10 times in a row. Then from the newly appeared "Developer options" menu I enabled "USB debugging".

Being a total novice to Android phones, I had to go through several attempts and internet searches, before being able to get root access on the device (which I needed in order to read the /etc/fstab file, for example): by default, adb only let me as normal user, and su didn't work.

In order to flash a rootkit on this phone, we need to use Heimdall, because on Samsung devices this is what is used instead of fastboot. I just followed the instructions in the Linux/README file and could easily build and install it.

We are going to flash TWRP and SuperSU; a link to the former can be found in the XDA post about the LineageOS port for our device (under the "Download" button) and the latter from its own homepage (in my case I had to download the "Recovery flashable" zip file, and for some weird reason I got the 2.76 version, but I bet that any newer version works as well). Once I got these files, I followed these steps:

• Extract TWRP in my host machine: cd /tmp && tar xvf ~/Downloads/sm-j320fgm_twrp_3.1.1_f2fs.tar.md5 (that leaves a recovery.img file in my /tmp)
• Upload SuperSU to the phone:

    adb push ~/Downloads/UPDATE-SuperSU-v2.76-20160630161323.zip /sdcard/supersu.zip

• Power off the phone
• Power it on by keeping the VOLUME DOWN and HOME keys pressed while pressing the power button. The phone will be brought up into Odin mode.
• Run

    sudo heimdall flash  --RECOVERY /tmp/recovery.img

• Before the phone reboots, start keeping the VOLUME UP (note: it's not the same as before!) and HOME keys, to make it reboot into recovery and not into the ordinary image. If the phone boots into the ordinary image it will overwrite the TWRP we just flashed, so, if that happens, you need to start from scratch.

• Find supersu (the zip file) and install it.

If you reboot the phone and use adb shell, you'll be able to run su and get root rights. At this point, I charged myself with hopes and got to the real work.

Let the Odyssey start

I just followed the "First steps" instructions step by step, with no major surprises.

Once done with that, I moved to the "Get sources" page. Here it was not just a matter of following the instructions: assembling the manifest file took me several iterations, so I had to come back to this step a few times. Anyway, I followed the instructions for the halium-7.1 version, given that my device has a LineageOS 14.1 port. Then:

• I created a git branch in the halium/devices/ tree, to keep track of my changes
• I added the manifest halium/devices/manifests/samsung_j3xlte.xml (you can see the final version in my repository, but initially I was missing some important lines)
• Run JOBS=2 ./halium/devices/setup j3xlte
• The makefile needed to be changed: the device/samsung/j3xlte/setup-makefiles.sh script (from the device tree) was missing a mkdir -p ../../../$OUTDIR near the beginning.

At this point I continued to the "Build sources" page. Here starts a long trial and error phase, and the longer part of this post. For the sake of documentation, I will show you all the commands I ran, including failed attempts (there are many of them), just to give you an idea of what problems you might run into with your own port, and hopefully help in finding a solution.

I first ran the breakfast command for my device:

$ breakfast j3xlte
including vendor/cm/vendorsetup.sh
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist.  Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist.  Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist.  Stop.
Device j3xlte not found. Attempting to retrieve device repository from LineageOS Github (http://github.com/LineageOS).
Repository for j3xlte not found in the LineageOS Github repository list. If this is in error, you may need to manually add it to your local_manifests/roomservice.xml.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist.  Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/samsung/sharkls-common/sharkls.mk" does not exist.  Stop.

** Don't have a product spec for: 'lineage_j3xlte'
** Do you have the right repo manifest?

It looks like the device/samsung/sharkls-common was not checked out properly. After re-doing with force-sync, it failed with other errors:

$ breakfast j3xlte
including vendor/cm/vendorsetup.sh
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist.  Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist.  Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist.  Stop.
Device j3xlte not found. Attempting to retrieve device repository from LineageOS Github (http://github.com/LineageOS).
Repository for j3xlte not found in the LineageOS Github repository list. If this is in error, you may need to manually add it to your local_manifests/roomservice.xml.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist.  Stop.
build/core/product_config.mk:254: *** _nic.PRODUCTS.[[device/samsung/j3xlte/lineage.mk]]: "device/common/gps/gps_eu_supl.mk" does not exist.  Stop.

** Don't have a product spec for: 'lineage_j3xlte'
** Do you have the right repo manifest?

Indeed, I had nothing under device/common. So, I looked under halium/devices/manifests to see if any other manifest was populating this directory, and found that the bq_krillin.xml file had such a line. So, I copied that line (the one adding the lineageos/android_device_common project) into my manifest. After doing that, the breakfast command succeeded:

$ breakfast j3xlte
including vendor/cm/vendorsetup.sh
Trying dependencies-only mode on a non-existing device tree?

============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=7.1.1
LINEAGE_VERSION=14.1-20190619-UNOFFICIAL-j3xlte
TARGET_PRODUCT=lineage_j3xlte
TARGET_BUILD_VARIANT=userdebug
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a-neon
TARGET_CPU_VARIANT=cortex-a7
TARGET_2ND_ARCH=
TARGET_2ND_ARCH_VARIANT=
TARGET_2ND_CPU_VARIANT=
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-4.15.0-51-generic-x86_64-with-Ubuntu-16.04-xenial
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=NOF27B
OUT_DIR=/mnt/Lavoro/halium/out
RECOVERY_VARIANT=twrp
WITH_SU=true
============================================

Building the kernel

According to the livestream, the proper script to check for the kernel config is not the Mer one, but halium/halium-boot/check-kernel-config (this is also documented in the Ubports - Building Halium boot documentation page).

At this point I realized that I didn't have a kernel: an entry starting with kernel/ should always be in the manifest file. So I added the kernel line taken from the LineageOS local manifest file.

I then looked for the config file, which I eventually found in arch/arm/configs/j3xlte_defconfig. So I ran

./halium/halium-boot/check-kernel-config kernel/samsung/sharkls/arch/arm/configs/j3xlte_defconfig -w

which reported no errors. However, in the command output I noticed this:

 CONFIG_TMPFS_POSIX_ACL is already set
 CONFIG_DEFAULT_SECURITY is set, but to y "selinux" y not "apparmor".
 Setting CONFIG_DEFAULT_SECURITY="apparmor" correctly
sed: -e expression #1, char 60: unterminated `s' command
 Setting CONFIG_SECURITY_APPARMOR_BOOTPARAM_VALUE=1

When I checked the config file, I noticed that AppArmor was not really enabled. So I changed it manually.

Next, the fstab. I had the Ubports and Halium guides both opened in my browser, and I went for the "Fix mounts" step from the Ubports guide. I used find to locate the fstab, and found two:

• device/samsung/sharkls-common/recovery/root/fstab.sc8830
• device/samsung/sharkls-common/rootdir/etc/fstab.sc8830

The recovery file seemed already fine to me, while the rootdir one had most entry duplicated; ones with f2fs and xattrs, the other ones with ext4. I commented out the f2fs ones and ran mka halium-boot:

[...]
/mnt/Lavoro/halium/out/build-aosp_arm.ninja is missing, regenerating...
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=7.1.1
LINEAGE_VERSION=
TARGET_PRODUCT=aosp_arm
TARGET_BUILD_VARIANT=eng
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a
TARGET_CPU_VARIANT=generic
TARGET_2ND_ARCH=
TARGET_2ND_ARCH_VARIANT=
TARGET_2ND_CPU_VARIANT=
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-4.15.0-51-generic-x86_64-with-Ubuntu-16.04-xenial
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=NOF27B
OUT_DIR=/mnt/Lavoro/halium/out
============================================
Checking build tools versions...
find: 'device/*/generic': No such file or directory
find: 'device/unknown': No such file or directory
find: 'device/android': No such file or directory
find: 'device/*/generic': No such file or directory
find: 'device/unknown': No such file or directory
find: 'device/android': No such file or directory
halium/hybris-boot/Android.mk:69: ********************* /boot appears to live on ERROR: *fstab* not found
halium/hybris-boot/Android.mk:70: ********************* /data appears to live on ERROR: *fstab* not found
halium/hybris-boot/Android.mk:73: *** There should be a one and only one device entry for HYBRIS_BOOT_PART and HYBRIS_DATA_PART.
build/core/ninja.mk:166: recipe for target '/mnt/Lavoro/halium/out/build-aosp_arm.ninja' failed
make: *** [/mnt/Lavoro/halium/out/build-aosp_arm.ninja] Error 1
make: Leaving directory '/mnt/Lavoro/halium'

#### make failed to build some targets (11 seconds) ####

this seems to be documented in the "Common system build errors" page: I had to rerun source build/envsetup.sh and breakfast j3xlte. After that, the mka halium-boot command failed with

device/samsung/sharkls-common/ims/sec_samsung/Android.mk:25: build/core/java_library.mk: No such file or directory
build/core/ninja.mk:166: recipe for target '/mnt/Lavoro/halium/out/build-lineage_j3xlte.ninja' failed
make: *** [/mnt/Lavoro/halium/out/build-lineage_j3xlte.ninja] Error 1
make: Leaving directory '/mnt/Lavoro/halium'

which looks like a progress. The issue about the missing java_library.mk has been raised in the #halium channel; possible solutions include:

• symlink from static_java_library.mk
• edit the Android.mk file

I went for the former; this probably needs to be fixed in Halium.

Reran mka halium-boot, got some improvement: fstab is found:

halium/hybris-boot/Android.mk:70: ********************* /boot appears to live on 
halium/hybris-boot/Android.mk:71: ********************* /data appears to live on /dev/block/platform/sdio_emmc/by-name/userdata
halium/hybris-boot/Android.mk:74: *** There should be a one and only one device entry for HYBRIS_BOOT_PART and HYBRIS_DATA_PART.
build/core/ninja.mk:166: recipe for target '/mnt/Lavoro/halium/out/build-lineage_j3xlte.ninja' failed
make: *** [/mnt/Lavoro/halium/out/build-lineage_j3xlte.ninja] Error 1
make: Leaving directory '/mnt/Lavoro/halium'

It looks like my fstab was not complete, but missing the /boot partition. given that none of the fstab I located in my source tree had a line for the boot partition, I checked out the Android's fstab from the phone, via adb shell. There is no fstab in /etc, but two fstab files right in the root directory of the filesystem: fstab.sc8830 with exactly the same contents as the recovery fstab I found before in device/samsung/sharkls-common/recovery/root/fstab.sc8830, and a fstab.goldfish:

127|root@j3xlte:/ # cat /fstab.goldfish                                        
# Android fstab file.
#<src>                                                  <mnt_point>         <type>    <mnt_flags and options>                              <fs_mgr_flags>
# The filesystem that contains the filesystem checker binary (typically /system) cannot
# specify MF_CHECK, and must come before any filesystems that do specify MF_CHECK
/dev/block/mtdblock0                                    /system             ext4      ro,noatime,barrier=1                                         wait
/dev/block/mtdblock1                                    /data               ext4      noatime,nosuid,nodev,barrier=1,nomblk_io_submit      wait,check
/dev/block/mtdblock2                                    /cache              ext4      noatime,nosuid,nodev  wait,check
/devices/platform/goldfish_mmc.0                        auto                vfat      defaults                                             voldmanaged=sdcard:auto

While mount says:

~ # mount
rootfs on / type rootfs (rw,seclabel)
tmpfs on /dev type tmpfs (rw,seclabel,nosuid,relatime,mode=755)
devpts on /dev/pts type devpts (rw,seclabel,relatime,mode=600)
proc on /proc type proc (rw,relatime,gid=3009,hidepid=2)
sysfs on /sys type sysfs (rw,seclabel,relatime)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,seclabel,relatime)
pstore on /sys/fs/pstore type pstore (rw,seclabel,relatime)
adb on /dev/usb-ffs/adb type functionfs (rw,relatime)
adb on /dev/usb-ffs/adb type functionfs (rw,relatime)
/dev/block/mmcblk0p27 on /data type ext4 (rw,seclabel,relatime,data=ordered)
/dev/block/mmcblk0p27 on /sdcard type ext4 (rw,seclabel,relatime,data=ordered)
/dev/block/mmcblk0p24 on /cache type ext4 (rw,seclabel,relatime,data=ordered)

None of the above helps in locating the boot partition. I then tried to look under /dev/block:

root@j3xlte:/dev/block/platform/sdio_emmc/by-name # ls -l
lrwxrwxrwx root     root              2019-06-26 19:13 CACHE -> /dev/block/mmcblk0p24
lrwxrwxrwx root     root              2019-06-26 19:13 FOTA_SIG -> /dev/block/mmcblk0p11
lrwxrwxrwx root     root              2019-06-26 19:13 HIDDEN -> /dev/block/mmcblk0p26
lrwxrwxrwx root     root              2019-06-26 19:13 KERNEL -> /dev/block/mmcblk0p20
lrwxrwxrwx root     root              2019-06-26 19:13 PARAM -> /dev/block/mmcblk0p16
lrwxrwxrwx root     root              2019-06-26 19:13 PERSDATA -> /dev/block/mmcblk0p23
lrwxrwxrwx root     root              2019-06-26 19:13 PERSISTENT -> /dev/block/mmcblk0p22
lrwxrwxrwx root     root              2019-06-26 19:13 RECOVERY -> /dev/block/mmcblk0p21
lrwxrwxrwx root     root              2019-06-26 19:13 RESERVED2 -> /dev/block/mmcblk0p19
lrwxrwxrwx root     root              2019-06-26 19:13 SBOOT -> /dev/block/mmcblk0p1
lrwxrwxrwx root     root              2019-06-26 19:13 SBOOT2 -> /dev/block/mmcblk0p2
lrwxrwxrwx root     root              2019-06-26 19:13 SYSTEM -> /dev/block/mmcblk0p25
lrwxrwxrwx root     root              2019-06-26 19:13 efs -> /dev/block/mmcblk0p17
lrwxrwxrwx root     root              2019-06-26 19:13 l_fixnv1 -> /dev/block/mmcblk0p3
lrwxrwxrwx root     root              2019-06-26 19:13 l_fixnv2 -> /dev/block/mmcblk0p4
lrwxrwxrwx root     root              2019-06-26 19:13 l_gdsp -> /dev/block/mmcblk0p9
lrwxrwxrwx root     root              2019-06-26 19:13 l_ldsp -> /dev/block/mmcblk0p7
lrwxrwxrwx root     root              2019-06-26 19:13 l_modem -> /dev/block/mmcblk0p8
lrwxrwxrwx root     root              2019-06-26 19:13 l_runtimenv1 -> /dev/block/mmcblk0p12
lrwxrwxrwx root     root              2019-06-26 19:13 l_runtimenv2 -> /dev/block/mmcblk0p13
lrwxrwxrwx root     root              2019-06-26 19:13 l_warm -> /dev/block/mmcblk0p10
lrwxrwxrwx root     root              2019-06-26 19:13 pm_sys -> /dev/block/mmcblk0p5
lrwxrwxrwx root     root              2019-06-26 19:13 prodnv -> /dev/block/mmcblk0p18
lrwxrwxrwx root     root              2019-06-26 19:13 rsvdfixnv1 -> /dev/block/mmcblk0p6
lrwxrwxrwx root     root              2019-06-26 19:13 td_runtimenv1 -> /dev/block/mmcblk0p14
lrwxrwxrwx root     root              2019-06-26 19:13 td_runtimenv2 -> /dev/block/mmcblk0p15
lrwxrwxrwx root     root              2019-06-26 19:13 userdata -> /dev/block/mmcblk0p27

This, combined from the PIT file which we can extract with Heimdall from the "Odin mode", gives us a better understanding of the device partitions:

$ sudo heimdall print-pit --no-reboot
Heimdall v1.4.2

Copyright (c) 2010-2017 Benjamin Dobell, Glass Echidna
http://www.glassechidna.com.au/

This software is provided free of charge. Copying and redistribution is
encouraged.

If you appreciate this software and you would like to support future
development please consider donating:
http://www.glassechidna.com.au/donate/

Initialising connection...
Detecting device...
Claiming interface...
Setting up interface...

Initialising protocol...
Protocol initialisation successful.

Beginning session...

Some devices may take up to 2 minutes to respond.
Please be patient!

Session begun.

Downloading device's PIT file...
PIT file download successful.

Entry Count: 31
Unknown 1: 1598902083
Unknown 2: 844251476
Unknown 3: 20563
Unknown 4: 17490
Unknown 5: 14136
Unknown 6: 13619
Unknown 7: 0
Unknown 8: 0


--- Entry #0 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 80
Attributes: 2 (STL Read-Only)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 0
Partition Block Count: 1024
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: BOOT
Flash Filename: spl.img
FOTA Filename: 


--- Entry #1 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 90
Attributes: 2 (STL Read-Only)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 0
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: BOOT2
Flash Filename: spl2.img
FOTA Filename: 


--- Entry #2 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 70
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 1024
Partition Block Count: 1024
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PIT
Flash Filename: J3XLTE.pit
FOTA Filename: 


--- Entry #3 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 71
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 2048
Partition Block Count: 6144
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: MD5HDR
Flash Filename: md5.img
FOTA Filename: 


--- Entry #4 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 1
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 8192
Partition Block Count: 4096
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: SBOOT
Flash Filename: sboot.bin
FOTA Filename: 


--- Entry #5 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 2
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 12288
Partition Block Count: 4096
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: SBOOT2
Flash Filename: sboot2.bin
FOTA Filename: 


--- Entry #6 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 3
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 16384
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_fixnv1
Flash Filename: nvitem1.bin
FOTA Filename: 


--- Entry #7 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 4
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 18432
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_fixnv2
Flash Filename: nvitem.bin
FOTA Filename: 


--- Entry #8 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 5
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 20480
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: pm_sys
Flash Filename: PM_sharkl_arm7.bin
FOTA Filename: 


--- Entry #9 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 6
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 22528
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: rsvdfixnv1
Flash Filename: 
FOTA Filename: 


--- Entry #10 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 7
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 24576
Partition Block Count: 8192
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_ldsp
Flash Filename: SPRDLTEDSP.img
FOTA Filename: 


--- Entry #11 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 8
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 32768
Partition Block Count: 32768
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_modem
Flash Filename: SPRDCP.img
FOTA Filename: 


--- Entry #12 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 9
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 65536
Partition Block Count: 8192
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_gdsp
Flash Filename: SPRDGDSP.img
FOTA Filename: 


--- Entry #13 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 10
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 73728
Partition Block Count: 8192
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_warm
Flash Filename: SPRDWDSP.img
FOTA Filename: 


--- Entry #14 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 11
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 81920
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: FOTA_SIG
Flash Filename: 
FOTA Filename: 


--- Entry #15 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 12
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 83968
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_runtimenv1
Flash Filename: 
FOTA Filename: 


--- Entry #16 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 13
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 86016
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: l_runtimenv2
Flash Filename: 
FOTA Filename: 


--- Entry #17 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 14
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 88064
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: td_runtimenv1
Flash Filename: 
FOTA Filename: 


--- Entry #18 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 15
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 90112
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: td_runtimenv2
Flash Filename: 
FOTA Filename: 


--- Entry #19 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 16
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 92160
Partition Block Count: 4096
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PARAM
Flash Filename: param.lfs
FOTA Filename: 


--- Entry #20 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 17
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 96256
Partition Block Count: 40960
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: efs
Flash Filename: efs.img
FOTA Filename: 


--- Entry #21 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 18
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 137216
Partition Block Count: 10240
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: prodnv
Flash Filename: prodnv.img
FOTA Filename: 


--- Entry #22 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 19
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 147456
Partition Block Count: 12288
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: RESERVED2
Flash Filename: 
FOTA Filename: 


--- Entry #23 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 20
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 159744
Partition Block Count: 40960
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: KERNEL
Flash Filename: boot.img
FOTA Filename: 


--- Entry #24 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 21
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 200704
Partition Block Count: 40960
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: RECOVERY
Flash Filename: recovery.img
FOTA Filename: 


--- Entry #25 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 22
Attributes: 5 (Read/Write)
Update Attributes: 1 (FOTA)
Partition Block Size/Offset: 241664
Partition Block Count: 2048
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PERSISTENT
Flash Filename: 
FOTA Filename: 


--- Entry #26 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 23
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 243712
Partition Block Count: 18432
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: PERSDATA
Flash Filename: persdata.img
FOTA Filename: 


--- Entry #27 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 24
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 262144
Partition Block Count: 409600
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: CACHE
Flash Filename: cache.img
FOTA Filename: 


--- Entry #28 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 25
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 671744
Partition Block Count: 4194304
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: SYSTEM
Flash Filename: system.img
FOTA Filename: 


--- Entry #29 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 26
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 4866048
Partition Block Count: 81920
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: HIDDEN
Flash Filename: hidden.img
FOTA Filename: 


--- Entry #30 ---
Binary Type: 0 (AP)
Device Type: 2 (MMC)
Identifier: 27
Attributes: 5 (Read/Write)
Update Attributes: 5 (FOTA)
Partition Block Size/Offset: 4947968
Partition Block Count: 0
File Offset (Obsolete): 0
File Size (Obsolete): 0
Partition Name: userdata
Flash Filename: userdata.img
FOTA Filename: remained

Ending session...
Releasing device interface...

Looking at the name, size and flash filename of the partitions, we can guess that the BOOT and BOOT2 are about splash screens, and what we need to use as our boot partition is in fact the KERNEL partition: /dev/block/mmcblk0p20 (note how the Identifier field from the PIT file matches what we got from block devices in the /dev filesystem). So, we edit our fstab file and add a line

/dev/block/platform/sdio_emmc/by-name/KERNEL   /boot    ext4 ro     wait

From the IRC #halium channel, I understood that the fixup mountpoints step is needed. The block I added to the halium/hybris-boot/fixup-mountpoints script is this:

    "j3xlte")
        sed -i \
            -e 's block/platform/sdio_emmc/by-name/SYSTEM mmcblk0p25 ' \
            -e 's block/platform/sdio_emmc/by-name/userdata mmcblk0p27 ' \
            -e 's block/platform/sdio_emmc/by-name/CACHE mmcblk0p24 ' \
            -e 's block/platform/sdio_emmc/by-name/efs mmcblk0p17 ' \
            -e 's block/platform/sdio_emmc/by-name/prodnv mmcblk0p18 ' \
            -e 's block/platform/sdio_emmc/by-name/KERNEL mmcblk0p20 ' \
            "$@"
        ;;

I then ran mka halium-boot again, and it finally started to build the kernel. But it soon stopped:

/mnt/Lavoro/halium/kernel/samsung/sharkls/kernel/cgroup.c: In function 'subsys_cgroup_allow_attach':
/mnt/Lavoro/halium/kernel/samsung/sharkls/kernel/cgroup.c:2138:37: error: invalid operands to binary != (have 'kuid_t' and 'kuid_t')
   if (current != task && cred->euid != tcred->uid &&
                                     ^
/mnt/Lavoro/halium/kernel/samsung/sharkls/kernel/cgroup.c:2139:18: error: invalid operands to binary != (have 'kuid_t' and 'kuid_t')
       cred->euid != tcred->suid)
                  ^

Luckily, this is one of the known build errors, so I didn't waste a lot of time on it: setting CONFIG_USER_NS=n in kernel/samsung/sharkls/arch/arm/configs/j3xlte_defconfig fixed it. Then the build failed with

  LD      init/built-in.o
mm/built-in.o: In function `set_AKSM_level':
/mnt/Lavoro/halium/kernel/samsung/sharkls/mm/ksm.c:1822: undefined reference to `get_minfree_high_value'
/mnt/Lavoro/halium/kernel/samsung/sharkls/Makefile:786: recipe for target 'vmlinux' failed
make[1]: *** [vmlinux] Error 1
Makefile:130: recipe for target 'sub-make' failed
make: *** [sub-make] Error 2

Looking at the code, I understood I could probably switch off this code path with CONFIG_ADAPTIVE_KSM=n. After doing that, got another error:

  LD      init/built-in.o
arch/arm/lib/lib.a(memcmpksm.o):(.ARM.exidx+0x0): undefined reference to `__aeabi_unwind_cpp_pr0'
/mnt/Lavoro/halium/kernel/samsung/sharkls/Makefile:786: recipe for target 'vmlinux' failed
make[1]: *** [vmlinux] Error 1
Makefile:130: recipe for target 'sub-make' failed
make: *** [sub-make] Error 2

Disabling also CONFIG_KSM_ASSEMBLY_MEMCMP did the trick:

  CC      /mnt/Lavoro/halium/vendor/sprd/wcn/wifi/sc2331/6.0/sprdwl.mod.o
  LD [M]  /mnt/Lavoro/halium/vendor/sprd/wcn/wifi/sc2331/6.0/sprdwl.ko
make[1]: Leaving directory '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/KERNEL_OBJ'
make: Leaving directory '/mnt/Lavoro/halium/vendor/sprd/wcn/wifi/sc2331/6.0'
[ 77% 7/9] Target dt image: /mnt/Lavoro/halium/out/target/product/j3xlte/dt.img
DTB combiner:
  Input directory: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/KERNEL_OBJ/arch/arm/boot/dts/'
  Output file: '/mnt/Lavoro/halium/out/target/product/j3xlte/dt.img'
Found file: sprd-scx35l_sharkls_j3xlte_rev03.dtb ... chipset: 9830, platform: 3, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev02.dtb ... chipset: 9830, platform: 2, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev06.dtb ... chipset: 9830, platform: 6, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev07.dtb ... chipset: 9830, platform: 7, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev04.dtb ... chipset: 9830, platform: 4, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev01.dtb ... chipset: 9830, platform: 1, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev05.dtb ... chipset: 9830, platform: 5, rev: 131072
Found file: sprd-scx35l_sharkls_j3xlte_rev00.dtb ... chipset: 9830, platform: 0, rev: 131072
=> Found 8 unique DTB(s)

Generating master DTB... completed
Made DT image: /mnt/Lavoro/halium/out/target/product/j3xlte/dt.img
[100% 9/9] Install: /mnt/Lavoro/halium/out/target/product/j3xlte/halium-boot.img
make: Leaving directory '/mnt/Lavoro/halium'

#### make completed successfully (01:28 (mm:ss)) ####

Building the Halium system image

Continuing following the Halium documentation, the next step was building the system image with the mka systemimage command. Of course, it failed:

make: Entering directory '/mnt/Lavoro/halium'
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=7.1.1
LINEAGE_VERSION=14.1-20190702-UNOFFICIAL-j3xlte
TARGET_PRODUCT=lineage_j3xlte
TARGET_BUILD_VARIANT=userdebug
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a-neon
TARGET_CPU_VARIANT=cortex-a7
TARGET_2ND_ARCH=
TARGET_2ND_ARCH_VARIANT=
TARGET_2ND_CPU_VARIANT=
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-4.15.0-52-generic-x86_64-with-Ubuntu-16.04-xenial
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=NOF27B
OUT_DIR=/mnt/Lavoro/halium/out
RECOVERY_VARIANT=twrp
WITH_SU=true
============================================
Running kati to generate build-lineage_j3xlte.ninja...
No need to regenerate ninja file
Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libcurl_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstrongswan_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
make: Leaving directory '/mnt/Lavoro/halium'

#### make failed to build some targets (2 seconds) ####

I found similar (though not exactly the same) errors in #halium: the solution is to locate the .mk file causing the build of the package requiring the missing file, and remove it. I ran a repo grep libstrongswan, and found that it's mentioned in device/samsung/sharkls-common/sharkls.mk. I removed it from there, but I got the same build error. Removing the whole block of PRODUCT_PACKAGES led to some progress:

build/core/binary.mk:1253: vendor/sprd/modules/libcamera/Android.mk: libcamisp2.0: Unused source files: isp2.0/calibration/backup isp2.0/third_lib/alc_ip
build/core/binary.mk:1253: vendor/sprd/open-source/libs/vpu/mmf/openmax/libomxil-bellagio-0.9.3/Android.mk: omxregister-bellagio: Unused source files: src/omxregister.h
No private recovery resources for TARGET_DEVICE j3xlte
build/core/Makefile:34: warning: overriding commands for target `/mnt/Lavoro/halium/out/target/product/j3xlte/system/etc/wifi/wpa_supplicant.conf'
build/core/base_rules.mk:316: warning: ignoring old commands for target `/mnt/Lavoro/halium/out/target/product/j3xlte/system/etc/wifi/wpa_supplicant.conf'
Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/STATIC_LIBRARIES/libbt-utils_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/EXECUTABLES/engpc_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

I then removed engpc from the same sharkls.mk file:

Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/STATIC_LIBRARIES/libbootloader_message_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/EXECUTABLES/vold_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

The last comment from here suggests that vold might not be needed (though some people in #halium claimed that it could be needed, and indeed in the Meizu Pro 5 by Canonical I do have that program under /system/bin/). Anyway, I edited build/target/product/base.mk and removed vold from the PRODUCT_PACKAGES variable. I also read that it's recommended to run the make command as LANG=C mka systemimage, since the outcome can be affected by the user locale. Next:

Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libfmjni_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengfm_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

This is mentioned in vendor/sprd/wcn/platform/wcnd/Android.mk, so let's remove it from there (LOCAL_SHARED_LIBRARIES variable):

Starting build with ninja
ninja: Entering directory `.'
ninja: error: 'vendor/samsung/j3xlte/proprietary/bin/at_distributor', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/system/bin/at_distributor', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

This is brought in by device/samsung/j3xlte/proprietary-files.txt, so I commented it out from there. This didn't seem to make any difference, and a mka clean didn't help either. Turns out it's also mentioned in vendor/samsung/j3xlte/j3xlte-vendor-blobs.mk, so I removed it from there too (PRODUCT_COPY_FILES variable). Reading the comment block on top of this makefile we understand that it's generated by the setup-makefiles.sh script based on the contents of proprietary-files.txt. So, one option is to re-run the setup-makefiles.sh script, but I preferred to take a no risk approach and edit both files by hand. The next failure was about mfgloader, which is also mentioned in the same file:

Starting build with ninja
ninja: Entering directory `.'
ninja: error: 'vendor/samsung/j3xlte/proprietary/bin/mfgloader', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/system/bin/mfgloader', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

Let's try to remove it as well, from the same two files:

Starting build with ninja
ninja: Entering directory `.'
ninja: error: 'vendor/samsung/j3xlte/proprietary/bin/sprdSleepLog', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/system/bin/sprdSleepLog', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

Next is modemd (exactly same logs). At this point I started suspecting that this was not the right way to proceed, that that instead I was just missing all the proprietary blobs. The device tree repository has an extract-files.sh which seems to get these binaries from a device. After looking at its source, it looks like it must be run from the device/samsung/j3xlte directory with the device connected and reachable via adb. Before running it, I reverted my changes to the proprietary-files.txt file, in order to retrieve all the needed blobs. Running the extraction tool failed once:

./extract-files.sh 
846 KB/s (92260 bytes in 0.106s)
108 KB/s (9664 bytes in 0.086s)
106 KB/s (9444 bytes in 0.086s)
279 KB/s (34092 bytes in 0.119s)
44 KB/s (1992 bytes in 0.043s)
115 KB/s (5296 bytes in 0.044s)
2 KB/s (265 bytes in 0.087s)
40 KB/s (3770 bytes in 0.090s)
25 KB/s (2247 bytes in 0.087s)
4 KB/s (427 bytes in 0.087s)
0 KB/s (70 bytes in 0.087s)
remote object '/system/bin/IPSecService' does not exist

but after removing the missing file from proprietary-files.txt, the extraction completed successfully. However, one file was still not retrieved, as this line from the output showed:

[...]
4779 KB/s (1386252 bytes in 0.283s)
405 KB/s (37928 bytes in 0.091s)
failed to copy '/system/vendor/firmware/vbc_eq' to '../../../vendor/samsung/j3xlte/proprietary/vendor/firmware/vbc_eq': Permission denied
3015 KB/s (415875 bytes in 0.134s)
[...]

I checked the situation by connecting to the device with adb shell, and the file was there but readable by root only. After copying the file to /data/user and calling chmod 0666 vbc_eq, I could manually retrieve it with

adb pull /data/user/vbc_eq ../../../vendor/samsung/j3xlte/proprietary/vendor/firmware/vbc_eq

I preferred not to touch the permissions directly on the original file, not to risk messing up with the system; and after retrieving the file, I removed my copy from the device. The I ran

JOBS=2 ./halium/devices/setup j3xlte

once again. That completed fine, and the next mka systemimage failed with another error:

Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/STATIC_LIBRARIES/libbootloader_message_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/EXECUTABLES/init_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

I played with the android_system_core repository, which is checked out in system/core/, to remove references to bootloader_message from init/builtins.cpp, and I also removed it from the LOCAL_STATIC_LIBRARIES variable in the system/core/init/Android.mk file. Next:

Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/STATIC_LIBRARIES/libminui_intermediates/export_includes', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/EXECUTABLES/healthd_intermediates/import_includes', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

This is listed in build/target/product/core_base.mk and build/target/product/embedded.mk and needs to be removed from both (it was disabled in Halium's init, so it's definitely not needed). It then failed with:

Starting build with ninja
ninja: Entering directory `.'
ninja: error: 'bootable/recovery-twrp/etc/init.rc', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/ramdisk-recovery.cpio', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

The recovery-twrp project is listed in device/samsung/sharkls-common/lineage.dependencies, so let's try adding a line

<project path="bootable/recovery-twrp" name="omnirom/android_bootable_recovery" remote="github" revision="android-7.1" />

to our manifest file and run JOBS=2 ./halium/devices/setup j3xlte once more. After that, mka systemimage gives another error:

Starting build with ninja
ninja: Entering directory `.'
ninja: error: '/mnt/Lavoro/halium/out/host/linux-x86/bin/imgdiff', needed by '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/recovery_patch_intermediates/recovery_from_boot.p', missing and no known rule to make it
build/core/ninja.mk:151: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1

Adding bootable/recovery-twrp to the subdirs variable in build/core/main.mk seems to get us further. But before showing you the next error, I need to stop for a second, because at this point I started suspecting that the previous error about the bootloader_message library could go away now that I started using the same recovery-twrp tree used by the LineageOS porter. So I reverted my changes about the bootloader_message library and indeed it looked like the previous error was gone forever. And it might be that some of the other stuff I disabled above might be restored back; but I decided to leave it for later (it's documented in this blog post, after all, so I can go back to it at any time) and proceed with the build:

Starting build with ninja
ninja: Entering directory `.'
[  0% 125/16364] host C++: libadb <= system/core/adb/usb_linux.cpp
FAILED: /bin/bash -c "(PWD=/proc/self/cwd  prebuilts/clang/host/linux-x86/clang-2690385/bin/clang++ -I device/samsung/sharkls-common/include -I system/core/adb -I /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates -I /mnt/Lavoro/halium/out/host/linux-x86/gen/STATIC_LIBRARIES/libadb_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem tools/include -isystem /mnt/Lavoro/halium/out/host/linux-x86/obj/include -c    -fno-exceptions -Wno-multichar -m64 -Wa,--noexecstack -fPIC -no-canonical-prefixes -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector -D__STDC_FORMAT_MACROS -D__STDC_CONSTANT_MACROS -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -O2 -g -fno-strict-aliasing -DNDEBUG -UDEBUG  -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics   --gcc-toolchain=prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8 -fstack-protector-strong    --gcc-toolchain=prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8 --sysroot prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/sysroot -Bprebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/bin -target x86_64-linux-gnu   -Wsign-promo  -Wno-inconsistent-missing-override   --gcc-toolchain=prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8 --sysroot prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/sysroot -isystem prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/include/c++/4.8 -isystem prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/include/c++/4.8/x86_64-linux -isystem prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/include/c++/4.8/backward -target x86_64-linux-gnu  -Wall -Wextra -Werror -Wno-unused-parameter -Wno-missing-field-initializers -Wvla -DADB_REVISION='\"80811505f1ff-android\"' -fvisibility=hidden -DADB_HOST=1 -fPIC -std=c++14 -Wexit-time-destructors -D_USING_LIBCXX -std=gnu++14 -nostdinc++  -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast  -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type    -MD -MF /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.d -o /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.o system/core/adb/usb_linux.cpp ) && (cp /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.d /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.d >> /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.P; rm -f /mnt/Lavoro/halium/out/host/linux-x86/obj/STATIC_LIBRARIES/libadb_intermediates/usb_linux.d )"
In file included from system/core/adb/usb_linux.cpp:25:
In file included from device/samsung/sharkls-common/include/linux/usb/ch9.h:35:
In file included from device/samsung/sharkls-common/include/uapi/linux/usb/ch9.h:36:
In file included from device/samsung/sharkls-common/include/linux/types.h:5:
In file included from device/samsung/sharkls-common/include/uapi/linux/types.h:13:
In file included from prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/sysroot/usr/include/linux/posix_types.h:4:
device/samsung/sharkls-common/include/linux/stddef.h:11:2: error: expected identifier
        false   = 0,
        ^
prebuilts/clang/host/linux-x86/clang-2690385/bin/../lib64/clang/3.8.256229/include/stdbool.h:38:15: note: expanded from macro 'false'
#define false false
              ^

(see here for the full log). After some resultless internet search and asking around, I tried to look into the issue myself, and noticed that in that failed command there's a mix of headers coming from the userspace and from the kernel (which usually are not used together). Since we are building userspace programs now, it seems a bit strange that there are also kernel headers being used (those under device/samsung/sharkls-common). Looking at system/core/adb/usb_linux.cpp, I saw that it contains a line #include <linux/usb/ch9.h> which causes the inclusion of device/samsung/sharkls-common/include/linux/usb/ch9.h; but maybe there are other ch9.h files in the build tree, provided by userspace libraries and not by the kernel? Indeed, a find showed that I had also a ./prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/sysroot/usr/include/linux/usb/ch9.h file, which should probably be used instead of the kernel's one. So I copied the failing command line (starting from what comes after FAILED:) into a terminal and ran it, and could reproduce the issue; but if after removing the first include directive (-I device/samsung/sharkls-common/include) the command succeeded.

The line is added in build/core/binary.mk:

my_c_includes := $(TOPDIR)$(TARGET_SPECIFIC_HEADER_PATH) $(my_c_includes)

(I found it out by modifying the file, and adding random text strings around the right-hand parts of these assignments, then running the build again and see which random strings were surrounding the device/samsung/sharkls-common/include text in the failing command line). So, we need to find why TARGET_SPECIFIC_HEADER_PATH is set. I ran a repo grep TARGET_SPECIFIC_HEADER_PATH and found that it's being set in device/samsung/sharkls-common/BoardConfigCommon.mk. Browsing the various directory trees, and double-checking the manifest file, I realized that the device/samsung/sharkls-common should not contain the kernel source tree, yet in my local disk it did (and all the files were untracked by git). It was probably the result of one of my previous attempts of compiling the manifest file, where some projects were checked out locally but I didn't clean them up after updating the manifest. I therefore ran a git clean -dfx in that directory, and the kernel source were gone. The next build continued for a while, then failed:

[ 76% 12454/16364] target thumb C: libengbt <= vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c
FAILED: /bin/bash -c "(PWD=/proc/self/cwd  prebuilts/clang/host/linux-x86/clang-2690385/bin/clang -I device/samsung/sharkls-common/include -I system/bt/stack/include -I system/bt/include -I vendor/sprd/open-source/apps/engmode/bt -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libengbt_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c    -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG  -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc  -target arm-linux-androideabi    -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin    -std=gnu99     -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing   -DHAS_NO_BDROID_BUILDCFG -DSPRD_WCNBT_MARLIN -fPIC -D_USING_LIBCXX   -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast  -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type  -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.o vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d )"
vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c:30:10: fatal error: 'bt_types.h' file not found
#include "bt_types.h"
         ^
1 error generated.

Indeed the bt_types.h was nowhere to be found in my source tree. A search in GitHub reveals that this file is provided by libncf-nci, so it's probably related to NFC. The local manifest from LineageOS indeed had a few lines with NFC-related projects (at least, judging from their name):

<project path="vendor/nxp-nfc/opensource/Nfc" name="LineageOS/android_vendor_nxp-nfc_opensource_Nfc" revision="cm-14.1" />
<project path="vendor/nxp-nfc/opensource/frameworks" name="LineageOS/android_vendor_nxp-nfc_opensource_frameworks" revision="cm-14.1" />
<project path="vendor/nxp-nfc/opensource/libnfc-nci" name="LineageOS/android_vendor_nxp-nfc_opensource_libnfc-nci" revision="cm-14.1" />

I initially excluded them from my manifest, given that my device doesn't have NFC, but at this point I thought I'd better include them, just for the sake of building the binaries (the alternative being removing the binaries required these libraries, or hacking on them to remove the dependency; this is a decision I might want to revisit later). So I added them to my manifest file (adding the proper remote attribute!), re-ran the setup script which fetched the newly-added repositories, but the build still failed with the same error, as the include lines were not updated. However, adding vendor/nxp-nfc/opensource/libnfc-nci/src/include to the TARGET_SPECIFIC_HEADER_PATH variable in device/samsung/sharkls-common/BoardConfigCommon.mk caused the build to fail even earlier. I then tried to add it to the local project being built (engmode), by changing the LOCAL_C_INCLUDES variable in vendor/sprd/open-source/apps/engmode/bt/Android.mk and that brought me just a little further:

[ 97% 1507/1552] target thumb C: libengbt <= vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c
FAILED: /bin/bash -c "(PWD=/proc/self/cwd  prebuilts/clang/host/linux-x86/clang-2690385/bin/clang -I device/samsung/sharkls-common/include -I system/bt/stack/include -I vendor/nxp-nfc/opensource/libnfc-nci/src/include -I system/bt/include -I vendor/sprd/open-source/apps/engmode/bt -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libengbt_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c    -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG  -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc  -target arm-linux-androideabi    -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin    -std=gnu99     -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing   -DHAS_NO_BDROID_BUILDCFG -DSPRD_WCNBT_MARLIN -fPIC -D_USING_LIBCXX   -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast  -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type  -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.o vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d )"
In file included from vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c:30:
vendor/nxp-nfc/opensource/libnfc-nci/src/include/bt_types.h:22:10: fatal error: 'data_types.h' file not found
#include "data_types.h"
         ^
1 error generated.

(note that in order to speed up the development I was no longer running mka systemimage, but mka libengbt, which is the library affected by the failure). I had three candidates:

vendor/nxp-nfc/opensource/libnfc-nci/p61-jcop-kit/inc/data_types.h
vendor/nxp-nfc/opensource/libnfc-nci/src/gki/ulinux/data_types.h
vendor/nxp-nfc/opensource/libnfc-nci/halimpl/bcm2079x/gki/ulinux/data_types.h

they didn't seem to be very different, so I picked the first one. I got then another failure:

[ 35% 12/34] target thumb C: libengbt <= vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c
FAILED: /bin/bash -c "(PWD=/proc/self/cwd  prebuilts/clang/host/linux-x86/clang-2690385/bin/clang -I device/samsung/sharkls-common/include -I system/bt/stack/include -I vendor/nxp-nfc/opensource/libnfc-nci/src/include -I vendor/nxp-nfc/opensource/libnfc-nci/p61-jcop-kit/inc/ -I system/bt/include -I vendor/sprd/open-source/apps/engmode/bt -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libengbt_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c    -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG  -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc  -target arm-linux-androideabi    -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin    -std=gnu99     -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing   -DHAS_NO_BDROID_BUILDCFG -DSPRD_WCNBT_MARLIN -fPIC -D_USING_LIBCXX   -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast  -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type  -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.o vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libengbt_intermediates/bt_cmd_executer.d )"
vendor/sprd/open-source/apps/engmode/bt/bt_cmd_executer.c:31:10: fatal error: 'btm_api.h' file not found
#include "btm_api.h"
         ^
1 error generated.

I search in the internet revealed that this is part of Bluedroid, which I assume is the Android bluetooth stack; so I looked in the Halium manifest directory for a repo line:

$ git grep blue
motorola_potter.xml:    <project path="external/bluetooth/bluedroid" name="android_external_bluetooth_bluedroid" remote="los" revision="cm-12.1" />

This one seemed promising, so I added it to my manifest. I ran the setup script again, and had a look at the newly downloaded external/bluetooth/bluedroid/stack/include/ directory: that not only included the btm_api.h file, but also the bt_types.h which we were needing before. So I went back and tried removing the two include lines related to the nxp-nfc projects and replaced them with this one from bluedroid. I got the same issue about the missing data_types.h file, so I added back the vendor/nxp-nfc/opensource/libnfc-nci/p61-jcop-kit/inc line. I then got an error about a missing tb_target.h file, which is found in external/bluetooth/bluedroid/include/, so I added this line as well. This let me build libengbt successfully. The systemimage target instead failed later:

[ 12% 502/3912] target thumb C++: audio.primary.sc8830 <= vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp
FAILED: /bin/bash -c "(PWD=/proc/self/cwd  prebuilts/clang/host/linux-x86/clang-2690385/bin/clang++ -I device/samsung/sharkls-common/include -I external/tinyalsa/include -I external/expat/lib -I system/media/audio_utils/include -I system/media/audio_effects/include -I vendor/sprd/open-source/apps/engmode -I vendor/sprd/modules/audio/normal/vb_pga -I vendor/sprd/modules/audio/normal/record_process -I vendor/sprd/modules/audio/normal/include -I vendor/sprd/modules/audio/normal/DumpData -I vendor/sprd/modules/audio/normal/audiotest -I vendor/sprd/modules/audio/normal/record_nr -I vendor/sprd/modules/audio/normal/skd -I vendor/sprd/modules/audio/normal/resample_api -I vendor/sprd/modules/audio/normal/custom_mmi -I vendor/sprd/modules/resampler -I vendor/sprd/modules/audio/normal/libaudioril -I vendor/sprd/open-source/libs/libatci -I vendor/sprd/modules/audio/normal/vb_effect/v2 -I vendor/sprd/modules/audio/normal -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/audio.primary.sc8830_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c    -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG  -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc  -target arm-linux-androideabi    -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin    -fvisibility-inlines-hidden -Wsign-promo  -Wno-inconsistent-missing-override -nostdlibinc  -target arm-linux-androideabi   -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing  -fno-rtti -D_POSIX_SOURCE -Wno-multichar -g -DAUDIO_SPIPE_TD -D_LPA_IRAM -DVOIP_DSP_PROCESS -DAUDIO_HAL_ANDROID_N_API -DFM_VERSION_IS_GOOGLE -fPIC -D_USING_LIBCXX -std=gnu++14  -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast  -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type    -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.o vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/audio.primary.sc8830_intermediates/audiotest/auto_audio_v2.d )"
vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp:21:9: warning: 'LOG_TAG' macro redefined [-Wmacro-redefined]
#define LOG_TAG "SPRD_AUDIOTRACK"
        ^
system/core/include/log/log.h:65:9: note: previous definition is here
#define LOG_TAG NULL
        ^
vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp:186:39: warning: unused parameter 'user' [-Wunused-parameter]
void playerCallback( int event, void* user, void *info )
                                      ^
vendor/sprd/modules/audio/normal/audiotest/auto_audio_v2.cpp:245:9: error: use of undeclared identifier 'AUDIO_DEVICE_OUT_FM_HEADSET'; did you mean 'AUDIO_DEVICE_OUT_WIRED_HEADSET'?
        AUDIO_DEVICE_OUT_FM_HEADSET,
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~
        AUDIO_DEVICE_OUT_WIRED_HEADSET
system/media/audio/include/system/audio.h:711:5: note: 'AUDIO_DEVICE_OUT_WIRED_HEADSET' declared here
    AUDIO_DEVICE_OUT_WIRED_HEADSET             = 0x4,
    ^

This is an issue reported here, The link mentioned there is for CyanogenMod 13, but I found one for LineageOS 14.1 here. So I ran

cd device/samsung/sharkls-common/patches/
./apply_sprd-diff.sh

This failed on a couple of patches, for which I reported an issue here, though it's likely not important for us (as it mainly regards Java code). The build proceeded:

[ 10% 436/4011] target thumb C++: libstagefright <= frameworks/av/media/libstagefright/CameraSource.cpp
FAILED: /bin/bash -c "(PWD=/proc/self/cwd  prebuilts/clang/host/linux-x86/clang-2690385/bin/clang++ -I device/samsung/sharkls-common/include -I ./frameworks/av/include/media/ -I ./frameworks/av/media/libavextensions -I ./frameworks/av/media/libstagefright/mpeg2ts -I ./frameworks/av/include/media/stagefright/timedtext -I ./frameworks/native/include/media/hardware -I ./frameworks/native/include/media/openmax -I ./external/flac/include -I ./external/tremolo -I ./external/libvpx/libwebm -I ./system/netd/include -I system/media/audio_utils/include -I hardware/qcom/media/sc8830/mm-core/inc -I ./external/stagefright-plugins/include -I frameworks/av/media/libstagefright -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libstagefright_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c    -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG  -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc  -target arm-linux-androideabi    -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin    -fvisibility-inlines-hidden -Wsign-promo  -Wno-inconsistent-missing-override -nostdlibinc  -target arm-linux-androideabi   -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing  -fno-rtti -Wno-multichar -Werror -Wno-error=deprecated-declarations -Wall -DENABLE_STAGEFRIGHT_EXPERIMENTS -DUSE_SPRD_COLORFORMAT -fPIC -D_USING_LIBCXX -fsanitize=unsigned-integer-overflow,signed-integer-overflow -fsanitize-trap=all -ftrap-function=abort -std=gnu++14  -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast  -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type    -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.o frameworks/av/media/libstagefright/CameraSource.cpp ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libstagefright_intermediates/CameraSource.d )"
frameworks/av/media/libstagefright/CameraSource.cpp:134:16: error: use of undeclared identifier 'OMX_SPRD_COLOR_FormatYVU420SemiPlanar'; did you mean 'OMX_QCOM_COLOR_FormatYVU420SemiPlanar'?
        return OMX_SPRD_COLOR_FormatYVU420SemiPlanar;
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
               OMX_QCOM_COLOR_FormatYVU420SemiPlanar
./frameworks/native/include/media/openmax/OMX_IVCommon.h:169:5: note: 'OMX_QCOM_COLOR_FormatYVU420SemiPlanar' declared here
    OMX_QCOM_COLOR_FormatYVU420SemiPlanar = 0x7FA30C00,
    ^
1 error generated.

I realized that there were more patches to be applied, and that the script stopped as soon as it stumbled into a failing patch. So, I took all the failing patches out of the way:

mv sprd-diff/frameworks_base.diff{,.skip}
mv sprd-diff/frameworks_opt.diff{,.skip}
mv sprd-diff/packages_apps.diff{,.skip}
mv sprd-diff/packages_services.diff{,.skip}
mv sprd-diff/system_bt.diff{,.skip} # This looks potentially worth a revisit: should we have a system/bt dir?

The next build failed here:

[ 81% 3506/4283] target thumb C: libbt-vendor <= vendor/sprd/wcn/bt/libbt/src/bt_vendor_sprd.c
FAILED: /bin/bash -c "(PWD=/proc/self/cwd  prebuilts/clang/host/linux-x86/clang-2690385/bin/clang -I device/samsung/sharkls-common/include -I vendor/sprd/wcn/bt/libbt/include -I system/bt/hci/include -I vendor/sprd/wcn/bt/libbt/conf/sprd/marlin/include -I vendor/sprd/wcn/bt/libbt -I /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates -I /mnt/Lavoro/halium/out/target/product/j3xlte/gen/SHARED_LIBRARIES/libbt-vendor_intermediates -I libnativehelper/include/nativehelper \$(cat /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/import_includes) -isystem system/core/include -isystem system/media/audio/include -isystem hardware/libhardware/include -isystem hardware/libhardware_legacy/include -isystem libnativehelper/include -isystem frameworks/native/include -isystem frameworks/native/opengl/include -isystem frameworks/av/include -isystem frameworks/base/include -isystem hardware/ril/include -isystem /mnt/Lavoro/halium/out/target/product/j3xlte/obj/include -isystem bionic/libc/arch-arm/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/common -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libm/include -isystem bionic/libm/include/arm -c    -fno-exceptions -Wno-multichar -msoft-float -ffunction-sections -fdata-sections -funwind-tables -fstack-protector-strong -Wa,--noexecstack -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-short-enums -no-canonical-prefixes -mcpu=cortex-a7 -mfpu=neon-vfpv4 -D__ARM_FEATURE_LPAE=1 -mfloat-abi=softfp -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -DNDEBUG -g -Wstrict-aliasing=2 -DNDEBUG -UDEBUG  -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -nostdlibinc  -target arm-linux-androideabi    -target arm-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin    -std=gnu99     -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing   -fPIC -D_USING_LIBCXX   -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast  -Werror=address-of-temporary -Werror=null-dereference -Werror=return-type  -MD -MF /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.d -o /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.o vendor/sprd/wcn/bt/libbt/src/bt_vendor_sprd.c ) && (cp /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.d /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.P; sed -e 's/#.*//' -e 's/^[^:]*: *//' -e 's/ *\\\\\$//' -e '/^\$/ d' -e 's/\$/ :/' < /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.d >> /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.P; rm -f /mnt/Lavoro/halium/out/target/product/j3xlte/obj/SHARED_LIBRARIES/libbt-vendor_intermediates/src/bt_vendor_sprd.d )"
In file included from vendor/sprd/wcn/bt/libbt/src/bt_vendor_sprd.c:31:
vendor/sprd/wcn/bt/libbt/include/bt_vendor_sprd.h:33:10: fatal error: 'bt_vendor_lib.h' file not found
#include "bt_vendor_lib.h"
         ^
1 error generated.

I had this file in external/bluetooth/bluedroid/hci/include/bt_vendor_lib.h, and I was about to add this directory to the vendor/sprd/wcn/bt/libbt/Android.mk file when I realized that this file was already including $(BDROID_DIR)/hci/include and defined BDROID_DIR := $(TOP_DIR)system/bt. So I understood that I should have changed my manifest file to check out the bluedroid repo under system/bt (which also meant that I was wrong in excluding the sprd-diff/system_bt.diff patch before). I also had a quick look at other manifest files, and I saw that they were typically getting system/bt from the LineageOS android_system_bt repo. Therefore I went to modify my manifest file, changing the bluedroid line to

<project path="system/bt" name="android_system_bt" remote="los" revision="cm-14.1" />

and removed the three vendor/nxp-nfc repositories I added a while before. Then I manually applied the patch in device/samsung/sharkls-common/patches/sprd-diff/system_bt.diff and I revisited the vendor/sprd/open-source/apps/engmode/bt/Android.mk to remove all the include lines I added before. The next build succeeded:

[...]
[ 99% 2250/2255] build /mnt/Lavoro/halium/out/target/product/j3xlte/obj/NOTICE.html
Combining NOTICE files into HTML
Combining NOTICE files into text
[ 99% 2254/2255] Target system fs image: /mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img
BuildImage: in_dir = /mnt/Lavoro/halium/out/target/product/j3xlte/system, out_file = /mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img
fs type is not ext4
Running:  mkuserimg.sh -s /mnt/Lavoro/halium/out/target/product/j3xlte/system /mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img ext4 system 2147483648 -D /mnt/Lavoro/halium/out/target/product/j3xlte/system -L system /mnt/Lavoro/halium/out/target/product/j3xlte/root/file_contexts.bin
make_ext4fs -s -T -1 -S /mnt/Lavoro/halium/out/target/product/j3xlte/root/file_contexts.bin -L system -l 2147483648 -a system /mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img /mnt/Lavoro/halium/out/target/product/j3xlte/system /mnt/Lavoro/halium/out/target/product/j3xlte/system
Creating filesystem with parameters:
    Size: 2147483648
    Block size: 4096
    Blocks per group: 32768
    Inodes per group: 8192
    Inode size: 256
    Journal blocks: 8192
    Label: system
    Blocks: 524288
    Block groups: 16
    Reserved block group size: 127
Created filesystem with 1381/131072 inodes and 55467/524288 blocks
Running ['mkuserimg.sh', '-s', '/mnt/Lavoro/halium/out/target/product/j3xlte/system', '/mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/systemimage_intermediates/system.img', 'ext4', 'system', '2147483648', '-D', '/mnt/Lavoro/halium/out/target/product/j3xlte/system', '-L', 'system', '/mnt/Lavoro/halium/out/target/product/j3xlte/root/file_contexts.bin'] command, exit code = 0
[ 99% 2254/2255] Construct recovery from boot
failed to reconstruct target deflate chunk 1 [(null)]; treating as normal
chunk 0: type 0 start 0 len 5582858
chunk 1: type 2 start 5582858 len 8611584
chunk 2: type 0 start 9205682 len 673886
Construct patches for 3 chunks...
patch   0 is 212 bytes (of 5582858)
patch   1 is 4311951 bytes (of 3622824)
patch   2 is 207 bytes (of 673886)
chunk   0: normal   (         0,    5582858)         212
chunk   1: deflate  (   5582858,    8574717)     4311951  (null)
chunk   2: normal   (  14157575,     674057)         207
[100% 2255/2255] Install system fs image: /mnt/Lavoro/halium/out/target/product/j3xlte/system.img
/mnt/Lavoro/halium/out/target/product/j3xlte/system.img+/mnt/Lavoro/halium/out/target/product/j3xlte/obj/PACKAGING/recovery_patch_intermediates/recovery_from_boot.p maxsize=2192424960 blocksize=135168 total=197470210 reserve=22167552
make: Leaving directory '/mnt/Lavoro/halium'

#### make completed successfully (01:22 (mm:ss)) ####

Just to be on the safe side (to make sure I didn't have stale files from previous build attempts), I ran

mka clean
mka halium-boot
LANG=C mka systemimage

again, without encountering any errors. I also ran mka hybris-boot as per the Halium documentation, as it might be a good idea to test the Halium reference image first. hybris-boot.img was built successfully at the first try.

Flashing and testing the Halium reference image

Following the Halium documentation, I downloaded the Halium rootfs image:

mkdir rootfs
cd rootfs
wget 'http://bshah.in/halium/halium-rootfs-20170630-151006.tar.gz'

A small section of the documentation is dedicated on how to install the boot image for Samsung devices; it closely matches the steps I took before to install TWRP, which is comforting.

adb reboot download    # much easier than the weird key combinations!
sudo heimdall flash --BOOT out/target/product/j3xlte/hybris-boot.img

The last command ended with this output:

Uploading BOOT
100%
BOOT upload successful

Ending session...
ERROR: Failed to receive session end confirmation!
Releasing device interface...

and the device was stuck on the same screen. I then thought of using TWRP to flash this device. I rebooted in download mode and installed TWRP as explained near the beginning of this blog post, then I uploaded my boot image to the SD card:

adb push out/target/product/j3xlte/hybris-boot.img /sdcard/

In TWRP, I did a wipe first: I wiped off the "System", "Data" and "cache" partitions (you can choose which partitions to wipe under the "Advanced wipe" option). Then I went back and pressed the "Install" button, then "Install Image" and picked my hybris-boot.img file (and chose "Boot" as target, of course). Make sure you don't reboot at this stage, or you'll lose adb access and the rest of the commands will fail.

I then installed the halium-install script:

git clone https://gitlab.com/JBBgameich/halium-install.git

I then downloaded the original Samsung ROM (see below for the link) and unzipped it in the rootfs directory I created before. That produced a file named J320FXXU0AQK2_J320FOXE0AQK2_J320FXXU0AQI1_HOME.tar.md5, from which I extracted the system.img and uploaded it to the device along with the halium rootfs, as explained in the "Halium guide":

tar xvf J320FXXU0AQK2_J320FOXE0AQK2_J320FXXU0AQI1_HOME.tar.md5 system.img
../halium-install/halium-install -p halium -v halium-rootfs-20170630-151006.tar.gz system.img

The last command complained about simg2img not being in $PATH, despite the fact that it was. A small change to the script fixed this. Once that was fixed it successfully pushed the files to the device, after asking me which password I wanted to set for the root user (on the phone): I chose a familiar "0000".

After rebooting, however, the device didn't go beyond the boot screen (with a red line warning about SELinux not being enabled). It just got stuck there, and was not even detected via USB.

Therefore, it's time to end this post with a "To be continued" message, and with a bonus section on how to restore the stock image below.

Restoring the Samsung stock ROM

The stock ROM can be downloaded from sammobile.com; you can restore the original image with this commands:

unzip J320*.zip
tar xvf J320*.tar.md5
sudo heimdall flash --KERNEL boot.img --CACHE cache.img \
    --HIDDEN hidden.img --l_fixnv2 nvitem.bin --PARAM param.lfs \
    --pm_sys PM_sharkl_arm7.bin --RECOVERY recovery.img \
    --SBOOT sboot.bin --SBOOT2 sboot2.bin --BOOT spl.img \
    --l_modem SPRDCP.img --l_gdsp SPRDGDSP.img --l_ldsp SPRDLTEDSP.img \
    --l_warm SPRDWDSP.img --SYSTEM system.img

If you were so crazy as to follow my steps, now running the command above is probably the wisest thing you could do. :-)

You know that I'm not an early adopter. That's why it was only a couple of weeks ago when I decided to give Qbs a try, by using the good old Mappero (and its spin-off, Mappero Geotagger) as a test bench. Yes, I know that the Qt company is not going to maintain Qbs anymore in the future, but the little I knew about Qbs was enough to convince me that it's a project worth supporting. So, better late than never -- and hopefully the community (me included) will do a good job in keeping Qbs thriving.

Having Mappero build with Qbs was the simplest thing ever. The only issue I met was in building the unit tests, because I'm used to set the rpath on test executables in order to make it easy to run them uninstalled, and with qmake I achieved that with this:

QMAKE_RPATHDIR = $${QMAKE_LIBDIR}

In turns out that with Qbs you can do it in almost the same way, but for some reason I couldn't figure it out and I even reported a bug to which I got some nice suggestions, before eventually settling on this:

import qbs 1.0

Test {
    name: "path-test"

    files: [
        "path-test.cpp",
        "path-test.h",
        "paths.qrc",
    ]

    Depends { name: "Mappero" }
    cpp.rpaths: cpp.libraryPaths    // <-- this does the trick!
}

It's surprisingly similar to how it's done in qmake, so it's not clear even to me why I didn't guess that immediately. Anyway, that was literally my only problem, and you can see the whole set of Qbs files I wrote by having a look at this commit.

Given how easy the migration was, I thought I should also try to add a code coverage report; that's not something I had in my qmake build either, but it's something I really want to have in all my newer projects.

Teaching Qbs to make a code coverage report

Unfortunately, my search for examples on how to have Qbs prepare a coverage report was mostly insuccessful, but thanks to some amazing help from Christian in the #qbs IRC channel, this was not hard to achieve. So, I hope to be of some help myself too, by sharing how this works.

First of all, it must be said that Qbs doesn't know anything about code coverage, at all. However, it's possible (and often easy) to extend Qbs by adding your own Product with its own set of build rules, so here's the CoverageReport item for Mappero (though, it should be general enough to be reusable in your own project):

import qbs

Product {
    name: "coverage"

    property string outputDirectory: "coverage-html"
    property stringList extractPatterns: []

    builtByDefault: false
    files: ["**"]
    type: ["coverage.html"]

    Depends { productTypes: ["autotest-result"] }

    Rule {
        multiplex: true
        explicitlyDependsOnFromDependencies: ["autotest-result"]
        outputFileTags: "coverage.html"
        requiresInputs: false
        prepare: {
            var commands = []
            var captureCmd = new Command("lcov", [
                "--directory", project.sourceDirectory,
                "--capture",
                "--output-file", "coverage.info",
                "--no-checksum",
                "--compat-libtool",
            ]);
            captureCmd.description = "Collecting coverage data";
            captureCmd.highlight = "coverage";
            captureCmd.silent = false;
            commands.push(captureCmd);

            var extractArgs = []
            for (var i = 0; i < product.extractPatterns.length; i++) {
                extractArgs.push("--extract");
                extractArgs.push("coverage.info");
                extractArgs.push(product.extractPatterns[i]);
            }
            if (product.extractPatterns.length > 0) {
                extractArgs.push("-o");
                extractArgs.push("coverage.info");
                var extractCmd = new Command("lcov", extractArgs);
                extractCmd.description = "Extracting coverage data";
                extractCmd.highlight = "coverage";
                extractCmd.silent = false;
                commands.push(extractCmd);
            }

            var filterCmd = new Command("lcov", [
                "--remove", "coverage.info", 'moc_*.cpp',
                "--remove", "coverage.info", 'qrc_*.cpp',
                "--remove", "coverage.info", '*/tests/*',
                "-o", "coverage.info",
            ]);
            filterCmd.description = "Filtering coverage data";
            filterCmd.highlight = "coverage";
            filterCmd.silent = false;
            commands.push(filterCmd);

            var genhtmlCmd = new Command("genhtml", [
                "--prefix", project.sourceDirectory,
                "--output-directory", product.outputDirectory,
                "--title", "Code coverage",
                "--legend",
                "--show-details",
                "coverage.info",
            ]);
            genhtmlCmd.description = "Generate HTML coverage report";
            genhtmlCmd.highlight = "coverage";
            genhtmlCmd.silent = false;
            commands.push(genhtmlCmd);

            return commands;
        }
    }
}

The most important thing here are the references to the autotest-result tag: this is the tag used by the AutotestRunner Qbs item, which is responsible for running the unit tests. Referencing its product's tag in the Depends item and in the explicitlyDependsOnFromDependencies properties ensures that "building" our product will cause the unit tests to run. Other needed bits are the requiresInputs: false property, which means that our rule doesn't have any required inputs, and the builtByDefault: false property, which says that our coverage report should not be generated when just typing qbs. Instead, to run the tests and get the code coverage report one will have to request it explicitly, by typing

qbs -p coverage

The prepare property of the Rule is where the commands to generate the code coverage report are defined. Here we can use the Command item to invoke external programs, and we return a list of such items, so that the commands will be executed in sequence. Note that here I'm using lcov and expecting to find the coverage data produced by gcov, so this is probably not portable outside of Linux/gcc.

Using the CoverageReport item is quite easy: you just need to declare it, and specify which paths contain the coverage data that you are interested in (otherwise, lcov will collect data from all object files that it find under the build directory, which might not be what you desire):

    CoverageReport {
        condition: project.enableCoverage
        extractPatterns: [ '*/src/*.cpp', '*/lib/*.cpp' ]
    }

There's little more than that to be done. Of course, you need to find a way to pass the --coverage option to gcc when building your products, and for this I created a small buildconfig module in qbs/modules/buildconfig/BuildConfig.qbs which I depend on in all products which I wish to build with coverage enabled:

import qbs

Module {
    cpp.cxxFlags: project.enableCoverage ? ["--coverage"] : undefined
    cpp.dynamicLibraries: project.enableCoverage ? ["gcov"] : undefined

    Depends { name: "cpp" }
}

If all this looks scary, you should probably have a look at the diff which shows how I added code coverage reporting to qbs: hopefully you'll find that it's not that complex, after all.

I hope that Qbs users will find this interesting, and possibly improving my setup. Ideally we should try to get something like this part of Qbs itself, but portability outside of Linux / gcc is going to be an issue.

More than one year has passed since the initial announcement of my plan to investigate using a different backend for contact storage. If you want to get a better understanding of the plan, that mail is still a good read -- not much has changed since them, planning wise.

The reason for this blog post is to give a small update on what has happened since then, and as a start nothing can be better than a couple of screenshots:

In other words, that means that contact synchonisation works, both with the new CardDAV protocol (for which we'll have preconfigured setups for NextCloud and OwnCloud accounts) and with Google Contacts, for which we are now using a different engine. What you see in the second screenshot (although indeed it's not obvious at all) is that the new qtcontacts-sqlite backend performs automatic contact merging based on some simple heuristics, meaning that when synchonising the same contact from multiple sources you should not happen to find a multitude of semi-identical copies of the contact, but a single one having all the aggregated details.

Before you get too excited, I have to say that this code is pre-alpha quality and that it's not even available for testing yet. The next step is indeed to setup CI so that the packaggqes get automatically built and published to a public repository, at which point I'll probably issue another update here in my blog.

The boring stuff

And now some detail for those who might wonder why this feature is not ready yet, or would like to get an idea on the time-frame for its completion.

Apart from a chronical lack of time from my part, the feature complexity is due to the large number of components involved:

• qtcontacts-sqlite: the QtContacts backend we are migrating to. This is a backend for the QtContacts API (used by our Addressbook application) which uses a SQLite database as storage for your contacts.
• buteo-sync-plugin-carddav: the CardDAV plugin for Buteo (our synchronisation manager). This plugin is loaded by Buteo and synchronises the contacts between a CardDAV remote source and the qtcontacts-sqlite database.
• buteo-sync-plugins-social: a Buteo plugin which can synchronise contacts from a multitude of sources, including Google, Facebook and Vk. At the moment we only care about Google, but once this feature has landed we can easily extend it to work with the other two as well.
• address-book-app: this is our well-known Contacts application. It needs some minor changes to adapt to the qtcontacts-sqlite backend and to support the creation of new CardDAV, NextCloud and OwnCloud accounts.
• QtPim: the contacts and calendar API developed by the Qt project. Our Contacts application is using the front-end side of this API, and the qtcontacts-sqlite component implements the backend side. There are some improvements proposed by Jolla, which we need to include in order to support grouping contacts by their initials.

The other tricky aspect is that the first three projects are maintained by Jolla as part of the Sailfish OS, and while on one side this means that we can share the development and maintenance burden with Jolla, on the other side of the coin it means that we need to apply extra care when submitting changes, in order not to step on each other's shoes. Specifically, Sailfish OS is using a much older version of QtPim than Ubports is, and the APIs between the two versions have changes in an incompatible version, so that it's nearly impossible to have a single code base working with both versions of QtPim. Luckily git supports branches, and Chris from Jolla was kind enough to create a branch for us in their upstream repository where I've proposed our changes (and they are a lot!).

However, this is not as bad as it sounds, and the fact that I have a roughly working version on my development device is a good sign that things are moving forwards.

At the beginning of this month I participated to the foss-north conference, in Gothenburg, and took the stage to give a short presentation of the blockchain technology. Given that my talk was somehow critical of the blockchain (or rather, of the projects using it without due reason) I was prepared to receive a wave of negative remarks, assuming that all the hype surrounding this technology would have infected a good part of my audience as well. I was therefore positively surprised when several people came to me afterwords to express their appreciation for my speech, appreciation that now makes me confident enough to share the video of the presentation here too:

I want to publicly thank Johan Thelin and all the other foss-north staff and volunteers who organized such a successful conference. They also managed to get the video recordings out in a surprisingly short time. Indeed, the above video is taken from the foss-north YouTube channel, which I recommend you to visit as there were a lot of good talks at the conference; the topics were so varied, that I'm sure you'll find at least a couple of talks of your interest.

Last November I was invited to talk at the LinuxPiter conference. I held a presentation of the Ubports project, to which I still contribute in my little spare time.

The video recording from the conference has finally been published:

(there's also a version in Russian)

There was not a big audience, to be honest, but those that were there expressed a lot of interest in the project.

It took a few days, but I've finally migrated my site to Nikola. I used to have blog.mardy.it served by Google's Blogger, the main sections of www.mardy.it generated with Jekyll, the image gallery served by the old and glorious Gallery2, plus a few leftovers from the old Drupal site.

discussion by Nicolas Alejandro, on Flickr

While Jekyll is cool, I was immediately captivated by Nikola's ease of use and by its developers' promptness in answering questions in the forum. Also, one nice thing about Nikola (and Pelican, too) which I forgot to mention in my previous post is it's support for multilingual sites. I guess I'll have to translate this post in interlingua too, to give you a demonstration. :-)

Anyway, while I've fallen in love with static site generators, I still would like to give people the chance of leaving comments. Services like Disqus are easy to integrate, but given the way they can be (ab)used to track the users, I prefered to go for something self hosted. So, enter Isso.

Isso is a Python server to handle comments; it's simple to install and configure, and offers some nice features like e-mail notifications on new replies.

My Isso setup

Integrating Isso with Nikola was relatively easy, but the desire to keep a multilingual site and some hosting limitation made the process worth spending a couple of words.

FastCGI

First, my site if hosted by Dreamhost with a very basic subscription that doesn't allow me to keep long-running processes. After reading Isso's quickstart guide I was left quite disappointed, because it seemed that the only way to use Isso is to have it running all the time, or have a nginx server (Dreamhost offers Apache). Luckily, that's not quite the case, and more deployment approach are described in a separate page, including one for FastCGI (which is supported by Dreamhost). Those instructions are a bit wrong, but yours truly submitted some amendments to the documentation which will hopefully go live soon.

Importing comments

Isso can import comments from other sites, but an importer for Blogger (a.k.a. blogspot.com) was missing. So I wrote a quick and dirty tool for that job, and shared it in case it could be useful to someone else, too.

Multilingual sites

The default configuration of Nikola + Isso binds the comments to the exact URL that they were entered into. What I mean is that if your site supports multiple languages, and a user has entered a comment to an entry while visiting the English version of the site, users visiting the Italian version of the site would see same blog entry, but without that comment. That happens regardless of whether the blog entry has been translated into multiple languages or not: it's enough that the site has been configured for multiple languages.

My solution to fix the issue could not be accepted into Nikola as it would break old comments in existing sites, but if you are starting a new multilingual site you should definitely consider it.

Testers welcome

Given that I've deployed Isso as a CGI, it's understandable that it's not the fastest thing ever: it takes some time to startup, so comments don't appear immediately when you open a page. However, once it's started it stays alive for several seconds, and that seems to help with performance when commenting.

Anyway, the real reason why I've written all this is to kindly ask you to write a comment on this post :-) Extra points if you leave your e-mail address and enable the reply notifications, and let me know if you receive a notification once I'll reply to your comment. As far as I understand, you won't get a notification when someone adds an unrelated comment, but only when the "reply" functionality is used.

But really, should the commenting system be completely broken, I'm sure you'll find a way to contact me, if you need to. :-)