fidzu : fedora

A few days ago I cut the 0.1.4 release of libxmlb, which is significant because it includes the last three features I needed in gnome-software to achieve the same search results as appstream-glib.

The first is something most users of database libraries will be familiar with: Bound variables. The idea is you prepare a query which is parsed into opcodes, and then at a later time you assign one of the ? opcode values to an actual integer or string. This is much faster as you do not have to re-parse the predicate, and also means you avoid failing in incomprehensible ways if the user searches for nonsense like ]@attr. Borrowing from SQL, the syntax should be familiar:

g_autoptr(XbQuery) query = xb_query_new (silo, "components/component/id[text()=?]/..", &error);
xb_query_bind_str (query, 0, "gimp.desktop", &error);

The second feature makes the caller jump through some hoops, but hoops that make things faster: Indexed queries. As it might be apparent to some, libxmlb stores all the text in a big deduplicated string table after the tree structure is defined. That means if you do <component component="component">component</component> then we only store just one string! When we actually set up an object to check a specific node for a predicate (for instance, text()='fubar' we actually do strcmp("fubar", "component") internally, which in most cases is very fast…

Unless you do it 10 million times…

Using indexed strings tells the XbMachine processing the predicate to first check if fubar exists in the string table, and if it doesn't, the predicate can't possibly match and is skipped. If it does exist, we know the integer position in the string table, and so when we compare the strings we can just check two uint32_t's which is quite a lot faster, especially on ARM for some reason. In the case of fwupd, it is searching for a specific GUID when returning hardware results. Using an indexed query takes the per-device query time from 3.17ms to about 0.33ms - which if you have a large number of connected updatable devices makes a big difference to the user experience. As using the indexed queries can have a negative impact and requires extra code it is probably only useful in a handful of cases. In case you do need this feature, this is the code you would use:

xb_silo_query_build_index (silo, "component/id", NULL, &error); // the cdata
xb_silo_query_build_index (silo, "component", "type", &error); // the @type attr
g_autoptr(XbNode) n = xb_silo_query_first (silo, "component/id[text()=$'test.firmware']", &error);

The indexing being denoted by $'' rather than the normal pair of single quotes. If there is something more standard to denote this kind of thing, please let me know and I'll switch to that instead.

The third feature is: <bStemming; which means you can search for "gaming mouse" and still get results that mention games, game and Gaming. This is also how you can search for words like Kongreßstraße which matches kongressstrasse. In an ideal world stemming would be computationally free, but if we are comparing millions of records each call to libstemmer sure adds up. Adding the stem() XPath operator took a few minutes, but making it usable took up a whole weekend.

The query we wanted to run would be of the form id[text()~=stem('?') but the stem() would be called millions of times on the very same string for each comparison. To fix this, and to make other XPath operators faster I implemented an opcode rewriting optimisation pass to the XbMachine parser. This means if you call lower-case(text())==lower-case('GIMP.DESKTOP') we only call the UTF-8 strlower function N+1 times, rather than 2N times. For lower-case() the performance increase is slight, but for stem it actually makes the feature usable in gnome-software. The opcode rewriting optimisation pass is kinda dumb in how it works ("lets try all combinations!"), but works with all of the registered methods, and makes all existing queries faster for almost free.

One common question I've had is if libxmlb is supposed to obsolete appstream-glib, and the answer is "it depends". If you're creating or building AppStream metadata, or performing any AppStream-specific validation then stick to the appstream-glib or appstream-builder libraries. If you just want to read AppStream metadata you can use either, but if you can stomach a binary blob of rewritten metadata stored somewhere, libxml is going to be a couple of orders of magnitude faster and use a ton less memory.

If you're thinking of using libxml in your project send me an email and I'm happy to add more documentation where required. At the moment libxmlb does everything I need for fwupd and gnome-software and so apart from bugfixes I think it's basically "done", which should make my manager somewhat happier. Comments welcome.

12 Nov 2018 3:49pm GMT

An issue was recently raised on libpod, the github repo for Podman.

"container_t isn't allowed to access container_var_lib_t"

Container policy is defined in the container-selinux package. By default containers run with the SELinux type "container_t" whether this is a container launched by just about any container engine like: podman, cri-o, docker, buildah, moby. And most people who use SELinux with containers from container runtimes like runc, systemd-nspawn use it also.

By default container_t is allowed to read/execute labels under /usr, read generically labeled content in the hosts /etc directory (etc_t).

The default label for content in /var/lib/docker and /var/lib/containers is container_var_lib_t, This is not accessible by containers, container_t, whether they are running under podman, cri-o, docker, buildah ... We specifically do not want containers to be able to read this content, because content that uses block devices like devicemapper and btrfs(I believe) is labeled container_var_lib_t, when the containers are not running.

For overlay content we need to allow containers to read/execute the content, we use the type container_share_t, for this content. So container_t is allowed to read/execute container_share_t files, but not write/modify them.

Content under /var/lib/containers/overlay* and /var/lib/docker/overlay* is labeled container_share_ by default.

$ grep overlay /etc/selinux/targeted/contexts/files/file_contexts
/var/lib/docker/overlay(/.*)? system_u:object_r:container_share_t:s0
/var/lib/docker/overlay2(/.*)? system_u:object_r:container_share_t:s0
/var/lib/containers/overlay(/.*)? system_u:object_r:container_share_t:s0
/var/lib/containers/overlay2(/.*)? system_u:object_r:container_share_t:s0
/var/lib/docker-latest/overlay(/.*)? system_u:object_r:container_share_t:s0
/var/lib/docker-latest/overlay2(/.*)? system_u:object_r:container_share_t:s0
/var/lib/containers/storage/overlay(/.*)? system_u:object_r:container_share_t:s0
/var/lib/containers/storage/overlay2(/.*)? system_u:object_r:container_share_t:s0

The label container_file_t is the only type that is writeable by containers. container_file_t is used when the overlay mount is created for the upper directory of an image. It is also used for content mounted from devicemapper and btrfs.

If you volume mount in a directory into a container and add a :z or :Z the container engines relabeled the content under the volumes to container_file_t.

Failure to read/write/execute content labeled container_var_lib_t is expected.

When I see this type of AVC, I expect that this is either a volume mounted in from /var/lib/container or /var/lib/docker or a mislabeled content under and overlay directory like /var/lib/containers/storage/overlay.

Solution:

To solve these, I usually recommend running

restorecon -R -v /var/lib/containers
restorecon -R -v /var/lib/docker

Or if it is a volume mount to use the :z, or :Z/

12 Nov 2018 2:01pm GMT

We're happy to announce Kiwi TCMS version 6.2.1! This is a small release that includes some improvements and bug-fixes. You can explore everything at https://demo.kiwitcms.org!

Supported upgrade paths:

5.3   (or older) -> 5.3.1
5.3.1 (or newer) -> 6.0.1
6.0.1            -> 6.1
6.1              -> 6.1.1
6.1.1            -> 6.2 (or newer)

Docker images:

kiwitcms/kiwi       latest  24338088bf46    956.8 MB
kiwitcms/kiwi       6.2     7870085ad415    957.6 MB
kiwitcms/kiwi       6.1.1   49fa42ddfe4d    955.7 MB
kiwitcms/kiwi       6.1     b559123d25b0    970.2 MB
kiwitcms/kiwi       6.0.1   87b24d94197d    970.1 MB
kiwitcms/kiwi       5.3.1   a420465852be    976.8 MB

cd Kiwi/
git pull
docker-compose down
docker pull kiwitcms/kiwi
docker pull centos/mariadb
docker-compose up -d
docker exec -it kiwi_web /Kiwi/manage.py migrate

# starting from an older Kiwi TCMS version
docker-compose down
docker pull kiwitcms/kiwi:<next_upgrade_version>
edit docker-compose.yml to use kiwitcms/kiwi:<next_upgrade_version>
docker-compose up -d
docker exec -it kiwi_web /Kiwi/manage.py migrate
# repeat until you have reached latest

12 Nov 2018 12:45pm GMT