22 Aug 2014

feedPlanet Gentoo

Andreas K. Hüttel: EAPI=5 adoption has reached 50% of ebuilds in the portage tree

As of today, more than 50% of the 37527 ebuilds in the Gentoo portage tree use the newest ebuild API (EAPI) version, EAPI=5!
The details of the various EAPIs can be found in the package manager specification (PMS); the most notable new feature of EAPI 5, which has sped up acceptance a lot is the introduction of so-called subslots. A package A can specify a subslot, another package B that depends on it can specify that it needs to be rebuilt when the subslot of A changes. This leads to much more elegant solutions for many of the the link or installation path problems that revdep-rebuild, emerge @preserved-rebuild, or e.g. perl-cleaner try to solve... Another useful new feature in EAPI=5 is the masking of use-flags specifically for stable-marked ebuilds.
You can follow the adoption of EAPIs in the portage tree on an automatically updated graph page.

22 Aug 2014 7:14pm GMT

19 Aug 2014

feedPlanet Gentoo

Sven Vermeulen: Switching to new laptop

I'm slowly but surely starting to switch to a new laptop. The old one hasn't completely died (yet) but given that I had to force its CPU frequency at the lowest Hz or the CPU would burn (and the system suddenly shut down due to heat issues), and that the connection between the battery and laptop fails (so even new battery didn't help out) so I couldn't use it as a laptop… well, let's say the new laptop is welcome ;-)

Building Gentoo isn't an issue (having only a few hours per day to work on it is) and while I'm at it, I'm also experimenting with EFI (currently still without secure boot, but with EFI) and such. Considering that the Gentoo Handbook needs quite a few updates (and I'm thinking to do more than just small updates) knowing how EFI works is a Good Thing ™.

For those interested - the EFI stub kernel instructions in the article on the wiki, and also in Greg's wonderful post on booting a self-signed Linux kernel (which I will do later) work pretty well. I didn't try out the "Adding more kernels" section in it, as I need to be able to (sometimes) edit the boot options (which isn't easy to accomplish with EFI stub-supporting kernels afaics). So I installed Gummiboot (and created a wiki article on it).

Lots of things still planned, so little time. But at least building chromium is now a bit faster - instead of 5 hours and 16 minutes, I can now enjoy the newer versions after little less than 40 minutes.

19 Aug 2014 8:11pm GMT

14 Aug 2014

feedPlanet Gentoo

Alexys Jacob: Using uWSGI and Consul to design a distributed application

Foreword

Let's say we have to design an application that should span across multiple datacenters while being able to scale as easily as firing up a new vm/container without the need to update any kind of configuration.

Facing this kind of challenge is exciting and requires us to address a few key scaffolding points before actually starting to code something :

Seeing the title of this article, the two components I'll demonstrate are obviously uWSGI and Consul which can now work together thanks to the uwsgi-consul plugin.

While this article example is written in python, you can benefit from the same features in all the languages supported by uWSGI which includes go, ruby, perl ad php !

Our first service discovering application

The application will demonstrate how simple it is for a client to discover all the available servers running a specific service on a given port. The best part is that the services will be registered and deregistered automatically by uWSGI as they're loaded and unloaded.

The demo application logic is as follows :

  1. uWSGI will load two server applications which are each responsible for providing the specified service on the given port
  2. uWSGI will automatically register the configured service into Consul
  3. uWSGI will also automatically register a health check for the configured service into Consul so that Consul will also be able to detect any failure of the service
  4. Consul will then respond to any client requesting the list of the available servers (nodes) providing the specified service
  5. The client will query Consul for the service and get either an empty response (no server available / loaded) or the list of the available servers

Et voilà, the client can dynamically detect new/obsolete servers and start working !

Setting up uWSGI and its Consul plugin

On Gentoo Linux, you'll just have to run the following commands to get started (other users refer to the uWSGI documentation or your distro's package manager). The plugin will be built by hand as I'm still not sure how I'll package the uWSGI external plugins…

$ sudo ACCEPT_KEYWORDS="~amd64" emerge uwsgi
$ cd /usr/lib/uwsgi/
$ sudo uwsgi --build-plugin https://github.com/unbit/uwsgi-consul
$ cd -

You'll have installed the uwsgi-consul plugin which you should see here :

$ ls /usr/lib/uwsgi/consul_plugin.so
/usr/lib/uwsgi/consul_plugin.so

That's all we need to have uWSGI working with Consul.

Setting up a Consul server

Gentoo users will need to add the ultrabug overlay (use layman) and then install consul (other users refer to the Consul documentation or your distro's package manager).

$ sudo layman -a ultrabug
$ sudo ACCEPT_KEYWORDS="~amd64" USE="web" emerge consul

Running the server and its UI is also quite straightforward. For this example, we will run it directly from a dedicated terminal so you can also enjoy the logs and see what's going on (Gentoo users have an init script and conf.d ready for them shall they wish to go further).

Open a new terminal and run :

$ consul agent -data-dir=/tmp/consul-agent -server -bootstrap -ui-dir=/var/lib/consul/ui -client=0.0.0.0

You'll see consul running and waiting for work. You can already enjoy the web UI by pointing your browser to http://127.0.0.1:8500/ui/.

Running the application

To get this example running, we'll use the uwsgi-consul-demo code that I prepared.

First of all we'll need the consulate python library (available on pypi via pip). Gentoo users can just install it (also from the ultrabug overlay added before) :

$ sudo ACCEPT_KEYWORDS="~amd64" emerge consulate

Now let's clone the demo repository and get into the project's directory.

$ git clone git@github.com:ultrabug/uwsgi-consul-demo.git
$ cd uwsgi-consul-demo

First, we'll run the client which should report that no server is available yet. We will keep this terminal open to see the client detecting in real time the appearance and disappearance of the servers as we start and stop uwsgi :

$ python client.py 
no consul-demo-server available
[...]
no consul-demo-server available

Open a new terminal and get inside the project's directory. Let's have uWSGI load the two servers and register them in Consul :

$ uwsgi --ini uwsgi-consul-demo.ini --ini uwsgi-consul-demo.ini:server1 --ini uwsgi-consul-demo.ini:server2
[...]
* server #1 is up on port 2001


* server #2 is up on port 2002

[consul] workers ready, let's register the service to the agent
[consul] service consul-demo-server registered succesfully
[consul] workers ready, let's register the service to the agent
[consul] service consul-demo-server registered succesfully

Now let's check back our client terminal, hooray it has discovered the two servers on the host named drakar (that's my local box) !

consul-demo-server found on node drakar (xx.xx.xx.xx) using port 2002
consul-demo-server found on node drakar (xx.xx.xx.xx) using port 2001

Expanding our application

Ok it works great on our local machine but we want to see how to add more servers to the fun and scale dynamically.

Let's add another machine (named cheetah here) to the fun and have servers running there also while our client is still running on our local machine.

On cheetah :

Run a Consul agent (no need of a server) and tell him to work with your already running consul server on your box (drakar in my case) :

$ /usr/bin/consul agent -data-dir=/tmp/consul-agent -join drakar -ui-dir=/var/lib/consul/ui -client=0.0.0.0

The -join <your host or IP> is the important part.

Now run uWSGI so it starts and registers two new servers on cheetah :

$ uwsgi --ini uwsgi-consul-demo.ini --ini uwsgi-consul-demo.ini:server1 --ini uwsgi-consul-demo.ini:server2

And check the miracle on your client terminal still running on your local box, the new servers have appeared and will disappear if you stop uwsgi on the cheetah node :

consul-demo-server found on node drakar (xx.xx.xx.xx) using port 2001
consul-demo-server found on node drakar (xx.xx.xx.xx) using port 2002
consul-demo-server found on node cheetah (yy.yy.yy.yy) using port 2001
consul-demo-server found on node cheetah (yy.yy.yy.yy) using port 2002

Go mad

Check the source code, it's so simple and efficient you'll cry ;)

I hope this example has given you some insights and ideas for your current or future application designs !

14 Aug 2014 9:10am GMT

11 Aug 2014

feedPlanet Gentoo

Gentoo Monthly Newsletter: Gentoo Monthly Newsletter: July 2014

Gentoo News

Trustee Election Results

The two open seats for the Gentoo Trustees for the 2014-2016 term will be:

Since there were only two nominees for the two seats up for election, there was no official election. They were appointed uncontested.

Council Election Results

The Gentoo Council for the 2014-2015 term will be:

Official announcement here.

Gentoo Developer Moves

Summary

Gentoo is made up of 242 active developers, of which 43 are currently away.
Gentoo has recruited a total of 803 developers since its inception.

Changes

The following developers have recently changed roles:

Additions

The following developers have recently joined the project:

Returning Dev

Moves

The following developers recently left the Gentoo project:
None this month

Portage

This section summarizes the current state of the portage tree.

Architectures 45
Categories 162
Packages 17595
Ebuilds 37628
Architecture Stable Testing Total % of Packages
alpha 3658 561 4219 23.98%
amd64 10863 6239 17102 97.20%
amd64-fbsd 0 1577 1577 8.96%
arm 2681 1743 4424 25.14%
arm64 559 32 591 3.36%
hppa 3061 482 3543 20.14%
ia64 3189 612 3801 21.60%
m68k 618 87 705 4.01%
mips 0 2402 2402 13.65%
ppc 6838 2353 9191 52.24%
ppc64 4326 866 5192 29.51%
s390 1477 331 1808 10.28%
sh 1670 403 2073 11.78%
sparc 4114 898 5012 28.49%
sparc-fbsd 0 317 317 1.80%
x86 11535 5288 16823 95.61%
x86-fbsd 0 3237 3237 18.40%

gmn-portage-stats-2014-08

Security

Package Removals/Additions

Removals

Package Developer Date
perl-core/Class-ISA dilfridge 05 Jul 2014
dev-python/argparse mgorny 06 Jul 2014
dev-python/ordereddict mgorny 06 Jul 2014
perl-core/Filter dilfridge 07 Jul 2014
app-text/qgoogletranslator grozin 09 Jul 2014
dev-lisp/openmcl grozin 09 Jul 2014
dev-lisp/openmcl-build-tools grozin 09 Jul 2014
net-libs/cyassl blueness 15 Jul 2014
dev-ruby/text-format graaff 18 Jul 2014
dev-ruby/jruby-debug-base graaff 18 Jul 2014
games-util/rubygfe graaff 18 Jul 2014
perl-core/PodParser dilfridge 20 Jul 2014
virtual/perl-PodParser dilfridge 21 Jul 2014
perl-core/digest-base dilfridge 22 Jul 2014
virtual/perl-digest-base dilfridge 22 Jul 2014
perl-core/i18n-langtags dilfridge 22 Jul 2014
virtual/perl-i18n-langtags dilfridge 22 Jul 2014
perl-core/locale-maketext dilfridge 23 Jul 2014
virtual/perl-locale-maketext dilfridge 23 Jul 2014
perl-core/net-ping dilfridge 23 Jul 2014
virtual/perl-net-ping dilfridge 23 Jul 2014
virtual/perl-Switch dilfridge 25 Jul 2014
perl-core/Switch dilfridge 25 Jul 2014
x11-misc/keytouch pacho 27 Jul 2014
x11-misc/keytouch-editor pacho 27 Jul 2014
media-video/y4mscaler pacho 27 Jul 2014
dev-python/manifestdestiny pacho 27 Jul 2014
dev-cpp/libsexymm pacho 27 Jul 2014

Additions

Package Developer Date
www-client/vimb radhermit 01 Jul 2014
dev-util/libsparse jauhien 01 Jul 2014
dev-python/docker-py chutzpah 01 Jul 2014
dev-util/ext4_utils jauhien 01 Jul 2014
dev-haskell/base16-bytestring gienah 02 Jul 2014
dev-haskell/boxes gienah 02 Jul 2014
dev-haskell/chell gienah 02 Jul 2014
dev-haskell/conduit-extra gienah 02 Jul 2014
dev-haskell/cryptohash-conduit gienah 02 Jul 2014
dev-haskell/ekg-core gienah 02 Jul 2014
dev-haskell/equivalence gienah 02 Jul 2014
dev-haskell/hastache gienah 02 Jul 2014
dev-haskell/options gienah 02 Jul 2014
dev-haskell/patience gienah 02 Jul 2014
dev-haskell/prelude-extras gienah 02 Jul 2014
dev-haskell/tf-random gienah 02 Jul 2014
dev-haskell/quickcheck-instances gienah 02 Jul 2014
dev-haskell/streaming-commons gienah 02 Jul 2014
dev-haskell/vector-th-unbox gienah 02 Jul 2014
dev-haskell/tasty-th gienah 02 Jul 2014
dev-haskell/dlist-instances gienah 02 Jul 2014
dev-haskell/temporary-rc gienah 02 Jul 2014
dev-haskell/stmonadtrans gienah 02 Jul 2014
dev-haskell/data-hash gienah 02 Jul 2014
dev-haskell/yesod-auth-hashdb gienah 02 Jul 2014
sci-mathematics/agda-lib-ffi gienah 02 Jul 2014
dev-haskell/lifted-async gienah 02 Jul 2014
dev-haskell/wai-conduit gienah 02 Jul 2014
dev-haskell/shelly gienah 02 Jul 2014
dev-haskell/chell-quickcheck gienah 03 Jul 2014
dev-haskell/tasty-ant-xml gienah 03 Jul 2014
dev-haskell/lcs gienah 03 Jul 2014
dev-haskell/tasty-golden gienah 03 Jul 2014
sec-policy/selinux-tcsd swift 04 Jul 2014
dev-perl/Class-ISA dilfridge 05 Jul 2014
net-wireless/gqrx zerochaos 06 Jul 2014
dev-perl/Filter dilfridge 07 Jul 2014
app-misc/abduco xmw 10 Jul 2014
virtual/perl-Math-BigRat dilfridge 10 Jul 2014
virtual/perl-bignum dilfridge 10 Jul 2014
dev-perl/Net-Subnet chainsaw 11 Jul 2014
dev-java/opencsv ercpe 11 Jul 2014
dev-java/trident ercpe 11 Jul 2014
dev-java/htmlparser-org ercpe 11 Jul 2014
dev-java/texhyphj ercpe 12 Jul 2014
dev-util/vmtouch dlan 12 Jul 2014
sys-block/megactl robbat2 14 Jul 2014
dev-python/fexpect jlec 14 Jul 2014
mail-filter/postfwd mschiff 15 Jul 2014
dev-python/wheel djc 15 Jul 2014
dev-ruby/celluloid-io mrueg 15 Jul 2014
sys-process/tiptop patrick 16 Jul 2014
dev-ruby/meterpreter_bins zerochaos 17 Jul 2014
sys-power/thermald dlan 17 Jul 2014
net-analyzer/check_mk dlan 17 Jul 2014
app-admin/fleet alunduil 19 Jul 2014
perl-core/Pod-Parser dilfridge 20 Jul 2014
virtual/perl-Pod-Parser dilfridge 21 Jul 2014
sci-libs/libcerf ottxor 21 Jul 2014
games-fps/enemy-territory-omnibot ottxor 22 Jul 2014
dev-libs/libflatarray slis 22 Jul 2014
perl-core/Digest dilfridge 22 Jul 2014
virtual/perl-Digest dilfridge 22 Jul 2014
net-libs/stem mrueg 22 Jul 2014
perl-core/I18N-LangTags dilfridge 22 Jul 2014
virtual/perl-I18N-LangTags dilfridge 22 Jul 2014
perl-core/Locale-Maketext dilfridge 22 Jul 2014
virtual/perl-Locale-Maketext dilfridge 23 Jul 2014
perl-core/Net-Ping dilfridge 23 Jul 2014
virtual/perl-Net-Ping dilfridge 23 Jul 2014
dev-libs/libbson ultrabug 23 Jul 2014
sci-libs/silo slis 24 Jul 2014
dev-python/pgpdump jlec 24 Jul 2014
net-libs/libasr zx2c4 25 Jul 2014
dev-libs/npth zx2c4 25 Jul 2014
net-wireless/bladerf-firmware zerochaos 25 Jul 2014
net-wireless/bladerf-fpga zerochaos 25 Jul 2014
net-wireless/bladerf zerochaos 25 Jul 2014
sci-libs/cgnslib slis 25 Jul 2014
sci-visualization/visit slis 25 Jul 2014
dev-perl/Switch dilfridge 25 Jul 2014
dev-util/objconv slyfox 28 Jul 2014
app-crypt/monkeysign k_f 29 Jul 2014
virtual/bitcoin-leveldb blueness 29 Jul 2014
dev-db/percona-server robbat2 29 Jul 2014
sys-cluster/galera robbat2 30 Jul 2014
dev-db/mariadb-galera robbat2 30 Jul 2014
net-im/corebird dlan 30 Jul 2014
dev-libs/libpfm slis 31 Jul 2014
dev-perl/ExtUtils-Config civil 31 Jul 2014
dev-libs/papi slis 31 Jul 2014
dev-perl/ExtUtils-Helpers civil 31 Jul 2014
sys-cluster/hpx slis 31 Jul 2014
dev-perl/ExtUtils-InstallPaths civil 31 Jul 2014
dev-perl/Module-Build-Tiny civil 31 Jul 2014
www-plugins/pipelight ryao 31 Jul 2014

Bugzilla

The Gentoo community uses Bugzilla to record and track bugs, notifications, suggestions and other interactions with the development team.

Activity

The following tables and charts summarize the activity on Bugzilla between 01 July 2014 and 31 July 2014. Not fixed means bugs that were resolved as NEEDINFO, WONTFIX, CANTFIX, INVALID or UPSTREAM.
gmn-activity-2014-07

Bug Activity Number
New 1405
Closed 958
Not fixed 164
Duplicates 180
Total 5912
Blocker 5
Critical 19
Major 69

Closed bug ranking

The following table outlines the teams and developers with the most bugs resolved during this period

Rank Team/Developer Bug Count
1 Gentoo KDE team 41
2 Gentoo Security 38
3 Java team 29
4 Gentoo's Team for Core System packages 28
5 Gentoo Linux Gnome Desktop Team 24
6 Gentoo Games 24
7 Netmon Herd 23
8 Qt Bug Alias 22
9 Perl Devs @ Gentoo 22
10 Others 706


gmn-closed-2014-07

Assigned bug ranking

The developers and teams who have been assigned the most bugs during this period are as follows.

Rank Team/Developer Bug Count
1 Gentoo Linux bug wranglers 85
2 Gentoo Linux Gnome Desktop Team 64
3 Gentoo Security 56
4 Gentoo's Team for Core System packages 53
5 Julian Ospald (hasufell) 48
6 Netmon Herd 47
7 Gentoo KDE team 47
8 Python Gentoo Team 31
9 media-video herd 30
10 Others 943


gmn-opened-2014-07

Tip of the month

(by Sven Vermeulen)
Launching commands in background once (instead of scheduled through cron)

Use things like:
~$ echo "egencache --update --repo=gentoo --jobs=4" | at now + 10 minutes

Heard in the community

Send us your favorite Gentoo script or tip at gmn@gentoo.org

Getting Involved?

Interested in helping out? The GMN relies on volunteers and members of the community for content every month. If you are interested in writing for the GMN or thinking of another way to contribute, please send an e-mail to gmn@gentoo.org.

Comments or Suggestions?

Please head over to this forum post.

11 Aug 2014 12:00am GMT

09 Aug 2014

feedPlanet Gentoo

Rafael Goncalves Martins: Introducing pyoembed

Warning: This is a (very) delayed announcement! ;-)

oEmbed is an open standard for embedded content. It allows users to embed some resource, like a picture or a video, in a web page using only the resource URL, without knowing the details of how to embed the resource in a web page.

oEmbed isn't new stuff. It was created around 2008, and despite not being widely supported by content providers, it is supported by some big players, like YouTube, Vimeo, Flickr and Instagram, making its usage highly viable.

To support the oEmbed standard, the content provider just needs to provide a simple API endpoint, that receives an URL and a few other parameters, like the maximum allowed height/width, and returns a JSON or XML object, with ready-to-use embeddable code.

The content provider API endpoint can be previously known by the oEmbed client, or auto-discovered using some meta tags added to the resource's HTML page. This is the point where the standard isn't precise enough: not all of the providers support auto-discovering of the API endpoint, neither all of the providers are properly listed on the oEmbed specification. Proper oEmbed clients should try both approaches, looking for known providers first, falling back to auto-discovered endpoints, if possible.

Each of the Python libraries for oEmbed decided to follow one of the mentioned approaches, without caring about the other one, failing to support relevant providers. And this is the reason why I decided to start writing pyoembed!

pyoembed is a simple and easy to use implementation of the oEmbed standard for Python, that supports both auto-discovered and explicitly defined providers, supporting most (if not all) the relevant providers.

pyoembed's architecture makes it easy to add new providers and supports most of the existing providers out of the box.

To install it, just type:

$ pip install pyoembed

Gentoo users can install it from gentoo-x86:

# emerge -av pyoembed

pyoembed is developed and managed using Github, the repository is publicly available:

https://github.com/rafaelmartins/pyoembed

A Jenkins instance runs the unit tests and the integration tests automatically, you can check the results here:

https://ci.rgm.io/view/pyoembed/

The integration tests are supposed to fail from time to time, because they rely on external urls, that may be unavailable while the tests are running.

pyoembed is released under a 3 clause BSD license.

Enjoy!

09 Aug 2014 9:46pm GMT

Sven Vermeulen: Some changes under the hood

In between conferences, technical writing jobs and traveling, we did a few changes under the hood for SELinux in Gentoo.

First of all, new policies are bumped and also stabilized (2.20130411-r3 is now stable, 2.20130411-r5 is ~arch). These have a few updates (mergers from upstream), and r5 also has preliminary support for tmpfiles (at least the OpenRC implementation of it), which is made part of the selinux-base-policy package.

The ebuilds to support new policy releases now are relatively simple copies of the live ebuilds (which always contain the latest policies) so that bumping (either by me or other developers) is easy enough. There's also a release script in our policy repository which tags the right git commit (the point at which the release is made), creates the necessary patches, uploads them, etc.

One of the changes made is to "drop" the BASEPOL variable. In the past, BASEPOL was a variable inside the ebuilds that pointed to the right patchset (and base policy) as we initially supported policy modules of different base releases. However, that was a mistake and we quickly moved to bumping all policies with every releaes, but kept the BASEPOL variable in it. Now, BASEPOL is "just" the ${PVR} value of the ebuild so no longer needs to be provided. In the future, I'll probably remove BASEPOL from the internal eclass and the selinux-base* packages as well.

A more important change to the eclass is support for the SELINUX_GIT_REPO and SELINUX_GIT_BRANCH variables (for live ebuilds, i.e. those with the 9999 version). If set, then they pull from the mentioned repository (and branch) instead of the default hardened-refpolicy.git repository. This allows for developers to do some testing on a different branch easily, or for other users to use their own policy repository while still enjoying the SELinux integration support in Gentoo through the sec-policy/* packages.

Finally, I wrote up a first attempt at our coding style, heavily based on the coding style from the reference policy of course (as our policy is still following this upstream project). This should allow the team to work better together and to decide on namings autonomously (instead of hours of discussing and settling for something as silly as an interface or boolean name ;-)

09 Aug 2014 7:45pm GMT

07 Aug 2014

feedPlanet Gentoo

Paweł Hajdan, Jr.: Can your distro compile Chromium?

Chromium is moving towards using C++11. Even more, it's going to require either gcc-4.8 or clang.

Distros like Ubuntu, Mageia, Fedora, openSUSE, Arch, CentOS, and Slackware are already using gcc-4.8 or later is their latest stable release.

On the other hand, Debian Wheezy (7.0) has gcc-4.7.2. Gentoo is using gcc-4.7.3 in stable.

I started a thread on gentoo-dev, gcc-4.8 may be needed in stable for www-client/chromium-38.x. There is a tracker for gcc-4.8 stabilization, bug #516152. There is also gcc-4.8 porting tracker, bug #461954.

Please consider testing gcc-4.8 on your stable Gentoo system, and file bugs for any package that fails to compile or needs to have a newer version stabilized to work with new gcc. I have recompiled all packages, the kernel, and GRUB without problems.

The title of this post is deliberately a bit similar to my earlier post Is your distro fast enough for Chromium? This browser project is pushing a lot towards shorter release cycles and latest software. I consider that a good thing. Now we just need to keep up with the updates, and any help is welcome.

07 Aug 2014 7:20am GMT

03 Aug 2014

feedPlanet Gentoo

Anthony Basile: Constructing a “Directed Linkage Graph” for an entire system: The usefulness of exporting /var/db/pkg (VDB) information for utilities other than the Package Management System (PMS).

When portage installs a package onto your system, it caches information about that package in a directory at /var/db/pkg/<cat>/<pkg>/, where <cat> is the category (ie ${CATEGORY}) and <pkg> is the package name, version number and revision number (ie. ${P}). This information can then be used at a later time to tell portage information about what's installed on a system: what packages were installed, what USE flags are set on each package, what CFLAGS were used, etc. Even the ebuild itself is cached so that if it is removed from the tree, and consequently from your system upon `emerge -sync`, you have a local copy in VDB to uninstall or otherwise continue working with the package. If you take look under /var/db/pkg, you'll find some interesting and some not so interesting files for each <cat>/<pkg>. Among the less interesting are files like DEPEND, RDPENED, FEATURES, IUSE, USE, which just contain the same values as the ebuild variables by the same name. This is redundant because that information is in the ebuild itself which is also cached but it is more readily available since one doesn't have to re-parse the ebuild to obtain them. More interesting is information gathered about the package as it is installed, like CONTENTS, which contains a list of all the regular files, directories, and sym link which belong to the package, along with their MD5SUM. This list is used to remove files from the system when uninstalling the package. Environment information is also cached, like CBUILD, CHOST, CFLAGS, CXXFLAGS and LDFLAGS which affects the build of compiled packages, and environment.bz2 which contains the entire shell environment that portage ran in, including all shell variables and functions from inherited eclasses. But perhaps the most interesting information, and the most expensive to recalculate is, cached in NEEDED and NEEDED.ELF.2. The later supersedes the former which is only kept for backward compatibility, so let's just concentrate on NEEDED.ELF.2. Its a list of every ELF object that is installed for a package, along with its ARCH/ABI information, its SONAME if it is a shared object (readelf -d <obj> | grep SONAME, or scanelf -S), any RPATH used to search for its needed shared objects (readelf -d <obj> | grep RPATH, or scanelf -r), and any NEEDED shared objects (the SONAMES of libraries) that it links against (readelf -d <obj> | grep NEEDED or scanelf -n). [1] Unless you're working with some exotic systems, like an embedded image where everything is statically linked, your user land utilities and applications depend on dynamic linking, meaning that when a process is loaded from the executable on your hard drive, the linker has to make sure that its needed libraries are also loaded and then do some relocation magic to make sure that unresolved symbols in your executable get mapped to appropriate memory locations in the libraries.

The subtleties of linking are beyond the scope of this blog posting [2], but I think its clear from the previous paragraph that one can construct a "directed linkage graph" [3] of dependencies between all the ELF objects on a system. An executable can link to a library which in turn links to another, and so on, usually back to your libc [4]. `readelf -d <obj> | grep NEEDED` only give you the immediate dependencies, but if you follow these through recursively, you'll get all the needed libraries that an executable needs to run. `ldd <obj>` is a shell script which provides this information, as does ldd.py from the pax-utils package, which also does some pretty indentation to show the depth of the dependency. If this is sounding vaguely familiar, its because portage's dependency rules "mimic" the underlying linking which is needed at both compile time and at run time. Let's take an example, curl compiled with polarssl as its SSL backend:

# ldd /usr/bin/curl | grep ssl
        libpolarssl.so.6 => /usr/lib64/libpolarssl.so.6 (0x000003a3d06cd000)
# ldd /usr/lib64/libpolarssl.so.6
        linux-vdso.so.1 (0x0000029c1ae12000)
        libz.so.1 => /lib64/libz.so.1 (0x0000029c1a929000)
        libc.so.6 => /lib64/libc.so.6 (0x0000029c1a56a000)
        /lib64/ld-linux-x86-64.so.2 (0x0000029c1ae13000)

Now let's see this dependency reflected in the ebuild:

# cat net-misc/curl/curl-7.36.0.ebuild
RDEPEND="
        ...
        ssl? (
                ...
                curl_ssl_polarssl? ( net-libs/polarssl:= app-misc/ca-certificates )
                ...
        )
        ...

Nothing surprising. However, there is one subtlety. What happens if you update polarssl to a version which is not exactly backwards compatible. Then curl which properly linked against the old version of polarssl doesn't quite work with the new version. This can happen when the library changes its public interface by either adding new functions, removing older ones and/or changing the behavior of existing functions. Usually upstream indicates this change in the library itself by bumping the SONAME:

# readelf -d /usr/lib64/libpolarssl.so.1.3.7 | grep SONAME
0x000000000000000e (SONAME) Library soname: [libpolarssl.so.6]

But how does curl know about the change when emerging an updated version of polarssl? That's where subslotting comes in. To communicate the reverse dependency, the DEPEND string in curl's ebuild has := as the slot indicator for polarssl. This means that upgrading polarssl to a new subslot will trigger a recompile of curl:

# emerge =net-libs/polarssl-1.3.8 -vp

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild r U ] net-libs/polarssl-1.3.8:0/7 [1.3.7:0/6] USE="doc sse2 static-libs threads%* zlib -havege -programs {-test}" ABI_X86="(64) (-32) (-x32)" 1,686 kB
[ebuild rR ] net-misc/curl-7.36.0 USE="ipv6 ldap rtmp ssl static-libs threads -adns -idn -kerberos -metalink -ssh {-test}" CURL_SSL="polarssl -axtls -gnutls -nss -openssl" 0 kB

Here the onus is on the downstream maintainer to know when the API breaks backwards compatibility and subslot accordingly. Going through with this build and then checking the new SONAME we find:

# readelf -d /usr/lib/libpolarssl.so.1.3.8 | grep SONAME
0x000000000000000e (SONAME) Library soname: [libpolarssl.so.7]

Aha! Notice the SONAME jumped from .6 for polarssl-1.3.7 to .7 for 1.3.8. Also notice the SONAME version number also follows the subslotting value. I'm sure this was a conscious effort by hasufell and tommyd, the ebuild maintainers, to make life easy.

So I hope my example has shown the importance of tracing forward and reverse linkage between the ELF objects in on a system [5]. Subslotting is relatively new but the need to trace linking has always been there. There was, and still is, revdep-rebuild (from gentoolkit) which uses output from ldd to construct a "directed linkage graph" [6] but is is relatively slow. Unfortunately, it recalculates all the NEEDED.ELF.2 information on the system in order to reconstruct and invert the directed linkage graph. Subslotting has partially obsoleted revdep-rebuild because portage can now track the reverse dependencies, but it has not completely obsoleted it. revdep-rebuild falls back on the SONAMEs in the shared objects themselves - an error here is an upstream error in which the maintainers of the library overlooked updating the value of CURRENT in the build system, usually in a line of some Makefile.am that looks like

LDFLAGS += -version-info $(CURRENT):$(REVISION):$(AGE)

But an error in subslotting is an downstream error where the maintainers didn't properly subslot their package and any dependencies to reflect upstream's changing API. So in some ways, these tools complement each other.

Now we come to the real point of the blog: there is no reason for revdep-rebuild to run ldd on every ELF object on the system when it can obtain that information from VDB. This doesn't save time on inverting the directed graph, but it does save time on running ldd (effectively /lib64/ld-linux-x86-64.so.2 -list) on every ELF object in the system. So guess what the python version does, revdep-rebuild.py? You guessed it, it uses VDB information which is exported by portage via something like

import portage
vardb = portage.db[portage.root]["vartree"].dbapi

So what's the difference in time? On my system right now, we're looking at a difference between approximately 5 minutes for revdep-rebuild versus about 20 seconds for revdep-rebuild.py. [7] Since this information is gathered at build time, there is no reason for any Package Management System (PMS) to not export it via some standarized API. portage does so in an awkward fashion but it does export it. paludis does not export NEEDED.ELF.2 although it does export other VDB stuff. I can't speak to future PMS's but I don't see why they should not be held to a standard.

Above I argued that exporting VDB is useful for utilities that maintain consistency between executibles and the shared objects that they consume. I suspect one could counter-argue that it doesn't need to be exported because "revdep-rebuild" can be made part of portage or whatever your PMS, but I hope my next point will show that exporting NEEDED.ELF.2 information has other uses besides "consistant linking". So a stronger point is that, not only should PMS export this information, but that it should provide some well documented API for use by other tools. It would be nice for every PMS to have the same API, preferably via python bindings, but as long as it is well documented, it will be useful. (Eg. webapp-config supports both portage and paludis. WebappConfig/wrapper.py has a simple little switch between "import portage; ... portage.settings['CONFIG_PROTECT'] ... " and "cave print-id-environment-variable -b --format '%%v\n' --variable-name CONFIG_PROTECT %s/%s ...".)

So besides consistent linking, what else could make use of NEEDED.ELF.2? In the world of Hardened Gentoo, to increase security, a PaX-patched kernel holds processes to much higher standards with respect to their use of memory. [8] Unfortunately, this breaks some packages which want to implement insecure methods, like RWX mmap-ings. Code is compiled "on-the-fly" by JIT compilers which typically create such mappings as an area to which they first write and then execute. However, this is dangerous because it can open up pathways by which arbitrary code can be injected into a running process. So, PaX does not allow RWX mmap-ings - it doesn't allow it unless that kernel is told otherwise. This is where the PaX flags come in. In the JIT example, marking the executables with `paxctl-ng -m` will turn off PaX's MPROTECT and allow the RWX mmap-ing. The issue of consistent PaX markings between executable and their libraries arises when it is the library that needs the markings. But when loaded, it is the markings of the executable, not the library, which set the PaX restrictions on the running process. [9] So if its the library needs the markings, you have to migrate the markings from the library to the executable. Aha! Here we go again: we need to answer the question "what are all the consumers of a particular library so we can migrate its flags to them?" We can, as revdep-rebuild does, re-read all the ELF objects on the system, reconstruct the directed linkage graph, then invert it; or we can just start from the already gathered VDB information and save some time. Like revdep-rebuild and revdep-rebuild.py, I wrote two utilities. The original, revdep-pax, did forward and reverse migration of PaX flags by gathering information with ldd. It was horribly slow, 5 to 10 minutes depending on the number of objects in $PATH and shared object reported by `ldconfig -p`. I then rewrote it to use VDB information and it accomplished the same task in a fraction of the time [10]. Since constructing and inverting the directed linkage graph is such a useful operation, I figured I'd abstract the bare essential code into a python class which you can get at [11]. The data structure containing the entire graph is a compound python dictionary of the form

{
        abi1 : { path_to_elf1 : [ soname1, soname2, ... ], ... },
        abi2 : { path_to_elf2 : [ soname3, soname4, ... ], ... },
        ...
}

whereas the inverted graph has form

{
        abi1 : { soname1 : [ path_to_elf1, path_to_elf2, ... ], ... },
        abi2 : { soname2 : [ path_to_elf3, path_to_elf4, ... ], ... },
        ...
}

Simple!

Okay, up to now I concentrated on exporting NEEDED.ELF.2 information. So what about rest of the VDB information? Is it useful? A lot of questions regarding Gentoo packages can be answered by "grepping the tree." If you use portage as your PMS, then the same sort of grep-sed-awk foo magic can be performed on /var/db/pkg to answer similar questions. However, this assumes that the PMS's cached information is in plain ASCII format. If a PMS decides to use something like Berkeley DB or sqlite, then we're going to need a tool to read the db format which the PMS itself should provide. Because I do a lot of release engineering of uclibc and musl stages, one need that often comes up is the need to compare of what's installed in the stage3 tarballs for the various arches and alternative libc's. So, I run some variation of the following script

#!/usr/bin/env python

import portage, re

portdb = portage.db[portage.root]["vartree"].dbapi

arm_stable = open('arm-stable.txt', 'w')
arm_testing = open('arm-testing.txt', 'w')

for pkg in portdb.cpv_all():
keywords = portdb.aux_get(pkg, ["KEYWORDS"])[0]

arches = re.split('\s+', keywords)
        for a in arches:
                if re.match('^arm$', a):
                        arm_stable.write("%s\n" % pkg)
                if re.match('^~arm$', a):
                        arm_testing.write("%s\n" % pkg)

arm_stable.close()
arm_testing.close()

in a stage3-amd64-uclibc-hardened chroot to see what stable packages in the amd64 tarball are ~arm. [12] I run similar scripts in other chroots to do pairwise comparisons. This gives me some clue as to what may be falling behind in which arches - to keep some consistency between my various stage3 tarballs. Of course there are other utilities to do the same, like eix, gentoolkit etc, but then one still has to resort to parsing the output of those utilities to get the answers you want. An API for VDB information allows you to write your own custom utility to answer the precise questions you need answers. I'm sure you can multiply these examples.

Let me close with a confession. The above is propaganda for the upcoming GLEP 64 which I just wrote [13]. The purpose of the GLEP is to delineate what information should be exported by all PMS's with particular emphasis on NEEDED.ELF.2 for the reasons stated above. Currently portage does provide NEEDED.ELF.2 but paludis does not. I'm not sure what future PMS's might or might not provide, so let's set a standard now for an important feature.

Notes:

[1] You can see where NEEDED.ELF.2 is generated for details. Take a look at line ~520 of /usr/lib/portage/bin/misc-functions.sh, or search for the comment "Create NEEDED.ELF.2 regardless of RESTRICT=binchecks".

[2] A simple hands on tutorial can be found at http://www.yolinux.com/TUTORIALS/LibraryArchives-StaticAndDynamic.html. It also includes dynamic linking via dlopen() which complicates the nice neat graph that can be constructed from NEEDED.ELF.2.

[3] I'm using the term "directed graph" as defined in graph theory. See http://en.wikipedia.org/wiki/Directed_graph. The nodes of the graph are each ELF object and the directed edges are from the consumer of the shared object to the shared object.

[4] Well, not quite. If you run readelf -d on readelf -d /lib/libc.so.6 you'll see that it links back to /lib/ld-linux-x86-64.so.2 which doesn't NEED anything else. The former is stricly your standard C library (man 7 libc) while the later is the dynamic linker/loader (man 8 ld.so).

[5] I should mention parenthatically that there are other executable/library file formats such as Mach-O used on MacOS X. The above arguments translate over to any executable formats which permit shared libraries and dynamic linking. My prejudice for ELF is because it is the primary executable format used on Linux and BSD systems.

[6] I'm coining this term here. If you read the revdep-rebuild code, you won't see reference to any graph there. Bash doesn't readily lend itself to the neat data structures that python does.

[7] Just a word of caution, revdep-rebuild.py is still in development and does warn when you run it "This is a development version, so it may not work correctly. The original revdep-rebuild script is installed as revdep-rebuild.sh".

[8] See https://wiki.gentoo.org/wiki/Hardened/PaX_Quickstart for an explanation of what PaX does as well as how it works.

[9] grep the contents of fs/binfmt_elf.c for PT_PAX_FLAGS and CONFIG_PAX_XATTR_PAX_FLAGS to see how these markings are used when the process is loaded from the ELF object. You can see the PaX protection on a running process by using `cat /proc/<pid>/maps | grep ^PaX` or `pspax` form the pax-utils package.

[10] The latest version off the git repo is at http://git.overlays.gentoo.org/gitweb/?p=proj/elfix.git;a=blob;f=scripts/revdep-pax.

[11] http://git.overlays.gentoo.org/gitweb/?p=proj/elfix.git;a=blob;f=pocs/link-graph/link_graph.py.

[12] These stages are distributed at http://distfiles.gentoo.org/releases/amd64/autobuilds/current-stage3-amd64-uclibc-hardened/ and http://distfiles.gentoo.org/experimental/arm/uclibc/.

[13] https://bugs.gentoo.org/show_bug.cgi?id=518630

03 Aug 2014 5:41pm GMT

Andreas K. Hüttel: Perl in Gentoo: Upgrading pains, perl-cleaner, and EAPI=5

In a previous post, we've already looked at the structure of Perl ebuilds in Gentoo Linux. Now, let's see what happens in the case of a major Perl update.

Does this look familiar?

UPDATE THE PERL MODULES:
After updating dev-lang/perl you must reinstall
the installed perl modules.
Use: perl-cleaner --all

Then maybe you have updated your major Perl version recently, since this important message is printed by emerge afterwards. So, what is it about? In short, a certain disconnect between the "Perl way" of doing things and the rest of the world. Both have their merits, they just don't play very well with each other... and the result is that major Perl updates in Gentoo have traditionally also been a major pain. (This will become much better in the future, see below.)

Let's see where a perl package stores its files.

caipi ~ # equery files dev-perl/Email-Address
* Searching for Email-Address in dev-perl ...
* Contents of dev-perl/Email-Address-1.898.0:
/usr
/usr/lib
/usr/lib/perl5
/usr/lib/perl5/vendor_perl
/usr/lib/perl5/vendor_perl/5.16.3
/usr/lib/perl5/vendor_perl/5.16.3/Email
/usr/lib/perl5/vendor_perl/5.16.3/Email/Address.pm
/usr/share
/usr/share/doc
/usr/share/doc/Email-Address-1.898.0
/usr/share/doc/Email-Address-1.898.0/Changes.bz2
/usr/share/doc/Email-Address-1.898.0/README.bz2
caipi ~ #

Interesting- the installation path contains the Perl version! The reasons for upstream to do this are pretty much obvious, the application binary interface for compiled modules can change and it's necessary to keep the installed modules for different versions apart. Also, in theory you can keep different Perl versions installed in parallel. Nice idea, however if you have only one "system Perl" installation, and you exchange that for a newer version (say, 5.18.1 instead of 5.16.3), the result is that the new version won't find the installed packages anymore.

The results are rather annoying. Imagine you haven't updated your system for a while, one of the many packages to be updated is dev-lang/perl, and later maybe (just picking an example at random) gnome-base/gsettings-desktop-schemas. Perl is updated fine, but when portage arrives at building the gnome package, the build fails with something like

checking for perl >= 5.8.1... 5.18.2
checking for XML::Parser... configure: error: XML::Parser perl module is required for intltool

Right. Perl is updated, dev-perl/XML-Parser is still installed in the old path, and Perl doesn't find it. Bah.

Enter perl-cleaner, the traditional "solution". This small program checks for files in "outdated" Perl installation paths, finds out which packages they belong to, and makes portage rebuild the corresponding packages. During the rebuild, the installation is run by the updated Perl, which makes the files go into the new, now correct path.

This sounds like a good solution, but there are a lot of details and potential problems hidden. For once, most likely you'll run perl-cleaner after a failed emerge command, and some unrelated packages still need updates. Portage will try to figure out how to do this, but blockers and general weirdness may happen. Then, sometimes a package isn't needed with the new Perl version anymore, but perl-cleaner can't know that. Again the result may be a blocker. We've added the following instructions to the perl-cleaner output, which may help cleaning up the most frequent difficulties:

* perl-cleaner is stopping here:
* Fix the problem and start perl-cleaner again.
*
* If you encounter blockers involving virtuals and perl-core, here are
* some things to try:
* Remove all perl-core packages from your world file
* emerge --deselect --ask $(qlist -IC 'perl-core/*')
* Update all the installed Perl virtuals
* emerge -uD1a $(qlist -IC 'virtual/perl-*')
* Afterwards re-run perl-cleaner

In the end, you may have to try several repeated emerge and perl-cleaner commands until you have an updated and consistent system again. So far, it always worked somehow with fiddling, but the situation was definitely not nice.

So what's the future? Well...

EAPI=5 brings the beautiful new feature of subslots and slot operator dependencies. In short, a package A may declare a subslot, and a package B that depends on A may declare "rebuild me if A changes subslot". This mechanism is now used to automate the Perl rebuilds directly from within emerge: dev-lang/perl declares a subslot corresponding to its major version, say "5.18", and every package that installs Perl modules needs to depend on it with the subslot-rebuild requested, e.g.

RDEPEND="dev-lang/perl:="

The good news about this is that portage now knows the dependency tree and can figure out the correct reinstallation order.

The bad news is, it can only work perfectly after all Perl packages have been converted to EAPI=5 and stabilized. perl-core is done, but with about 2100 ebuilds that use perl-module.eclass in the portage tree still quite some work remains. I've plotted the current EAPI distribution of ebuilds using perl-module.eclass in a pie chart for illustration... Maybe we're done when Perl 5.20 goes stable. Who knows. :)

03 Aug 2014 10:14am GMT

01 Aug 2014

feedPlanet Gentoo

Sven Vermeulen: Gentoo Hardened July meeting

I failed to show up myself (I fell asleep - kids are fun, but deplete your energy source quickly), but that shouldn't prevent me from making a nice write-up of the meeting.

Toolchain

GCC 4.9 gives some issues with kernel compilations and other components. Lately, breakage has been reported with GCC 4.9.1 compiling MySQL or with debugging symbols. So for hardened, we'll wait this one out until the bugs are fixed.

For GCC 4.10, the -enable-default-pie patch has been sent upstream. If that is accepted, the SSP one will be sent as well.

In uclibc land, stages are being developed for PPC. This is the final architecture that is often used in embedded worlds that needed support for it in Gentoo, and that's now being finalized. Go blueness!

SELinux

A libpcre upgrade broke relabeling operations on SELinux enabled systems. A fix for this has been made part of libselinux, but a little too late, so some users will be affected by the problem. It's easily worked around (removing the *.bin files in the contexts/files/ directory of the SELinux configuration) and hopefully will never occur again.

The 2.3 userland has finally been stabilized (we had a few dependencies that we were waiting for - and we were a dependency ourselves for other packages as well).

Finally, some thought discussion is being done (not that there's much feedback on it, but every documented step is a good step imo) on the SELinux policy within Gentoo (and the principles that we'll follow that are behind it).

Kernel and grsecurity / PaX

Due to some security issues, the Linux kernel sources have been stabilized more rapidly than usual, which left little time for broad validation and regression testing. Updates and fixes have been applied since and new stabilizations occurred. Hopefully we're now at the right, stable set again.

The C-based install-xattr application (which is performance-wise a big improvement over the Python-based one) is working well in "lab environments" (some developers are using it exclusively). It is included in the Portage repository (if I understand the chat excerpts correctly) but as such not available for broader usage yet.

An update against elfix is made as well as there was a dependency mismatch when building with USE=-ptpax. This will be corrected in elfix-0.9.

Finally, blueness is also working on a GLEP (Gentoo Linux Enhancement Proposal) to export VDB information (especially NEEDED.ELF.2) as this is important for ELF/library graph information (as used by revdep-pax, migrate-pax, etc.). Although Portage already does this, this is not part of the PMS and as such other package managers might not do this (such as Paludis).

Profiles

Updates on the profiles has been made to properly include multilib related variables and other metadata. For some profiles, this went as easy as expected (nice stacking), but other profiles have inheritance troubles making it much harder to include the necessary information. Although some talks have arised on the gentoo-dev mailinglist about refactoring how Gentoo handles profiles, there hasn't been done much more than just talking :-( But I'm sure we haven't heard the last of this yet.

Documentation

Blueness has added information on EMULTRAMP in the kernel configuration, especially noting to the user that it is needed for Python support in Gentoo Hardened. It is also in the PaX Quickstart document, although this document is becoming a very large one and users might overlook it.

01 Aug 2014 7:48pm GMT

27 Jul 2014

feedPlanet Gentoo

Andreas K. Hüttel: Perl in Gentoo: dev-lang/perl, virtuals, and perl-core packages


We've got the stabilization of Perl 5.18 upcoming, so what better chance is there to explain a bit how the Perl-related ebuilds in Gentoo work...

First of all, there is dev-lang/perl. This contains the Perl core distribution, installing the binaries and all the Perl modules that are bundled with Perl itself.

Then, there is the perl-core category. It contains independent ebuilds for Perl modules that are also present in the core Perl distribution. Most Perl modules that are bundled with Perl are also in addition released as independent tarballs. If any of these packages is installed from perl-core, its files are placed such that the perl-core download overrides the bundled copy. This means you can also update part of the bundled Perl modules, e.g. in case of a bug, without updating Perl itself.

Next, there are a lot of virtuals "virtual/perl-..." in the virtual category of the portage tree. What are these good for? Well, imagine you want to depend on a specific version of a module that is usually bundled with Perl. For example, you need Module::CoreList at at least version 3. This can either be provided by a new enough Perl (for example, now hardmasked Perl 5.20 contains Module::CoreList 3.10), or by a separate package from perl-core (where we have Module::CoreList 5.021001 as perl-core/Module-CoreList-5.21.1).
To make sure that everything works, you should never directly depend on a perl-core package, but always on the corresponding virtual (here virtual/perl-Module-CoreList; any perl-core package must have a corresponding virtual). Then both ways to fulfil the dependency are automatically taken into account. (Future repoman versions will warn if you directly depend on perl-core. Also you should never have anything perl-core in your world file!)

Last, we have lots of lots of modules in the dev-perl category. Most of them are from CPAN, and the only thing they have in common is that they have no copy inside core Perl.

I hope this clarifies things a bit. More Perl posts coming...

27 Jul 2014 12:45pm GMT

22 Jul 2014

feedPlanet Gentoo

Diego E. Pettenò: LibreSSL: drop-in and ABI leakage

There has been some confusion on my previous post with Bob Beck of LibreSSL on whether I would advocate for using a LibreSSL shared object as a drop-in replacement for an OpenSSL shared object. Let me state this here, boldly: you should never, ever, for no reason, use shared objects from different major/minor OpenSSL versions or implementations (such as LibreSSL) as a drop-in replacement for one another.

The reason is, obviously, that the ABI of these libraries differs, sometimes subtly enought that they may actually load and run, but then perform abysmally insecure operations, as its data structures will have changed, and now instead of reading your random-generated key, you may be reading the master private key. nd in general, for other libraries you may even be calling the wrong set of functions, especially for those written in C++, where the vtable content may be rearranged across versions.

What I was discussing in the previous post was the fact that lots of proprietary software packages, by bundling a version of Curl that depends on the RAND_egd() function, will require either unbundling it, or keeping along a copy of OpenSSL to use for runtime linking. And I think that is a problem that people need to consider now rather than later for a very simple reason.

Even if LibreSSL (or any other reimplementation, for what matters) takes foot as the default implementation for all Linux (and not-Linux) distributions, you'll never be able to fully forget of OpenSSL: not only if you have proprietary software that you maintain, but also because a huge amount of software (and especially hardware) out there will not be able to update easily. And the fact that LibreSSL is throwing away so much of the OpenSSL clutter also means that it'll be more difficult to backport fixes - while at the same time I think that a good chunk of the black hattery will focus on OpenSSL, especially if it feels "abandoned", while most of the users will still be using it somehow.

But putting aside the problem of the direct drop-in incompatibilities, there is one more problem that people need to understand, especially Gentoo users, and most other systems that do not completely rebuild their package set when replacing a library like this. The problem is what I would call "ABI leakage".

Let's say you have a general libfoo that uses libssl; it uses a subset of the API that works with both OpenSSL. Now you have a bar program that uses libfoo. If the library is written properly, then it'll treat all the data structures coming from libssl as opaque, providing no way for bar to call into libssl without depending on the SSL API du jour (and thus putting a direct dependency on libssl for the executable). But it's very well possible that libfoo is not well-written and actually treats the libssl API as transparent. For instance a common mistake is to use one of the SSL data structures inline (rather than as a pointer) in one of its own public structures.

This situation would be barely fine, as long as the data types for libfoo are also completely opaque, as then it's only the code for libfoo that relies on the structures, and since you're rebuilding it anyway (as libssl is not ABI-compatible), you solve your problem. But if we keep assuming a worst-case scenario, then you have bar actually dealing with the data structures, for instance by allocating a sized buffer itself, rather than calling into a proper allocation function from libfoo. And there you have a problem.

Because now the ABI of libfoo is not directly defined by its own code, but also by whichever ABI libssl has! It's a similar problem as the symbol table used as an ABI proxy: while your software will load and run (for a while), you're really using a different ABI, as libfoo almost certainly does not change its soname when it's rebuilt against a newer version of libssl. And that can easily cause crashes and worse (see the note above about dropping in LibreSSL as a replacement for OpenSSL).

Now honestly none of this is specific to LibreSSL. The same is true if you were to try using OpenSSL 1.0 shared objects for software built against OpenSSL 0.9 - which is why I cringed any time I heard people suggesting to use symlink at the time, and it seems like people are giving the same suicidal suggestion now with OpenSSL, according to Bob.

So once again, don't expect binary-compatibility across different versions of OpenSSL, LibreSSL, or any other implementation of the same API, unless they explicitly aim for that (and LibreSSL definitely doesn't!)

22 Jul 2014 11:09pm GMT

20 Jul 2014

feedPlanet Gentoo

Diego E. Pettenò: LibreSSL and the bundled libs hurdle

It was over five years ago that I ranted about the bundling of libraries and what that means for vulnerabilities found in those libraries. The world has, since, not really listened. RubyGems still keep insisting that "vendoring" gems is good, Go explicitly didn't implement a concept of shared libraries, and let's not even talk about Docker or OSv and their absolutism in static linking and bundling of the whole operating system, essentially.

It should have been obvious how this can be a problem when Heartbleed came out, bundled copies of OpenSSL would have needed separate updates from the system libraries. I guess lots of enterprise users of such software were saved only by the fact that most of the bundlers ended up using older versions of OpenSSL where heartbeat was not implemented at all.

Now that we're talking about replacing the OpenSSL libraries with those coming from a different project, we're going to be hit by both edges of the proprietary software sword: bundling and ABI compatibility, which will make things really interesting for everybody.

If you've seen my (short, incomplete) list of RAND_egd() users which I posted yesterday. While the tinderbox from which I took this is out of date and needs cleaning, it is a good starting point to figure out the trends, and as somebody already picked up, the bundling is actually strong.

Software that bundled Curl, or even Python, but then relied on the system copy of OpenSSL, will now be looking for RAND_egd() and thus fail. You could be unbundling these libraries, and then use a proper, patched copy of Curl from the system, where the usage of RAND_egd() has been removed, but then again, this is what I've been advocating forever or so. With caveats, in the case of Curl.

But now if the use of RAND_egd() is actually coming from the proprietary bits themselves, you're stuck and you can't use the new library: you either need to keep around an old copy of OpenSSL (which may be buggy and expose even more vulnerability) or you need a shim library that only provides ABI compatibility against the new LibreSSL-provided library - I'm still not sure why this particular trick is not employed more often, when the changes to a library are only at the interface level but still implements the same functionality.

Now the good news is that from the list that I produced, at least the egd functions never seemed to be popular among proprietary developers. This is expected as egd was vastly a way to implement the /dev/random semantics for non-Linux systems, while the proprietary software that we deal with, at least in the Linux world, can just accept the existence of the devices themselves. So the only problems have to do with unbundling (or replacing) Curl and possibly the Python SSL module. Doing so is not obvious though, as I see from the list that there are at least two copies of libcurl.so.3 which is the older ABI for Curl - although admittedly one is from the scratchbox SDKs which could just as easily be replaced with something less hacky.

Anyway, my current task is to clean up the tinderbox so that it's in a working state, after which I plan to do a full build of all the reverse dependencies on OpenSSL, it's very possible that there are more entries that should be in the list, since it was built with USE=gnutls globally to test for GnuTLS 3.0 when it came out.

20 Jul 2014 9:55am GMT

19 Jul 2014

feedPlanet Gentoo

Paweł Hajdan, Jr.: Recovering from removed libgcc_s.so.1 (and missing busybox)

I was experimenting in my arm chroot, and after a gcc upgrade and emerge --depclean --ask that removed the old gcc I got the following error:

# ls -l
ls: error while loading shared libraries: libgcc_s.so.1: cannot open shared object file: No such file or directory

Fortunately the newer working gcc was present, so the steps to make things work again were:

# LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/lib/gcc/armv7a-hardfloat-linux-gnueabi/4.8.2/" gcc-config -l
* gcc-config: Active gcc profile is invalid!

[1] armv7a-hardfloat-linux-gnueabi-4.8.2

# LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/lib/gcc/armv7a-hardfloat-linux-gnueabi/4.8.2/" gcc-config 1
* Switching native-compiler to armv7a-hardfloat-linux-gnueabi-4.8.2 ...

Actually my first thought was using busybox. The unexpected breakage during a routine gcc upgrade made me do some research in case I can't rely on /bin/busybox being present and working.

I highly recommend the following links for further reading:
http://lambdaops.com/rm-rf-remains
http://eusebeia.dyndns.org/bashcp
http://www.reddit.com/r/linux/comments/27is0x/rm_rf_remains/ci199bk

Read more »

19 Jul 2014 5:19pm GMT

Diego E. Pettenò: LibreSSL is taking a beating, and that's good

When I read about LibreSSL coming from the OpenBSD developers, my first impression was that it was a stunt. I did not change my impression of it drastically still. While I know at least one quite good OpenBSD developer, my impression of the whole set is still the same: we have different concepts of security, and their idea of "cruft" is completely out there for me. But this is a topic for some other time.

So seeing the amount of scrutiny from other who are, like me, skeptical of the OpenBSD people left on their own is a good news. It keeps them honest, as they say. But it also means that things that wouldn't be otherwise understood by people not used to Linux don't get shoved under the rug.

This is not idle musings: I still remember (but can't find now) an article in which Theo boasted not ever having used Linux. And yet kept insisting that his operating system was clearly superior. I was honestly afraid that the way the fork-not-a-fork project was going to be handled was the same, I'm positively happy to be proven wrong up to now.

I actually have been thrilled to see that finally there is movement to replace the straight access to /dev/random and /dev/urandom: Ted's patch to implement a getrandom() system call that can be made compatible with OpenBSD's own getentropy() in user space. And even more I'm happy to see that at least one of the OpenBSD/LibreSSL developers pitching in to help shape the interface.

Dropping out the egd support made me puzzled for a moment, but then I realized that there is no point in using egd to feed the randomness to the process, you just need to feed entropy to the kernel, and let the process get it normally. I have had, unfortunately, quite a bit of experience with entropy-generating daemons, and I wonder if this might be the right time to suggest getting a new multi-source daemon out.

So a I going to just blindly trust the OpenBSD people because "they have a good track record"? No. And to anybody that suggest that you can take over lines and lines of code from someone else's crypto-related project, remove a bunch of code that you think is useless, and have an immediate result, my request is to please stop working with software altogether.

Security Holes Copyright © Randall Munroe.

I'm not saying that they would do it on purpose, or that they wouldn't be trying to do the darndest to make LibreSSL a good replacement for OpenSSL. What I'm saying is that I don't like the way, and the motives, the project was started from. And I think that a reality check, like the one they already got, was due and a good news.

On my side, once the library gets a bit more mileage I'll be happy to run the tinderbox against it. For now, I'm re-gaining access to Excelsior after a bad kernel update, and I'll just go and search with elfgrep for which binaries do use the egd functionalities and need to be patched, I'll post it on Twitter/G+ once I have it. I know it's not much, but this is what I can do.

19 Jul 2014 4:23pm GMT

14 Jul 2014

feedPlanet Gentoo

Richard Freeman: Quick systemd-nspawn guide

I switched to using systemd-nspawn in place of chroot and wanted to give a quick guide to using it. The short version is that I'd strongly recommend that anybody running systemd that uses chroot switch over - there really are no downsides as long as your kernel is properly configured.

Chroot should be no stranger to anybody who works on distros, and I suspect that the majority of Gentoo users have need for it from time to time.

The Challenges of chroot

For most interactive uses it isn't sufficient to just run chroot. Usually you need to mount /proc, /sys, and bind mount /dev so that you don't have issues like missing ptys, etc. If you use tmpfs you might also want to mount the new tmp, var/tmp as tmpfs. Then you might want to make other bind mounts into the chroot. None of this is particularly difficult, but you usually end up writing a small script to manage it.

Now, I routinely do full backups, and usually that involves excluding stuff like tmp dirs, and anything resembling a bind mount. When I set up a new chroot that means updating my backup config, which I usually forget to do since most of the time the chroot mounts aren't running anyway. Then when I do leave it mounted overnight I end up with backups consuming lots of extra space (bind mounts of large trees).

Finally, systemd now by default handles bind mounts a little differently when they contain other mount points (such as when using -rbind). Apparently unmounting something in the bind mount will cause systemd to unmount the corresponding directory on the other side of the bind. Imagine my surprise when I unmounted my chroot bind to /dev and discovered /dev/pts and /dev/shm no longer mounted on the host. It looks like there are ways to change that, but this isn't the point of my post (it just spurred me to find another way).

Systemd-nspawn's Advantages

Systemd-nspawn is a tool that launches a container, and it can operate just like chroot in its simplest form. By default it automatically sets up most of the overhead like /dev, /tmp, etc. With a few options it can also set up other bind mounts as well. When the container exits all the mounts are cleaned up.

From the outside of the container nothing appears different when the container is running. In fact, you could spawn 5 different systemd-nspawn container instances from the same chroot and they wouldn't have any interaction except via the filesystem (and that excludes /dev, /tmp, and so on - only changes in /usr, /etc will propagate across). Your backup won't see the bind mounts, or tmpfs, or anything else mounted within the container.

The container also has all those other nifty container benefits like containment - a killall inside the container won't touch anything outside, and so on. The security isn't airtight - the intent is to prevent accidental mistakes.

Then, if you use a compatible sysvinit (which includes systemd, and I think recent versions of openrc), you can actually boot the container, which drops you to a getty inside. That means you can use fstab to do additional mounts inside the container, run daemons, and so on. You get almost all the benefits of virtualization for the cost of a chroot (no need to build a kernel, and so on). It is a bit odd to be running systemctl poweroff inside what looks just like a chroot, but it works.

Note that unless you do a bit more setup you will share the same network interface with the host, so no running sshd on the container if you have it on the host, etc. I won't get into this but it shouldn't be hard to run a separate network namespace and bind the interfaces so that the new instance can run dhcp.

How to do it

So, getting it actually working will likely be the shortest bit in this post.

You need support for namespaces and multiple devpts instances in your kernel:

CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_USER_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_DEVPTS_MULTIPLE_INSTANCES=y

From there launching a namespace just like a chroot is really simple:

systemd-nspawn -D .

That's it - you can exit from it just like a chroot. From inside you can run mount and see that it has taken care of /dev and /tmp for you. The "." is the path to the chroot, which I assume is the current directory. With nothing further it runs bash inside.

If you want to add some bind mounts it is easy:

systemd-nspawn -D . -bind /usr/portage

Now your /usr/portage is bound to your host, so no need to sync/etc. If you want to bind to a different destination add a ":dest" after the source, relative to the root of the chroot (so -bind foo is the same as -bind foo:foo).

If the container has a functional init that can handle being run inside, you can add a -b to boot it:

systemd-nspawn -D . -bind /usr/portage -b

Watch the init do its job. Shut down the container to exit.

Now, if that container is running systemd you can direct its journal to the host journal with -h:

systemd-nspawn -D . -bind /usr/portage -j -b

Now, nspawn registers the container so that it shows up in machinectl. That makes it easy to launch a new getty on it, or ssh to it (if it is running ssh - see my note above about network namespaces), or power it off from the host.

That's it. If you're running systemd I'd suggest ditching chroot almost entirely in favor of nspawn.


Filed under: foss, gentoo, linux

14 Jul 2014 8:31pm GMT