02 Sep 2015

feedPlanet Gentoo

Sven Vermeulen: Maintaining packages and backporting

A few days ago I committed a small update to policycoreutils, a SELinux related package that provides most of the management utilities for SELinux systems. The fix was to get two patches (which are committed upstream) into the existing release so that our users can benefit from the fixed issues without having to wait for a new release.

Getting the patches

To capture the patches, I used git together with the commit id:

~$ git format-patch -n -1 73b7ff41
0001-Only-invoke-RPM-on-RPM-enabled-Linux-distributions.patch
~$ git format-patch -n -1 4fbc6623
0001-Set-self.sename-to-sename-after-calling-semanage_seu.patch

The two generated patch files contain all information about the commit. Thanks to the epatch support in the eutils.eclass, these patch files are immediately usable within Gentoo's ebuilds.

Updating the ebuilds

The SELinux userspace ebuilds in Gentoo all have live ebuilds available which are immediately usable for releases. The idea with those live ebuilds is that we can simply copy them and commit in order to make a new release.

So, in case of the patch backporting, the necessary patch files are first moved into the files/ subdirectory of the package. Then, the live ebuild is updated to use the new patches:

@@ -88,6 +85,8 @@ src_prepare() {
                epatch "${FILESDIR}/0070-remove-symlink-attempt-fails-with-gentoo-sandbox-approach.patch"
                epatch "${FILESDIR}/0110-build-mcstrans-bug-472912.patch"
                epatch "${FILESDIR}/0120-build-failure-for-mcscolor-for-CONTEXT__CONTAINS.patch"
+               epatch "${FILESDIR}/0130-Only-invoke-RPM-on-RPM-enabled-Linux-distributions-bug-534682.patch"
+               epatch "${FILESDIR}/0140-Set-self.sename-to-sename-after-calling-semanage-bug-557370.patch"
        fi

        # rlpkg is more useful than fixfiles

The patches themselves do not apply for the live ebuilds themselves (they are ignored there) as we want the live ebuilds to be as close to the upstream project as possible. But because the ebuilds are immediately usable for releases, we add the necessary information there first.

Next, the new release is created:

~$ cp policycoreutils-9999.ebuild policycoreutils-2.4-r2.ebuild

Testing the changes

The new release is then tested. I have a couple of scripts that I use for automated testing. So first I update these scripts to also try out the functionality that was failing before. On existing systems, these tests should fail:

Running task semanage (Various semanage related operations).
  ...
    Executing step "perm_port_on   : Marking portage_t as a permissive domain                              " -> ok
    Executing step "perm_port_off  : Removing permissive mark from portage_t                               " -> ok
    Executing step "selogin_modify : Modifying a SELinux login definition                                  " -> failed

Then, on a test system where the new package has been installed, the same testset is executed (together with all other tests) to validate if the problem is fixed.

Pushing out the new release

Finally, with the fixes in and validated, the new release is pushed out (into ~arch first of course) and the bugs are marked as RESOLVED:TEST-REQUEST. Users can confirm that it works (which would move it to VERIFIED:TEST-REQUEST) or we stabilize it after the regular testing period is over (which moves it to RESOLVED:FIXED or VERIFIED:FIXED).

I do still have to get used to Gentoo using git as its repository now. The workflow to use is documented though. Luckily, because I often get that the git push fails (due to changes to the tree since my last pull). So I need to run git pull --rebase=preserve followed by repoman full and then the push again sufficiently quick after each other).

This simple flow is easy to get used to. Thanks to the existing foundation for package maintenance (such as epatch for patching, live ebuilds that can be immediately used for releases and the ability to just cherry pick patches towards our repository) we can serve updates with just a few minutes of work.

02 Sep 2015 6:33pm GMT

Bernard Cafarelli: LLVM 3.7.0 released, Gentoo package changes

As announced here, the new LLVM release is now available. If you are interested after reading the release notes (or the clang ones), I just added the corresponding packages in Gentoo, although still masked for some additional testing. mesa-11.0 release candidates compile fine with it though.

On the Gentoo side, this version bump has some nice changes:

On the TODO list, some tests are still failing on 32bit multilib (bug #558604), ocaml doc compilation/installation has a strange bug (see the additional call on docs/ocaml_doc target in the ebuild). And let's not forget adding other tools, like polly

Also, if people were still using the dragonegg GCC plugin, note that it is not released anymore upstream. As 3.6.x versions had compilation problems, this package received its last rites.

02 Sep 2015 11:53am GMT

28 Aug 2015

feedPlanet Gentoo

Michael Palimaka: There’s no such thing as KDE 5, and its status on Gentoo

Background

For those that are not already aware, KDE's release structure has evolved. The familiar all-in-one release of KDE SC 4 has been split into three distinct components, each with their own release cycles - KDE Frameworks 5, KDE Plasma 5, and KDE Applications 5.

This means there's no such thing as KDE 5!

KDE Frameworks 5

KDE Frameworks 5 is a collection of libraries upon which Plasma and Applications are built. Each framework is distinct in terms of functionality, allowing consumers to depend on smaller individual libraries. This has driven adoption in other Qt-based projects such as LXQt as they no longer have to worry about "pulling in KDE".

We ship the latest version of KDE Frameworks 5 in the main tree, and plan to target it for stabilisation shortly.

KDE Plasma 5

KDE Plasma 5 is the next generation of the Plasma desktop environment. While some might not consider it as mature as Plasma 4, it is in a good state for general use and is shipped as stable by a number of other distributions.

We ship the latest version of KDE Plasma 5 in the main tree, and would expect to target it for stabilisation within a few months.

KDE Applications 5

KDE Applications 5 consists of the remaining applications and supporting libraries. Porting is gradual process, with each new major release containing more KF5-based and fewer KDE 4-based packages.

Unfortunately, current Applications releases are not entirely coherent - some packages have features that require unreleased dependencies, and some cannot be installed at the same time as others. This situation is expected to improve in future releases as porting efforts progress.

Because of this, it's not possible to ship KDE Applications in its entirety. Rather, we are in the process of cherry-picking known-good packages into the main tree. We have not discussed any stabilisation plan yet.

Migrating

As Frameworks are just libraries, they are automatically pulled in as required by consuming packages, and no user intervention is required.

To upgrade to Plasma 5, please follow the upgrade guide. Unfortunately it's not possible to have both Plasma 4 and Plasma 5 installed at the same time, due to an upstream design decision.

Applications appear in the tree as a regular version bump, so will upgrade automatically.

Ongoing KDE 4 support

Plasma 4 has reached end-of-life upstream, and no further releases are expected. As per usual, we will keep it for a reasonable time before removing it completely.

As each Applications upgrade should be invisible, there's less need to retain old versions. It is likely that the existing policy of removing old shortly after stabilisation will continue.

What is this 15.08.0 stuff, or, why is it upgrading to KDE 5?

As described above, Applications are now released separately from Plasma, following a yy.mm.xx versioning scheme. This means that, regardless of whether they are KDE 4 or KF5-based, they will work correctly in both Plasma 4 and Plasma 5, or any other desktop environment.

It is the natural upgrade path, and there is no longer a "special relationship" between Plasma and Applications the way there was in KDE SC 4.

Feedback

As always, feedback is appreciated - especially during major transitions like this. Sharing your experience will help improve it for the next person, and substantial improvements have already been made made thanks to the contributions of early testers.

Feel free to file a bug, send a mail, or drop by #gentoo-kde for a chat any time. Thanks for flying Gentoo KDE!

28 Aug 2015 8:06pm GMT

27 Aug 2015

feedPlanet Gentoo

Alexys Jacob: py3status v2.6

Ok I was a bit too hasty in my legacy module support code clean up and I broke quite a few things on the latest version 2.5 release sorry ! :(

highlights

thanks !

27 Aug 2015 5:04pm GMT

25 Aug 2015

feedPlanet Gentoo

Sven Vermeulen: Slowly converting from GuideXML to HTML

Gentoo has removed its support of the older GuideXML format in favor of using the Gentoo Wiki and a new content management system for the main site (or is it static pages, I don't have the faintest idea to be honest). I do still have a few GuideXML pages in my development space, which I am going to move to HTML pretty soon.

In order to do so, I make use of the guidexml2wiki stylesheet I developed. But instead of migrating it to wiki syntax, I want to end with HTML.

So what I do is first convert the file from GuideXML to MediaWiki with xsltproc.

Next, I use pandoc to convert this to restructured text. The idea is that the main pages on my devpage are now restructured text based. I was hoping to use markdown, but the conversion from markdown to HTML is not what I hoped it was.

The restructured text is then converted to HTML using rst2html.py. In the end, I use the following function (for conversion, once):

# Convert GuideXML to RestructedText and to HTML
gxml2html() {
  basefile=${1%%.xml};

  # Convert to Mediawiki syntax
  xsltproc ~/dev-cvs/gentoo/xml/htdocs/xsl/guidexml2wiki.xsl $1 > ${basefile}.mediawiki

  if [ -f ${basefile}.mediawiki ] ; then
    # Convert to restructured text
    pandoc -f mediawiki -t rst -s -S -o ${basefile}.rst ${basefile}.mediawiki;
  fi

  if [ -f ${basefile}.rst ] ; then
    # Use your own stylesheet links (use full https URLs for this)
    rst2html.py  --stylesheet=link-to-bootstrap.min.css,link-to-tyrian.min.css --link-stylesheet ${basefile}.rst ${basefile}.html
  fi
}

Is it perfect? No, but it works.

25 Aug 2015 9:30am GMT

17 Aug 2015

feedPlanet Gentoo

Alexys Jacob: py3status v2.5

This new py3status comes with an amazing number of contributions and new modules !

24 files changed, 1900 insertions(+), 263 deletions(-)

I'm also glad to say that py3status becomes my first commit in the new git repository of Gentoo Linux !

IMPORTANT

Please note that this version has deprecated the legacy implicit module loading support to favour and focus on the generic i3status order += module loading/ordering !

New modules

Changelog

As usual, full changelog is available here.

Contributors

Along with all those who reported issues and helped fixed them, quick and surely not exhaustive list:

What's next ?

Well something tells me @Horgix is working hard on some standardization and on the core of py3status ! I'm sure some very interesting stuff will emerge from this, so thank you !

17 Aug 2015 3:05pm GMT

13 Aug 2015

feedPlanet Gentoo

Sven Vermeulen: Finding a good compression utility

I recently came across a wiki page written by Herman Brule which gives a quick benchmark on a couple of compression methods / algorithms. It gave me the idea of writing a quick script that tests out a wide number of compression utilities available in Gentoo (usually through the app-arch category), with also a number of options (in case multiple options are possible).

The currently supported packages are:

app-arch/bloscpack      app-arch/bzip2          app-arch/freeze
app-arch/gzip           app-arch/lha            app-arch/lrzip
app-arch/lz4            app-arch/lzip           app-arch/lzma
app-arch/lzop           app-arch/mscompress     app-arch/p7zip
app-arch/pigz           app-arch/pixz           app-arch/plzip
app-arch/pxz            app-arch/rar            app-arch/rzip
app-arch/xar            app-arch/xz-utils       app-arch/zopfli
app-arch/zpaq

The script should keep the best compression information: duration, compression ratio, compression command, as well as the compressed file itself.

Finding the "best" compression

It is not my intention to find the most optimal compression, as that would require heuristic optimizations (which has triggered my interest in seeking such software, or writing it myself) while trying out various optimization parameters.

No, what I want is to find the "best" compression for a given file, with "best" being either

For me personally, I think I would use it for the various raw image files that I have through the photography hobby. Those image files are difficult to compress (the Nikon DS3200 I use is an entry-level camera which applies lossy compression already for its raw files) but their total size is considerable, and it would allow me to better use the storage I have available both on my laptop (which is SSD-only) as well as backup server.

But next to the best compression ratio, the efficiency is also an important metric as it shows how efficient the algorithm works in a certain time aspect. If one compression method yields 80% reduction in 5 minutes, and another one yields 80,5% in 45 minutes, then I might want to prefer the first one even though that is not the best compression at all.

Although the script could be used to get the most compression (without resolving to an optimization algorithm for the compression commands) for each file, this is definitely not the use case. A single run can take hours for files that are compressed in a handful of seconds. But it can show the best algorithms for a particular file type (for instance, do a few runs on a couple of raw image files and see which method is most succesful).

Another use case I'm currently looking into is how much improvement I can get when multiple files (all raw image files) are first grouped in a single archive (.tar). Theoretically, this should improve the compression, but by how much?

How the script works

The script does not contain much intelligence. It iterates over a wide set of compression commands that I tested out, checks the final compressed file size, and if it is better than a previous one it keeps this compressed file (and its statistics).

I tried to group some of the compressions together based on the algorithm used, but as I don't really know the details of the algorithms (it's based on manual pages and internet sites) and some of them combine multiple algorithms, it is more of a high-level selection than anything else.

The script can also only run the compressions of a single application (which I use when I'm fine-tuning the parameter runs).

A run shows something like the following:

Original file (test.nef) size 20958430 bytes
      package name                                                 command      duration                   size compr.Δ effic.:
      ------------                                                 -------      --------                   ---- ------- -------
app-arch/bloscpack                                               blpk -n 4           0.1               20947097 0.00054 0.00416
app-arch/bloscpack                                               blpk -n 8           0.1               20947097 0.00054 0.00492
app-arch/bloscpack                                              blpk -n 16           0.1               20947097 0.00054 0.00492
    app-arch/bzip2                                                   bzip2           2.0               19285616 0.07982 0.03991
    app-arch/bzip2                                                bzip2 -1           2.0               19881886 0.05137 0.02543
    app-arch/bzip2                                                bzip2 -2           1.9               19673083 0.06133 0.03211
...
    app-arch/p7zip                                      7za -tzip -mm=PPMd           5.9               19002882 0.09331 0.01592
    app-arch/p7zip                             7za -tzip -mm=PPMd -mmem=24           5.7               19002882 0.09331 0.01640
    app-arch/p7zip                             7za -tzip -mm=PPMd -mmem=25           6.4               18871933 0.09955 0.01551
    app-arch/p7zip                             7za -tzip -mm=PPMd -mmem=26           7.7               18771632 0.10434 0.01364
    app-arch/p7zip                             7za -tzip -mm=PPMd -mmem=27           9.0               18652402 0.11003 0.01224
    app-arch/p7zip                             7za -tzip -mm=PPMd -mmem=28          10.0               18521291 0.11628 0.01161
    app-arch/p7zip                                       7za -t7z -m0=PPMd           5.7               18999088 0.09349 0.01634
    app-arch/p7zip                                7za -t7z -m0=PPMd:mem=24           5.8               18999088 0.09349 0.01617
    app-arch/p7zip                                7za -t7z -m0=PPMd:mem=25           6.5               18868478 0.09972 0.01534
    app-arch/p7zip                                7za -t7z -m0=PPMd:mem=26           7.5               18770031 0.10442 0.01387
    app-arch/p7zip                                7za -t7z -m0=PPMd:mem=27           8.6               18651294 0.11008 0.01282
    app-arch/p7zip                                7za -t7z -m0=PPMd:mem=28          10.6               18518330 0.11643 0.01100
      app-arch/rar                                                     rar           0.9               20249470 0.03383 0.03980
      app-arch/rar                                                 rar -m0           0.0               20958497 -0.00000        -0.00008
      app-arch/rar                                                 rar -m1           0.2               20243598 0.03411 0.14829
      app-arch/rar                                                 rar -m2           0.8               20252266 0.03369 0.04433
      app-arch/rar                                                 rar -m3           0.8               20249470 0.03383 0.04027
      app-arch/rar                                                 rar -m4           0.9               20248859 0.03386 0.03983
      app-arch/rar                                                 rar -m5           0.8               20248577 0.03387 0.04181
    app-arch/lrzip                                                lrzip -z          13.1               19769417 0.05673 0.00432
     app-arch/zpaq                                                    zpaq           0.2               20970029 -0.00055        -0.00252
The best compression was found with 7za -t7z -m0=PPMd:mem=28.
The compression delta obtained was 0.11643 within 10.58 seconds.
This file is now available as test.nef.7z.

In the above example, the test file was around 20 MByte. The best compression compression command that the script found was:

~$ 7za -t7z -m0=PPMd:mem=28 a test.nef.7z test.nef

The resulting file (test.nef.7z) is 18 MByte, a reduction of 11,64%. The compression command took almost 11 seconds to do its thing, which gave an efficiency rating of 0,011, which is definitely not a fast one.

Some other algorithms don't do bad either with a better efficiency. For instance:

   app-arch/pbzip2                                                  pbzip2           0.6               19287402 0.07973 0.13071

In this case, the pbzip2 command got almost 8% reduction in less than a second, which is considerably more efficient than the 11-seconds long 7za run.

Want to try it out yourself?

I've pushed the script to my github location. Do a quick review of the code first (to see that I did not include anything malicious) and then execute it to see how it works:

~$ sw_comprbest -h
Usage: sw_comprbest --infile=<inputfile> [--family=<family>[,...]] [--command=<cmd>]
       sw_comprbest -i <inputfile> [-f <family>[,...]] [-c <cmd>]

Supported families: blosc bwt deflate lzma ppmd zpaq. These can be provided comma-separated.
Command is an additional filter - only the tests that use this base command are run.

The output shows
  - The package (in Gentoo) that the command belongs to
  - The command run
  - The duration (in seconds)
  - The size (in bytes) of the resulting file
  - The compression delta (percentage) showing how much is reduced (higher is better)
  - The efficiency ratio showing how much reduction (percentage) per second (higher is better)

When the command supports multithreading, we use the number of available cores on the system (as told by /proc/cpuinfo).

For instance, to try it out against a PDF file:

~$ sw_comprbest -i MEA6-Sven_Vermeulen-Research_Summary.pdf
Original file (MEA6-Sven_Vermeulen-Research_Summary.pdf) size 117763 bytes
...
The best compression was found with zopfli --deflate.
The compression delta obtained was 0.00982 within 0.19 seconds.
This file is now available as MEA6-Sven_Vermeulen-Research_Summary.pdf.deflate.

So in this case, the resulting file is hardly better compressed - the PDF itself is already compressed. Let's try it against the uncompressed PDF:

~$ pdftk MEA6-Sven_Vermeulen-Research_Summary.pdf output test.pdf uncompress
~$ sw_comprbest -i test.pdf
Original file (test.pdf) size 144670 bytes
...
The best compression was found with lrzip -z.
The compression delta obtained was 0.27739 within 0.18 seconds.
This file is now available as test.pdf.lrz.

This is somewhat better:

~$ ls -l MEA6-Sven_Vermeulen-Research_Summary.pdf* test.pdf*
-rw-r--r--. 1 swift swift 117763 Aug  7 14:32 MEA6-Sven_Vermeulen-Research_Summary.pdf
-rw-r--r--. 1 swift swift 116606 Aug  7 14:32 MEA6-Sven_Vermeulen-Research_Summary.pdf.deflate
-rw-r--r--. 1 swift swift 144670 Aug  7 14:34 test.pdf
-rw-r--r--. 1 swift swift 104540 Aug  7 14:35 test.pdf.lrz

The resulting file is 11,22% reduced from the original one.

13 Aug 2015 5:15pm GMT

12 Aug 2015

feedPlanet Gentoo

Gentoo News: Gentoo Package Repository now using Git

Good things come to those who wait: The main Gentoo package repository (also known as the Portage tree or by its historic name gentoo-x86) is now based on Git.

Timeline

The Gentoo Git migration has arrived and is expected to be completed soon. As previously announced, the CVS freeze occurred on 8 August and Git commits for developers were opened soon after. As a last step, rsync mirrors are expected to have the updated changelogs again on or after 12 August. Read-only access to gentoo-x86 (and write to the other CVS repositories) was restored on 9 August following the freeze.

History

Work on migrating the repository from CVS to Git began in 2006 with a proof-of-concept migration project during the Summer of Code. Back then, migrating the repository took a week and using Git was considerably slower than using CVS. While plans to move were shelved for a while, things improved over the coming years. Several features were implemented in Git, Portage, and other tools to meet the requirements of a migrated repository.

What changes?

The repository can be checked out from git.gentoo.org and is available via our Git web interface.

For users of our package repository, nothing changes: Updates continue to be available via the established mechanisms (rsync, webrsync, snapshots). Options to fetch the package tree via Git are to be announced later.

The migration facilitates the process of new contributors getting involved as proxy maintainers and eventually Developers. Alternate places for users to submit pull requests, such as GitHub, can be expected in the future.

In regards to package signing, the migration will streamline how GPG keys are used. This will allow end-to-end signature tracking from the developer to the final repository, as outlined in GLEP 57 et seq.

While the last issues are being worked on, join us in thanking everyone involved in the project. As always, you can discuss this on our Forums or hit up @Gentoo.

12 Aug 2015 12:00am GMT

07 Aug 2015

feedPlanet Gentoo

Nathan Zachary: Apache 2.4, the Event MPM, PHP via mod_proxy_fcgi and PHP-FPM – with vhosts!

Important!

My tech articles-especially Linux ones-are some of the most-viewed on The Z-Issue. If this one has helped you, please consider a small donation to The Parker Fund by using the top widget at the right. Thanks!

Recently, I've spent a huge amount of time working on Apache and PHP-FPM in order to allow for a threaded Apache MPM whilst still using PHP and virtual hosts (vhosts). As this article is going to be rather lengthy, I'm going to split it up into sections below. It's my hope that after reading the article, you'll be able to take advantage of Apache's newer Event MPM and mod_proxy_fcgi in order to get the best performance out of your web server.

1. Definitions:

Before delving into the overarching problem, it might be best to have some definitions in place. If you're new to the whole idea of web servers, programming languages used for web sites, and such, these definitions may help you become more acquainted with the problem that this article addresses. If you're familiar with things like Apache, PHP, threading, preforking, and so on, feel free to skip this basic section.

Defining threads versus processes in detail is beyond the scope of this article, but basically, a thread is much smaller and less resource-intense than a process. There are many benefits to using threads instead of full processes for smaller tasks.

2. Introduction:

Okay, so now that we have a few operational definitions in place, let's get to the underlying problem that this article addresses. When running one or more busy websites on a server, it's desirable to reduce the amount of processing power and memory that is needed to serve those sites. Doing so will, consequently, allow the sites to be served more efficiently, rapidly, and to more simultaneous clients. One great way to do that is to switch from the Prefork MPM to the new Event MPM in Apache 2.4. However, that requires getting rid of mod_php and switching to a PHP-FPM backend proxied via mod_proxy_fcgi. All that sounds fine and dandy, except that there have been several problems with it in the past (such as .htaccess files not being honoured, effectively passing the processed PHP file back to Apache, and making the whole system work with virtual hosts [vhosts]). Later, I will show you a method that addresses these problems, but beforehand, it might help to see some diagrams that outline these two different MPMs and their respective connections to PHP process.

The de facto way to use PHP and Apache has been with the embedded PHP processor via mod_php:

Apache Prefork MPM with mod_php embedded
Apache with the Prefork MPM and embedded mod_php
Click to enlarge

This new method involves proxying the PHP processing to PHP-FPM:

Apache Event MPM to PHP-FPM via mod_proxy_fcgi
Apache with the Event MPM offloading to PHP-FPM via mod_proxy_fcgi
Click to enlarge

3. The Setup:

On the majority of my servers, I use Gentoo Linux, and as such, the exact file locations or methods for each task may be different on your server(s). Despite the distro-specific differences, though, the overall configuration should be mostly the same.

3a. OS and Linux kernel:

Before jumping into the package rebuilds needed to swap Apache to the Event MPM and to use PHP-FPM, a few OS/kernel-related items should be confirmed or modified: 1) epoll support, 2) the maximum number of open file descriptors, and 3) the number of backlogged sockets.

Epoll support
The Apache Event MPM and PHP-FPM can use several different event mechanisms, but on a Linux system, it is advisable to use epoll. Even though epoll has been the default since the 2.6 branch of the Linux kernel, you should still confirm that it is available on your system. That can be done with two commands (one for kernel support, and one for glibc support):

Kernel support for epoll (replace /usr/src/linux/.config with the actual location of your kernel config):
# grep -i epoll /usr/src/linux/.config
CONFIG_EPOLL=y

glibc support for epoll
# nm -D /lib/libc.so.6 | grep -i epoll
00000000000e7dc0 T epoll_create
00000000000e7df0 T epoll_create1
00000000000e7e20 T epoll_ctl
00000000000e7a40 T epoll_pwait
00000000000e7e50 T epoll_wait

Max open file descriptors
Another aspect you will need to tune based on your specific needs is the maximum number of open file descriptors. Basically, since there will be files opened for the Apache connections, the UNIX socket to PHP-FPM, and for the PHP-FPM processes themselves, it is a good idea to have a high maximum number of open files. You can check the current kernel-imposed limit by using the following command:

# cat /proc/sys/fs/file-max
3291373

Though that number is usually very high, you also need to check the limits based on user. For sake of ease, and seeing as servers running many vhosts may frequently add and modify user accounts, I generally set the max file descriptors pretty high for *all* users.

# grep nofile /etc/security/limits.conf
# - nofile - max number of open file descriptors
* soft nofile 16384
* hard nofile 16384

The specifics of modifying the limits.conf file are outside of the scope of this article, but the above changes will set the soft and hard limits to 16,384 open files for any user on the system.

Backlogged socket connections
The last OS configuration that you might want to initially configure is the maximum number of backlogged socket connections. This number is important because we will be proxying Apache to PHP-FPM via UNIX sockets, and this value sets the maximum number of connections per socket. It should be noted that this is the maximum number per socket, so setting this kernel parameter to some outlandish value doesn't really make a lot of sense. As you will see later in this tutorial, I create a socket for each virtual host, so unless you plan on having more than 1024 simultaneous connections to PHP-FPM per site, this value is sufficient. You can change it permanently in /etc/sysctl.conf:

# grep somaxconn /etc/sysctl.conf
net.core.somaxconn = 1024

and remember to run sysctl -p thereafter in order to make the changes active.

3b. Package rebuilds:

Now that you've checked (and possibly modified) some OS-related configurations, a few packages needed to be recompiled in order to change from the Apache Prefork MPM to the Event MPM and to use PHP-FPM. Here's a list of the changes that I needed to make:

Now that those changes have been made, and you've recompiled the appropriate packages (Apache, and PHP), it's time to start configuring the packages before reloading them. In Gentoo, these changes won't take effect until you restart the running instances. For me, that meant that I could compile everything ahead of time, and then make the configuration changes listed in the remaining 3.x subsections before restarting. That procedure allowed for very little downtime!

3c. Apache:

Within the configurations for the main Apache application, there are several changes that need to be made due to switching to a threaded version with the Event MPM. The first three bullet points for httpd.conf listed below relate to the proxy modules, and the fourth bullet point relates to the directives specific to the Event MPM. For more information about the MPM directives, see Apache's documentation.

You will also need to make sure that Apache loads certain modules at runtime. In Gentoo, that configuration is done in /etc/conf.d/apache, and needs to contain at least the two modules listed below (note that your configuration will likely have other modules loaded as well):

3d. Apache PHP module:

The last change that needs to be made for Apache is to the configuration file for how it handles PHP processing (in Gentoo, this configuration is found in /etc/apache2/modules.d/70_mod_php5.conf. I would suggest making a backup of the existing file, and making changes to a copy so that it can be easily reverted, if necessary.

70_mod_php5.conf BEFORE changes
<IfDefine PHP5>
<IfModule !mod_php5.c>
LoadModule php5_module modules/libphp5.so
</IfModule>
<FilesMatch "\.(php|php5|phtml)$">
SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch "\.phps$">
SetHandler application/x-httpd-php-source
</FilesMatch>

DirectoryIndex index.php index.phtml
</IfDefine>

70_mod_php5.conf AFTER changes
<IfDefine PHP5>
## Define FilesMatch in each vhost
<IfModule mod_mime.c>
AddHandler application/x-httpd-php .php .php5 .phtml
AddHandler application/x-httpd-php-source .phps
</IfModule>

DirectoryIndex index.php index.phtml
</IfDefine>

Basically, you're getting rid of the LoadModule reference for mod_php, and all the FileMatch directives. Now, if you're not using virtual hosts, you can put the FilesMatch and Proxy directives from the vhosts section (3e) below in your main PHP module configuration. That being said, the point of this article is to set up everything for use with vhosts, and as such, the module should look like the example immediately above.

As mentioned in section 3b above, the reason you still need the 70_mod_php5.conf at all is so that you can definite the MIME types for PHP and the DirectoryIndex. If you would rather not have this file present, (since you're not really using mod_php), you can just place these directives (the AddHandler and DirectoryIndex ones) in your main Apache configuration-your choice.

3e. vhosts:

For each vhost, I would recommend creating a user and corresponding group. How you manage local users and groups on your server is beyond the scope of this document. For this example, though, we're going to have site1.com and site2.com (very creative, I know 😎 ). To keep it simple, the corresponding users will be 'site1′ and 'site2′, respectively. For each site, you will have a separate vhost configuration (and again, how you manage your vhosts is beyond the scope of this document). Within each vhost, the big change that you need to make is to add the FilesMatch and Proxy directives. Here's the generic template for the vhost configuration additions:

Generic template for vhost configs
<FilesMatch "\.php$">
SetHandler "proxy:unix:///var/run/php-fpm/$pool.sock|fcgi://$pool/"
</FilesMatch>

<Proxy fcgi://$pool/ enablereuse=on max=10>
</Proxy>

In that template, you will replace $pool with the name of each PHP-FPM pool, which will be explained in the next subsection (3f). In my opinion, it is easiest to keep the name of the pool the same as the user assigned to each site, but you're free to change those naming conventions so that they make sense to you. Based on my site1.com / site2.com example, the vhost configuration additions would be:

site1.com vhost config
<FilesMatch "\.php$">
SetHandler "proxy:unix:///var/run/php-fpm/site1.sock|fcgi://site1/"
</FilesMatch>

<Proxy fcgi://site1/ enablereuse=on max=10>
</Proxy>

site2.com vhost config
<FilesMatch "\.php$">
SetHandler "proxy:unix:///var/run/php-fpm/site2.sock|fcgi://site2/"
</FilesMatch>

<Proxy fcgi://site2/ enablereuse=on max=10>
</Proxy>

3f. PHP-FPM:

Since you re-compiled PHP with FPM support (in step 3b), you need to actually start the new process with /etc/init.d/php-fpm start and add it to the default runlevel so that it starts automatically when the server boots (note that this command varies based on your init system):

rc-update add php-fpm default (Gentoo with OpenRC)

Now it's time to actually configure PHP-FPM via the php-fpm.conf file, which, in Gentoo, is located at /etc/php/fpm-php$VERSION/php-fpm.conf. There are two sections to this configuration file: 1) global directives, which apply to all PHP-FPM instances, and 2) pool directives, which can be changed for each pool (e.g. each vhost). For various non-mandatory options, please see the full list of PHP-FPM directives.

At the top of php-fpm.conf are the global directives, which are:
[global]
error_log = /var/log/php-fpm.log
events.mechanism = epoll
emergency_restart_threshold = 0

These directives should be fairly self-explanatory, but here's a quick summary:

Thereafter are the pool directives, for which I follow this template, with one pool for each vhost:
;; $SITE.$TLD
[$USER]
listen = /var/run/php-fpm/$USER.sock
listen.owner = $USER
listen.group = apache
listen.mode = 0660
user = $USER
group = apache
pm = dynamic
pm.start_servers = 3
pm.max_children = 100
pm.min_spare_servers = 2
pm.max_spare_servers = 5
pm.max_requests = 10000
request_terminate_timeout = 300

It may look a bit intimidating at first, but only the $SITE, $TLD, and $USER variables change based on your vhosts and corresponding users. When the template is used for the site1.com / site2.com example, the pools look like:

;; site1.com
[site1]
listen = /var/run/php-fpm/site1.sock
listen.owner = site1
listen.group = apache
listen.mode = 0660
user = site1
group = apache
pm = dynamic
pm.start_servers = 3
pm.max_children = 100
pm.min_spare_servers = 2
pm.max_spare_servers = 5
pm.max_requests = 10000
request_terminate_timeout = 300

;; site2.com
[site2]
listen = /var/run/php-fpm/site2.sock
listen.owner = site2
listen.group = apache
listen.mode = 0660
user = site2
group = apache
pm = dynamic
pm.start_servers = 3
pm.max_children = 100
pm.min_spare_servers = 2
pm.max_spare_servers = 5
pm.max_requests = 10000
request_terminate_timeout = 300

Since the UNIX sockets are created in /var/run/php-fpm/, you need to make sure that that directory exists and is owned by root. The listen.owner and listen.group directives will make the actual UNIX socket for the pool owned by $USER:apache, and will make sure that the apache group can write to it (which is necessary).

The following directives are ones that you will want to change based on site traffic and server resources:

These directives should all be adjusted based on the needs of each site. For instance, a really busy site with many, many, simultaneous PHP connections may need 3 servers each with 100 children, whilst a low-traffic site may only only need 1 server with 10 children. The thing to remember is that those children are only active whilst actually processing PHP, which could be a very short amount of time (possibly measured in milliseconds). Contrapositively, setting the numbers too high for the server to handle (in terms of available memory and processor) will result in poor performance when sites are under load. As with anything, tuning the pool requirements will take time and analysis to get it right.

Remember to reload (or restart) PHP-FPM after making any changes to its configuration (global or pool-based):

/etc/init.d/php-fpm reload

4. Verification:

After you have configured Apache, your vhosts, PHP, PHP-FPM, and the other components mentioned throughout this article, you will need to restart them all (just for sake of cleanness). To verify that things are as they should be, you can simply browse to one of the sites configured in your vhosts and make sure that it functions as intended. You can also verify some of the individual components with the following commands:

Apache supports threads, and is using the Event MPM
# apache2 -V | grep 'Server MPM\|threaded'
Server MPM: event
threaded: yes (fixed thread count)

Apache / vhost syntax
# apache2ctl configtest
* Checking apache2 configuration ... [ ok ]

5. Troubleshooting:

So you followed the directions here, and it didn't go flawlessly?! Hark! 😮 In all seriousness, there are several potential points of failure for setting up Apache to communicate with PHP-FPM via mod_proxy_fcgi, but fortunately, there are some methods for troubleshooting (one of the beauties of Linux and other UNIX derivatives is logging).

By default, mod_proxy and mod_proxy_fcgi will report errors to the log that you have specified via the ErrorLog directive, either in your overall Apache configuration, or within each vhost. Those logs serve as a good starting point for tracking down problems. For instance, before I had appropriately set the UNIX socket permission options in php-fpm.conf, I noticed these errors:

[Wed Jul 08 00:03:26.717538 2015] [proxy:error] [pid 2582:tid 140159555143424] (13)Permission denied: AH02454: FCGI: attempt to connect to Unix domain socket /var/run/php-fpm/site1.sock (*) failed
[Wed Jul 08 00:03:26.717548 2015] [proxy_fcgi:error] [pid 2582:tid 140159555143424] [client 52350] AH01079: failed to make connection to backend: httpd-UDS

That error in bold text indicated that Apache didn't have permission to write to the UNIX socket. Setting the permissions accordingly-see the listen.owner and listen.group directives portion of subsection 3f-fixed the problem.

Say that the errors don't provide detailed enough information, though. You can set the LogLevel directive (again, either in your overall Apache configuration, or per vhost) to substantially increase the verbosity-to debug or even trace levels. You can even change the log level just for certain modules, so that you don't get bogged down with information that is irrelevant to your particular error. For instance:

LogLevel warn proxy:debug proxy_fcgi:debug

will set the overall LogLevel to "warn," but the LogLevel for mod_proxy and mod_proxy_fcgi to "debug," in order to make them more verbose.

6. Conclusion:

If you've stuck with me throughout this gigantic post, you're either really interested in systems engineering and the web stack, or you may have some borderline masochistic tendencies. 😛 I hope that you have found the article helpful in setting up PHP to proxy PHP interpretation to PHP-FPM via mod_proxy_fcgi, and that you're able to see the benefits of this type of offloading. Not only does this method of processing PHP files free up resources so that Apache can serve more simultaneous connections with fewer resources, but it also more closely adheres to the UNIX philosophy of having applications perform only one task, and do so in the best way possible.

If you have any questions, comments, concerns, or suggestions, please feel free to leave a comment.

Cheers,
Zach

07 Aug 2015 1:28am GMT

04 Aug 2015

feedPlanet Gentoo

Anthony Basile: alt-libc: The state of uClibc and musl in Gentoo (part 1)

About five years ago, I became interested in alternative C standard libraries like uClibc and musl and their application to more than just embedded systems. Diving into the implementation details of those old familiar C functions can be tedious, but also challenging especially under constraints of size, performance, resource consumption, features and correctness. Yet these details can make a big difference as you can see from this comparison of glibc, uClibc, dietlibc and musl. I first encountered libc performance issues when I was doing number crunching work for my ph.d. in physics at Cornell in the late 80's. I was using IBM RS6000's (yes!) and had to jump into the assembly. It was lots of fun and I've loved this sort of low level stuff ever since.

Over the past four years, I've been working towards producing stage3 tarballs for both uClibc and musl systems on various arches, and with help from generous contributors (thanks guys!), we now have a pretty good selection on the mirrors. These stages are not strictly speaking embedded in that they do not make use of busybox to provide their base system. Rather, they employ the same packages as our glibc stages and use coreutils, util-linux, net-tools and friends. Except for small details here and there, they only differ from our regular stages in the libc they use.

If you read my last blog posting on this new release engineering tool I'm developing called GRS, you'll know that I recently hit a milestone in this work. I just released three hardened, fully featured XFCE4 desktop systems for amd64. These systems are identical to each other except for their libc, again modulo a few details here and there. I affectionately dubbed these Bluemoon for glibc, Lilblue for uClibc, and Bluedragon for musl. (If you're curious about the names, read their homepages.) You can grab all three off my dev space , or off the mirrors under experimental/amd64/{musl,uclibc} if you're looking for just Lilblue or Bluedragon - the glibc system is too boring to merit mirror space. I've been maintaining Lilblue for a couple of years now, but with GRS, I can easily maintain all three and its nice to have them for comparison.

If you play with these systems, don't expect to be blown away by some amazing differences. They are there and they are noticeable, but they are also subtle. For example, you're not going to notice code correctness in, say, pthread_cancel() unless you're coding some application and expect certain behavior but don't get it because of some bad code in your libc. Rather, the idea here is push the limits of uClibc and musl to see what breaks and then fix it, at least on amd64 for now. Each system includes about 875 packages in the release tarballs, and an extra 5000 or so binpkgs built using GRS. This leads to lots of breakage which I can isolate and address. Often the problem is in the package itself, but occasionally it's the libc and that's where the fun begins! I've asked Patrick Lauer for some space where I can set up my GRS builds and serve out the binpkgs. Hopefully he'll be able to set me up with something. I'll also be able to make the build.log's available for packages that fail via the web, so that GRS will double as a poor man's tinderbox.

In a future article I'll discuss musl, but in the remainder of this post, I want to highlight some big ticket items we've hit in uClibc. I've spent a lot of time building up machinery to maintain the stages and desktops, so now I want to focus my attention on fixing the libc problems. The following laundry list is as much a TODO for me as it is for your entertainment. I won't blame you if you want to skip it. The selection comes from Gentoo's bugzilla and I have yet to compare it to upstream's bugs since I'm sure there's some overlap.

Currently there are thirteen uClibc stage3's being supported:

Here hardened and vanilla refer to the toolchain hardening as we do for regular glibc systems. Some bugs are arch specific some are common. Let's look at each in turn.

* amd64 and i686 are the only stages considered stable and are distributed along side our regular stage3 releases in the mirrors. However, back in May I hit a rather serious bug (#548950) in amd64 which is our only 64-bit arch. The problem was in the the implementation of pread64() and pwrite64() and was triggered by a change in the fsck code with e2fsprogs-1.42.12. The bug led to data corruption of ext3/ext4 filesystem which is a very serious issue for a release advertised as stable. The problem was that the wrong _syscall wrapper was being used for amd64. If we don't require 64-bit alignment, and you don't on a 64-bit arch (see uClibc/libc/sysdeps/linux/x86_64/bits/uClibc_arch_features.h), then you need to use _syscall4, not _syscall6.

The issue was actually fixed by Mike Frysinger (vapier) in uClibc's git HEAD but not in the 0.9.33 branch which is the basis of the stages. Unfortunately, there hasn't been a new release of uClibc in over three year so backporting meant disentangling the fix from some new thread cancel stuff and was annoying.

* The armv7a stages are close to being stable, but they are still being distributed in the mirrors under experimental. The problem here is not uClibc specific, but due to hardened gcc-4.8 and it affects all our hardened arm stages. With gcc-4.8, we're turning on -fstack-check=yes by default in the specs and this breaks alloca(). The workaround for now is to use gcc-4.7, but we should turn off -fstack-check for arm until bug #518598 - (PR65958) is fixed.

* I hit this one weird bug when building the mips stages, bug #544756. An illegal instruction is encountered when building any version of python using gcc-4.9 with -O1 optimization or above, yet it succeeds with -O0. What I suspect happened here is some change in the optimization code for mips between gcc-4.8 and 4.9 introduced the problem. I need to distill out some reduced code before I submit to gcc upstream. For now I've p.masked >=gcc-4.9 in default/linux/uclibc/mips. Since mips lacks stable keywords, this actually brings the mips stages in better line with the other stages that use gcc-4.8 (or 4.7 in the case of hardened arm).

* Unfortunately, ppc is plagued with bug #517160. PIE code is broken on ppc and causes a seg fault in plt_pic32.__uClibc_main (). Since PIE is an integral part of how we do hardening in Gentoo, there' s no hardened ppc-uclibc stage. Luckily, there are no known issues with the vanilla ppc stage3.

Finally, there are five interesting bugs which are common to all arches. These problems lie in the implementation of functions in uClibc and deviate from the expected behavior. I'm particularly grateful to René Rhéaume who found them by running the test suite for various packages.

* Bug 527954 - One of wget's tests makes use of fnmatch(3) which intelligently matches file names or paths. On uClibc, there is an unexpected failure in a test where it should return a non-match when fnmatch's options contains FNM_PATHNAME and a matching slash is not present in both strings. Apparently this is a duplicate of a really old bug (#181275).

* Bug 544118 - René noticed this problem in an e2fsprogs test for libss.so. The failure here is due to the fact that setbuf(3) is ineffective at changing the buffer size of stdout if it is run after some printf(1). Output to stdout is buffered while output to stderr is not. This particular test tries to preserve the order of output from a sequence of writes to stdout and stderr by setting the buffer size of stdout to zero. But setbuf() only works on uClibc if it is invoked before any output to stdout. As soon as there is even one printf(), all invocations to setbuf(stdout, …) are ineffecitve!

* Bug 543972 - This one came up in the gzip test suite. One of the tests there checks to make sure that gzip properly fails if it runs out of disk space. It uses /dev/full, which is a pseudo-device provided by the kernel that pretends to always be full. The expectation is that fclose() should set errno = ENOSPC when closing /dev/full. It does on glibc but it doesn't in uClibc. It actually happens when piping stdout to /dev/full, so the problem may even be in dup(2).

* Bug 543668 - There is some problem in uClibc's dopen()/dlclose() code. I wrote about this in a previous blog post and David Flogeras also hit it with syslog-ng's plugin system. A seg fault occurs when unmapping a plugin from the memory space of a process during do_close() in uClibc/ldso/libdl/libdl.c. My guess is that the problem lies in uClibc's accounting of the mappings and when it tries to unmap an area of memory which is not mapped or was previously unmapped, it seg faults.

04 Aug 2015 12:58pm GMT

31 Jul 2015

feedPlanet Gentoo

Anthony Basile: The “Gentoo Reference System” suite: a new Release Engineering tool.

The other day I installed Ubuntu 15.04 on one of my boxes. I just needed something where I could throw in a DVD, hit install and be done. I didn't care about customization or choice, I just needed a working Linux system from which I could chroot work. Thousands of people around the world install Ubuntu this way and when they're done, they have a stock system like any other Ubuntu installation, all identical like frames in a Andy Warhol lithograph. Replication as a form of art.

In contrast, when I install a Gentoo system, I enjoy the anxiety of choice. Should I use syslog-ng, metalog, or skip a system logger altogether? If I choose syslog-ng, then I have a choice of 14 USE flags for 2^14 possible configurations for just that package. And that's just one of some 850+ packages that are going to make up my desktop. In contrast to Ubuntu where every installation is identical (whatever "idem" means in this context), the shear space of possibilities make no two Gentoo systems the same unless there is some concerted effort to make them so. In fact, Gentoo doesn't even have a notion of a "stock" system unless you count the stage3s which are really bare bones. There is no "stock" Gentoo desktop.

With the work I am doing with uClibc and musl, I needed a release tool that would build identical desktops repeatedly and predictably where all the choices of packages and USE flags were layed out a priori in some specifications. I considered catalyst stage4, but catalyst didn't provide the flexibility I wanted. I initially wrote some bash scripts to build an XFCE4 desktop from uClibc stage3 tarballs (what I dubbed "Lilblue Linux"), but this was very much ad hoc code and I needed something that could be generalized so I could do the same for a musl-based desktop, or indeed any Gentoo system I could dream up.

This led me to formulate the notion of what I call a "Gentoo Reference System" or GRS for short - maybe we could make stock Gentoo systems available. The idea here is that one should be able to define some specs for a particular Gentoo system that will unambiguously define all the choices that go into building that system. Then all instances built according to those particular GRS specs would be identical in much the same way that all Ubuntu systems are the same. In a Warholian turn, the artistic choices in designing the system would be pushed back into the specs and become part of the automation. You draw one frame of the lithograph and you magically have a million.

The idea of these systems being "references" was also important for my work because, with uClibc or musl, there's a lot of package breakages - remember you pushing up against actual implementations of C functions and nearly everything in your systems written in C. So, in the space of all possible Gentoo systems, I needed some reference points that worked. I needed that magical combinations of flags and packages that would build and yield useful systems. It was also important that these references be easily kept working over time since Gentoo systems evolve as the main tree, or overlays, are modified. Since on some successive build something might break, I needed to quickly identify the delta and address it. The metaphor that came up in my head from my physics background is that of phase space. In the swirling mass of evolving dynamical systems, I pictured these "Gentoo Reference Systems" as markers etching out a well defined path over time.

Enough with the metaphors, how does GRS work? There are two main utilities, grsrun and grsup. The first is run on a build machine and generates the GRS release as well as any extra packages and updates. These are delivered as binpkgs. In contrast, grsup is run on an installed GRS instance and its used for package management. Since we're working in a world of identical systems, grsup prefers working with binpkgs that are downloaded from some build machine, but it can revert to building locally as well.

The GRS specs for some system are found on a branch of a git repository. Currently the repo at https://gitweb.gentoo.org/proj/grs.git/ has four branches, each for one of the four GRS specs housed there. grsrun is then directed to sync the remote repo locally, check out the branch of the GRS system we want to build and begin reading a script file called build which directs grsrun on what steps to take. The scripting language is very simple and contains only a handful of different directives. After a stage tarball is unpacked, build can direct grsrun to do any of the following:

mount and umount - Do a bind mount of /dev/, /dev/pts/ and other directories that are required to get a chroot ready.

populate - Selectively copy files from the local repo to the chroot. Any files can be copied in, so, for example, you can prepare a pristine home directory for some user with a pre-configured desktop. Or you can add customized configuration files to /etc for services you plan to run.

runscript - This will run some bash or python script in the chroots. The scripts are copied from the local repo to /tmp of the chroot and executed there. These scripts can be like the ones that catalyst runs during stage1/2/3 but can also be scripts to add users and groups, to add services to runlevels, etc. Think of anything you would do when growing a stage3 into the system you want, script it up and GRS will automated it for you.

kernel - This looks for a kernel config file in the local repo, parses it for the version, builds it and both bundles it as a packages called linux-image-<version>.tar.xz for later distribution as well as installs it into the chroot. grsup knows how to work with these linux-image-<version>.tar.xz files and can treat them like binpkgs.

tarit and hashit - These directives create a release tarball of the entire chroot and generate the digests.

pivot - If you built a chroot within a chroot, like catalyst does during stage1, then this pivots the inner chroot out so that further building can make use of it.

From an implementation point of view, the GRS suite is written in python and each of the above directives is backed by a simple python class. Its easy, for instance, to implement more directives this way. E.g. if you want to build a bootable CD image, you can include a directive called isoit, write a python class for what's required to construct the iso image and glue this new class into the grs module.

If you're familiar with catalyst, at this point you might be wondering what's the difference? Can't you do all of this with catalyst? There is a lot of overlap, but the emphasis is different. For example, I wanted to be able to drop in a pre-configured desktop for a user. How would I do that with catalyst? I guess I could create an overlay with packages for some pre-built home directory but that's a perversion of what ebuilds are for - we should never be installing into /home. Rather with grsrun I can just populate the chroot with whatever files I like anywhere in the filesystem. More importantly, I want to be able control what USE flags are set and, in general, manage all of /etc/portage/. catalyst does provide portage_configdir which populates /etc/portage when building stages, but its pretty static. Instead, grsup and two other utilities, install-worldconf and clean-worldconf, can dynamically manage files under /etc/portage/ according to a configuration file called world.conf.

Lapsing back into metaphor, I see catalyst as rigid and frozen whereas grsrun is loose and fluid. You can use grsrun to build stage1/2/3 tarballs which are identical to those built with catalyst, and in fact I've done so for hardened amd64 mutlilib stages so I could compare. But with grsrun you have too much freedom in writing the scripts and file that go into the GRS specs and chances are you'll get something wrong, whereas with catalyst the build is pretty regimented and you're guaranteed to get uniformity across arches and profiles. So while you can do the same things with each tool, its not recommended that you use grsrun to do catalyst stage builds - there's too much freedom. Whereas when building desktops or servers you might welcome that freedom.

Finally, let me close with how grsup works. As mentioned above, the GRS specs for some system include a file called world.conf. Its in configparser format and it specifies files and their contents in the /etc/portage/ directory. An example section in the file looks like:

[app-crypt/gpgme:1]
package.use : app-crypt/gpgme:1 -common-lisp static-libs
package.env : app-crypt/gpgme:1 app-crypt_gpgme_1
env : LDFLAGS=-largp

This says, for package app-crypt/gpgme:1, drop a file called app-crypt_gpgme_1 in /etc/portage/package.use/ that contains the line "app-crypt/gpgme:1 -common-lisp static-libs", drop another file by the same name in /etc/portage/package.env/ with line "app-crypt/gpgme:1 app-crypt_gpgme_1″, and finally drop a third file by the same name in /etc/portage/env/ with line "LDFLAGS=-largp". grsup is basically a wrapper to emerge which first populates /etc/portage/ according to the world.conf file, then emerges the requested pkg(s) preferring the use of binpkgs over building locally as stated above, and finally does a clean up on /etc/portage/. install-worldconf and clean-worldconf isolate the populate and clean up steps so they can be used in scripts run by grsrun when building the release. To be clear, you don't have to use grsup to maintain a GRS system. You can maintain it just like any other Gentoo system, but if you manage your own /etc/portage/, then you are no longer tracking the GRS specs. grsup is meant to make sure you update, install or remove packages in a manner that keeps the local installation in compliance with the GRS specs for that system.

All this is pretty alpha stuff, so I'd appreciate comments on design and implementation before things begin to solidify. I am using GRS to build three desktop systems which I'll blog about next. I've dubbed these systems Lilblue which is a hardened amd64 XFCE4 desktop with uClibc as its standard libc, Bluedragon that uses musl, and finally Bluemoon which uses good old glibc. (Lilblue is actually a few years old, but the latest release is the first built using GRS.) All three desktops are identical with respect to the choice of packages and USE flags, and differ only in their libc's so one can compare the three. Lilbue and Bluedragon are on the mirrors, or you can get all three from my dev space at http://dev.gentoo.org/~blueness/theblues/. I didn't push out bluemoon on the mirrors because a glibc based desktop is nothing special. But since building with GRS is as simple as cloning a git branch and tweaking, and since the comparison is useful, why not?

The GRS home page is at https://wiki.gentoo.org/wiki/Project:RelEng_GRS.

31 Jul 2015 2:21pm GMT

23 Jul 2015

feedPlanet Gentoo

Johannes Huber: Tasty calamares in Gentoo

First of all it's nothing to eat. So what is it then? This is the introduction by upstream:

Calamares is an installer framework. By design it is very customizable, in order to satisfy a wide variety of needs and use cases. Calamares aims to be easy, usable, beautiful, pragmatic, inclusive and distribution-agnostic. Calamares includes an advanced partitioning feature, with support for both manual and automated partitioning operations. It is the first installer with an automated "Replace Partition" option, which makes it easy to reuse a partition over and over for distribution testing. Got a Linux distribution but no system installer? Grab Calamares, mix and match any number of Calamares modules (or write your own in Python or C++), throw together some branding, package it up and you are ready to ship!

I have just added newest release version (1.1.2) to the tree and in my dev overlay a live version (9999). The underlaying technology stack is mainly Qt5, KDE Frameworks, Python3, YAML and systemd. It's picked up and of course in evaluation process by several Linux distributions.

You may asking why i have added it to Gentoo then where we have OpenRC as default init system?! You are right at the moment it is not very useful for Gentoo. But for example Sabayon as a downstream of us will (maybe) use it for the next releases, so in the first place it is just a service for our downstreams.

The second reason, there is a discussion on gentoo-dev mailing list at the moment to reboot the Gentoo installer. Instead of creating yet another installer implementation, we have two potential ways to pick it up, which are not mutual exclusive:

1. Write modules to make it work with sysvinit aka OpenRC
2. Solve Bug #482702 - Provide alternative stage3 tarballs using sys-apps/systemd

Have fun!

[1] https://calamares.io/about/
[2] johu dev overlay
[3] gentoo-dev ml - Rebooting the Installer Project
[4] Bug #482702 - Provide alternative stage3 tarballs using sys-apps/systemd

23 Jul 2015 8:36pm GMT

22 Jul 2015

feedPlanet Gentoo

Rafael Goncalves Martins: Hello, World!

Hi all,

I'm starting a new blog!

Actually, it is almost the same blog, but powered by a new "blogging engine", and I don't want to spend time migrating the old posts, that are mostly outdated right now.

The old content is archived here, if you need it due to some crazy reason: http://old.rafaelmartins.eng.br/.

For Gentoo planet readers, everything should be working just fine. I created rewrite rules to keep the old atom feeds working.

I'll publish another blog post soon, talking about the "blogging engine" and my next plans.

Thanks.

22 Jul 2015 6:45am GMT

20 Jul 2015

feedPlanet Gentoo

Andreas K. Hüttel: VMware Workstation 11 and kwin - hangs and hiccups (solved?)

Since updating to VMware Workstation 11 (from the Gentoo vmware overlay), I've experienced a lot of hangs of my KDE environment whenever a virtual machine was running. Basically my system became unusable, which is bad if your workflow depends on accessing both Linux and (gasp!) Windows 7 (as guest). I first suspected a dbus timeout (doing the "stopwatch test" for 25s waits), but it seems according to some reports that this might be caused by buggy behavior in kwin (4.11.21). Sadly I haven't been able to pinpoint a specific bug report.

Now, I'm not sure if the problem is really 100% fixed, but at least now the lags are much smaller- and here's how to do it (kudos to matthewls and vrenn):

I'll leave it as an exercise to the reader to figure out what these settings do. (Feel free to explain it in a comment. :) No guarantees of any kind. If this kills kittens you have been warned. Cheers.

20 Jul 2015 8:28pm GMT

18 Jul 2015

feedPlanet Gentoo

Richard Freeman: Running cron jobs as units automatically

I just added sys-process/systemd-cron to the Gentoo repository. Until now I've been running it from my overlay and getting it into the tree was overdue. I've found it to be an incredibly useful tool.

All it does is install a set of unit files and a crontab generator. The unit files (best used by starting/enabling cron.target) will run jobs from /etc/cron.* at the appropriate times. The generator can parse /etc/crontab and create timer units for every line dynamically.

Note that the default Gentoo install runs the /etc/cron.* jobs from /etc/crontab, so if you aren't careful you might end up running them twice. The simplest solutions this are to either remove those lines from /etc/crontab, or install systemd-cron using USE=etc-crontab-systemd which will have the generator ignore /etc/crontab and instead look for /etc/crontab-systemd where you can install jobs you'd like to run using systemd.

The generator works like you'd expect it to - if you edit the crontab file the units will automatically be created/destroyed dynamically.

One warning about timer units compared to cron jobs is that the jobs are run as services, which means that when the main process dies all its children will be killed. If you have anything in /etc/cron.* which forks you'll need to have the main script wait at the end.

On the topic of race conditions, each cron.* directory and each /etc/crontab line will create a separate unit. Those units will all run in parallel (to the extent that one is still running when the next starts), but within a cron.* directory the scripts will run in series. That may be a bit different from some cron implementations which may limit the number of simultaneous jobs globally.

All the usual timer unit logic applies. stdout goes to the journal, systemctl list-timers shows what is scheduled, etc.


Filed under: gentoo, linux, systemd

18 Jul 2015 4:00pm GMT

16 Jul 2015

feedPlanet Gentoo

Agostino Sarubbo: libav: divide-by-zero in ff_h263_decode_mba()

Description:
Libav is an open source set of tools for audio and video processing.

After talking with Luca Barbato which is both a Gentoo and Libav developer, I spent a bit of my time fuzzing libav and in particular I fuzzed libavcodec though avplay.
I hit a crash and after I reported it to upstream, they confirmed the issue as a divide-by-zero.

The complete gdb output:

ago@willoughby $ gdb --args /usr/bin/avplay avplay.crash 
GNU gdb (Gentoo 7.7.1 p1) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/avplay...Reading symbols from /usr/lib64/debug//usr/bin/avplay.debug...done.
done.
(gdb) run
Starting program: /usr/bin/avplay avplay.crash
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
avplay version 11.3, Copyright (c) 2003-2014 the Libav developers
  built on Jun 19 2015 09:50:59 with gcc 4.8.4 (Gentoo 4.8.4 p1.6, pie-0.6.1)
[New Thread 0x7fffec4c7700 (LWP 7016)]
[New Thread 0x7fffeb166700 (LWP 7017)]
INFO: AddressSanitizer ignores mlock/mlockall/munlock/munlockall
[New Thread 0x7fffe9e28700 (LWP 7018)]
[h263 @ 0x60480000f680] Format detected only with low score of 25, misdetection possible!
[h263 @ 0x60440001f980] Syntax-based Arithmetic Coding (SAC) not supported
[h263 @ 0x60440001f980] Reference Picture Selection not supported
[h263 @ 0x60440001f980] Independent Segment Decoding not supported
[h263 @ 0x60440001f980] header damaged

Program received signal SIGFPE, Arithmetic exception.
[Switching to Thread 0x7fffe9e28700 (LWP 7018)]
0x00007ffff21e3313 in ff_h263_decode_mba (s=s@entry=0x60720005a100) at /tmp/portage/media-video/libav-11.3/work/libav-11.3/libavcodec/ituh263dec.c:142
142     /tmp/portage/media-video/libav-11.3/work/libav-11.3/libavcodec/ituh263dec.c: No such file or directory.
(gdb) bt
#0  0x00007ffff21e3313 in ff_h263_decode_mba (s=s@entry=0x60720005a100) at /tmp/portage/media-video/libav-11.3/work/libav-11.3/libavcodec/ituh263dec.c:142
#1  0x00007ffff21f3c2d in ff_h263_decode_picture_header (s=0x60720005a100) at /tmp/portage/media-video/libav-11.3/work/libav-11.3/libavcodec/ituh263dec.c:1112
#2  0x00007ffff1ae16ed in ff_h263_decode_frame (avctx=0x60440001f980, data=0x60380002f480, got_frame=0x7fffe9e272f0, avpkt=) at /tmp/portage/media-video/libav-11.3/work/libav-11.3/libavcodec/h263dec.c:444
#3  0x00007ffff2cd963e in avcodec_decode_video2 (avctx=0x60440001f980, picture=0x60380002f480, got_picture_ptr=got_picture_ptr@entry=0x7fffe9e272f0, avpkt=avpkt@entry=0x7fffe9e273b0) at /tmp/portage/media-video/libav-11.3/work/libav-11.3/libavcodec/utils.c:1600
#4  0x00007ffff44d4fb4 in try_decode_frame (st=st@entry=0x60340002fb00, avpkt=avpkt@entry=0x601c00037b00, options=) at /tmp/portage/media-video/libav-11.3/work/libav-11.3/libavformat/utils.c:1910
#5  0x00007ffff44ebd89 in avformat_find_stream_info (ic=0x60480000f680, options=0x600a00009e80) at /tmp/portage/media-video/libav-11.3/work/libav-11.3/libavformat/utils.c:2276
#6  0x0000000000431834 in decode_thread (arg=0x7ffff7e0b800) at /tmp/portage/media-video/libav-11.3/work/libav-11.3/avplay.c:2268
#7  0x00007ffff0284b08 in ?? () from /usr/lib64/libSDL-1.2.so.0
#8  0x00007ffff02b4be9 in ?? () from /usr/lib64/libSDL-1.2.so.0
#9  0x00007ffff4e65aa8 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.4/libasan.so.0
#10 0x00007ffff0062204 in start_thread () from /lib64/libpthread.so.0
#11 0x00007fffefda957d in clone () from /lib64/libc.so.6
(gdb)

Affected version:
11.3 (and maybe past versions)

Fixed version:
11.5 and 12.0

Commit fix:
https://git.libav.org/?p=libav.git;a=commitdiff;h=0a49a62f998747cfa564d98d36a459fe70d3299b;hp=6f4cd33efb5a9ec75db1677d5f7846c60337129f

Credit:
This bug was discovered by Agostino Sarubbo of Gentoo.

CVE:
CVE-2015-5479

Timeline:
2015-06-21: bug discovered
2015-06-22: bug reported privately to upstream
2015-06-30: upstream commit the fix
2015-07-14: CVE assigned
2015-07-16: advisory release

Note:
This bug was found with American Fuzzy Lop.
This bug does not affect ffmpeg.

Permalink:
http://blogs.gentoo.org/ago/2015/07/16/libav-divide-by-zero-in-ff_h263_decode_mba

16 Jul 2015 9:57am GMT