03 Jul 2015

feedPlanet Gentoo

Andreas K. Hüttel: KDEPIM without Akonadi

As you know, Gentoo is all about flexibility. You can run bleeding edge code (portage, our package manager, even provides you with installation from git master KF5 and friends) or you can focus on stability and trusted code. This is why we've been offering our users for the last years KDEPIM 4.4.11.1 (the version where KMail e-mail storage was not integrated with Akonadi yet, also known as KMail1) as a drop-in replacement for the newer versions.
Recently the Nepomuk search framework has been replaced by Baloo, and after some discussion we decided that for the Nepomuk-related packages it's now time to go. Problem is, the old KDEPIM packages still depend on it via their Akonadi version. This is why - for those of our users who prefer to run KDEPIM 4.4 / KMail1 - we've decided to switch to Pali Rohár's kdepim-noakonadi fork (see also his 2013 blog post and the code).The packages are right now in the KDE overlay, but will move to the main tree after a few days of testing and be treated as an update of KDEPIM 4.4.11.1.
The fork is essentially KDEPIM 4.4 including some additional bugfixes from the KDE/4.4 git branch, with KAddressbook patched back to KDEPIM 4.3 state and references to Akonadi removed elsewhere. This is in some ways a functionality regression since the integration of e.g. different calendar types is lost, however in that version it never really worked perfectly anyway.

For now, you will still need the akonadi-server package, since kdepimlibs (outside kdepim and now at version 4.14.9) requires it to build, but you'll never need to start the Akonadi server. As a consequence, Nepomuk support can be disabled everywhere, and the Nepomuk core and client and Akonadi client packages can be removed by the package manager (--depclean, make sure to first globally disable the nepomuk useflag and rebuild accordingly).

You might ask "Why are you still doing this?"... well. I've been told Akonadi and Baloo is working very nicely, and again I've considered upgrading all my installations... but then on my work desktop where I am using newest and greatest KDE4PIM bug 338658 pops up regularly and stops syncing of important folders. I just don't have the time to pointlessly dig deep into the Akonadi database every few days. So KMail1 it is, and I'll rather spend some time occasionally picking and backporting bugfixes.

03 Jul 2015 2:16pm GMT

26 Jun 2015

feedPlanet Gentoo

Mike Pagano: kdbus in gentoo-sources

Keeping with the theme of 'Gentoo is about choice" I've added the ability for users to include kdbus into their gentoo-sources kernel. I wanted an easy way for gentoo users to test the patchset while maintaining the default installation of not having it at all.

In order to include the patchset on your gentoo-sources you'll need the following:

1. A kernel version >= 4.1.0-r1

2. the 'experimental' use flag

3. the 'kdbus' use flag

I am not a systemd user, but from the ebuild it looks like if you build systemd with the 'kdbus' use flag it will use it.

Please send all kdbus bugs upstream by emailing the developers and including linux-kernel@vger.kernel.org in the CC .

Read as much as you can about kdbus before you decided to build it into your kernel. There have been security concerns mentioned (warranted or not), so following the upstream patch review at lkml.org would probably be prudent.

When a new version is released, wait a week before opening a bug. Unless I am on vacation, I will most likely have it included before the week is out. Thanks!

NOTE: This is not some kind of Gentoo endorsement of kdbus. Nor is it a Mike Pagano endorsement of kdbus. This is no different then some of the other optional and experimental patches we carry. I do all the genpatches work which includes the patches, the ebuilds and the bugs therefore since I don't mind the extra work of keeping this up to date, then I can't see any reason not to include it as an option.

26 Jun 2015 11:35pm GMT

25 Jun 2015

feedPlanet Gentoo

Johannes Huber: KDE Plasma 5.3.1 testing

After several month of packaging in kde overlay and almost a month in tree, we have lifted the mask for KDE Plasma 5.3.1 today. If you want to test it out, now some infos how to get it.

For easy transition we provide two new profiles, one for OpenRC and the other for systemd.

root # eselect profile list
...
[8] default/linux/amd64/13.0/desktop/plasma
[9] default/linux/amd64/13.0/desktop/plasma/systemd
...

Following example activates the Plasma systemd profile:

root # eselect profile set 9

On stable systems you need to unmask the qt5 use flag:

root # echo "-qt5" >> /etc/portage/profile/use.stable.mask

Any packages affected by the profile change need to be rebuilt:

root # emerge -avuND @world

For stable users, you also need to keyword the required packages. You can let portage handle it with autokeyword feature or just grep the keyword files for KDE Frameworks 5.11 and KDE Plasma 5.3.1 from kde overlay.

Now just install it (this is full Plasma 5, the basic desktop would be kde-plasma/plasma-desktop):

root # emerge -av kde-plasma/plasma-meta

KDM is not supported for Plasma 5 anymore, so if you have installed it kill it with fire. Possible and tested login managers are SDDM and LightDM.

For detailed instructions read the full upgrade guide. Package bugs can be filed to bugs.gentoo.org and about the software to bugs.kde.org.

Have fun,
the Gentoo KDE Team

25 Jun 2015 11:15pm GMT

23 Jun 2015

feedPlanet Gentoo

Hanno Böck: The tricky security issue with FollowSymLinks and Apache

tl;dr Most servers running a multi-user webhosting setup with Apache HTTPD probably have a security problem. Unless you're using Grsecurity there is no easy fix.

I am part of a small webhosting business that I run as a side project since quite a while. We offer customers user accounts on our servers running Gentoo Linux and webspace with the typical Apache/PHP/MySQL combination. We recently became aware of a security problem regarding Symlinks. I wanted to share this, because I was appalled by the fact that there was no obvious solution.

Apache has an option FollowSymLinks which basically does what it says. If a symlink in a webroot is accessed the webserver will follow it. In a multi-user setup this is a security problem. Here's why: If I know that another user on the same system is running a typical web application - let's say Wordpress - I can create a symlink to his config file (for Wordpress that's wp-config.php). I can't see this file with my own user account. But the webserver can see it, so I can access it with the browser over my own webpage. As I'm usually allowed to disable PHP I'm able to prevent the server from interpreting the file, so I can read the other user's database credentials. The webserver needs to be able to see all files, therefore this works. While PHP and CGI scripts usually run with user's rights (at least if the server is properly configured) the files are still read by the webserver. For this to work I need to guess the path and name of the file I want to read, but that's often trivial. In our case we have default paths in the form /home/[username]/websites/[hostname]/htdocs where webpages are located.

So the obvious solution one might think about is to disable the FollowSymLinks option and forbid users to set it themselves. However symlinks in web applications are pretty common and many will break if you do that. It's not feasible for a common webhosting server.

Apache supports another Option called SymLinksIfOwnerMatch. It's also pretty self-explanatory, it will only follow symlinks if they belong to the same user. That sounds like it solves our problem. However there are two catches: First of all the Apache documentation itself says that "this option should not be considered a security restriction". It is still vulnerable to race conditions.

But even leaving the race condition aside it doesn't really work. Web applications using symlinks will usually try to set FollowSymLinks in their .htaccess file. An example is Drupal which by default comes with such an .htaccess file. If you forbid users to set FollowSymLinks then the option won't be just ignored, the whole webpage won't run and will just return an error 500. What you could do is changing the FollowSymLinks option in the .htaccess manually to SymlinksIfOwnerMatch. While this may be feasible in some cases, if you consider that you have a lot of users you don't want to explain to all of them that in case they want to install some common web application they have to manually edit some file they don't understand. (There's a bug report for Drupal asking to change FollowSymLinks to SymlinksIfOwnerMatch, but it's been ignored since several years.)

So using SymLinksIfOwnerMatch is neither secure nor really feasible. The documentation for Cpanel discusses several possible solutions. The recommended solutions require proprietary modules. None of the proposed fixes work with a plain Apache setup, which I think is a pretty dismal situation. The most common web server has a severe security weakness in a very common situation and no usable solution for it.

The one solution that we chose is a feature of Grsecurity. Grsecurity is a Linux kernel patch that greatly enhances security and we've been very happy with it in the past. There are a lot of reasons to use this patch, I'm often impressed that local root exploits very often don't work on a Grsecurity system.

Grsecurity has an option like SymlinksIfOwnerMatch (CONFIG_GRKERNSEC_SYMLINKOWN) that operates on the kernel level. You can define a certain user group (which in our case is the "apache" group) for which this option will be enabled. For us this was the best solution, as it required very little change.

I haven't checked this, but I'm pretty sure that we were not alone with this problem. I'd guess that a lot of shared web hosting companies are vulnerable to this problem.

Here's the German blog post on our webpage and here's the original blogpost from an administrator at Uberspace (also German) which made us aware of this issue.

23 Jun 2015 9:41pm GMT

21 Jun 2015

feedPlanet Gentoo

Andreas K. Hüttel: Perl 5.22 testers needed!

Gentoo users rejoice, for a few days already we have Perl 5.22.0 packaged in the main tree. Since we don't know yet how much stuff will break because of the update, it is masked for now. Which means, we need daring testers (preferably running ~arch systems, stable is also fine but may need more work on your part to get things running) who unmask the new Perl, upgrade, and file bugs if needed!!!
Here's what you need in /etc/portage/package.unmask (and possibly package.accept_keywords) to get started (download); please always use the full block, since partial unmasking will lead to chaos. We're looking forward to your feedback!

# Perl 5.22.0 mask / unmask block
=dev-lang/perl-5.22.0
=virtual/perl-Archive-Tar-2.40.0
=virtual/perl-Attribute-Handlers-0.970.0
=virtual/perl-B-Debug-1.230.0
=virtual/perl-CPAN-2.110.0
=virtual/perl-CPAN-Meta-2.150.1
=virtual/perl-CPAN-Meta-Requirements-2.132.0
=virtual/perl-Carp-1.360.0
=virtual/perl-Compress-Raw-Bzip2-2.68.0
=virtual/perl-Compress-Raw-Zlib-2.68.0
=virtual/perl-DB_File-1.835.0
=virtual/perl-Data-Dumper-2.158.0
=virtual/perl-Devel-PPPort-3.310.0
=virtual/perl-Digest-MD5-2.540.0
=virtual/perl-Digest-SHA-5.950.0
=virtual/perl-Exporter-5.720.0
=virtual/perl-ExtUtils-CBuilder-0.280.221
=virtual/perl-ExtUtils-Command-1.200.0
=virtual/perl-ExtUtils-Install-2.40.0
=virtual/perl-ExtUtils-MakeMaker-7.40.100_rc
=virtual/perl-ExtUtils-ParseXS-3.280.0
=virtual/perl-File-Spec-3.560.0
=virtual/perl-Filter-Simple-0.920.0
=virtual/perl-Getopt-Long-2.450.0
=virtual/perl-HTTP-Tiny-0.54.0
=virtual/perl-IO-1.350.0
=virtual/perl-IO-Compress-2.68.0
=virtual/perl-IO-Socket-IP-0.370.0
=virtual/perl-JSON-PP-2.273.0
=virtual/perl-Locale-Maketext-1.260.0
=virtual/perl-MIME-Base64-3.150.0
=virtual/perl-Math-BigInt-1.999.700
=virtual/perl-Math-BigRat-0.260.800
=virtual/perl-Module-Load-Conditional-0.640.0
=virtual/perl-Module-Metadata-1.0.26
=virtual/perl-Perl-OSType-1.8.0
=virtual/perl-Pod-Escapes-1.70.0
=virtual/perl-Pod-Parser-1.630.0
=virtual/perl-Pod-Simple-3.290.0
=virtual/perl-Safe-2.390.0
=virtual/perl-Scalar-List-Utils-1.410.0
=virtual/perl-Socket-2.18.0
=virtual/perl-Storable-2.530.0
=virtual/perl-Term-ANSIColor-4.30.0
=virtual/perl-Term-ReadLine-1.150.0
=virtual/perl-Test-Harness-3.350.0
=virtual/perl-Test-Simple-1.1.14
=virtual/perl-Text-Balanced-2.30.0
=virtual/perl-Text-ParseWords-3.300.0
=virtual/perl-Time-Piece-1.290.0
=virtual/perl-Unicode-Collate-1.120.0
=virtual/perl-XSLoader-0.200.0
=virtual/perl-autodie-2.260.0
=virtual/perl-bignum-0.390.0
=virtual/perl-if-0.60.400
=virtual/perl-libnet-3.50.0
=virtual/perl-parent-0.232.0
=virtual/perl-threads-2.10.0
=virtual/perl-threads-shared-1.480.0
=dev-perl/Test-Tester-0.114.0
=dev-perl/Test-use-ok-0.160.0

# end of the Perl 5.22.0 mask / unmask block

After the update, first run

emerge --depclean --ask

and afterwards

perl-cleaner --all

perl-cleaner should not need to do anything, ideally. If you have depcleaned first and it still wants to rebuild something, that's a bug. Please file a bug report for the package that is getting rebuilt (but check our wiki page on known Perl 5.22 issues first to avoid duplicates).


21 Jun 2015 8:14am GMT

19 Jun 2015

feedPlanet Gentoo

Nathan Zachary: Amavisd – debugging and logging startup failures

Important!

My tech articles-especially Linux ones-are some of the most-viewed on The Z-Issue. If this one has helped you, please consider a small donation to The Parker Fund by using the top widget at the right. Thanks!

Recently, I wrote an article about amavisd not running with Postfix, and getting a "Connection refused to 127.0.0.1″ error message that wasn't easy to diagnose. Yesterday, I ran into another problem with amavisd refusing to start properly, and I wasn't readily able to figure out why. By default, amavisd logs to your mail log, which for me is located at /var/log/mail.log, but could be different for you based on your syslogger preferences. The thing is, though, that it will not log start-up errors there. So basically, one is seemingly left in the dark if you start amavisd and then realise it isn't running immediately thereafter.

I decided to take a look at the init script for amavisd, and saw that there were some non-standard functions in it:


# grep 'extra_commands' /etc/init.d/amavisd
extra_commands="debug debug_sa"

These extra commands map to the following functions:


debug() {
ebegin "Starting ${progname} in debug mode"
"${prog}" debug
eend $?
}

debug_sa() {
ebegin "Starting ${progname} in debug-sa mode"
"${prog}" debug-sa
eend $?
}

Though these extra commands may be Gentoo-specific, they are pretty easy to implement on other distributions by directly calling the binary itself. For instance, if you wanted the debug function, it would be the location of the binary with 'debug' appended to it. On my system, that would be:


/usr/sbin/amavisd -c $LOCATION_OF_CONFIG_FILE debug

replacing the $LOCATION_OF_CONFIG_FILE with your actual config file location.

When I started amavisd in debug mode, the start-up problem that it was having became readily apparent:


# /etc/init.d/amavisd debug
* Starting amavisd-new in debug mode ...
Jun 18 12:48:21.948 /usr/sbin/amavisd[4327]: logging initialized, log level 5, syslog: amavis.mail
Jun 18 12:48:21.948 /usr/sbin/amavisd[4327]: starting. /usr/sbin/amavisd at amavisd-new-2.10.1 (20141025), Unicode aware, LANG="en_GB.UTF-8"

Jun 18 12:48:22.200 /usr/sbin/amavisd[4327]: Net::Server: 2015/06/18-12:48:22 Amavis (type Net::Server::PreForkSimple) starting! pid(4327)
Jun 18 12:48:22.200 /usr/sbin/amavisd[4327]: (!)Net::Server: 2015/06/18-12:48:22 Unresolveable host [::1]:10024 - could not load IO::Socket::INET6: Can't locate Socket6.pm in @INC (you may need to install the Socket6 module) (@INC contains: lib /etc/perl /usr/local/lib64/perl5/5.20.2/x86_64-linux /usr/local/lib64/perl5/5.20.2 /usr/lib64/perl5/vendor_perl/5.20.2/x86_64-linux /usr/lib64/perl5/vendor_perl/5.20.2 /usr/local/lib64/perl5 /usr/lib64/perl5/vendor_perl/5.20.1/x86_64-linux /usr/lib64/perl5/vendor_perl/5.20.1 /usr/lib64/perl5/vendor_perl /usr/lib64/perl5/5.20.2/x86_64-linux /usr/lib64/perl5/5.20.2) at /usr/lib64/perl5/vendor_perl/5.20.1/Net/Server/Proto.pm line 122.\n\n at line 82 in file /usr/lib64/perl5/vendor_perl/5.20.1/Net/Server/Proto.pm
Jun 18 12:48:22.200 /usr/sbin/amavisd[4327]: Net::Server: 2015/06/18-12:48:22 Server closing!

In that code block, the actual error (in bold text) indicates that it couldn't find the Perl module IO:Socket::INET6. This problem was easily fixed in Gentoo with emerge -av dev-perl/IO-Socket-INET6, but could be rectified by installing the module from your distribution's repositories, or by using CPAN. In my case, it was caused by my recent compilation and installation of a new kernel that, this time, included IPV6 support.

The point of my post, however, wasn't about my particular problem with amavisd starting, but rather how one can debug start-up problems with the daemon. Hopefully, if you run into woes with amavisd logging, these debug options will help you track down the problem.

Cheers,
Zach

19 Jun 2015 8:19pm GMT

10 Jun 2015

feedPlanet Gentoo

Sven Vermeulen: Live SELinux userspace ebuilds

In between courses, I pushed out live ebuilds for the SELinux userspace applications: libselinux, policycoreutils, libsemanage, libsepol, sepolgen, checkpolicy and secilc. These live ebuilds (with Gentoo version 9999) pull in the current development code of the SELinux userspace so that developers and contributors can already work with in-progress code developments as well as see how they work on a Gentoo platform.

That being said, I do not recommend using the live ebuilds for anyone else except developers and contributors in development zones (definitely not on production). One of the reasons is that the ebuilds do not apply Gentoo-specific patches to the ebuilds. I would also like to remove the Gentoo-specific manipulations that we do, such as small Makefile adjustments, but let's start with just ignoring the Gentoo patches.

Dropping the patches makes sure that we track upstream libraries and userspace closely, and allows developers to try and send out patches to the SELinux project to fix Gentoo related build problems. But as not all packages can be deployed successfully on a Gentoo system some patches need to be applied anyway. For this, users can drop the necessary patches inside /etc/portage/patches as all userspace ebuilds use the epatch_user method.

Finally, observant users will notice that "secilc" is also provided. This is a new package, which is probably going to have an official release with a new userspace release. It allows for building CIL-based SELinux policy code, and was one of the drivers for me to create the live ebuilds as I'm experimenting with the CIL constructions. So expect more on that later.

10 Jun 2015 6:07pm GMT

07 Jun 2015

feedPlanet Gentoo

Sebastian Pipping: gentoo.de relaunched

Hi!

Two months ago, the gentoo.org website redesign was announced. For a few hours now, gentoo.de is following. The similarities in look are not by coincidence: Both design were done by Alex Legler.

The new website is based on Jekyll rather than GuideXML previously. Any bugs you find on the website can be reported on GitHub where the website sources are hosted.

I would like to take the opportunity to thank both Manitu and SysEleven for providing the hardware running gentoo.de (and other Gentoo e.V. services) free of charge. Very kind!

Best, Sebastian

07 Jun 2015 11:02pm GMT

06 Jun 2015

feedPlanet Gentoo

Diego E. Pettenò: LibreSSL, OpenSSL, collisions and problems

Some time ago, on the gentoo-dev mailing list, there has been an interesting thread on the state of LibreSSL in Gentoo. In particular I repeated some of my previous concerns about ABI and API compatibility, especially when trying to keep both libraries on the same system.

While I hope that the problems I pointed out are well clear to the LibreSSL developers, I thought reiterating them again clearly in a blog post would give them a wider reach and thus hope that they can be addressed. Please feel free to reshare in response to people hand waving the idea that LibreSSL can be either a drop-in, or stand-aside replacement for OpenSSL.

Last year, when I first blogged about LibreSSL, I had to write a further clarification as my post was used to imply that you could just replace the OpenSSL binaries with LibreSSL and be done with it. This is not the case and I won't even go back there. What I'm concerned about this time is whether you can install the two in the same system, and somehow decide which one you want to use on a per-package basis.

Let's start with the first question: why would you want to do that? Everybody at this point knows that LibreSSL was forked from the OpenSSL code and started removing code that has been needed unnecessary or even dangerous - a very positive thing, given the amount of compatibility kludges around OpenSSL! - and as such it was a subset of the same interface as its parent, thus there would be no reason to wanting the two libraries on the same system.

But then again, LibreSSL never meant to be considered a drop-in replacement, so they haven't cared as much for the evolution of OpenSSL, and just proceeded in their own direction; said direction included building a new library, libtls, that implements higher-level abstractions of TLS protocol. This vaguely matches the way NSS (the Netscape-now-Mozilla TLS library) is designed, and I think it makes sense: it reduces the amount of repetition that needs to be coded in multiple parts of the software stack to implement HTTPS for instance, reducing the chance of one of them making a stupid mistake.

Unfortunately, this library was originally tied firmly to LibreSSL and there was no way for it to be usable with OpenSSL - I think this has changed recently as a "portable" build of libtls should be available. Ironically, this wouldn't have been a problem at all if it wasn't that LibreSSL is not a superset of OpenSSL, as this is where the core of the issue lies.

By far, this is not the first time a problem like this happens in Open Source software communities: different people will want to implement the same concept in different ways. I like to describe this as software biodiversity and I find it generally a good thing. Having more people looking at the same concept from different angles can improve things substantially, especially in regard to finding safe implementations of network protocols.

But there is a problem when you apply parallel evolution to software: if you fork a project and then evolve it on your own agenda, but keep the same library names and a mostly compatible (thus conflicting) API/ABI, you're going to make people suffer, whether they are developers, consumers, packagers or users.

LibreSSL, libav, Ghostscript, ... there are plenty of examples. Since the features of the projects, their API and most definitely their ABIs are not the same, when you're building a project on top of any of these (or their originators), you'll end up at some point making a conscious decision on which one you want to rely on. Sometimes you can do that based only on your technical needs, but in most cases you end up with a compromise based on technical needs, licensing concerns and availability in the ecosystem.

These projects didn't change the name of their libraries, that way they can be used as drop-rebuild replacement for consumers that keep to the greatest common divisor of the interface, but that also means you can't easily install two of them in the same system. And since most distributions, with the exception of Gentoo, would not really provide the users with choice of multiple implementations, you end up with either a fractured ecosystem, or one that is very much non-diverse.

So if all distributions decide to standardize on one implementation, that's what the developers will write for. And this is why OpenSSL will likely to stay the standard for a long while still. Of course in this case it's not as bad as the situation with libav/ffmpeg, as the base featureset is going to be more or less the same, and the APIs that have been dropped up to now, such as the entropy-gathering daemon interface, have been considered A Bad Idea™ for a while, so there are not going to be OpenSSL-only source projects in the future.

What becomes an issue here is that software is built against OpenSSL right now, and you can't really change this easily. I've been told before that this is not true, because OpenBSD switched, but there is a huge difference between all of the BSDs and your usual Linux distributions: the former have much more control on what they have to support.

In particular, the whole base system is released in a single scoop, and it generally includes all the binary packages you can possibly install. Very few third party software providers release binary packages for OpenBSD, and not many more do for NetBSD or FreeBSD. So as long as you either use the binaries provided by those projects or those built by you on the same system, switching the provider is fairly easy.

When you have to support third-party binaries, then you have a big problem, because a given binary may be built against one provider, but depend on a library that depends on the other. So unless you have full control of your system, with no binary packages at all, you're going to have to provide the most likely provider - which right now is OpenSSL, for good or bad.

Gentoo Linux is, once again, in a more favourable position than many others. As long as you have a full source stack, you can easily choose your provider without considering its popularity. I have built similar stacks before, and my servers deploy stacks similarly, although I have not tried using LibreSSL for any of them yet. But on the desktop it might be trickier, especially if you want to do things like playing Steam games.

But here's the harsh reality, even if you were to install the libraries in different directories, and you would provide a USE flag to choose between the two, it is not going to be easy to apply the right constraints between final executables and libraries all the way into the tree.

I'm not sure if I have an answer to balance the ability to just make the old software use the new library and the side-installation. I'm scared that a "solution" that can be found to solve this problem is bundling and you can probably figure out that doing so for software like OpenSSL or LibreSSL is a terrible idea, given how fast you should update in response to a security vulnerability.

06 Jun 2015 12:33pm GMT

05 Jun 2015

feedPlanet Gentoo

Nathan Zachary: RHEL/CentOS Kickstart partition scheme with disk by-path IDs and automation via PXE

Important!

This article is one of the most-viewed on The Z-Issue, and is sometimes read thousands of times per day. If it has helped you, please consider a small donation to The Parker Fund by using the top widget at the right. Thanks!

So, for several weeks now, I've been struggling with disk naming and UDEV as they relate to RHEL or CentOS Kickstart automation via PXE. Basically, what used to be a relatively straight-forward process of setting up partitions based on disk name (e.g. /dev/sda for the primary disk, /dev/sdb for the secondary disk, and so on) has become a little more complicated due to the order in which the kernel may identify disks with UDEV implementations. The originally-used naming conventions of /dev/sdX may no longer be consistent across reboots or across machines. How this problem came to my attention was when I was Kickstarting a bunch of hosts, and realised that several of them were indicating that there wasn't enough space on the drives that I had referenced. After doing some digging, I realised that some servers were identifying KVM-based media (like via a Cisco UCS CIMC interface), or other USB media as primary disks instead of the actual primary SCSI-based disks (SATA or SAS drives).

Though I agree that identifying disks by their path, their UUID, or an otherwise more permanent name is preferred to the ambiguous /dev/sdX naming scheme, it did cause a problem, and one that didn't have a readily-identifiable workaround for my situation. My first thought was to use UUID, or gather the disk ID from /dev/disk/by-id/. However, that wouldn't work because it wouldn't be the same across all servers, so automatic installations via PXE wouldn't be feasible. Next, I looked at referencing disks by their PCI location, which can be found within /dev/disk/by-path/. Unfortunately, I found that the PCI location might vary between servers as well. For example, I found:

Server 1:
/dev/disk/by-path/pci-0000:82:00.0-scsi-0:2:0:0 -> primary SCSI disk
/dev/disk/by-path/pci-0000:82:00.0-scsi-0:2:0:0 -> secondary SCSI disk

Server 2:
/dev/disk/by-path/pci-0000:0b:00.0-scsi-0:2:0:0 -> primary SCSI disk
/dev/disk/by-path/pci-0000:0b:00.0-scsi-0:2:0:0 -> secondary SCSI disk

That being said, I did notice one commonality between the servers that I used for testing. The primary and secondary disks (which, by the way, are RAID arrays attached to the same controller) all ended with 'scsi-0:2:0:0 ' for the primary disk, and 'scsi-0:2:1:0 ' for the secondary disk. Thinking that I could possibly just specify using a wildcard, I tried:

part /boot --fstype ext2 --size=100 --ondisk=disk/by-path/*scsi-0:2:0:0

but alas, that caused Anaconda to error out stating that the specified drive did not exist. At this point, I thought that all hope was lost in terms of automation. Then, however, I had a flash of genius (they don't happen all that often). I could probably gather the full path ID in a pre-installation script, and then get the FULL path ID into the Kickstart configuration file with an include. Here are the code snippets that I wrote:


%pre --interpreter=/bin/bash
## Get the full by-path ID of the primary and secondary disks
osdisk=$(ls -lh /dev/disk/by-path/ | grep 'scsi-0:2:0:0 ' | awk '{print $9}')
datadisk=$(ls -lh /dev/disk/by-path/ | grep 'scsi-0:2:1:0 ' | awk '{print $9}')


## Create a temporary file with the partition scheme to be included in the KS config
echo "# Partitioning scheme" > /tmp/partition_layout
echo "clearpart --all" >> /tmp/partition_layout
echo "zerombr" >> /tmp/partition_layout
echo "part /boot --fstype ext2 --size=100 --ondisk=disk/by-path/$osdisk" >> /tmp/partition_layout
echo "part swap --recommended --ondisk=disk/by-path/$osdisk" >> /tmp/partition_layout
echo "part pv.01 --size=100 --grow --ondisk=disk/by-path/$osdisk" >> /tmp/partition_layout
echo "part pv.02 --size=100 --grow --ondisk=disk/by-path/$datadisk" >> /tmp/partition_layout
echo "volgroup vg_os pv.01" >> /tmp/partition_layout
echo "volgroup vg_data pv.02" >> /tmp/partition_layout
echo "logvol / --vgname=vg_os --size=100 --name=lv_os --fstype=ext4 --grow" >> /tmp/partition_layout
echo "logvol /data --vgname=vg_data --size=100 --name=lv_data --fstype=ext4 --grow" >> /tmp/partition_layout
echo "# Bootloader location" >> /tmp/partition_layout
echo "bootloader --location=mbr --driveorder=/dev/disk/by-path/$osdisk,/dev/disk/by-path/$datadisk --append=\"rhgb\"" >> /tmp/partition_layout
%end

Some notes about the pre-installation script would be: 1) yes, I know that it is a bit clumsy, but it functions despite the lack of elegance; 2) in lines 3 and 4, make sure that the grep statements include a final space. The final space (e.g. grep 'scsi-0:2:0:0 ' instead of just grep 'scsi-0:2:0:0') is important because the machine may have partitions already set up, and in that case, those partitions would be identified as '*scsi-0:2:0:0-part#' where # is the partition number.

So, this pre-installation script generates a file called /tmp/partition_layout that essentially has a normal partition scheme for a Kickstart configuration file, but references the primary and secondary SCSI disks (again, as RAID arrays attached to the same controller) by their full 'by-path' IDs. Then, the key is to include that file within the Kickstart configuration via:


# Partitioning scheme information gathered from pre-install script below
%include /tmp/partition_layout

I am happy to report that for the servers that I have tested thus far, this method works. Is it possible that there will be primary and secondary disks that are identified via different path IDs not caught by my pre-installation script? Yes, that is possible. It's also possible that your situation will require a completely different pre-installation script to gather the unique identifier of your choice. That being said, I hope that this post will help you devise your method of reliably identifying disks so that you can create the partition scheme automatically via your Kickstart configuration file. That way, you can then deploy it to many servers via PXE (or any other implementation that you're currently utilising for mass deployment).

Cheers,
Zach

05 Jun 2015 6:33pm GMT

Robin Johnson: gnupg-2.1 mutt

For the mutt users with GnuPG, depending on your configuration, you might notice that mutt's handling of GnuPG mail stopped working with GnuPG. There were a few specific cases that would have caused this, which I'll detail, but if you just want it to work again, put the below into your Muttrc, and make the tweak to gpg-agent.conf. The underlying cause for most if it is that secret key operations have moved to the agent, and many Mutt users used the agent-less mode, because Mutt handled the passphrase nicely on it's own.

set pgp_autosign = yes
set pgp_use_gpg_agent = no
set pgp_timeout = 600
set pgp_sign_as="(your key here)"
set pgp_ignore_subkeys = no

set pgp_decode_command="gpg %?p?--pinentry-mode loopback  --passphrase-fd 0? --verbose --no-auto-check-trustdb --batch --output - %f"
set pgp_verify_command="gpg --pinentry-mode loopback --verbose --batch --output - --no-auto-check-trustdb --verify %s %f"
set pgp_decrypt_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --output - %f"
set pgp_sign_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --output - --armor --textmode %?a?-u %a? --detach-sign %f"
set pgp_clearsign_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --output - --armor --textmode %?a?-u %a? --detach-sign %f"
set pgp_encrypt_sign_command="pgpewrap gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --textmode --trust-model always --output - %?a?-u %a? --armor --encrypt --sign --armor -- -r %r -- %f"
set pgp_encrypt_only_command="pgpewrap gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --trust-model always --output --output - --encrypt --textmode --armor -- -r %r -- %f"
set pgp_import_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --import -v %f"
set pgp_export_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --export --armor %r"
set pgp_verify_key_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --fingerprint --check-sigs %r"
set pgp_list_pubring_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --with-colons --list-keys %r"
set pgp_list_secring_command="gpg %?p?--pinentry-mode loopback --passphrase-fd 0? --verbose --batch --with-colons --list-secret-keys %r"

This entry was originally posted at http://robbat2.dreamwidth.org/238770.html. Please comment there using OpenID.

05 Jun 2015 5:25pm GMT

26 May 2015

feedPlanet Gentoo

Alexys Jacob: MongoDB 3.0 upgrade in production : step 2 failed

In my previous post regarding the migration of our production cluster to mongoDB 3.0 WiredTiger, we successfully upgraded all the secondaries of our replica-sets with decent performances and (almost, read on) no breakage.

Step 2 plan

The next step of our migration was to test our work load on WiredTiger primaries. After all, this is where the new engine would finally demonstrate all its capabilities.

The failure

After all the wait and expectation, I can't express our level of disappointment at work when we saw that the WiredTiger engine could not handle our work load. Our application started immediately to throw 50 to 250 WriteConflict errors per minute !

Turns out that we are affected by this bug and that, of course, we're not the only ones. So far it seems that it affects collections with :

The breakages

We also discovered that we're affected by a weird mongodump new behaviour where the dumped BSON file does not contain the number of documents that mongodump said it was exporting. This is clearly a new problem because it happened right after all our secondaries switched to WiredTiger.

Since we have to ensure a strong consistency of our exports and that the mongoDB guys don't seem so keen on moving on the bug (which I surely can understand) there is a large possibility that we'll have to roll back even the WiredTiger secondaries altogether.

Not to mention that since the 3.0 version, we experience some CPU overloads crashing the entire server on our MMAPv1 primaries that we're still trying to tackle before opening another JIRA bug…

Sad panda

Of course, any new major release such as 3.0 causes its headaches and brings its lot of bugs. We were ready for this hence the safety steps we took to ensure that we could roll back on any problem.

But as a long time advocate of mongoDB I must admit my frustration, even more after the time it took to get this 3.0 out and all the expectations that came with it.

I hope I can share some better news on the next blog post.

26 May 2015 4:11pm GMT

23 May 2015

feedPlanet Gentoo

Sebastian Pipping: (German) Fwd: Gentoo-Workshop Samstag 01.08.2015 Troisdorf (nahe Köln/Bonn)

Hallo!

Die Troisdorfer Linux User Group (kurz TroLUG) veranstaltet

am Samstag den 01.08.2015
in Troisdorf nahe Köln/Bonn

einen Gentoo-Workshop, der sich an fortgeschrittene User richtet.

Mehr Details, die genaue Adresse und der Ablauf finden sich auf der entsprechenden Seite der TroLUG.

Grüße,

Sebastian

23 May 2015 10:54pm GMT

Alexys Jacob: Gevent : SSL support fixed for python 2.7.9

Good news for gevent users blocked on python < 2.7.9 due to broken SSL support since python upstream dropped the private API _ssl.sslwrap that eventlet was using.

This issue was starting to get old and problematic since GLSA 2015-0310 but I'm happy to say that almost 6 hours after the gevent-1.0.2 release, it is already available on portage !

We were also affected by this issue at work so I'm glad that the tension between ops and devs this issue was causing will finally be over ;)

23 May 2015 6:38pm GMT

20 May 2015

feedPlanet Gentoo

Nathan Zachary: Amavisd isn’t running with Postfix – connection refused to 127.0.0.1

Important!

My tech articles-especially Linux ones-are some of the most-viewed on The Z-Issue. If this one has helped you, please consider a small donation to The Parker Fund by using the top widget at the right. Thanks!

Late last night, I decided to apply some needed updates to my personal mail server, which is running Gentoo Linux (OpenRC) with a mail stack of Postfix & Dovecot with AMaViS (filtering based on SpamAssassin, ClamAV, and Vipul's Razor). After applying the updates, and restarting the necessary components of the mail stack, I ran my usual test of sending an email from one of my accounts to another one. It went through without a problem.

However, I realised that it isn't a completely valid test to send an email from one internal account to another because I have amavisd configured to not scan anything coming from my trusted IPs and domains. I noticed several hundred mails in the queue when I ran postqueue -p, and they all had notices similar to:


status=deferred (delivery temporarily suspended:
connect to 127.0.0.1[127.0.0.1]:10024: Connection refused)

That indicated to me that it wasn't a problem with Postfix (and I knew it wasn't a problem with Dovecot, because I could connect to my accounts via IMAP). Seeing as amavisd is running on localhost:10024, I figured that that is where the problem had to be. A lot of times, when there is a "connection refused" notification, it is because no service is listening on that port. You can test to see what ports are in a listening state and what processes, applications, or daemons are listening by running:

netstat -tupan | grep LISTEN

When I did that, I noticed that amavisd wasn't listening on port 10024, which made me think that it wasn't running at all. That's when I ran into the strange part of the problem: the init script output:

# /etc/init.d/amavisd start
* WARNING: amavisd has already been started
# /etc/init.d/amavisd stop
The amavisd daemon is not running [ !! ]
* ERROR: amavisd failed to start

So, apparently it is running and not running at the same time (sounds like a Linux version of Schrödinger's cat to me)! It was obvious, though, that it wasn't actually running (which could be verified with 'ps -elf | grep -i amavis'). So, what to do? I tried manually removing the PID file, but that actually just made matters a bit worse. Ultimately, this combination is what fixed the problem for me:


sa-update
/etc/init.d/amavisd zap
/etc/init.d/amavisd start

It seems that the SpamAssassin rules file had gone missing, and that was causing amavisd to not start properly. Manually updating the rules file (with 'sa-update') regenerated it, and then I zapped amavisd completely, and lastly restarted the daemon.

Hope that helps anyone running into the same problem.

Cheers,
Zach

EDIT: I have included a new post about debugging amavisd start-up problems.

20 May 2015 2:32pm GMT

13 May 2015

feedPlanet Gentoo

Alexys Jacob: uWSGI, gevent and pymongo 3 threads mayhem

This is a quick heads-up post about a behaviour change when running a gevent based application using the new pymongo 3 driver under uWSGI and its gevent loop.

I was naturally curious about testing this brand new and major update of the python driver for mongoDB so I just played it dumb : update and give a try on our existing code base.

The first thing I noticed instantly is that a vast majority of our applications were suddenly unable to reload gracefully and were force killed by uWSGI after some time !

worker 1 (pid: 9839) is taking too much time to die...NO MERCY !!!

uWSGI's gevent-wait-for-hub

All our applications must be able to be gracefully reloaded at any time. Some of them are spawning quite a few greenlets on their own so as an added measure of making sure we never loose any running greenlet we use the gevent-wait-for-hub option, which is described as follow :

wait for gevent hub's death instead of the control greenlet

… which does not mean a lot but is explained in a previous uWSGI changelog :

During shutdown only the greenlets spawned by uWSGI are taken in account,
and after all of them are destroyed the process will exit.

This is different from the old approach where the process wait for
ALL the currently available greenlets (and monkeypatched threads).

If you prefer the old behaviour just specify the option gevent-wait-for-hub

pymongo 3

Compared to its previous 2.x versions, one of the overall key aspect of the new pymongo 3 driver is its intensive usage of threads to handle server discovery and connection pools.

Now we can relate this very fact to the gevent-wait-for-hub behaviour explained above :

the process wait for ALL the currently available greenlets
(and monkeypatched threads)

This explained why our applications were hanging until the reload-mercy (force kill) timeout option of uWSGI hit the fan !

conclusion

When using pymongo 3 with the gevent-wait-for-hub option, you have to keep in mind that all of pymongo's threads (so monkey patched threads) are considered as active greenlets and will thus be waited for termination before uWSGI recycles the worker !

Two options come in mind to handle this properly :

  1. stop using the gevent-wait-for-hub option and change your code to use a gevent pool group to make sure that all of your important greenlets are taken care of when a graceful reload happens (this is how we do it today, the gevent-wait-for-hub option usage was just over protective for us).
  2. modify your code to properly close all your pymongo connections on graceful reloads.

Hope this will save some people the trouble of debugging this ;)

13 May 2015 2:56pm GMT