I'd like to invite members of the Arch Linux community to join the new Gittip Community I set up for Arch Linux. Communities are a great idea that helps make Gittip more global and more local at the same time. Gittip is a platform to use generosity to crowdfund kickass content creators, from musicians to [...]

21 May 2013 2:51pm GMT

While my old hosting with Laughing Squid was did its job, I was very limited in what I could do with it. Access only via sftp, 1GB of storage and I was breaking my monthly 25GB traffic limit fairly frequently… So it was time to look for something new. For me, the primary consideration when [...]

21 May 2013 12:44pm GMT

{% img left http://dl.dropbox.com/u/261312/Blog-images/woodcutting.jpg 'Cutting wood' %} I posted a couple of weeks ago about Building Vim and how, using ABS and makepkg it is possible to customize packages in the repositories to suit your individual requirements, in that case with a specific feature set.

One of Arch's real strengths is in the flexibility that makepkg and PKGBUILDs provide the community; the ability to adapt official packages-or unofficial ones in the AUR-as you see fit. As PKGBUILDs are just shell scripts, the entry level to start playing around with them is quite low¹.

A fairly standard, and simple, example of the type of customization that I might make is with dmenu, the suckless dynamic menu, where the standard package in the Arch repositories is not patched for Xft support. There is a patch for this on the suckless wiki, so it is just a case of making the requisite changes in the PKGBUILD from ABS and building it.

As you can see from the diff below, there is not a lot involved in this exercize; essentially, adding libxft as a dependency, sourcing the patch from the suckless site (and including the hash for it), and then in the build function ensuring that the patch is applied and the Makefile updated with the new library:

{% codeblock lang:diff %} --- PKGBUILD 2013-05-18 09:33:07.156328812 +1200 +++ PKGBUILD 2012-11-14 09:25:15.915335588 +1300 @@ -11,16 +6,22 @@ pkgdesc="A generic menu for X" url="http://tools.suckless.org/dmenu/" arch=('i686' 'x86_64') +groups=('modified') license=('MIT') -depends=('sh' 'libxinerama') -source=(http://dl.suckless.org/tools/$pkgname-$pkgver.tar.gz) -md5sums=('9c46169ed703732ec52ed946c27d84b4') +depends=('sh' 'libxinerama' 'libxft') +source=(http://dl.suckless.org/tools/$pkgname-$pkgver.tar.gz +http://tools.suckless.org/dmenu/patches/$pkgname-$pkgver-xft.diff) +md5sums=('9c46169ed703732ec52ed946c27d84b4' + 'd448ec9120718b0aedbdb338f4fa69ba')

build(){ cd $srcdir/$pkgname-$pkgver + patch -p1 < ../$pkgname-$pkgver-xft.diff + sed -i 's:-I/usr/local/include/freetype2:-I/usr/include/freetype2:' config.mk + make \ {% endcodeblock %}

Running makepkg -i will build and install dmenu with Xft support. This is the most straightforward approach. I also, primarily by way of experimentation and in an effort to try an understand how this actually works, have slightly more convoluted examples. msmtp, the SMTP client has a couple of makedepends in libgnome-keyring and texlive-core; the former I have zero use for and the latter is only installed on my desktop, so I have no wish to install it on my laptop just to be able to send emails…

In this case, I modified the PKGBUILD to completely remove the libgnome-keyring dependency and to only build the msmtp documentation in .pdf and .html if texlive-core was already installed on the machine. Unfortunately, I wasn't able to test for the presence of texlive-core with the standard utilities like type or which, so-as it is installed on all my boxes-I went with expac (pacman -Q would also work):

{% codeblock lang:diff %} --- PKGBUILD 2013-05-18 09:32:07.393095131 +1200 +++ PKGBUILD 2013-05-18 09:31:55.449986364 +1200 @@ -1,7 +1,8 @@ arch=('i686' 'x86_64') +groups=('modified') license=('GPL3') url="http://msmtp.sourceforge.net" -makedepends=('texlive-core' 'gsasl' 'libgnome-keyring') +makedepends=('gsasl') source=(http://download.sourceforge.net/sourceforge/msmtp/${pkgbase}-${pkgver}.tar.bz2) sha1sums=('c0edce1e1951968853f15209c8509699ff9e9ab5')

@@ -12,19 +13,24 @@

build() { cd ${pkgbase}-${pkgver} - ./configure --prefix=/usr --sysconfdir=/etc --with-ssl=gnutls + ./configure --prefix=/usr --sysconfdir=/etc --with-ssl=gnutls --without-gnome-keyring make - make -C doc html pdf + if [[ -n $(expac -Q '%n' texlive-core) ]]; then + make -C doc html pdf + fi }

package_msmtp() { pkgdesc="A mini smtp client" - depends=('gsasl' 'libgnome-keyring') + depends=('gsasl') install=msmtp.install

cd ${pkgbase}-${pkgver} make DESTDIR="${pkgdir}" install + + if [[ -n $(expac -Q '%n' texlive-core) ]]; then make DESTDIR="${pkgdir}" -C doc install-html install-pdf + fi

# Installing example configs and scripts to /usr/share/doc/msmtp {% endcodeblock %}

It isn't necessarily an attractive solution, but it works for me… On the subject of unattractive solutions, as of pacman 4.1, released last month, the packaging standards for VCS PKGBUILDs have been changed, principally around how sources and versioning is handled. For the couple of VCS packages I maintain in the AUR², I have been experimenting with how to capture the pkgver in a way that conforms to the standards and provides people with a meaningful version number.

By default, the version number for these projects from their git repos is not that helpful:

{% codeblock lang:sh %} git describe --always 4861046 {% endcodeblock %}

After looking through the git logs, and playing around with awk to filter the results, I came up with this:

{% codeblock lang:sh %} pkgver() { cd "$gitname" printf '%s\n' "$(awk '/^ / {print $2}' <(git log --grep=version -1))\ $(git describe --always)" } {% endcodeblock %}

This prints a more, for me anyway, intelligble package version: vimprobable2-git 1.2.1_c5936cc-1 that relates back to the last stable release and appends the current commit. I'm sure that this could be improved upon; suggestions are welcome.

The other change to note in all of these PKGBUILDs is the inclusion of the groups variable. By adding all of the modified packages from the official repositories to the-imaginatively titled-modified group, I can then add a line to /etc/pacman.conf that prevents those packages from being overwritten on upgrade³:

{% codeblock lang:sh %} IgnoreGroup = modified {% endcodeblock %}

Issuing pacman -Syu, or running checkupdates from a cron job, will notify you that the packages have had a version bump and that they need to be rebuilt. The PKGBUILDs for all of these packages can be found in my bitbucket repo.

Notes

1. As my experiments attest…
2. Vimprobable2-git, Surfraw-git and ruby-build-git.
3. All credit to ataraxia for this idea.

Creative Commons image on Flickr by Wells P. Wilson

17 May 2013 10:10pm GMT

I have deleted the [allanbroke] repository. It was started mainly to test the PGP signing implementation in pacman, which is now well established. Also, I would delay any packaging of release candidates or beta releases for this repository until I had enough free time and often official releases were made before that happened. The repository [...]

12 May 2013 5:00am GMT

What's that? It is now May and it has been for a few days… Better do this post then! News from the world of distributions: This relatively unknown distro got a release…. (Raring Ringtail) Although there might be a lack of co-operation between them and Debian Fedora 19 was slightly delayed. openSUSE is doing lots [...]

03 May 2013 4:21am GMT

I recently ordered a new batch of Arch Linux laptop stickers. However, because they have been selling so well, I ordered three times the number that I usually stock! I was able to get a much better price by ordering a larger bulk quantity and I'm excited to pass the savings onto all the loyal [...]

28 Apr 2013 5:23pm GMT

The first 64 Bit x86 CPU was released ten years ago: the AMD Opteron. Intel followed with compatible processors about one year later. Another two years later, in 2006, a first port of Arch Linux to the x86_64 platform was available. Starting outside the official repositories, the new architecture was merged into the Arch project […]

28 Apr 2013 4:14pm GMT

Security in computer programs is often based on some kind of secret. It might be a key, passphrase or the popular password. It is also equally important that this key is not only secret but also random and cannot be easily guessed. The same concept applies to our package signing method. The security model which […]

19 Apr 2013 4:58pm GMT

From a gender equality point of view, I'd call Pycon 2013 a success, though perhaps a better word is "progress". The gender ratio apparently doubled to 20% this year. If that continues (depending if it's a constant, linear, or quadratic curve) we could be at parity in as little as two to three years. Another [...]

10 Apr 2013 10:17pm GMT

Florian Pritz wrote:

Meet netctl: a profile based networking CLI using systemd. In the near future, the old netcfg will be removed from [core]. Anyone using it is urged to move to netctl. Migration is a manual process during which you might not have access to the Internet, so take care and read the man pages (netctl(1), netctl.profile(5) and netctl.special(7)).

The design of netctl is so that systemd enthusiasts will appreciate its usage and netcfg users will be familiar with its profile files. Shipped with netctl comes a ncurses-based wifi connection assistant called wifi-menu.

As you install netctl, netcfg will be removed, but the enabled systemd services will remain until manually disabled. The netctl wiki holds some additional information on migrating from netcfg.

10 Apr 2013 2:09pm GMT

{% img left http://dl.dropbox.com/u/261312/Blog-images/crutches.jpg 'Crutches on Flickr' %}

Or, "Why you should uninstall Yaourt and embrace makepkg…"

The release of Pacman 4.1 saw the same flurry of posts on the boards, in IRC and the mailing lists about people being "unable" to upgrade or, worse, claiming that pacman was "broken" because their upgrade was failing due to unsatisfied dependencies, that pretty much every pacman upgrade ocassions. How is it possible that so many people can run an operating system designed for competent users without having even a basic understanding of how the package manager-one of the single most critical components of the distribution-works?

Even a cursory perusal of the resulting threads on the boards will quickly identify the common denominator in these cases:

{% blockquote %} resolving dependencies... looking for inter-conflicts... error: failed to prepare transaction (could not satisfy dependencies) :: package-query: requires pacman<4.1 {% endblockquote %}

package-query is required by yaourt; so these upgrades have been stymied by a package that is in the (unsupported) AUR? There are a multitude of AUR helpers, but yaourt is most commonly used by people who are new to Arch¹ for two reasons. First, it is one of the most "featureful" and secondly, and more to the point of my argument, it can be installed by simply adding an unsupported repo to pacman.conf; thereby effectively bypassing the need for the hapless user ever having to use or understand makepkg.

Consequently, over time, people who are habitually using yaourt -Syu --aur to update both the packages in the supported repositories and those they have installed from the AUR lose the conceptual distinction between the two. yaourt obscures this from them and-if they are completely reliant upon it, as these threads attest they are-they have abnegated responsibility for managing those unsupported packages and in doing so have found themselves incapable of understanding the bind they are in.

This sort of obscuring of fundamental operating principles in the pursuit of "convenience" is anathema to Arch and is precisely the reason I moved away from using yaourt (and indeed from using Ubuntu when I jumped from that sinking ship). Any convenience is purely illusory, in reality it just fosters learned helplessness.

I understand that the yaourt developer(s) was scratching their own itch, and this post is not about maligning the project; but there are significant unintended consequences of giving people a tool that abstracts such a fundamental element of the distribution away from the user, especially for a distribution where you are expected to have complete control and responsbility over your system.

To be clear, I don't have a complaint with the concept of AUR helpers. I used yaourt initially before switching to aurget and then alighting on cower, or more particularly, a partial wrapper for cower:

{% codeblock lang:sh cowerd %}

!/bin/sh

install AUR packages with cower

cd $HOME/Build && cower -d "$1" builddir="$_" cd "$builddir" && ${EDITOR:-vi} PKGBUILD

makepkg -si && cd - &>/dev/null

read -p "Remove Build directory? [Y/n]? " yn if [ "$yn" = "y" ]; then

printf "\n%s\n" "Removing build directory..." 
rm -rf "$builddir"

else

printf "%s\n" "Build completed."

fi {% endcodeblock %}

This provides me the minimum level of automation I require-essentially only around downloading and installing a package. It doesn't automatically handle dependencies, nor manage updating the packages; that remains, rightly in my view, my responsibility.

If I were to look to a more fully featured wrapper, I would undoubtedly choose meat, however as on my desktop machine, I only have ~30 AUR packages installed, I don't really need anything more sophisticated.

So by all means, use an AUR helper. But recognize that it is intended to help you, not preclude you from being able to accomplish the most simple and critical task of system maintenance, updating your package manager. Uninstall yaourt if you are using it and familiarize yourself with makepkg; once you do understand the relationship between the official repositories and the AUR, download cower or meat, they are both much better solutions.

Notes

1. Myself included…

Creative Commons image of crutches by net_efekt on Flickr.

09 Apr 2013 7:07am GMT

I will clarify this just because I have had several people ask me already. No, we did not remove the SyncFirst option in pacman to deliberately cause issues for Manjaro Linux. In fact, it was first discussed in Feburary 2012 and, as far as I can tell, Manjaro has only been around from late March [...]

05 Apr 2013 11:34am GMT

metrics often seem to lack details, such as units and metric types

looking at a metric name, it's often hard to know

• the unit a metric is measured in (bits, queries per second, jiffies, etc)
• the "type" (a rate, an ever increasing counter, gauge, etc)
• the scale/prefix (absolute, relative, percentage, mega, milli, etc)

structured_metrics solves this by adding these tags to graphite metrics:

• what
  

  what is being measured?: bytes, queries, timeouts, jobs, etc

• target_type
  

  must be one of the existing clearly defined target_types (count, rate, counter, gauge)
  These match statsd metric types (i.e. rate is per second, count is per flushInterval)

In Graph-Explorer these tags are mandatory, so that it can show the unit along with the prefix (i.e. 'Gb/s') on the axis.
This will also allow you to request graphs in a different unit and the dashboard will know how to convert (say, Mbps to GB/day)

tree navigation/querying is cumbersome, metrics search is broken. How do I organize the tree anyway?

the tree is a simplistic model. There is simply too much dimensionality that can't be expressed in a flat tree. There's no way you can organize it so that will it satisfy all later needs. A tag space like structured_metrics makes it obsolete. with Graph-Explorer you can do (full-text) search on metric name, by any of their tags, and/or by added metadata. So practically you can filter by things like server, service, unit (e.g. anything expressed in bits/bytes per second, or anything denoting errors). All this irrespective of the source of a metric or the "location in the tree".

no interactivity with graphs

timeserieswidget allows you to easily add interactive graphite graph objects to html pages. You get modern features like togglable/reorderable metrics, realtime switching between lines/stacked, information popups on hoover, highlighting, smoothing, and (WIP) realtime zooming. It has a canvas (flot) and svg (rickshaw/d3) backend. So it basically provides a simpler api to use these libraries specifically with graphite.
There's a bunch of different graphite dashboards with different takes on graph composition/configuration and workflow, but the actual rendering of graphs usually comes down to plotting some graphite targets with a legend. timeserieswidget aims to be a drop-in plugin that brings all modern features so that different dashboards can benefit from a common, shared codebase, because static PNGs are a thing from the past

screenshot:

events lack text annotations, they are simplistic and badly supported

Graphite is a great system for time series metrics. Not for events. metrics and events are very different things across the board. drawAsInFinite() is a bit of a hack.

• anthracite is designed specifically to manage events.
  It brings extra features such as different submission scripts, outage annotations, various ways to see events and reports with uptime/MTTR/etc metrics.
• timeserieswidget displays your events on graphs along with their metadata (which can be just some text or even html code).
  this is where client side rendering shines

screenshots:

cumbersome to compose graphs

There's basically two approaches:

• interactive composing: with the graphite composer, you navigate through the tree and apply functions. This is painfull, dashboards like descartes and graphiti can make this easier
• use a dashboard that uses predefined templates (gdash and others) They often impose a strict navigation path to reach pages which may or may not give you the information you need (usually less or way more)

With both approaches, you usually end up with an ever growing pile of graphs that you created and then keep for reference.
This becomes unwieldy but is useful for various use cases and needs.
However, neither approach is convenient for changing information needs.
Especially when troubleshooting, one day you might want to compare the rate of increase of open file handles on a set of specific servers to the traffic on given network switches, the next day it's something completely different.
With Graph-Explorer:

• GE gives you a query interface on top of structured_metric's tag space. this enables a bunch of things (see above)
• you can yield arbitrary targets for each metric, to look at the same thing from a different angle (i.e. as a rate with `derivative()` or as a daily summary), and you can of course filter by angle
• You can group metrics into graphs by arbitrary tags (e.g. you can see bytes used of all filesystems on a graph per server, or compare servers on a graph per filesystem). This feature always results in the "wow that's really cool" every time I show it
• GE includes 'what' and 'target_type' in the group_by tags by default so basically, if things are in a different unit (B/s vs B vs b etc) it'll put them in separate graphs (controllable in query)
• GE automatically generates the graph title and vertical title (always showing the 'what' and the unit), and shows all metrics' extra tags. This also gives you a lot of inspiration to modify or extend your query

limited options to request a specific time range

GE's query language supports freeform `from` and `to` clauses.

Referenced projects

• anthracite:
  event/change logging/management with a bunch of ingestion scripts and outage reports
• timeserieswidget:
  jquery plugin to easily get highly interactive graphite graphs onto html pages (dashboards)
• structured_metrics:
  python library to convert graphite metrics tree into a tag space with clearly defined units and target types, and arbitrary metadata.
• graph-explorer:
  dashboard that provides a query language so you can easily compose graphs on the fly to satisfy varying information needs.

All tools are designed for integration with other tools and each other. Timeserieswidget gets data from anthracite, graphite and elasticsearch. Graph-Explorer uses structured_metrics and timeserieswidget.

Future work

There's a whole lot going on in the monitoring space, but I'd like to highlight a few things I personally want to work more on:

• I spoke with Michael Leinartas at Monitorama (and there's also a launchpad thread). We agreed that native tags in graphite are the way forward. This will address some of the pain points I'm already fixing with structured_metrics but in a more native way. I envision submitting metrics would move from:

  stats.serverdb123.mysql.queries.selects 895 1234567890
  

  to something more along these lines:

  host=serverdb123 service=mysql type=select what=queries target_type=rate 895 1234567890
  host=serverdb123 service=mysql type=select unit=Queries/s 895 1234567890
  h=serverdb123 s=mysql t=select queries r 895 1234567890
  

• switch Anthracite backend to ElasticSearch for native integration with logstash data (and allow you to use kibana)

04 Apr 2013 12:54pm GMT

Pacman 4.1 has just been released to [testing] with some fantastic and long awaited features, including:

• tight integration between the package manager and systemd

• colour support added

• checkupdates merged

• and more...

Check out Allan's blog post for all the details:
http://allanmcrae.com/2013/04/pacman-4-1-released/

01 Apr 2013 7:09am GMT

I have just released pacman-4.1 and packages are now in the [testing] repo. This is the first time I have made a release for any software project, so I was glad to have released a 4.1RC a few weeks back to learn everything that needed to be done. It has been over a year since [...]

01 Apr 2013 6:37am GMT

And we come to the end of another month. And not surprisingly, more stuff happened… Software news first: There was lots and lots and lots and lots of talk about Mir - Ubuntu's new anti-Wayland. And if that was not enough, here are more comments from people on the issue Speaking of Wayland, here is [...]

31 Mar 2013 3:57am GMT