12 Jul 2014

feedPlanet Arch Linux

MariaDB 10.0 enters [extra]

Bartłomiej Piotrowski wrote:

A new major release of MariaDB will be moved to [extra] soon. The change in versioning scheme has been made to clearly distinguish provided functionality from MySQL 5.6. From now on, it won't be possible to easily move between various MySQL implementations provided in the official repositories.

Due to major changes in MariaDB 10.0, it is recommended (although not necessary) to dump the tables before upgrading and reloading the dump file afterwards. After upgrading to the new version don't forget to restart mysqld.service and run mysql_upgrade to check the databases for possible errors.

Additionally TokuDB storage engine has been disabled because of repeating build failures. I'm sorry for any inconvenience caused.

For detailed information on changes and upgrade steps, please refer to MariaDB Knowledge Base and MySQL Reference Manual.

12 Jul 2014 2:38pm GMT

10 Jul 2014

feedPlanet Arch Linux

Install scripts

It is now almost exactly two years since the AIF was put out to pasture. At the time, it caused a degree of consternation, inexplicably causing some to believe that it presaged the demise of-if not Arch itself, then certainly the community around it. And I think it would be fair to say that it was the signal event that launched a number of spin-offs, the first of which from memory was Archbang; soon followed by a number of others that promised "Arch Linux with an easy installation," or something to that effect…

Of course, if you look back at the Installation Guide immediately before the move to the new scripts, for example the version that shipped with the last AIF in October, 2011, it is pretty evident that the current approach is a lot simpler. Sure, there is no curses GUI to step you through each part of the install but the introduction of pacstrap and arch-chroot meant that you no longer need those prompts.

There is also the added advantage that these scripts are useful outside the installation process itself; they can be used for system maintenance and, in the rare event that your recent bout of experimentation at 2am after a few drinks doesn't pan out the way you anticipated, repair.

One of the other responses to the new approach, however, has been the steady proliferation of "helpful" install scripts. These are essentially bash scripts that emulate the behaviour of the AIF and walk people through an automated install of their system. Well, not really their system, more accurately a system. So you run one of these scripts, answer a few prompts and then, when you reboot, you have a brand spanking new Arch Linux install running KDE with the full panoply of software and, in a few special cases, some customized dot files to "enhance" your experience.

From a certain perspective, I can see how these things appeal. "I wonder if I could script an entire install, from the partitioning right through to the desktop environment?" That sounds like a fun project, right? Where it all comes unstuck, unfortunately, is when the corollary thought appears that suggests sharing it with the wider community would be a good idea. It is at this point that a rigorous bout of self-examination about the outcomes that you are seeking and your base motivations for this act of selflessness are called for.

Whatever those motivations are, whether driven by altruism or the naked desire for fame and fortune that have-from time to time-alighted on these projects when they appear on /r/archlinux and the adoring throngs bestow their favours in equal measures of upvotes and bitcoin, they are grotesquely misplaced. No good comes from these things, I tell you; none.

Why not? Because, in the guise of being helpful, you are effectively depriving people of the single most important part of using Arch: building it themselves. It's like inviting someone to a restaurant for an amazing haute cuisine meal, sitting them down across the table from you and then them watching as the staff bring out a mouth-watering array of dishes, each of which you ostentatiously savour before vomiting it all back into their mouth.

Now, I am sure there is a small minority (well, at least from my own sheltered experience I imagine it is small) who would relish this scenario, but for most it would qualify as a pretty disappointing date.

Then, after the technicolour table d'hôte, there is the matter of the clean up. Recently, we had someone show up on the Arch boards who had "installed Arch" but who did not understand how to edit a text file; literally had no clue how to open a file like /etc/fstab make some changes and then save it. This is beyond stupid; it is a drain on the goodwill of the community that has to deal with this ineptitude, it is unfair on people to put them in a position where they feel they are at the mercy of their technology, rather than in control of it, and it does nothing to advance the interests of Arch.

If you want to write something like this to improve your scripting skills, by all means proceed. If you want to contribute to Arch, then pick a project to contribute to, some bugs to squash, wiki edits, whatever; just don't publish another one of these idiotic scripts, because you aren't doing anyone any favours, quite the contrary.

Notes

Flickr Creative Commons image, Measuring spoons by Theen Moy.

10 Jul 2014 9:49pm GMT

BBS, Wiki, and AUR maintenance

Florian Pritz wrote:

The BBS, Wiki and AUR will be moving to a new server starting today at 13:30 UTC.

Expected downtime is 2 hours, this post will be updated when the migration is complete.

Update: The new server is online and services seem to work just fine. Most nameservers should have already picked up the changes, but if you still can't connect (especially to the AUR) please wait a couple hours.

(If you want to check with host, dig or nslookup the new IPs are 5.9.250.164 and 2a01:4f8:160:3033::2)

10 Jul 2014 12:51pm GMT

04 Jul 2014

feedPlanet Arch Linux

The latest TalkingArch is out, born on the 4th of July

Hot off the presses for the 4th of July, we are pleased to bring you the latest TalkingArch iso. Rockin' the latest 3.15.3 Linux kernel and a new irssi configuration that connects new users to live humans ready to help get them set up, this TalkingArch promises to be better than ever. If you have [...]

04 Jul 2014 9:21pm GMT

29 Jun 2014

feedPlanet Arch Linux

Local Euler

Now with all 476 puzzles and images

29 Jun 2014 5:02am GMT

27 Jun 2014

feedPlanet Arch Linux

OpenVPN and Time on the Raspberry Pi

After relieving my Pi of seedbox duties, I was looking around for some other use for it. I decided, after looking over the Arch wiki article on OpenVPN, that the Pi would be a terrific VPN server; when I am out and about I can access a secure connection to my home network, thereby significantly reducing the risk of my privacy being compromised while using connectivity to the Internet provided by the notoriously security conscious sysadmins that run networks in hotels and other public places.

Setting it up was straightforward enough, the wiki is typically clear and thorough; the only bottleneck in the process was waiting for the Pi's tiny chip to chug through the creation of public keys. Once it was done, and I had tested that it was indeed working as intended, a more vexing issue presented itself. The service wouldn't come up after rebooting. Not a deal breaker, I could always just SSH in and manually bring the server up, but that sort of defeats the purpose of being able to have the thing running reliably.

The reason that it fails on boot is that, without a hardware clock, the Pi resets its time to UNIX time until the NTP daemon can start, which in turn depends upon the network being up. So, after rebooting, the journal would show the VPN server as having failed as the date was nearly half a century ago.

There are a variety of fixes floating around, the most amusing being a wrapper for init. Suffice to say, this wasn't an option for me. Looking at the problem logically, it occurred to me that the issue was actually a trivial one: the correct sequencing of different services post boot. Isn't this, I asked myself, one of the issues systemd was supposed to address?

I just had to ensure that the network came up as quickly as possible after boot, that the time was reset correctly once there was a viable connection, and that the openvpn.service waited for those things to happen before launching.

I have fitted the Pi with a wireless dongle, so the first step was to ditch netctl (the default network manager on the ARM image) and replace it with systemd-networkd. This is the point at which all the wingnuts that think that systemd is some sort of conspiracy to overthrow the old UNIX gods and replace them with false idols in chapeau rouge start foaming at their retrognathic mouths about "viral takeovers" and-seriously what fucking planet do these imbeciles hail from?-"an abhorrent and violent slap in the face to the Unix philosophy."1

For those of us that accept the theory of evolution, this technology is both effective and non-threatening; in fact, it is quite an improvement over its by now ageing predecessor. So, a few minutes later, /etc/systemd/network/wlan0.network and /etc/wpa_supplicant/wpa_supplicant-wlan0.conf had pretty much written themselves and then it was just a matter of enabling the eponymous services as well as systemd-resolved.service. Reboot and the network is up seemingly instantly.

Compounding my heresy, I then switched out NTP for systemd-timesyncd and the Pi's clock was now reset with the same alacrity. The final piece, ensuring that the openvpn service waited for this to happen, was to add two lines to the service file:

After=network-online.target
Wants=network-online.target

That is all there is to it. Reboot and the network comes up, the clock is reset and then the OpenVPN server starts. Like magic. The sort of heathen magic that threatens to sap and impurify all of our precious bodily fluids.

Notes

  1. No, Virginia, I did not make that up… And I don't really understand how you can slap a philosophy in the face, but then rationality is anathema to zealots; irrespective of which chimæra they prostrate before.

27 Jun 2014 9:57pm GMT

25 Jun 2014

feedPlanet Arch Linux

Recovering Windows 7 Backup on Windows 8.1

My wife's laptop recently died and was replaced. Unlike last time, there were regular backups with the last performed only a day or two before the failure. So all is fine… Right? Turns out not so much. The new laptop Continue reading

25 Jun 2014 11:56am GMT

24 Jun 2014

feedPlanet Arch Linux

Bringing back my data

The past days have been very "FOSS" for me.

Let's start from the beginning: Tuesday I managed (= it means I found money) to became a supporter of the FSFE! This is something that I had in mind since the last Linux Day (I'm sorry, that post is in Italian!) because I meet the FSFE Bari guys there and we had a nice chat.

I already knew the FreeYourAndroid campaign, but on Thursday I stumbled upon Nikon Roussos' post and he really motivated me to start replacing closed app with FOSS app on my phone.
Thanks to F-Droid after some hour I realized there's a good FOSS alternative for almost every application I use. Notably, I switched from Navigator to OsmAnd~, from Google Keep to Mirakel, from Google Translator to QuickDic, from Google Hangout (SMS) to TextSecure, from Twitter to Twidere.
Luckily I was already using Apollo, ConnectBot, Document Viewer, KeePassDroid, Barcode Scanner, Diode, Episodes...
The list was not so short as I thought initially and this scared me because there were too many Google apps. Yes, that's because it has the best services out there, but this means it had my location, my SMSs, my todos, my translations, my mails, my contacts, my events,... even my website stats! - I bet Google knows me better than I do.

On Monday, next step was to move away from Google Analytics. I did a quick search and found Piwik. The setup was simple and faster and it even allows (through a third-parthy script) to import your data from Google Analytics.

Piwik

Tuesday. It was time to remove the Dropbox client and switch to ownCloud. I never really used Dropbox to store my private files, I just did install the client because others co-workers did and we needed a quick way to work on a folder at the same time. Luckily I don't have to share a folder with co-workers anymore, and even if I had I would use the website instead - Yep, downloading and uploading the file everytime.
I used this day to remove that package from my system and setup an ownCloud server that I'm using for remote backups. Now I can put my private files on the cloud because I trust the service this time :-).

Only GMail and Google+ left and then I free-ed my Android.

Google+ is harder. I don't want to quit Google+ because of the FOSS development around it. It's a great place where to look for FOSS news, look for people reactions and even people issues. It's a real pity Pump.IO or even Diaspora* aren't at the same level. However, I always publish my staff as 'Public' audience, so Google doesn't really own anything.

On the contrary, GMail is a bit easier because I already use my aliases everywhere so I just need to open a Kolab account and update my forwarding rules. However we're all Gmail users now.

Let me say that I don't want to quit every closed service and use my own instance (it would be cool, but you know it isn't realistic), instead I guess we should look for a FOSS alternative first and in the case there's none we could split our data between many companies...and start making the FOSS implementation!

Oh I almost forgot, keep an eye upon the Tox Project! It's the free alternative to Skype/Hangout.

FOSS <3

24 Jun 2014 12:11am GMT

21 Jun 2014

feedPlanet Arch Linux

The #talkingarch IRC channel is moving

For the past few months, TalkingArch has had an IRC channel called #talkingarch on the Freenode network. This arrangement worked out pretty well, since the network was already set up. However, Kyle was recently invited to try running an IRC server linked with another, and enjoyed the experience so much that a new network was [...]

21 Jun 2014 4:54am GMT

13 Jun 2014

feedPlanet Arch Linux

Writing with Vim

Vim is not just an editor (and not in the way that Emacs is more than just an editor); it is for all intents and purposes a universal design pattern. The concept of using Vim's modes and keybinds extends from the shell through to file managers and browsers. If you so choose (and I do), then a significant amount of your interactions with your operating system are mediated by Vim's design principles. This is undeniably a good thing™ as it goes some way to standardising your command interface (whether at the command line or in a GUI).

To that end, I have been spending more time working with Vim and trying to improve both my understanding of it's capabilities, the environment in which I use it and how I can optimise both of those conditions to not necessarily just be more productive, but to minimise friction in my work flows.

A large part of my job involves writing and, happily, so does a good deal of what constitutes my leisure activity. Whether it is emails written in Mutt, these blog posts, longer documents written in LaTeX, or just roboposting to forums; it is Vim all the way down. So I have spent some effort setting up Vim to make that experience as comfortable as possible.

The first step, and one I took several years ago now, was to write custom colour schemes, depending on whether I am in the console or X. Several weeks ago, I came across a Vim plugin called DistractFree, which is described as "enabling a distraction free writing mode." I had always been slightly (well, perhaps scathingly would be more accurate) cynical when reading about these sorts of things when they first started to appear, but-after playing with two or three of them-this one has really grown on me (see the screenshot on the right).

I adapted my colour scheme for it, added a line to my ~/.vimrc to enable it for specific document types, and have not looked back.

autocmd BufRead <em>.markdown,</em>tex call DistractFree#DistractFreeToggle() | wincmd w

The final piece was to set up spell checking to work properly. As well as using English, I wanted to share my personal whitelist of words between all of my machines, so I define a location in ~/.vimrc as well:

set spelllang=en_gb                               " real English spelling
set dictionary+=/usr/share/dict/words             " use standard dictionary
set spellfile=$HOME/Sync/vim/spell/en.utf-8.add   " my whitelist

Then it is just a matter of ensuring that misspelled or other flagged words (like those that should be capitalised) are highlighted correctly. This required a couple of lines in my colour schemes:

" Spell checking  ---
if version >= 700
  hi clear SpellBad
  hi clear SpellCap
  hi clear SpellRare
  hi clear SpellLocal
  hi SpellBad    ctermfg=9
  hi SpellCap    ctermfg=3    cterm=underline
  hi SpellRare   ctermfg=13   cterm=underline
  hi SpellLocal  cterm=None
endif

The most significant change, however, was that I recently purchased a hard copy of Drew Neill's excellent Practical Vim. I have long been a fan of Drew's excellent podcast, Vimcasts, and the book is every bit as impressive as the collection of those episodes, and more. Reading it on my commute over the last week, I have been constantly amazed at the power and flexibility of this editor and Drew's ability to clearly articulate how to work economically and efficiently with it. I'm convinced this is an invaluable resource for anyone that currently uses Vim or wants to learn.

I expect that, over time, as I become more proficient with Vim, that I will adapt some of these settings, but for the time being they feel very comfortable and allow me to focus on hacking words together (and to do the occasional bit of scripting on the side)…

Notes

Image is from the cover of Drew Neill's Practical Vim, by Ben Cormack.

13 Jun 2014 9:30pm GMT

05 Jun 2014

feedPlanet Arch Linux

2014.06-2 archboot "2k14-R2" ISO hybrid image released

Hi Arch community,

Arch Linux (archboot creation tool) 2014.06-2 "2k14-R2" has been released.

Homepage and for more information on archboot:
http://wiki.archlinux.org/index.php/Archboot

Please get it from your favorite arch linux mirror:
https://downloads.archlinux.de/iso/archboot/2014.06
<yourmirror>/iso/archboot/2014.06/

Further documentation can be found on-disk and on the wiki.
Thanks to all testers, who reported bugs in last release.
Have fun!

greetings
tpowa

05 Jun 2014 1:19pm GMT

04 Jun 2014

feedPlanet Arch Linux

Perl updated to 5.20

Florian Pritz wrote:

Perl 5.20, as any other new perl version, requires all modules that are not purely perl code to be rebuilt. We did that for all packages in our repos.

The last couple major updates printed an error message when trying to load modules built against the wrong version, 5.20 seems to cause segfaults. Please make sure to rebuild all custom installed CPAN modules and binaries that link to libperl.so.

Refer to my post on arch-dev-public for a script that helps to find those packages.

04 Jun 2014 2:00pm GMT

02 Jun 2014

feedPlanet Arch Linux

The June 2014 TalkingArch iso is here

The TalkingArch team is pleased to announce the release of the June 2014 iso. This release includes version 3.14.4 of the linux kernel and all the latest base packages that are included on the official Arch Linux iso. As always, you can download the latest TalkingArch iso from its download page, either via BitTorrent or [...]

02 Jun 2014 3:32pm GMT

29 May 2014

feedPlanet Arch Linux

Browser tunnels

Using a Socks proxy over an SSH tunnel is a well documented and simple if much less flexible stand in for a full-blown VPN. It can provide a degree of comfort when accessing private or sensitive information over a public Internet connection, or you might use it to get around the terminally Canutian1 construct that is known as geo-blocking; that asinine practice of pretending that the Internet observes political boundaries…

By way of a digression, it occurred to me at some point while I was wrestling with setting this up that, over the last seven or so years, much of the "entertainment" provided by corporate content distributors has been in the form of encouraging me to spend hundreds? thousands? of hours researching and implementing ways to circumvent their litany of failed and defective technological restrictions: region codes, DRM and the like. It is worth noting that, in the vast majority of cases, I was just seeking access to content that I already owned (in another format), or was prepared to pay for.

My move to GNU/Linux in 2007 was in large part motivated by the awful realisation that the music I had bought in iTunes was stuck in there. The combined intellectual effort globally expended trying to legitimately route around broken copyright law would have comfortably powered another golden age of the sciences; it's not entirely implausible to think that the only reason we still have to deal with cancer is the malignant legacy of Sonny Bono2.

Now, back to our regular programming… One of my approaches to get around this sort of economic and policy rigor mortis has been to use a basic script to create a proxy tunnel to my home server. It assumes that you have public key authentication set up and your passphrase loaded in keychain, or something similar.

</p>

<h1>!/usr/bin/env bash</h1>

<p>SSH_HOST="jason@XXX.XXX.XXX.XXX -p XXX -i $HOME/.ssh/box1"</p>

<p>up(){</p>

<pre><code>ssh -f -N -D 8080 -M -S /tmp/ssh_tunnel_%h.sock -o ExitOnForwardFailure=yes $SSH_HOST &amp;&amp; \
printf '%s\n' "ssh tunnel started successfully" || \
printf '%s\n' "ssh tunnel failed to start"
</code></pre>

<p>}</p>

<p>down(){</p>

<pre><code>ssh -S /tmp/ssh_tunnel_%h.sock -O exit $SSH_HOST
</code></pre>

<p>}</p>

<p>if [[ $1 = up ]]; then</p>

<pre><code>up &amp;&amp; chromium --proxy-server="socks://127.0.0.1:8080" &amp;
</code></pre>

<p>elif [[ $1 = down ]]; then</p>

<pre><code>down
</code></pre>

<p>else</p>

<pre><code>printf '%s\n' "Fail…"
exit 1
</code></pre>

<p>fi

Over the last couple of weeks, though, while I have been setting up and playing with Syncthing, I found this script wanting. With six nodes and, depending if I was on the LAN or not, as many as four of those hosts only accessible via SSH, then having the ability to quickly and painlessly open a browser on any one of the nodes without having to edit the script suddenly seemed like quite a good idea.

Accordingly I went to work on the script, including a test to determine if I was on my home network and passing the name of the desired host as an argument. With this approach, I simply type tunnel $host and chromium opens tunneled to that host, where I can the happily open Hulu the Syncthing GUI.

The updated script is posted as a gist, and as you can see, still needs some work to make it a little more generic. You will need, for example, to hand edit in the hosts and ports in get_host(). It is also the first time I have played with named pipes and I am not convinced that my use of mkfifo here is either the correct approach or implementation; but it works. Comments enlightening me would be gratefully received.

Notes

  1. The good king was, appropriately enough, actually called Cnut the Great…
  2. And, no, I am not referring to his musical corpus, which is as carcinogenic as his political career was as a Myrmidon for Big Content.

Flickr Creative Commons image, The Tunnel, by Lawrence Whitmore.

29 May 2014 9:32pm GMT

Monitorama PDX & my metrics 2.0 presentation

Earlier this month we had another iteration of the Monitorama conference, this time in Portland, Oregon.


(photo by obfuscurity)

I think the conference was better than the first one in Boston, much more to learn. Also this one was quite focused on telemetry (timeseries metrics processing), lots of talks on timeseries analytics, not so much about things like sensu or nagios. Adrian Cockroft's keynote brought some interesting ideas to the table, like building a feedback loop into the telemetry to drive infrastructure changes (something we do at Vimeo, I briefly give an example in the intro of my talk) or shortening the time from fault to alert (which I'm excited to start working on soon)
My other favorite was Noah Kantrowitz's talk about applying audio DSP techniques to timeseries, I always loved audio processing and production. Combining these two interests hadn't occurred to me so now I'm very excited about the applications.
The opposite, but just as interesting of an idea - conveying information about system state as an audio stream - came up in puppetlab's monitorama recap and that seems to make a lot of sense as well. There's a lot of information in a stream of sound, it is much denser than text, icons and perhaps even graph plots. Listening to an audio stream that's crafted to represent various information might be a better way to get insights into your system.


I'm happy to see the idea reinforced that telemetry is a key part of modern monitoring. For me personally, telemetry (the tech and the process) is the most fascinating part of modern technical operations, and I'm glad to be part of the movement pushing this forward. There's also a bunch of startups in the space (many stealthy ones), validating the market. I'm curious to see how this will play out.


I had the privilege to present metrics 2.0 and Graph-Explorer. As usual the slides are on slideshare and the footage on the better video sharing platform ;-) .
I'm happy with all the positive feedback, although I'm not aware yet of other tools and applications adopting metrics 2.0, and I'm looking forward to see some more of that, because ultimately that's what will show if my ideas are any good.

29 May 2014 2:39pm GMT

18 May 2014

feedPlanet Arch Linux

On Graphite, Whisper and InfluxDB

Graphite, and the storage Achilles heel

Graphite is a neat timeseries metrics storage system that comes with a powerful querying api, mainly due to the whole bunch of available processing functions.
For medium to large setups, the storage aspect quickly becomes a pain point. Whisper, the default graphite storage format, is a simple storage format, using one file per metric (timeseries).
  • It can't keep all file descriptors in memory so there's a lot of overhead in constantly opening, seeking, and closing files, especially since usually one write comes in for all metrics at the same time.
  • Using the rollups feature (different data resolutions based on age) causes a lot of extra IO.
  • The format is also simply not optimized for writes. Carbon, the storage agent that sits in front of whisper has a feature to batch up writes to files to make them more sequential but this doesn't seem to help much.
  • Worse, due to various implementation details the carbon agent is surprisingly inefficient and cpu-bound. People often run into cpu limitations before they hit the io bottleneck. Once the writeback queue hits a certain size, carbon will blow up.
Common recommendations are to run multiple carbon agents and running graphite on SSD drives.
If you want to scale out across multiple systems, you can get carbon to shard metrics across multiple nodes, but the complexity can get out of hand and manually maintaining a cluster where nodes get added, fail, get phased out, need recovery, etc involves a lot of manual labor even though carbonate makes this easier. This is a path I simply don't want to go down.

These might be reasonable solutions based on the circumstances (often based on short-term local gains), but I believe as a community we should solve the problem at its root, so that everyone can reap the long term benefits.


In particular, running Ceres instead of whisper, is only a slight improvement, that suffers from most of the same problems. I don't see any good reason to keep working on Ceres, other than perhaps that it's a fun exercise. This probably explains the slow pace of development.
However, many mistakenly believe Ceres is "the future".
Switching to LevelDB seems much more sensible but IMHO still doesn't cut it as a general purpose, scalable solution.

The ideal backend

I believe we can build a backend for graphite that
  • can easily scale from a few metrics on my laptop in power-save mode to millions of metrics on a highly loaded cluster
  • supports nodes joining and leaving at runtime and automatically balancing the load across them
  • assures high availability and heals itself in case of disk or node failures
  • is simple to deploy. think: just run an executable that knows which directories it can use for storage, elasticsearch-style automatic clustering, etc.
  • has the right read/write optimizations. I've never seen a graphite system that is not write-focused, so something like LSM trees seems to make a lot of sense.
  • can leverage cpu resources (e.g. for compression)
  • provides a more natural model for phasing out data. Optional, runtime-changeable rollups. And an age limit (possibly, but not necessarily round robin)
While we're at it. pub-sub for realtime analytics would be nice too. Especially when it allows to use the same functions as the query api.
And getting rid of the metric name restrictions such as inability to use dots or slashes.

InfluxDB

There's a lot of databases that you could hook up to graphite. riak, hdfs based (opentsdb), Cassandra based (kairosdb, blueflood, cyanite), etc. Some of these are solid and production ready, and would make sense depending on what you already have and have experience with. I'm personally very interested in playing with Riak, but decided to choose InfluxDB as my first victim.

InfluxDB is a young project that will need time to build maturity, but is on track to meet all my goals very well. In particular, installing it is a breeze (no dependencies), it's specifically built for timeseries (not based on a general purpose database), which allows them to do a bunch of simplifications and optimizations, is write-optimized, and should meet my goals for scalability, performance, and availability well. And they're in NYC so meeting up for lunch has proven to be pretty fruitful for both parties. I'm pretty confident that these guys can pull off something big.

Technically, InfluxDB is a "timeseries, metrics, and analytics" databases with use cases well beyond graphite and even technical operations. Like the alternative databases, graphite-like behaviors such as rollups management and automatically picking the series in the most appropriate resolutions, is something to be implemented on top of it. Although you never know, it might end up being natively supported.

Graphite + InfluxDB

InfluxDB developers plan to implement a whole bunch of processing functions (akin to graphite, except they can do locality optimizations) and add a dashboard that talks to InfluxDB natively (or use Grafana), which means at some point you could completely swap graphite for InfluxDB. However, I think for quite a while, the ability to use the Graphite api, combine backends, and use various graphite dashboards is still very useful. So here's how my setup currently works:
  • carbon-relay-ng is a carbon relay in Go. It's a pretty nifty program to partition and manage carbon metrics streams. I use it in front of our traditional graphite system, and have it stream - in realtime - a copy of a subset of our metrics into InfluxDB. This way I basically have our unaltered Graphite system, and in parallel to it, InfluxDB, containing a subset of the same data.
    With a bit more work it will be a high performance alternative to the python carbon relay, allowing you to manage your streams on the fly. It doesn't support consistent hashing, because CH should be part of a strategy of a highly available storage system (see requirements above), using CH in the relay still results in a poor storage system, so there's no need for it.
  • I contributed the code to InfluxDB to make it listen on the carbon protocol. So basically, for the purpose of ingestion, InfluxDB can look and act just like a graphite server. Anything that can write to graphite, can now write to InfluxDB. (assuming the plain-text protocol, it doesn't support the pickle protocol, which I think is a thing to avoid anyway because almost nothing supports it and you can't debug what's going on)
  • graphite-api is a fork/clone of graphite-web, stripped of needless dependencies, stripped of the composer. It's conceived for many of the same reasons behind graphite-ng (graphite technical debt, slow development pace, etc) though it doesn't go to such extreme lengths and for now focuses on being a robust alternative for the graphite server, api-compatible, trivial to install and with a faster pace of development.
  • That's where graphite-influxdb comes in. It hooks InfluxDB into graphite-api, so that you can query the graphite api, but using data in InfluxDB. It should also work with the regular graphite, though I've never tried. (I have no incentive to bother with that, because I don't use the composer. And I think it makes more sense to move the composer into a separate project anyway).
With all these parts in place, I can run our dashboards next to each other - one running on graphite with whisper, one on graphite-api with InfluxDB - and simply look whether the returned data matches up, and which dashboards loads graphs faster. Later i might do more extensive benchmarking and acceptance testing.

If all goes well, I can make carbon-relay-ng fully mirror all data, make graphite-api/InfluxDB the primary, and turn our old graphite box into a live "backup". We'll need to come up with something for rollups and deletions of old data (although it looks like by itself influx is already more storage efficient than whisper too), and I'm really looking forward to the InfluxDB team building out the function api, having the same function api available for historical querying as well as realtime pub-sub. (my goal used to be implementing this in graphite-ng and/or carbon-relay-ng, but if they do this well, I might just abandon graphite-ng)

To be continued..

18 May 2014 5:22pm GMT