02 Sep 2015

feedPlanet Arch Linux

Transplanting Go packages for fun and profit

crazy Gopher scientist

A while back I read coders at work, which is a book of interviews with some great computer scientists who earned their stripes, the questions just as thoughtful as the answers. For one thing, it re-ignited my interest in functional programming, for another I got interested in literate programming but most of all, it struck me how common of a recommendation it was to read other people's code as a means to become a better programmer. (It also has a good section of Brad Fitzpatrick describing his dislike for programming languages, and dreaming about his ideal language. This must have been shortly before Go came about and he became a maintainer.)

I hadn't been doing a good job reading/studying other code out of fear that inferior patterns/style would rub off on me. But I soon realized that was an irrational, perhaps slightly absurd excuse. So I made the decision to change. Contrary to my presumption I found that by reading code that looks bad you can challenge and re-evaluate your mindset and get out with a more nuanced understanding and awareness of the pros and cons of various approaches.

I also realized if code is proving too hard to get into or is of too low quality, you can switch to another code base with negligible effort and end up spending almost all of your time reading code that is worthwhile and has plenty of learnings to offer. There is a lot of high quality Go code, easy to find through sites like Github or Golang weekly, just follow your interests and pick a project to start reading.

It gets really interesting though once you find bodies of code that are not only a nice learning resource, but can be transplanted into your code with minimal work to solve a problem you're having, but in a different context then the author of the code originally designed it for. Components often grow and mature in the context of an application without being promoted as reusable libraries, but you can often use them as if they were. I would like to share 2 such success cases below.

Nsq's diskqueue code

I've always had an interest in code that manages the same binary data both in memory and on a block device. Think filesystems, databases, etc. There's some interesting concerns like robustness in light of failures combined with optimizing for performance (infrequent syncs to disk, maintaining the hot subset of data in memory, etc), combined with optimizing for various access patterns, this can be a daunting topic to get into.

Luckily there's a use case that I see all the time in my domain (telemetry systems) and that covers just enough of the problems to be interesting and fun, but not enough to be overwhelming. And that is: for each step in a monitoring data pipeline, you want to be able to buffer data if the endpoint goes down, in memory and to disk if the amount of data gets too much. Especially to disk if you're also concerned with your software crashing or the machine power cycling.

This is such a common problem that applies to all metrics agents, relays, etc that I was longing for a library that just takes care of spooling data to disk for you without really affecting much of the rest of your software. All it needs to do is sequentially write pieces of data to disk and have a sequential reader catching up and read newer data as it finishes processing the older.

NSQ is a messaging platform from bitly, and it has diskqueue code that does exactly that. And it does so oh so elegantly. I had previously found a beautiful pattern in bitly's go code that I blogged about and again I found a nice and elegant design that builds further on this pattern, with concurrent access to data protected via a single instance of a for loop running a select block which assures only one piece of code can make changes to data at the same time (see bottom of the file), not unlike ioloops in other languages. And method calls such as Put() provide a clean external interface, though their implementation simply hooks into the internal select loop that runs the code that does the bulk of the work. Genius.

func (d *diskQueue) Put(data []byte) error {
  // some details
  d.writeChan <- data
  return <-d.writeResponseChan

In addition the package came with extensive tests and benchmarks out of the box.

After finding and familiarizing myself with this diskqueue code about a year ago I had an easy time introducing disk spooling to Carbon-relay-ng, by transplanting the code into it. The only change I had to make was capitalizing the Diskqueue type to export it outside of the package. It has proven a great fit, enabling a critical feature through little work of transplanting mature, battle-tested code into a context that original authors probably never thought of.

Note also how the data unit here is the []byte, the queue does not deal with the higher level nsq.Message (!). The authors had the foresight of keeping this generic, enabling code reuse and rightfully shot down a PR of mine that had a side effect of making the queue aware of the Message type. In NSQ you'll find thoughtful and deliberate api design and pretty sound code all around. Also, they went pretty far in detailing some lessons learned and providing concrete advice, a very interesting read, especially around managing goroutines & synchronizing their exits, and performance optimizations. At Raintank, we had a need for a messaging solution for metrics so we will so be rolling out NSQ as part of the raintank stack. This is an interesting case where my past experience with the NSQ code and ideas helped to adopt the full solution.

Bosun expression package

I'm a fan of the bosun alerting system which came out of Stack Exchange. It's a full-featured alerting system that solves a few problems like no other tool I've seen does (see my linked post), and timeseries data storage aside, comes with basically everything built in to the one program. I've used it with success. However, for litmus I needed an alerting handler that integrated well into the Grafana backend. I needed the ability to do arbitrarily complex computations. Graphite's api only takes you so far. We also needed (desired) reduction functions, boolean logic, etc. This is where bosun's expression language is really strong. I found the expression package quite interesting, they basically built their own DSL for metrics processing. so it deals with expression parsing, constructing AST's, executing them, dealing with types (potentially mixed types in the same expression), etc.

But bosun also has incident management, contacts, escalations, etc. Stuff that we either already had in place, or didn't want to worry about just yet. So we could run bosun standalone and talk to it as a service via its API which I found too loosely coupled and risky, hook all its code into our binary at once - which seemed overkill - or the strategy I chose: gradually familiarize ourself and adopt pieces of Bosun on a case by case basis, making sure there's a tight fit and without ever building up so much technical debt that it would become a pain to move away from the transplanted code if it becomes clear it's not/no longer well suited. For the foreseeable future we only need one piece, the expression package. Potentially ultimately we'll adopt the entire thing, but without the upfront commitment and investment.

So practically, our code now simply has one line where we create a bosun expression object from a string, and another where we ask bosun to execute the expression for us, which takes care of parsing the expression, querying for the data, evaluating and processing the results and distilling everything down into a final result. We get all the language features (reduction functions, boolean logic, nested expressions, …) for free.

This transplantation was again probably not something the bosun authors expected, but for us it was tremendously liberating. We got a lot of power for free. The only thing I had to do was spend some time reading code, and learning in the process. And I knew the code was well tested so we had zero issues using it.

Much akin to the NSQ example above, there was another reason the transplantation went so smoothly: the expression package is not tangled into other stuff. It just needs a string expression and a graphite instance. To be precise, any struct instance that satisfies the graphiteContext interface that is handily defined in the bosun code. While the bosun design aims to make its various clients (graphite, opentsdb, …) applicable for other projects, it also happens to let us do opposite: reuse some of its core code - the expression package - and pass in a custom graphite Context, such as our implementation which has extensive instrumentation. This lets us use the bosun expression package as a "black box" and still inject our own custom logic into the part that queries data from graphite. Of course, once we want to change the logic of anything else in the black box, we will need come up with something else, perhaps fork the package, but it doesn't seem like we'll need that any time soon.


If you want to become a better programmer I highly recommend you go read some code. There's plenty of good code out there. Pick something that deals with a topic that is of interest to you and looks mature. You typically won't know if code is good before you start reading but you'll find out really fast, and you might be pleasantly surprised, as was I, several times. You will learn a bunch, possibly pretty fast. However, don't go for the most advanced, complex code straight away. Pick projects and topics that are out of your comfort zone and do things that are new to you, but nothing too crazy. Once you truly grok those, proceed to other, possibly more advanced stuff.

Often you'll read reusable libraries that are built to be reused, or you might find ways to transplant smaller portions of code into your own projects. Either way is a great way to tinker and learn, and solve real problems. Just make sure the code actually fits in so you don't end up with the software version of Frankenstein's monster. It is also helpful to have the authors available to chat if you need help or have issues understanding something, though they might be surprised if you're using their code in a way they didn't envision and might not be very inclined to provide support to what they consider internal implementation details. So that could be a hit or miss. Luckily the people behind both nsq and bosun were supportive of my endeavors but I also made sure to try to figure out things by myself before bothering them. Another reason why it's good to pick mature, documented projects.

Gopher frankenstein

Part of the original meaning of hacking, extended into open source, is a mindset and practice of seeing how others solve a problem, discussion and building on top of it. We've gotten used to - and fairly good at - doing this on a project and library level but forgot about it on the level of code, code patterns and ideas. I want to see these practices come back to life.

We also apply this at Raintank: not only are we trying to build the best open source monitoring platform by reusing (and often contributing to) existing open source tools and working with different communities, we realize it's vital to work on a more granular level, get to know the people and practice cross-pollination of ideas and code.

Next stuff I want to read and possibly implement or transplant parts of: dgryski/go-trigram, armon/go-radix, especially as used in the dgryski/carbonmem server to search through Graphite metrics. Other fun stuff by dgryski: an implementation of the ARC caching algorithm and bloom filters. (you might want to get used to reading Wikipedia pages also). And mreiferson/wal, a write ahead log by one of the nsqd authors, which looks like it'll become the successor of the beloved diskqueue code.

Go forth and transplant!

Also posted on the Raintank blog

02 Sep 2015 4:25pm GMT

22 Aug 2015

feedPlanet Arch Linux

Python 3 Object-oriented Programming Second Edition

One of several reasons this blog has been so quiet this year is the time I've invested in the second edition of my first book. I am extremely proud of the end result. The first edition of Python 3 Object Oriented Programming was great and garnered 30 five star reviews on Amazon. This edition is […]

22 Aug 2015 4:43am GMT

14 Aug 2015

feedPlanet Arch Linux

openssh-7.0p1 deprecates ssh-dss keys

In light of recently discovered vulnerabilities, the new openssh-7.0p1 release deprecates keys of ssh-dss type, also known as DSA keys. See the upstream announcement for details.

Before updating and restarting sshd on a remote host, make sure you do not rely on such keys for connecting to it. To enumerate DSA keys granting access to a given account, use:

    grep ssh-dss ~/.ssh/authorized_keys

If you have any, ensure you have alternative means of logging in, such as key pairs of a different type, or password authentication.

Finally, host keys of ssh-dss type being deprecated too, you might have to confirm a new fingerprint (for a host key of a different type) when connecting to a freshly updated server.

14 Aug 2015 5:10am GMT

07 Aug 2015

feedPlanet Arch Linux

Firefox vulnerability

Hi everyone,

A serious vulnerability has been found in firefox yesterday (in the PDFjs module).
You will find more details at:
- https://www.mozilla.org/en-US/security/ … sa2015-78/
- https://blog.mozilla.org/security/2015/ … -the-wild/

The second link mentions that an exploit already existed before mozilla was informed about this problem, and this exploit was found to read the ssh files among others.
Archlinux repo already contains a fixed version of firefox (39.0.3-1).

Note for moderators: I am not sure wether the Archlinux forum is the right place for this topic, many security blogs and website have already published about this, but I thought quite a lot of us might be interested by this issue. Please feel free to close the topic if you wish.

07 Aug 2015 7:09pm GMT

28 Jul 2015

feedPlanet Arch Linux

Hardening Android’s Bionic libc

Good post by Daniel Micay on hardening Android's Bionic libc: https://copperhead.co/2015/07/27/hardening-bionic

28 Jul 2015 12:53pm GMT

20 Jul 2015

feedPlanet Arch Linux

Climbing in Gimmeldingen (2)

It wasn't the perfect weather for climbing yesterday. It was totally a contrary weather with sun, cloudy and in the morning hours some rain. Nevertheless we tried it in the evening and hoped that the rock will be dry enough for climbing. Luckily the rock was dry 1-2 meters above the ground.

We started with "Allgäu" (UIAA: 6-, French: 5b, USA: 5.8) to warm up. Last time we climbed it as last route, yesterday it was our first one. Really a nice route, but because of the rain it was a littble bit sloppy in the start section. On the left photo it's the route which starts on the left plate and goes up the corner in the wall.

Next route was "Sonnenaufgang (Sunrise)" (UIAA: 6+, French: 6a, USA: 5.10a), which is a little crack in the wall. You can see it in the left photo on the right side where the rope starts and if you follow up the rope you see the route. Very nice route, but it was a little bit too hard in the beginning for a 6+ (at least this was my impression). Therefor it will be easier in the last section, maybe that's the compensation for the hard start.

The third one for this evening was "Tugarez" (UIAA: 6-, French: 5b, USA: 5.8). The start and the last move is harder, the rest is more or less easy. It's a good route, but nothing special. You can see it in the right photo. Just follow the rope there and you see the route.

The last one has no name in our climbing guide, but it was rated with UIAA 5 (French: 4c, USA: 5.6), personally I would say it's a 5+ (French: 5a, USA: 5.7). Perfect route to end a climbing session in the evening, a short route with a relative easy difficulty.

It's not me on the photos, it's my climbing partner Christoph, just to mention it.

20 Jul 2015 6:49am GMT

16 Jul 2015

feedPlanet Arch Linux

Climbing in Gausbach

Yesterday I was climbing in Gausbach (Murg valley, Northern Black Forest). Gausbach has a nice climbing area with 56 routes and a height of 40 metres. The area is directly at an old and nearly unused road, no long walking distance from the parking place or train station.

I helped out my girlfriend who was climbing with her school class. I had to prepare the routes for top roping, so there was a wide range of difficulty.

So, we started with the easiest route there because the children weren't climbing on a rock before, which was "Komm mit Ski (Come with ski)" (UIAA: 3+, French: 3, USA: 5.3). A really really easy one, but perfect for those who weren't climbing on a rock before. The next ones were a little bit harder, "Signo di oro" (UIAA: 5-/5, French: 4c, USA: 5.6), "Toprope ahoi" (UIAA: 5-, French: 4c, USA: 5.6) and "Via bambini" (UIAA: 5, French: 4c, USA: 5.6). For some children we also prepared two harder routes: "Stairway to heaven" (UIAA: 5+, French: 5a, USA: 5.7) which is a very nice and long route (about 28 meters), "Komm mit Ski (links) (Come with ski (left alternative))" (UIAA: 6/6+, French: 5c/6a, USA: 5.9/5.10a) which is good route of slab climbing.

After the children are gone and the official part was over, I tried "Veni vidi vici" (UIAA: 7-, French: 6a+, USA: 5.10b). Sadly I had to abort it after 3/4 of the route. I think it was the key part of the route, or maybe not (I don't know it), but it was just a little bit too hard for me this day. I definitely have to go back there and try it again, I'm sure that this route isn't that complicated at this point.

A small hint at the end, climb in the morning hours, after midday you are climbing without any shadow and it will be definitely hot there.

16 Jul 2015 6:00am GMT

13 Jul 2015

feedPlanet Arch Linux

Bug squashing evening for Arch Linux

This evening I started my personal bug squashing day for Arch Linux. After a very long delay and no contribution the last several weeks for Arch Linux, I decided to start again and what will be better than fixing long outstanding bugs.

So, I decided to fix the following bugs:

All bugs are now fixed, at least I hope so. The dhcp package has now seperate configuration files for IPv4 and IPv6, also the systemd service files are referring to the correct configuration file.

MonoDevelop has now xsp as optional dependency and the Razor solution wizards works now and all necessary files are on its place.

mod_mono's configuration file is now located in the correct directory and must be changed in the httpd.conf file if you refer to the old directory.

Additionally I have updated the Mono package and MonoDevelop package to the latest stable release.

I hope that I will find another evening like today to do my next personal bug squashing day.

The next bug squashing day will come... ;-)

13 Jul 2015 9:41pm GMT

12 Jul 2015

feedPlanet Arch Linux

Anime Guide 2014

After many repeated requests (which was kind of surprising), here is my overview of the 2014 anime I watched. I am told that I was too negative in previous lists and I want to avoid that this time… probably not Continue reading

12 Jul 2015 12:54pm GMT

Rolling Release Pizza

/u/XSSpants on reddit: There's a lot to be said for rolling release pizza. (without going hardcore "pizza fountain" like Arch)

12 Jul 2015 4:19am GMT

10 Jul 2015

feedPlanet Arch Linux

Climbing in Gimmeldingen

Starting from today, I will write down some notes and information about my sport activities. This is more or less for my personal recording (no, I will not use a smartphone app for this) than for information. Nevertheless, there will also some public information which anyone can use from these posts. Maybe you will find new climbing places or hiking routes or whatever. Maybe you just enjoy to read my posts.

Yesterday I was climbing at the quarry of Gimmeldingen (near Neustadt an der Weinstraße, Rhineland-Palatinate) after work. The quarry is a sandstone climbing area with about 70 routes and a height of 20 metres.

In total we "just" climb three routes there, but all three routes are great for itself. We started with "Mitten im Winter (In the middle of winter)" which has a difficulty of 6 in the UIAA scale (in French it's a 5c and in USA (YDS) 5.9). The next route was "Ponyexpress" (UIAA: 6+, French: 6a, USA: 5.10a) which is a really nice crack and corner in the wall, you can see route in the left photo. I can really recommend this route, I have climbed it the first time yesterday, but it has become one of my favourites there already. The last route was "Allgäu" (UIAA: 6-, French: 5b, USA: 5.8). It's a great route to end a climbing session. It has one hard spot and the rest is more or less easy.

This climbing session was kind of after work climbing to end the day with some sport activity and to relax.

10 Jul 2015 7:49am GMT

08 Jul 2015

feedPlanet Arch Linux

July 2015 TalkingArch

The TalkingArch team is happy to announce the availability of the TalkingArch release for July 2015. This release includes Linux Kernel 4.0.7 and all the latest base and rescue packages. Additionally, there is a new release info section on the Download pabe,, which shows the download size, included Linux kernel and the current release date, [...]

08 Jul 2015 6:33pm GMT

03 Jul 2015

feedPlanet Arch Linux

Focusing on open source monitoring. Joining raintank.

Goodbye Vimeo

It's never been as hard saying goodbye to the people and the work environment as it is now.
Vimeo was created by dedicated film creators and enthusiasts, just over 10 years ago, and today it still shows. From the quirky, playful office culture, the staff created short films, to the tremendous curation effort and staff picks including monthly staff screenings where we get to see the best of the best videos on the Internet each month, to the dedication towards building the best platform and community on the web to enjoy videos and the uncompromising commitment to supporting movie creators and working in their best interest.
Engineering wise, there has been plenty of opportunity to make an impact and learn.
Nonetheless, I have to leave and I'll explain why. First I want to mention a few more things.

vimeo goodbye drink

In Belgium I used to hitchhike to and from work so that each day brought me opportunities to have conversations with a diverse, fantastic assortment of people. I still fondly remember some of those memories. (and it was also usually faster than taking the bus!)
Here in NYC this isn't really feasible, so I tried the next best thing. A mission to have lunch with every single person in the company, starting with those I don't typically interact with. I managed to have lunch with 95 people, get to know them a bit, find some gems of personalities and anecdotes, and have conversations on a tremendous variety of subjects, some light-hearted, some deep and profound. It was fun and I hope to be able to keep doing such social experiments in my new environment.

Vimeo is also part of my life in an unusually personal way. When I came to New York (my first ever visit to the US) in 2011 to interview, I also met a pretty fantastic woman in a random bar in Williamsburg. We ended up traveling together in Europe, I decided to move the US and we moved in together. I've had the pleasure of being submerged in both American and Greek culture for the last few years, but the best part is that today we are engaged and I feel like the luckiest guy in the world. While I've tried to keep work and personal life somewhat separate, Vimeo has made an undeniable ever lasting impact on my life that I'm very grateful for.

At Vimeo I found an area where a bunch of my interests converge: operational best practices, high performance systems, number crunching, statistics and open source software. Specifically, timeseries metrics processing in the context of monitoring. While I have enjoyed my opportunity to make contributions in this space to help our teams and other companies who end up using my tools, I want to move out of the cost center of the company, I want to be in the department that creates the value. If I want to focus on open source monitoring, I should align my incentives with those of my employer. Both for my and their sake. I want to make more profound contributions to the space. The time has come for me to join a company for which the main focus is making open source monitoring better.

Hello raintank!

Over the past two years or so I've talked to many people in the industry about monitoring, many of them trying to bring me into their team. I never found a perfect fit but as we transitioned from 2014 into 2015, the stars seemingly aligned for me. Here's why I'm very excited to join the raintank crew:

OK, so what am I really up to?

Grafana is pretty much the leading open source metrics dashboard right now. So it only makes sense that raintank is a heavy Grafana user and contributor. My work, logically, revolves around codifying some of the experience and ideas I have, and making them accessible through the polished interface that is Grafana, which now also has a full time UX designer working on it. Since according to the Grafana user survey alerting is the most sorely missed non-feature of Grafana, we are working hard on rectifying this and it is my full-time focus. If you've followed my blog you know I have some thoughts on where the sweet spot lies in clever alerting. In short, take the claims of anomaly detection via machine learning with a big grain of salt, and instead, focus on enabling operators to express complex logic simply, quickly, and in an agile way. My latest favorite project, bosun exemplifies this approach (highly recommend giving this a close look).

The way I'm thinking of it now, the priorities (and sequence of focus) for alerting within Grafana will probably be something like this:

There's a lot of thought work, UX and implementation details around this topic, I've created a github ticket to kick off a discussion and am curious to hear your thoughts. Finally, if any of this sounds interesting to you, you can sign up to the grafana newsletter or the raintank newsletter which will get you info on the open source platform as well as the SaaS product. Both are fairly low volume.

office sausolito It may look like I'm not doing much from my temporary Mill Valley office, but trust me, cool stuff is coming!

03 Jul 2015 4:22pm GMT

02 Jul 2015

feedPlanet Arch Linux

Moved blog to hugo, fastly and comma

02 Jul 2015 11:35pm GMT

18 Jun 2015

feedPlanet Arch Linux

TalkingArch June 2015

It's been a rocky road, but it's actually been out for some time. It's the TalkingArch iso for June 2015. The delay in the posting of the release has been due to a server OS upgrade that took some rather unexpected turns, resulting in breakage of the GNU Social node that hosts the social media [...]

18 Jun 2015 1:55am GMT

08 Jun 2015

feedPlanet Arch Linux

The Case of GCC-5.1 and the Two C++ ABIs

Recently, Arch Linux updated to gcc-5.1. This brought a lot of new features, but the one that I am focusing on today is the new C++ ABI that appears when the compiler is built using default option. Supporting the C++11 Continue reading

08 Jun 2015 12:58am GMT