19 May 2016

feedPlanet Twisted

Hynek Schlawack: Conditional Python Dependencies

Since the inception of wheels that install Python packages without executing arbitrary code, we need a static way to encode conditional dependencies for our packages. Thanks to PEP 508 we do have a blessed way but sadly the prevalence of old setuptools versions makes it a minefield to use.

19 May 2016 12:00am GMT

18 May 2016

feedPlanet Twisted

Twisted Matrix Laboratories: Twisted 16.2 Released

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 16.2!

Just in time for PyCon US, this release brings a few headlining features (like the haproxy endpoint) and the continuation of the modernisation of the codebase. More Python 3, less deprecated code, what's not to like?

For more information, check the NEWS file (link provided below).

You can find the downloads on PyPI (or alternatively our website). The NEWS file is also available on GitHub.

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,
Amber Brown (HawkOwl)

18 May 2016 5:10pm GMT

06 May 2016

feedPlanet Twisted

Moshe Zadka: Forking Skip-level Dependencies

I have recently found I explain this concept over and over to people, so I want to have a reference.

Most modern languages comes with a "dependency manager" of sorts that helps manage the 3rd party libraries a given project uses. Rust has Cargo, Node.js has npm, Python has pip and so on. All of these do some things well and some things poorly. But one thing that can be done (well or poorly) is "support forking skip-level dependencies".

In order to explain what I mean, here as an example: our project is PlanetLocator, a program to tell the user which direction they should face to see a planet. It depends on a library called Astronomy. Astronomy depends on Physics. Physics depends on Math.

PlanetLocator is a SaaS, running on our servers. One day, we find Math has a critical bug, leading to a remote execution vulnerability. This is pretty bad, because it can be triggered via our application by simply asking PlanetLocator for the location of Mars at a specific date in the future. Luckily, the bug is simple - in Math's definition of Pi, we need to add a couple of significant digits.

How easy is it to fix?

Well, assume PlanetLocator is written in Go, and not using any package manager. A typical import statement in PlanetLocator is

import "github.com/astronomy/astronomy"

A typical import statement in Astronomy is

import "github.com/physics/physics

..and so on.

We fork Math over to "github.com/planetlocator/math" and fix the vulnerability. Now we have to fork over physics to use the forked math, and astronomy to use the forked physics and finally, change all of our imports to import the forked astronomy - and Physics, Astronomy and PlanetLocator had no bugs!

Now assume, instead, we had used Python. In our requirements.txt file, we could put


and voila! even though Physics' "setup.py" said "install_requires=['math']", it will get our forked math.

When starting to use a new language/dependency manager, the first question to ask is: will it support me forking skip-level dependencies? Because every upstream maintainer is, effectively, an absent-maintainer if rapid response is at stake (for any reason - I chose security above, but it might be beating the competition to a deadline, or fulfilling contractual obligations).

06 May 2016 4:17am GMT

03 May 2016

feedPlanet Twisted

Glyph Lefkowitz: Letters To The Editor: Re: Email

Since I removed comments from this blog, I've been asking y'all to email me when you have feedback, with the promise that I'd publish the good bits. Today I'm making good on that for the first time, with this lovely missive from Adam Doherty:

I just wanted to say thank you. As someone who is never able to say no, your article on email struck a chord with me. I have had Gmail since the beginning, since the days of hoping for an invitation. And the day I received my invitation was the the last day my inbox was ever empty.

Prior to reading your article I had over 40,000 unread messages. It used to be a sort of running joke; I never delete anything. Realistically though was I ever going to do anything with them?

With 40,000 unread messages in your inbox, you start to miss messages that are actually important. Messages that must become tasks, tasks that must be completed.

Last night I took your advice; and that is saying something - most of the things I read via HN are just noise. This however spoke to me directly.

I archived everything older than two weeks, was down to 477 messages and kept pruning. So much of the email we get on a daily basis is also noise. Those messages took me half a second to hit archive and move on.

I went to bed with zero messages in my inbox, woke up with 21, archived 19, actioned 2 and then archived those.

Seriously, thank you so very much. I am unburdened.

First, I'd like to thank Adam for writing in. I really do appreciate the feedback.

Second, I wanted to post this here not in service of showcasing my awesomeness1, but rather to demonstrate that getting to the bottom of your email can have a profound effect on your state of mind. Even if it's a running joke, even if you don't think it's stressing you out, there's a good chance that, somewhere in the back of your mind, it is. After all, if you really don't care, what's stopping you from hitting select all / archive right now?

At the very least, if you did that, your mail app would load faster.

  1. although, let there be no doubt, I am awesome

03 May 2016 6:06am GMT

27 Apr 2016

feedPlanet Twisted

Itamar Turner-Trauring: How you should choose which technology to learn next

Keeping up with the growing software ecosystem - new databases, new programming languages, new web frameworks - becomes harder and harder every year as more and more software is written. It is impossible to learn all existing technologies, let alone the new ones being released every day. If you want to learn another programming language you can choose from Dart, Swift, Go, Idris, Futhark, Ceylon, Zimbu, Elm, Elixir, Vala, OCaml, LiveScript, Oz, R, TypeScript, PureScript, Haskell, F#, Scala, Dylan, Squeak, Julia, CoffeeScript... and about a thousand more, if you're still awake. This stream of new technologies can be overwhelming, a constant worry that your skills are getting rusty and out of date.

Luckily you don't need to learn all technologies, and you are likely to use only a small subset during your tenure as a programmer. Instead your goal should be to maximize your return on investment: learn the most useful tools, with the least amount of effort. How then should you choose which technologies to learn?

Don't spend too much time on technologies which are either too close or too far from your current set of knowledge. If you are an expert on PostgreSQL then learning another relational database like MySQL won't teach you much. Your existing knowledge is transferable for the most part, and you'd have no trouble applying for a job requiring MySQL knowledge. On the other hand a technology that is too far from your current tools will be much more difficult to learn, e.g. switching from web development to real-time embedded devices.

Focus on technologies that can build on your existing knowledge while still being different enough to teach you something new. Learning these technologies provides multiple benefits:

There are three ways you can build on your existing knowledge of tools and technologies:

  1. Branch out to nearby technologies: If you're a backend web developer you are interacting with a database, with networking via the HTTP protocol, with a browser running Javascript. You will end up knowing at least a little bit about these technologies, and you have some sense of how they interact with the technology you already know well. These are all great candidates for a new technology to learn next.
  2. Alternative solutions for a problem you understand: If you are an expert on the PostgreSQL database you might want to learn MongoDB. It's still a database, solving a problem whose parameters you already understand: how to store and search structured data. But the way MongoDB solves this problem is fundamentally different than PostgreSQL, which means you will learn a lot.
  3. Enhance your usage of existing tools: Tools for testing your existing technology stack can make you a better programmer by providing faster feedback and a broader view of software quality and defects. Learning how to better use a sophisticated text editor like Emacs/Vim or an IDE like Eclipse with your programming language of choice can make you a more productive programmer.

Neither you nor any other programmer will ever be able to learn all the technologies in use today: there are just too many. What you can and should do is learn those that will help with your current projects, and those that you can learn more easily. The more technologies you know, the broader the range of technologies you have at least partial access to, and the easier it will be to learn new ones.

27 Apr 2016 4:00am GMT

24 Apr 2016

feedPlanet Twisted

Glyph Lefkowitz: Email Isn’t The Thing You’re Bad At

I've been using the Internet for a good 25 years now, and I've been lucky enough to have some perspective dating back farther than that. The common refrain for my entire tenure here:

We all get too much email.

A New, New, New, New Hope

Luckily, something is always on the cusp of replacing email. AOL instant messenger will totally replace it. Then it was blogging. RSS. MySpace. Then it was FriendFeed. Then Twitter. Then Facebook.

Today, it's in vogue to talk about how Slack is going to replace email. As someone who has seen this play out a dozen times now, let me give you a little spoiler:

Slack is not going to replace email.

But Slack isn't the problem here, either. It's just another communication tool.

The problem of email overload is both ancient and persistent. If the problem were really with "email", then, presumably, one of the nine million email apps that dot the app-stores like mushrooms sprouting from a globe-spanning mycelium would have just solved it by now, and we could all move on with our lives. Instead, it is permanently in vogue1 to talk about how overloaded we all are.

If not email, then what?

If you have twenty-four thousand unread emails in your Inbox, like some kind of goddamn animal, what you're bad at is not email, it's transactional interactions.

Different communication media have different characteristics, but the defining characteristic of email is that it is the primary mode of communication that we use, both professionally and personally, when we are asking someone else to perform a task.

Of course you might use any form of communication to communicate tasks to another person. But other forms - especially the currently popular real-time methods - appear as a bi-directional communication, and are largely immutable. Email's distinguishing characteristic is that it is discrete; each message is its own entity with its own ID. Emails may also be annotated, whether with flags, replied-to markers, labels, placement in folders, archiving, or deleting. Contrast this with a group chat in IRC, iMessage, or Slack, where the log is mostly2 unchangeable, and the only available annotation is "did your scrollbar ever move down past this point"; each individual message has only one bit of associated information. Unless you have catlike reflexes and an unbelievably obsessive-compulsive personality, it is highly unlikely that you will carefully set the "read" flag on each and every message in an extended conversation.

All this makes email much more suitable for communicating a task, because the recipient can file it according to their system for tracking tasks, come back to it later, and generally treat the message itself as an artifact. By contrast if I were to just walk up to you on the street and say "hey can you do this for me", you will almost certainly just forget.

The word "task" might seem heavy-weight for some of the things that email is used for, but tasks come in all sizes. One task might be "click this link to confirm your sign-up on this website". Another might be "choose a time to get together for coffee". Or "please pass along my resume to your hiring department". Yet another might be "send me the final draft of the Henderson report".

Email is also used for conveying information: here are the minutes from that meeting we were just in. Here is transcription of the whiteboard from that design session. Here are some photos from our family vacation. But even in these cases, a task is implied: read these minutes and see if they're accurate; inspect this diagram and use it to inform your design; look at these photos and just enjoy them.

So here's the thing that you're bad at, which is why none of the fifty different email apps you've bought for your phone have fixed the problem: when you get these messages, you aren't making a conscious decision about:

  1. how important the message is to you
  2. whether you want to act on them at all
  3. when you want to act on them
  4. what exact action you want to take
  5. what the consequences of taking or not taking that action will be

This means that when someone asks you to do a thing, you probably aren't going to do it. You're going to pretend to commit to it, and then you're going to flake out when push comes to shove. You're going to keep context-switching until all the deadlines have passed.

In other words:

The thing you are bad at is saying 'no' to people.

Sometimes it's not obvious that what you're doing is saying 'no'. For many of us - and I certainly fall into this category - a lot of the messages we get are vaguely informational. They're from random project mailing lists, perhaps they're discussions between other people, and it's unclear what we should do about them (or if we should do anything at all). We hang on to them (piling up in our Inboxes) because they might be relevant in the future. I am not advocating that you have to reply to every dumb mailing list email with a 5-part action plan and a Scrum meeting invite: that would be a disaster. You don't have time for that. You really shouldn't have time for that.

The trick about getting to Inbox Zero3 is not in somehow becoming an email-reading machine, but in realizing that most email is worthless, and that's OK. If you're not going to do anything with it, just archive it and forget about it. If you're subscribed to a mailing list where only 1 out of 1000 messages actually represents something you should do about it, archive all the rest after only answering the question "is this the one I should do something about?". You can answer that question after just glancing at the subject; there are times when checking my email I will be hitting "archive" with a 1-second frequency. If you are on a list where zero messages are ever interesting enough to read in their entirety or do anything about, then of course you should unsubscribe.

Once you've dug yourself into a hole with thousands of "I don't know what I should do with this" messages, it's time to declare email bankruptcy. If you have 24,000 messages in your Inbox, let me be real with you: you are never, ever going to answer all those messages. You do not need a smartwatch to tell you exactly how many messages you are never going to reply to.

We're In This Together, Me Especially

A lot of guidance about what to do with your email addresses email overload as a personal problem. Over the years of developing my tips and tricks for dealing with it, I certainly saw it that way. But lately, I'm starting to see that it has pernicious social effects.

If you have 24,000 messages in your Inbox, that means you aren't keeping track or setting priorities on which tasks you want to complete. But just because you're not setting those priorities, that doesn't mean nobody is. It means you are letting availability heuristic - whatever is "latest and loudest" - govern access to your attention, and therefore your time. By doing this, you are rewarding people (or #brands) who contact you repeatedly, over inappropriate channels, and generally try to flood your attention with their priorities instead of your own. This, in turn, creates a culture where it is considered reasonable and appropriate to assume that you need to do that in order to get someone's attention.

Since we live in the era of subtext and implication, I should explicitly say that I'm not describing any specific work environment or community. I used to have an email startup, and so I thought about this stuff very heavily for almost a decade. I have seen email habits at dozens of companies, and I help people in the open source community with their email on a regular basis. So I'm not throwing shade: almost everybody is terrible at this.

And that is the one way that email, in the sense of the tools and programs we use to process it, is at fault: technology has made it easier and easier to ask people to do more and more things, without giving us better tools or training to deal with the increasingly huge array of demands on our time. It's easier than ever to say "hey could you do this for me" and harder than ever to just say "no, too busy".

Mostly, though, I want you to know that this isn't just about you any more. It's about someone much more important than you: me. I'm tired of sending reply after reply to people asking to "just circle back" or asking if I've seen their email. Yes, I've seen your email. I have a long backlog of tasks, and, like anyone, I have trouble managing them and getting them all done4, and I frequently have to decide that certain things are just not important enough to do. Sometimes it takes me a couple of weeks to get to a message. Sometimes I never do. But, it's impossible to be mad at somebody for "just checking in" for the fourth time when this is probably the only possible way they ever manage to get anyone else to do anything.

I don't want to end on a downer here, though. And I don't have a book to sell you which will solve all your productivity problems. I know that if I lay out some incredibly elaborate system all at once, it'll seem overwhelming. I know that if I point you at some amazing gadget that helps you keep track of what you want to do, you'll either balk at the price or get lost fiddling with all its knobs and buttons and not getting a lot of benefit out of it. So if I'm describing a problem that you have here, here's what I want you to do.

Step zero is setting aside some time. This will probably take you a few hours, but trust me; they will be well-spent.

Email Bankruptcy

First, you need to declare email bankruptcy. Select every message in your Inbox older than 2 weeks. Archive them all, right now. In the past, you might have to worry about deleting those messages, but modern email systems pretty much universally have more storage than you'll ever need. So rest assured that if you actually need to do anything with these messages, they'll all be in your archive. But anything in your Inbox right now older than a couple of weeks is just never going to get dealt with, and it's time to accept that fact. Again, this part of the process is not about making a decision yet, it's just about accepting a reality.

Mailbox Three

One extra tweak I would suggest here is to get rid of all of your email folders and filters. It seems like many folks with big email problems have tried to address this by ever-finer-grained classification of messages, ever more byzantine email rules. At least, it's common for me, when looking over someone's shoulder to see 24,000 messages, it's common to also see 50 folders. Probably these aren't helping you very much.

In older email systems, it was necessary to construct elaborate header-based filtering systems so that you can later identify those messages in certain specific ways, like "message X went to this mailing list". However, this was an incomplete hack, a workaround for a missing feature. Almost all modern email clients (and if yours doesn't do this, switch) allow you to locate messages like this via search.

Your mail system ought to have 3 folders:

  1. Inbox, which you process to discover tasks,
  2. Drafts, which you use to save progress on replies, and
  3. Archive, the folder which you access only by searching for information you need when performing a task.

Getting rid of unnecessary folders and queries and filter rules will remove things that you can fiddle with.

Moving individual units of trash between different heaps of trash is not being productive, and by removing all the different folders you can shuffle your messages into before actually acting upon them you will make better use of your time spent looking at your email client.

There's one exception to this rule, which is filters that do nothing but cause a message to skip your Inbox and go straight to the archive. The reason that this type of filter is different is that there are certain sources or patterns of message which are not actionable, but rather, a useful source of reference material that is only available as a stream of emails. Messages like that should, indeed, not show up in your Inbox. But, there's no reason to file them into a specific folder or set of folders; you can always find them with a search.

Make A Place For Tasks

Next, you need to get a task list. Your email is not a task list; tasks are things that you decided you're going to do, not things that other people have asked you to do5. Critically, you are going to need to parse e-mails into tasks. To explain why, let's have a little arithmetic aside.

Let's say it only takes you 45 seconds to go from reading a message to deciding what it really means you should do; so, it only takes 20 seconds to go from looking at the message to remembering what you need to do about it. This means that by the time you get to 180 un-processed messages that you need to do something about in your Inbox, you'll be spending an hour a day doing nothing but remembering what those messages mean, before you do anything related to actually living your life, even including checking for new messages.

What should you use for the task list? On some level, this doesn't really matter. It only needs one really important property: you need to trust that if you put something onto it, you'll see it at the appropriate time. How exactly that works depends heavily on your own personal relationship with your computers and devices; it might just be a physical piece of paper. But for most of us living in a multi-device world, something that synchronizes to some kind of cloud service is important, so Wunderlist or Remember the Milk are good places to start, with free accounts.

Turn Messages Into Tasks

The next step - and this is really the first day of the rest of your life - start at the oldest message in your Inbox, and work forward in time. Look at only one message at a time. Decide whether this message is a meaningful task that you should accomplish.

If you decide a message represents a task, then make a new task on your task list. Decide what the task actually is, and describe it in words; don't create tasks like "answer this message". Why do you need to answer it? Do you need to gather any information first?

If you need to access information from the message in order to accomplish the task, then be sure to note in your task how to get back to the email. Depending on what your mail client is, it may be easier or harder to do this6, but in the worst case, following the guidelines above about eliminating unnecessary folders and filing in your email client, just put a hint into your task list about how to search for the message in question unambiguously.

Once you've done that:

Archive the message immediately.

The record that you need to do something about the message now lives in your task list, not your email client. You've processed it, and so it should no longer remain in your inbox.

If you decide a message doesn't represent a task, then:

Archive the message immediately.

Do not move on to the next message until you have archived this message. Do not look ahead7. The presence of a message in your Inbox means you need to make a decision about it. Follow the touch-move rule with your email. If you skip over messages habitually and decide you'll "just get back to it in a minute", that minute will turn into 4 months and you'll be right back where you were before.

Circling back to the subject of this post; once again, this isn't really specific to email. You should follow roughly the same workflow when someone asks you to do a task in a meeting, or in Slack, or on your Discourse board, or wherever, if you think that the task is actually important enough to do. Note the slack timestamp and a snippet of the message so you can search for it again, if there is a relevant attachment. The thing that makes email different is really just the presence of an email box.

Banish The Blue Dot

Almost all email clients have a way of tracking "unread" messages; they cheerfully display counters of them. Ignore this information; it is useless. Messages have two states: in your inbox (unprocessed) and in your archive (processed). "Read" vs. "Unread" can be, at best, of minimal utility when resuming an interrupted scanning session. But, you are always only ever looking at the oldest message first, right? So none of the messages below it should be unread anyway...

Be Ruthless

As you try to start translating your flood of inbound communications into an actionable set of tasks you can actually accomplish, you are going to notice that your task list is going to grow and grow just as your Inbox was before. This is the hardest step:

Decide you are not going to do those tasks, and simply delete them. Sometimes, a task's entire life-cycle is to be created from an email, exist for ten minutes, and then have you come back to look at it and then delete it. This might feel pointless, but in going through that process, you are learning something extremely valuable: you are learning what sorts of things are not actually important enough to do you do.

If every single message you get from some automated system provokes this kind of reaction, that will give you a clue that said system is wasting your time, and just making you feel anxious about work you're never really going to get to, which can then lead to you un-subscribing or filtering messages from that system.

Tasks Before Messages

To thine own self, not thy Inbox, be true.

Try to start your day by looking at the things you've consciously decided to do. Don't look at your email, don't look at Slack; look at your calendar, and look at your task list.

One of those tasks, probably, is a daily reminder to "check your email", but that reminder is there more to remind you to only do it once than to prevent you from forgetting.

I say "try" because this part is always going to be a challenge; while I mentioned earlier that you don't want to unthinkingly give in to availability heuristic, you also have to acknowledge that the reason it's called a "cognitive bias" is because it's part of human cognition. There will always be a constant anxious temptation to just check for new stuff; for those of us who have a predisposition towards excessive scanning behavior have it more than others.

Why Email?

We all need to make commitments in our daily lives. We need to do things for other people. And when we make a commitment, we want to be telling the truth. I want you to try to do all these things so you can be better at that. It's impossible to truthfully make a commitment to spend some time to perform some task in the future if, realistically, you know that all your time in the future will be consumed by whatever the top 3 highest-priority angry voicemails you have on that day are.

Email is a challenging social problem, but I am tired of email, especially the user interface of email applications, getting the blame for what is, at its heart, a problem of interpersonal relations. It's like noticing that you get a lot of bills through the mail, and then blaming the state of your finances on the colors of the paint in your apartment building's mail room. Of course, the UI of an email app can encourage good or bad habits, but Gmail gave us a prominent "Archive" button a decade ago, and we still have all the same terrible habits that were plaguing Outlook users in the 90s.

Of course, there's a lot more to "productivity" than just making a list of the things you're going to do. Some tools can really help you manage that list a lot better. But all they can help you to do is to stop working on the wrong things, and start working on the right ones. Actually being more productive, in the sense of getting more units of work out of a day, is something you get from keeping yourself healthy, happy, and well-rested, not from an email filing system.

You can't violate causality to put more hours into the day, and as a frail and finite human being, there's only so much work you can reasonably squeeze in before you die.

The reason I care a lot about salvaging email specifically is that it remains the best medium for communication that allows you to be in control of your own time, and by extension, the best medium for allowing people to do creative work.

Asking someone to do something via SMS doesn't scale; if you have hundreds of unread texts there's no way to put them in order, no way to classify them as "finished" and "not finished", so you need to keep it to the number of things you can fit in short term memory. Not to mention the fact that text messaging is almost by definition an interruption - by default, it causes a device in someone's pocket to buzz. Asking someone to do something in group chat, such as IRC or Slack, is similarly time-dependent; if they are around, it becomes an interruption, and if they're not around, you have to keep asking and asking over and over again, which makes it really inefficient for the asker (or the asker can use a @highlight, and assume that Slack will send the recipient, guess what, an email).

Social media often comes up as another possible replacement for email, but its sort order is even worse than "only the most recent and most frequently repeated". Messages are instead sorted by value to advertisers or likeliness to increase 'engagement'", i.e. most likely to keep you looking at this social media site rather than doing any real work.

For those of us who require long stretches of uninterrupted time to produce something good - "creatives", or whatever today's awkward buzzword for intersection of writers, programmers, graphic designers, illustrators, and so on, is - we need an inbound task queue that we can have some level of control over. Something that we can check at a time of our choosing, something that we can apply filtering to in order to protect access to our attention, something that maintains the chain of request/reply for reference when we have to pick up a thread we've had to let go of for a while. Some way to be in touch with our customers, our users, and our fans, without being constantly interrupted. Because if we don't give those who need to communicate with such a tool, they'll just blast @everyone messages into our slack channels and @mentions onto Twitter and texting us Hey, got a minute? until we have to quit everything to try and get some work done.

Questions about this post?

Go ahead and send me an email.


As always, any errors or bad ideas are certainly my own.

First of all, Merlin Mann, whose writing and podcasting were the inspiration, direct or indirect, for many of my thoughts on this subject; and who sets a good example because he won't answer your email.

Thanks also to David Reid for introducing me to Merlin's work, as well as Alex Gaynor, Tristan Seligmann, Donald Stufft, Cory Benfield, Piët Delport, Amber Brown, and Ashwini Oruganti for feedback on drafts.

  1. Email is so culturally pervasive that it is literally in Vogue, although in fairness this is not a reference to the overflowing-Inbox problem that I'm discussing here.

  2. I find the "edit" function in Slack maddening; although I appreciate why it was added, it's easy to retroactively completely change the meaning of an entire conversation in ways that make it very confusing for those reading later. You don't even have to do this intentionally; sometimes you make a legitimate mistake, like forgetting the word "not", and the next 5 or 6 messages are about resolving that confusion; then, you go back and edit, and it looks like your colleagues correcting you are a pedantic version of Mr. Magoo, unable to see that you were correct the first time.

  3. There, I said it. Are you happy now?

  4. Just to clarify: nothing in this post should be construed as me berating you for not getting more work done, or for ever failing to meet any commitment no matter how casual. Quite the opposite: what I'm saying you need to do is acknowledge that you're going to screw up and rather than hold a thousand emails in your inbox in the vain hope that you won't, just send a quick apology and move on.

  5. Maybe you decided to do the thing because your boss asked you to do it and failing to do it would cost you your job, but nevertheless, that is a conscious decision that you are making; not everybody gets to have "boss" priority, and unless your job is a true Orwellian nightmare, not everything your boss says in email is an instant career-ending catastrophe.

  6. In Gmail, you can usually just copy a link to the message itself. If you're using OS X's Mail.app, you can use this Python script to generate links that, when clicked, will open the Mail app:

    from __future__ import (print_function, unicode_literals,
                            absolute_import, division)
    from ScriptingBridge import SBApplication
    import urllib
    mail = SBApplication.applicationWithBundleIdentifier_("com.apple.mail")
    for viewer in mail.messageViewers():
        for message in viewer.selectedMessages():
            for header in message.headers():
                name = header.name()
                if name.lower() == "message-id":
                    content = header.content()
                    print("message:" + urllib.quote(content))

    You can then paste these links into just about any task tracker; if they don't become clickable, you can paste them into Safari's URL bar or pass them to the open command-line tool.

  7. The one exception here is that you can look ahead in the same thread to see if someone has already replied.

24 Apr 2016 11:54pm GMT

Moshe Zadka: Use virtualenv

In a conversation recently with a friend, we agreed that "something the instructions tell you to do 'sudo pip install'…which is good, because then you know to ignore them".

There is never a need for "sudo pip install", and doing it is an anti-pattern. Instead, all installation of packages should go into a virtualenv. The only exception is, of course, virtualenv (and arguably, pip and wheel). I got enough questions about this that I wanted to write up an explanation about the how, why and why the counter-arguments are wrong.

What is virtualenv?

The documentation says:

virtualenv is a tool to create isolated Python environments.

The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into/usr/lib/python2.7/site-packages (or whatever your platform's standard location is), it's easy to end up in a situation where you unintentionally upgrade an application that shouldn't be upgraded.

Or more generally, what if you want to install an application and leave it be? If an application works, any change in its libraries or the versions of those libraries can break the application.

The tl:dr; is:

The first problem is the one the "sudo" comment addresses - but the real issues stem from the second and third: not using a virtual environment leads to the potential of conflicts and dependency hell.

How to use virtualenv?

Creating a virtual environment is easy:

$ virtualenv dirname

will create the directory, if it does not exist, and then create a virtual environment in it. It is possible to use it either activated or unactivated. Activating a virtual environment is done by

$ . dirname/bin/activate

this will make python, as well as any script installed using setuptools' "console_scripts" option in the virtual environment, on the command-execution path. The most important of those is pip, and so using pip will install into the virtual environment.

It is also possible to use a virtual environment without activating it, by directly calling dirname/bin/python or any other console script. Again, pip is an example of those, and used for installing into the virtual environment.

Installing tools for "general use"

I have seen a couple of times the argument that when installing tools for general use it makes sense to install them into the system install. I do not think that this is a reasonable exception for two reasons:

There are a few good alternatives for this:

Exploratory programming

People often use Python for exploratory programming. That's great! Note that since pip 7, pip is building and caching wheels by default. This means that creating virtual environments is even cheaper: tearing down an environment and building a new one will not require recompilation. Because of that, it is easy to treat virtual environments as disposable except for configuration: activate a virtual environment, explore - and whenever needing to move things into production, 'pip freeze' will allow easy recreation of the environment.

24 Apr 2016 4:54am GMT

20 Apr 2016

feedPlanet Twisted

Glyph Lefkowitz: Far too many things can stop the BLOB

It occurs to me that the lack of a standard, well-supported, memory-efficient interface for BLOBs in multiple programming languages is one of the primary driving factors of poor scalability characteristics of open source SaaS applications.

Applications like Gitlab, Redmine, Trac, Wordpress, and so on, all need to store potentially large files ("attachments"). Frequently, they elect to store these attachments (at least by default) in a dedicated filesystem directory. This leads to a number of tricky concurrency issues, as the filesystem has different (and divorced) concurrency semantics from the backend database, and resides only on the individual API nodes, rather than in the shared namespace of the attached database.

Some databases do support writing to BLOBs like files. Postgres, SQLite, and Oracle do, although it seems MySQL lags behind in this area (although I'd love to be corrected on this front). But many higher-level API bindings for these databases don't expose support for BLOBs in an efficient way.

Directly using the filesystem, as opposed to a backing service, breaks the "expected" scaling behavior of the front-end portion of a web application. Using an object store, like Cloud Files or S3, is a good option to achieve high scalability for public-facing applications, but that creates additional deployment complexity.

So, as both a plea to others and a note to myself: if you're writing a database-backed application that needs to store some data, please consider making "store it in the database as BLOBs" an option. And if your particular database client library doesn't support it, consider filing a bug.

20 Apr 2016 1:01am GMT

15 Apr 2016

feedPlanet Twisted

Itamar Turner-Trauring: Improving your skills as a 9 to 5 programmer

Do you only code 9 to 5, but wonder if that's good enough? Do you see other programmers working on personal projects or open source projects, going to hackathons, and spending all their spare time writing software? You might think that as someone who only writes software at their job, who only works 9-5, you will never be as good. You might believe that only someone who eats, sleeps and breathes code can excel. But actually it's possible to stick to a 40-hour week and still be a valuable, skilled programmer.

Working on personal or open source software projects doesn't automatically make you better programmer. Hackathons might even be a net negative if they give you the impression that building software to arbitrary deadlines while exhausted is a reasonable way to produce anything of value. There are inherent limits to your productive working hours. If you don't feel like spending more time coding when you get home, then don't: you'll be too tired or unfocused to gain anything.

Spending time on side projects does have some value, but the most useful result is not so much practice as knowledge. Established software projects tend to use older technology and techniques, simply because they've been in existence for a while. The main value you get from working on other software projects and interacting with developers outside of work is knowledge of:

  1. A broader range of technologies and tools.
  2. New techniques and processes. Perhaps your company doesn't do much testing, but you can learn about test-driven development elsewhere.

Having a broad range of tools and techniques to reach for is a valuable skill both at your job and when looking for a new job. But actual coding is not an efficient way to gain this knowledge. You don't actually need to use new tools and techniques, and you'll never really have to time to learn all tools and all techniques in detail anyway. You get the most value just from having some sense of what tools and techniques are out there, what they do and when they're useful. If a new tool you discover is immediately relevant to your job you can just learn it during working hours, and if it's not you can should just file it away in your brain for later.

Learning about new tools can also help you find a new job, even when you don't actually use them. I was once asked at an interview about the difference between NoSQL and traditional databases. At the time I'd never used MongoDB or any other NoSQL database, but I knew enough to answer satisfactorily. Being able to answer that question told the interviewer I'd be able to use that tool, if necessary, even if I hadn't done it before.

Instead of coding in your spare time you can get similar benefits, and more efficiently, by directly focusing on acquiring knowledge of new tools and techniques. And since this knowledge will benefit your employer and you don't need to spend significant time on it, you can acquire it during working hours. You're never actually working every single minute of your day, you always have some time when you're slacking off on the Internet. Perhaps you're doing so right now! You can use that time to expand your knowledge.

Each week you should allocate one hour of your time at work to learning about new tools and techniques. Choosing a particular time will help you do this on a regular basis. Personally I'd choose Friday afternoons, since by that point in the week I'm not achieving much anyway. Don't skip this hour just because of deadlines or tiredness. You'll do better at deadlines, and be less tired, if you know of the right tools and techniques to efficiently solve the problems you encounter at your job.

15 Apr 2016 4:00am GMT

13 Apr 2016

feedPlanet Twisted

Glyph Lefkowitz

I think I'm using GitHub wrong.

I use a hodgepodge of https: and : (i.e. "ssh") URL schemes for my local clones; sometimes I have a remote called "github" and sometimes I have one called "origin". Sometimes I clone from a fork I made and sometimes I clone from the upstream.

I think the right way to use GitHub would instead be to always fork first, make my remote always be "origin", and consistently name the upstream remote "upstream". The problem with this, though, is that forks rapidly fall out of date, and I often want to automatically synchronize all the upstream branches.

Is there a script or a github option or something to synchronize a fork with upstream automatically, including all its branches and tags? I know there's no comment field, but you can email me or reply on twitter.

13 Apr 2016 9:11pm GMT

04 Apr 2016

feedPlanet Twisted

Twisted Matrix Laboratories: Twisted 16.1 Released

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 16.1!

This release is hot off the heels of 16.0 released last month, including some nice little tidbits. The highlights include:

For more information, check the NEWS file (link provided below).

You can find the downloads on PyPI (or alternatively our website). The NEWS file is also available on GitHub.

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,
Amber Brown (HawkOwl)

04 Apr 2016 5:14pm GMT

Itamar Turner-Trauring: Vanquish whiteboard interview puzzles with test-driven development

Whiteboard coding puzzles are of course utterly terrifying and totally unrealistic, which does not recommend them as an interview procedure. Yet they are still commonly used, which means the next time you interview for a programming job you might find yourself asked to solve an algorithmic problem with a 30 minute deadline, no text editor and a stranger who will decide your future employment staring at you expectantly. Personally this is where I start to panic, just a little: am I fraud? Can I really write software? I can, in fact, and so can you, and one of the techniques you can use to develop software will also allow you to go from a blank whiteboard to an impressed interviewer: test-driven development.

The most important thing you need to understand is this: the puzzle is a distraction. You want to spend the next 30 minutes impressing the interviewer. Ideally you will also solve the puzzle, but that is not your goal; your goal is to impress the interviewer so you can get a job offer. And if you do it right, sometimes you can even fail to solve the puzzle and still succeed at your goal.

Simply trying to solve the puzzle may fail to impress your interviewer:

Interviewer: ... and so you need to find the optimal route for the salesman between all the cities.
You: OK, so... <stare at the whiteboard>
Interviewer: I'm getting bored. Here's a hint!
You: <write some code>... OK, maybe that works?
Interviewer: Foolish interviewee, you have missed a bug that is obvious to me since I have given this puzzle to dozens of other candidates, most of whom were more impressive than you.
You: Let me fix that.
Interviewer: You are out of time. Next up, lunch interview!

Impressing the interviewer requires more: you need to show off your thinking and process, and take control of the conversation. The details are important, but at a high level you must follow a four-step process:

  1. Explain the steps of your process to the interviewer.
  2. Write some test cases on the whiteboard.
  3. Try to implement a solution.
  4. Run all the test cases against the implementation. If the implementation fails the tests go back to step 3.

Step 1: Explain what you're about to do

You don't want the interviewer interrupting you mid-thought, or deciding you're floundering. The first thing you do then is explain to the interviewer the process you are about to follow: writing tests, then doing a first pass implementation, then validating the implementation against the tests, then fixing the bugs you find.

Step 2: Write some tests

Write a series of test cases on the whiteboard: this input gives that output. Start with simple edge cases, and try to find some more interesting cases as well. Thinking through the transformations will help you get a better sense of how to solve the problem, and writing them on the whiteboard will demonstrate progress to the interviewer. Make sure to narrate your thought process.

Step 3: Initial implementation

Start solving the puzzle by writing code. To ensure you don't get interrupted too soon, remind the interviewer that next you will be testing the code with your pre-written tests. If you get stuck, return to your test cases and see how you can change the code to do what they suggest. Make sure to narrate your thought process.

Step 4: Testing and bug fixing

For each test case, compare the output you expect to what your current code does. This will help you catch bugs on your own and again demonstrate progress to your interviewer. If you find a bug then rewrite the code to fix it, and then start testing again.

Let's listen to an interview that follows these four steps; your code may be the same, but you sound more confident and professional:

Interviewer: ... and so you need to find the optimal route for the salesman between all the cities.
You: Let me explain how I will go about solving this. First, I will write down some test cases. Second, I will write a first pass implementation; it will probably be buggy, so third, I will run the test cases against my implementation to find the bugs and fix them. Interviewer: OK, sounds good.
You: <write some test cases>... OK, next I will write a first pass of code. Remember that when I'm done I won't be finished yet, I still will need to run my tests.
Interviewer: <This candidate will write high-quality code!>
You: <write some code>... and now to run my test cases. Feed this input, x is multiplied, run through the loop... this one gives expected result! OK, next test case... hm, that doesn't match. Let me see... ah! OK, that's fixed, let's run the tests again...
Interviewer: Unfortunately we are out of time, but I can see where you're going with this.

Next time you're interviewing for a programming job remember these four steps: explain, test, code, debug with tests. You will face your next puzzle armed with a process, helping you overcome your anxiety. By writing tests you are more likely to solve the puzzle correctly. And most importantly you will impress your interviewer, and maybe even get a job offer.

04 Apr 2016 4:00am GMT

26 Mar 2016

feedPlanet Twisted

Moshe Zadka: Weak references, caches and immutable objects

Consider the following situation:

We often would like to cache the function. As an example, consider a function to serialize an object - if the same objects serialized several times, we would like to avoid recomputing the serialization.

One naive solution would be to implement a cache:

cache = {}
def serialize(obj):
    if obj not in cache:
        cache[obj] = _really_serialize(obj)
    return cache[obj]

The problem with that is that the cache would keep references to our objects long after they should have died. We can try and use an LRU (for example, repoze.lru) so that only a certain number of objects would extend their lifetimes in that way, but the size of the LRU would trade-off space overhead and time overhead.

An alternative is to use weak references. Weak references are references that do not keep objects from being collected. There are several ways to use weak references, but here one is ideally suited:

import weakref
cache = weakref.WeakKeyDictionary()
def serialize(obj):
    if obj not in cache:
        cache[obj] = _really_serialize(obj)
    return cache[obj]

Note that this is the same code as before - except that the cache is a weak key dictionary. A weak key dictionary keeps weak references to the keys, but strong references to the value. When a key is garbage collected, the entry in the dictionary disappears.

>>> import weakref
>>> a=weakref.WeakKeyDictionary()
>>> fs = frozenset([1,2,3])
>>> a[fs] = "three objects"
>>> print a[fs]
three objects
>>> len(a)
>>> fs = None
>>> len(a)

26 Mar 2016 4:36am GMT

22 Mar 2016

feedPlanet Twisted

Itamar Turner-Trauring: Learning from the programmers who built your tools

The software you use was written by developers just like you. They made mistakes just like you do, learned and improved just like you do. But you have a huge advantage: you can learn from the mistakes and discoveries they have already made. I've discussed comparing and contrasting a single task across multiple alternative technologies, using resource cleanup idioms of C++, Go and Python as an example. Another technique for gaining technical depth is examining a single technology and seeing how its support for a single task evolved over time. As an example, let's consider resource cleanup in Python.

To recap the previous post, resource cleanup is a problem any programming language needs to solve: you've opened a file, and eventually you will need to close it. However, in the interim your code might return, or throw an exception. You want to have that file cleaned up no matter what, so you don't leak resources:

def write():
    f = open("myfile", "w")
    # If this throws an exception, e.g. when disk is full,
    # then f.close() will never be run:

In Python the clean up idiom started as an extension to the exception handling syntax:

def write():
    f = open("myfile", "w")

Whether the code in the try block returns, throws an exception or continues, the finally block will always be called.

What problems can we spot in this idiom? Why would the Python developers try to improve on it? Let's make a list; notice that these are all focused on solving problems with humans, not with computers:

  1. You still have to remember the particular function name to clean up each kind of resource, e.g. close() for files and release() for locks.
  2. It's repetitive: every single time you open a file you have to call close() on it, meaning more code to write and more code to read.
  3. The cleanup code happens long after the resource is initialized, interrupting your flow of reading the code.

The Python developers eventually came up with a new, improved language feature that solves these problems:

def write():
    with open("myfile", "w") as f:

This solves all three problems:

  1. Each resource knows how to clean itself up.
  2. Clean up is done automatically, no need to explicitly call the method.
  3. Clean up is done at the right time, but without extra code to read.

Can this be improved? I think so. Consider the following example:

>>> with open("/tmp/file", "w") as f:
...     f.write("hello")
>>> print(f)
<_io.TextIOWrapper name='/tmp/file' mode='w' encoding='UTF-8'>

Even though the the file has been closed, the resource has been cleaned up, the variable referring to the object persists outside the with block. This is "namespace pollution", extra unnecessary variables being added that can potentially introduce bugs. Better if f only existed inside the with block.

By examining the improvements to a particular feature you can take advantage of all the hard work the developers put into coming with alternatives: instead of just learning from the latest version you can see how their thinking evolved over time. You can then try to come up with improvements on your own, to exercise your new understanding. While I've examined a language feature in this post you can apply the same technique to any form of technology. Pick your favorite database, for example, and a feature that has evolved over time: why did it change, and what could be improved?

The developers who wrote the software you use had to find their solutions the hard way. You should respect their work by building on what they have learned.

22 Mar 2016 4:00am GMT

19 Mar 2016

feedPlanet Twisted

Itamar Turner-Trauring: Book Review: Become a better learner by discovering "How Learning Works"

Learning is never easy, but it is even harder when you are learning on your own. When in school you have teachers to rely on, when you are new to the profession you have experienced programmers to guide you. But eventually you reach a point where you have to learn on your own, without help or support. You must then know how to teach yourself, which is to say you must first learn how to teach. To acquire such a skill you would need an expert on human learning, and one who is also an able teacher. And if such a teacher were not available the next best thing would be a book that they wrote, a book summarizing what they know and how you can apply it.

"How Learning Works" is the unexpected child of the two somewhat conflicting goals of academia: research and teaching. Research universities value research far more than teaching, even as teaching is required of faculty. But some faculty do care about teaching, and some scientists study learning and teaching. This book is the result of intersection of those two interests: practical principles for teaching based on what scientists have discovered about learning. And these principles can also be applied by learners themselves, as the book's conclusion points out, learners like me and you.

The book is organized around seven principles, reviewing the relevant academic research and then providing practical teaching advice based on the findings. The evidence for the principles is quickly summarized and for the most part is fairly plausible, with the usual caveats about the difficulty of such research. And since the book is driven by scientific evidence its lessons often go far beyond naive common sense. While discussing how students organize their knowledge the book discusses research regarding "expert blind spot." While experts are much better at organizing knowledge, their very different ways of understanding can actually make it more difficult for them to teach novices. This is one reason why you will hear so much seemingly contradictory advice on subjects like testing: experts often assume the scope and limitations of their advice are obvious, even when they're not.

While the book is organized around high-level principles, the explanations and advice it gives are detailed, specific and very hands-on. When discussing the principle of targeted feedback, for example, the book reviews research suggesting:

  1. "Feedback is more effective when it identifies particular aspects [students need to improve]."
  2. "Too much feedback tends to overwhelm students."
  3. "Even minimal feedback can lead to better results."
  4. Immediate feedback can be less helpful than delayed feedback.

The chapter then follows up with at least eight different ways to improve the relevance, frequency and timeliness of feedback. E.g. the book suggests asking students to to explain how they applied feedback in later work, a suggestion well-worth following even without a teacher's requirement. The detailed breakdown and suggestions are just one part of the overall principle the chapter covers; an additional section covers more research and corresponding advice. The advice itself quite obviously comes from practiced teachers, although it is somewhat over-focused on academic teaching involving a large class and a semester schedule. Even so, I have found the book full of advice and ideas relevant to far more than just academic teaching.

Both the principles covered and the resulting advice show up continuously in the previous blog posts I've written. For example, I previously talked about how providing the solutions you've come up with is important when asking for help. We can see how some of the principles discussed in the book apply in this situation. First, "prior knowledge can help or hinder learning." By providing your solutions you give your respondent a much more detailed understanding of what you know and how it affected your approach to solving the problem. When they provide feedback by helping you solve the problem they will be able to tailor it to your particular understanding, as in the principle mentioned earlier which stresses the importance of targeted feedback.

"How Learning Works" is a wonderful way to understand how you learn and how to improve your learning. It has helped me immensely in writing this blog, by making me a better teacher and better learner. I hope it will also help you take the next step in learning on your own. Grab a copy from your local library or buy it from Amazon.

19 Mar 2016 4:00am GMT

Itamar Turner-Trauring: An apology

I previously promised I would post a review of a book on writing, but I've had to reject my original planned recommendation. Meanwhile I've written a review of another book that is well worth your while, How Learning Works. Enjoy!

19 Mar 2016 4:00am GMT