20 Mar 2017

feedPlanet Twisted

Itamar Turner-Trauring: Dear recruiter, "open floor space" is not a job benefit

I was recently astonished by a job posting for a company that touted their "open floor space." Whoever wrote the ad appeared to sincerely believe that an open floor office was a reason to work for the company, to be mentioned alongside a real benefit like work/life balance.

While work/life balance and an open floor plan can co-exist, an open floor plan is a management decision much more akin to requiring long hours: both choose control over productivity.

The fundamental problem facing managers is that productivity is hard to measure. Faced with the inability to measure productivity, managers may feel compelled to measure time spent working. Never mind that it's counter-productive: at least it gives management control, even if it's control over the wrong thing.

Here is a manager explaining the problem:

I [would] like to manage [the] team's output rather than managing their time, because if they are forced to spend time inside the office, it doesn't mean they are productive or even working. At the same time it's hard to manage output, because software engineering tasks are hard to estimate and things can go out of the track easily.

In this case at least the manager involved understands that what matters is output, not hours in the office. But not everyone realizes is as insightful.

Choosing control

In cases where management does fall into the trap of choosing control over productivity, the end result is a culture where the only thing that matters is hours in the office. Here's a story I heard from a friend about a startup they used to work at:

People would get in around 8 or 9, because that's when breakfast is served. They work until lunch, which is served in the office, then until dinner, which is served in the office. Then they do social activities in the office, video games or board games, and then keep working until 10PM or later. Their approach was that you can bring your significant other in for free dinner, and therefore why leave work? Almost like your life is at the office.

Most of the low level employees, especially engineers, didn't feel that this was the most productive path. But everyone knew this was the most direct way to progress, to a higher salary, to becoming a team lead. The number of hours in the office is a significant part of how your performance is rated.

I don't think people can work 12 hours consistently. And when you're not working and you're in an open plan office, you distract people. It's not about hours worked, it's about hours in office, there's ping pong tables... so there's always someone asking you to play ping pong or distracting you with a funny story. They're distracting you, their head wasn't in the zone, but they had to be in the office.

A team of 10 achieved what a team of 3 should achieve.

Control through visibility

Much like measuring hours in the office, an open floor office is designed for control rather than productivity. A manager can easily see what each developer is doing: are they coding? Are they browsing the web? Are they spending too much time chatting to each other?

In the company above the focus on working hours was apparently so strong that the open floor plan was less relevant. But I've no doubt there are many companies where you'll start getting funny looks from your manager if you spend too much time appearing to be "unproductive."

To be fair, sometimes open floor plans are just chosen for cheapness, or thoughtlessly copied from other companies because all the cool kids are doing it. Whatever the motivation, they're still bad for productivity. Programming requires focus, and concentrated thought, and understanding complex systems that cannot fit in your head all at once and so constantly need to be swapped in and out.

Open floor spaces create exactly the opposite of the environment you need for programming: they're full of noise and distraction, and headphones only help so much. I've heard of people wearing hoodies to block out not just noise but also visual distraction.

Dear recruiter

Work/life balance is a real job benefit, for both employers and employees: it increases productivity while allowing workers space to live outside their job. But "open office space" is not a benefit for anyone.

At worst it means your company is perversely sabotaging its employees in order to control them better. At best it means your company doesn't understand how to enable its employees to do their job.

20 Mar 2017 4:00am GMT

Moshe Zadka: Shipping Python Applications in Docker

Posted on my new blog as an experiment!

20 Mar 2017 1:36am GMT

12 Mar 2017

feedPlanet Twisted

Itamar Turner-Trauring: Unit testing, Lean Startup, and everything in-between

This is a preliminary version of a talk I'll be giving at PyCon 2017. I would appreciate feedback, comments and contrasting points of view.

Why do we test our software?

I used to think this was in order to produce high-quality code: you should always test you code because you always want to write high-quality code. But this viewpoint has some problems.

Sometimes quality is besides the point. In his book "Lean Startup" Eric Ries talks about building software only to discover that no one actually wanted to use it. This was one of his motivations to develop a better methodology for creating startups, a way to figure out if a product will succeed before putting the time into building a quality product. Ensuring high quality is a waste of time if no one will ever use your software.

Even if quality is necessary, the connection between quality and testing is vague. How does testing by a QA team differ from an automated unit test? They're obviously quite different, but exactly what kind of quality does each give? When should you use a particular kind of testing?

In addition, tests have costs: how can you decide if the costs outweigh the benefits? For example, consider a company that writes tax preparation software (I've changed the actual details a little.) They wrote Selenium tests for their web UI... but their application was still buggy, and every time they changed the UI the tests would break. The tests didn't seem to improve quality, and they wasted developer time maintaining them. What did they do wrong?

Saying we should all just write high-quality software doesn't help address these issues. So let's take a step back and think more deeply about testing.

What does it mean to test?

The best English dictionary tells us that to test is "to put to the proof; to prove the truth, genuineness, or quality of by experiment, or by some principle or standard." Quality is in there, yes, but there's much more to it than that.

This is only an English definition, to be sure, and programming is not limited to English-speakers. But I don't mean to argue that a dictionary definition can dictate what we ought to do. Human language is an accumulation of centuries of looking at the world and trying to understand it, a repository of ideas that we can draw on.

Let's take this definition as a starting point and see what we can learn.

The first dimension of testing

Is the following code a test?

def test_add():
    assert add(2, 2) == 5

I would say that, yes, that's clearly a test. Says so right in the function name, even. The test proves that add() does what it ought to do: add two numbers and give us the result.

You've noticed, of course, that this test is wrong. Luckily our development process has another step: code review. You, dear reader, can act as a code reviewer and tell me that my code is wrong, that 2 + 2 is 4, not 5.

Is code review a form of testing?

According to the dictionary definition it is: the code reviewer is validating the code's "truth, genuineness and quality" by comparing it to a "standard", the arithmetic we all learned as children.

Let's imagine code review as a form of testing, alongside automated unit tests. Even if they're both tests, they're also quite different. What is the core difference between them?

One form of testing is automated, the other is done by a human.

An automated test is consistent and repeatable. You can write this:

def test_add_twice():
    for i in range(10000000):
        assert add(i, i) == 2 * i

And the computer will run the exact same code every time. The code will make sure add() consistently returns that particular result for those particular inputs. A human would face some difficulties in manually verifying ten million different computations: boredom, distraction, errors, slowness.

On the other hand, a human can read this code and tell you it's buggy:

def add(a, b):
    return a + b + 1

Where the computer does what it's told, for good or for bad, a human can provide meaning. Only a human can tell what the software is for.

Now we have one way of understanding what testing means, and how to organize it: humans test for meaning, whereas automated tests ensure consistency.

The second dimension of testing

Let's consider another aspect of testing. "A/B testing" is a form of testing where you try out two variations and see which produces a better result. Perhaps you are testing a redesign of your website: you show the current design to 90% of your visitors, the new design to 10% of your visitors, and see which results in more signups for your product.

Is this a test? It's called "A/B testing", so it seems like it ought to be.

Let's look at the dictionary definition again: "to put to the proof; to prove the truth, genuineness, or quality of by experiment, or by some principle or standard."

The dictionary thinks it's a test too, a test "by experiment". We're running an experiment to find out which version of the website will do best.

Unit tests and code review, in contrast, are tests "by some principle or standard". We have some intended specification for the software, some way we want it to behave, and we are trying to ensure it meets that specification.

Now we have a second way of understanding and organizing tests: testing by experiment vs. testing against a specification.

The testing quadrants

Put both of these together and we get the following chart showing four different types of testing:

Testing by experiment
Feedback from humans
(Meaning)
User Behavior
- Usability tests
- A/B tests
- MVP tests
- User bug reports
Software Behavior
- Stress tests
- Soak tests
- Error logs
Feedback from automation
(Consistency)
Correct Functionality
- Manual UI tests
- Code review
Stable Functionality
- Unit tests
- Compiler checks
- Automated UI tests
- Linters
Testing against a specification

Let's go through these four types of testing one by one.

User Behavior

These are all questions that cannot be answered by comparing your software to a specification. Instead you need empirical knowledge: you need to observe what actual human beings do when presented with your software.

Software Behavior

These questions can't be answered by comparing your software to a specification. You need to actually run your software and observe what happens.

Correct Functionality

It's tempting to say that automated tests can prove this, but remember the unit test that checked that 2 + 2 is 5. On a more fundamental level, software can technically match a specification and completely fail to achieve the goal of the specification. Only a human can understand the meaning of the specification and decide if the software matches it.

Stable Functionality

Humans are not a good way to test this. Humans are pretty good at ignoring small changes: if a button changes from "Send Now" to "Send now" you might not even notice at all. In contrast, software will break if your API changes from sendNow() to send_now(), or if the return type changes subtly.

This means a public API, an API that other software relies on, needs stability in order to be correct. Writing automated tests for private APIs, or code that is changing rapidly, will result in high maintenance costs as you continuously update your tests.

Applying the model

How can you use this model?

Choosing how to test

First, the model can help you choose what form of testing to do based on your goals.

Consider a startup building a product no one wants. Writing automated tests is even more of a waste of a time, since it focuses on implementing a specification before figuring out what users actually want.

One way of doing that is the Lean Startup methodology, which focuses on experiments whose goal is finding what product will meet customers' needs. That means focusing on the User Behavior quadrant. Only later is it worth spending the time to have more than the most minimal of specifications, and therefore do more than a minimal amount of testing for Correct Functionality or Stable Functionality.

Recognize when you've chosen the wrong type of testing

Second, the model can help you change course if you're using the wrong type of testing. Consider the tax preparation startup that had automated UI tests which didn't find bugs, and broke every time they changed the UI. Their problem was that their system really had two parts:

  1. The tax engine, which is fairly stable: the tax code changes only once per year. This suggests the need for Stable Functionality tests, e.g. unit tests talking directly to the tax calculation engine. Correct Functionality could be ensured by code review and feedback from a tax accountant.
  2. The web-based user interface The UI was changing constantly, which suggests Stable Functionality wasn't a goal. Correct Functionality was still a goal, so the UI should have been tested manually by humans (e.g. the programmers as they wrote the code.)

A basis for discussing testing

Finally, the model provides a shared terminology that can help you discuss testing in its broadest sense and its many differing goals.

Summary

Want to discuss this? Email me at itamar@codewithoutrules.com.

12 Mar 2017 5:00am GMT

05 Mar 2017

feedPlanet Twisted

Itamar Turner-Trauring: Why you're (not) failing at your new job

It's your first month at your new job, and you're worried you're on the verge of getting fired. You don't know what you're doing, everyone is busy and you need to bother them with questions, and you're barely writing any code. Any day now your boss will notice just how bad a job you're doing... what will you do then?

Luckily, you're unlikely to be fired, and in reality you're likely doing just fine. What you're going through happens to almost everyone when they start a new job, and your panicked feeling will eventually pass.

New jobs make you incompetent

Just like you, every time I've started a new job I've had to deal with feeling incompetent.

Do you notice the theme?

Every time you start a new job you are leaving behind the people, processes and tools you understand, and starting from scratch. A new language, new frameworks, new tools, new codebases, new ways of doing things, people you don't know, business logic you don't understand, processes you're unfamiliar with... of course it's scary and uncomfortable.

Luckily, for the most part this is a temporary feeling.

This too will pass

Like a baby taking their first steps, or a child on their first bike ride, those first months on a new job will make you feel incompetent. Soon the baby will be a toddler, the child will be riding with confidence. You too will soon be productive and competent.

Given some time and effort you will eventually learn what you need to know:

...the codebase.

...the processes, how things are done and maybe even why.

...the programming language.

...who to ask and when to ask them.

...the business you are operating in.

Since that's a lot to learn, it will take some time, but unless you are working for an awful company that is expected and normal.

What you can do

While the incompetent phase is normal and unavoidable, there is still something you can do about it: learn how to learn better. Every time you start a new job you're going to be learning new technologies, new processes, new business logic. The most important skill you can learn is how to learn better and faster.

The faster you learn the faster you'll get past the feeling of incompetence when you start a new job. The faster you learn the faster you can become a productive employee or valued consultant.

Some skills are specific to programming. For example, when learning new programming languages, I like skimming a book or tutorial first before jumping in: it helps me understand the syntax and basic concepts. Plus having a mental map of the book helps me know where to go back to when I'm stuck. Other skills are more generic, e.g. there is considerable research on how learning works that can help you learn better.

Finally, another way I personally try to learn faster is by turning my mistakes into educational opportunities. Past and present, coding or career, every mistake I make is a chance to figure out what I did wrong and how I can do better. If you'd like to avoid my past mistakes, sign up to get a weekly email with one of my mistakes and what you can learn from it.

05 Mar 2017 5:00am GMT

19 Feb 2017

feedPlanet Twisted

Itamar Turner-Trauring: When AI replaces programmers

The year is 2030, and artificial intelligence has replaced all programmers. Let's see how this brave new world works out:

Hi! I'm John the Software Genie, here to help you with all your software needs.

Hi, I'd like some software to calculate the volume of my house.

Awesome! May I have access to your location?

Why do you need to access my location?

I will look up your house in your city's GIS database and use its dimensions to calculate its volume.

Sorry, I didn't quite state that correctly. I want some software that will calculate the dimensions of any house.

Awesome! What is the address of this house?

No, look, I don't want anything with addresses. You can have multiple apartments in a house, and anyway some structures don't have an address, or are just being designed... and the attic and basement doesn't always count... How about... I want software that calculates the volume of an abstract apartment.

Awesome! What's an abstract apartment?

Grrrr. I want software that calculates the sum of the volumes of some rooms.

Awesome! Which rooms?

You know what, never mind, I'll use a spreadsheet.

I'm sorry Dave, I can't let you do that.

What.

Just a little joke! I'm sorry you decided to go with a spreadsheet. Your usage bill for $153.24 will be charged to your credit card. Have a nice day!

Back to the present: I've been writing software for 20 years, and I find the idea of being replaced by an AI laughable.

Processing large amounts of data? Software's great at that. Figuring out what a human wants, or what a usable UI is like, or what the real problem you need to solve is... those are hard.

Imagine what it would take for John the Software Genie to learn from that conversation. I've made my share of mistakes over the years, but I've learned enough that these days I can gather requirements decently. How do you teach an AI to gather requirements?

We might one day have AI that is as smart as a random human, an AI that can learn a variety of skills, an AI that can understand what those strange and pesky humans are talking about. Until that day comes, I'm not worried about being replaced by an AI, and you shouldn't worry either.

Garbage collection didn't make programmers obsolete just because it automated memory management. In the end automation is a tool to be controlled by human understanding. As a programmer you should focus on building the skills that can't be automated: figuring out the real problems and how to solve them.

19 Feb 2017 5:00am GMT

11 Feb 2017

feedPlanet Twisted

Twisted Matrix Laboratories: Twisted 17.1.0 Released

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 17.1!

The highlights of this release are:


For more information, check the NEWS file (link provided below).

You can find the downloads on PyPI (or alternatively our website). The NEWS file is also available on GitHub.

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,
Amber Brown (HawkOwl)

11 Feb 2017 10:08am GMT

10 Feb 2017

feedPlanet Twisted

Glyph Lefkowitz: Make Time For Hope

Pandora hastened to replace the lid! but, alas! the whole contents of the jar had escaped, one thing only excepted, which lay at the bottom, and that was HOPE. So we see at this day, whatever evils are abroad, hope never entirely leaves us; and while we have THAT, no amount of other ills can make us completely wretched.

It's been a rough couple of weeks, and it seems likely to continue to be so for quite some time. There are many real and terrible consequences of the mistake that America made in November, and ignoring them will not make them go away. We'll all need to find a way to do our part.

It's not just you - it's legit hard to focus on work right now. This is especially true if, as many people in my community are, you are trying to motivate yourself to work on extracurricular, after-work projects that you used to find exciting, and instead find it hard to get out of bed in the morning.

I have no particular position of authority to advise you what to do about this situation, but I need to give a little pep talk to myself to get out of bed in the morning these days, so I figure I'd share my strategy with you. This is as much in the hope that I'll follow it more closely myself as it is that it will be of use to you.

With that, here are some ideas.

It's not over.

The feeling that nothing else is important any more, that everything must now be a life-or-death political struggle, is exhausting. Again, I don't want to minimize the very real problems that are coming or the need to do something about them, but, life will go on. Remind yourself of that. If you were doing something important before, it's still important. The rest of the world isn't going away.

Make as much time for self-care as you need.

You're not going to be of much use to anyone if you're just a sobbing wreck all the time. Do whatever you can do to take care of yourself and don't feel guilty about it. We'll all do what we can, when we can.1

You need to put on your own oxygen mask first.

Make time, every day, for hope.

"You can stand anything for 10 seconds. Then you just start on a new 10 seconds."

Every day, set aside some time - maybe 10 minutes, maybe an hour, maybe half the day, however much you can manage - where you're going to just pretend everything is going to be OK.2

Once you've managed to securely fasten this self-deception in place, take the time to do the things you think are important. Of course, for my audience, "work on your cool open source code" is a safe bet for something you might want to do, but don't make the mistake of always grimly setting your jaw and nose to the extracurricular grindstone; that would just be trading one set of world-weariness for another.

After convincing yourself that everything's fine, spend time with your friends and family, make art, or heck, just enjoy a good movie. Don't let the flavor of life turn to ash on your tongue.

Good night and good luck.

Thanks for reading. It's going to be a long four years3; I wish you the best of luck living your life in the meanwhile.


  1. I should note that self-care includes just doing your work to financially support yourself. If you have a job that you don't feel is meaningful but you need the wages to survive, that's meaningful. It's OK. Let yourself do it. Do a good job. Don't get fired.

  2. I know that there are people who are in desperate situations who can't do this; if you're an immigrant in illegal ICE or CBP detention, I'm (hopefully obviously) not talking to you. But, luckily, this is not yet the majority of the population. Most of us can, at least some of the time, afford to ignore the ongoing disaster.

  3. Realistically, probably more like 20 months, once the Rs in congress realize that he's completely destroyed their party's credibility and get around to impeaching him for one of his numerous crimes.

10 Feb 2017 7:58am GMT

Itamar Turner-Trauring: Buggy Software, Loyal Users: Why Bug Reporting is Key To User Retention

Your software has bugs. Sorry, mine does too.

Doesn't matter how much you've tested it or how much QA has tested it, some bugs will get through. And unless you're NASA, you probably can't afford to test your software enough anyway.

That means your users will be finding bugs for you. They will discover that your site doesn't work on IE 8.2. They clicked a button and a blank screen came up. Where is that feature you promised? WHY IS THIS NOT WORKING?!

As you know from personal experience, users don't enjoy finding bugs. Now you have buggy software and unhappy users. What are you going to do about it?

Luckily, in 1970 the economist Albert O. Hirschman came up with an answer to that question.

Exit, Voice and Loyalty

In his classic treatise Exit, Voice and Loyalty, Hirschman points out that users who are unhappy with a product have exactly two options. Just two, no more:

  1. Exiting, i.e. giving up on your product.
  2. Voicing their concerns.

Someone who has given up on your software isn't likely to tell you about their issues. And someone who cares enough to file a bug is less likely to switch away from your software. Finally, loyal users will stick around and use their voice when otherwise they would choose exit.

Now, your software has no purpose without users. So chances are you want to keep them from leaving (though perhaps you're better off without some of them - see item #2).

And here's the thing: there's only two choices, voice or exit. If you can convince your users to use their voice to complain, and they feel heard, your users are going to stick around.

Now at this point you might be thinking, "Itamar, why are you telling me obvious things? Of course users will stick around if we fix their bugs." But that's not how you keep users around. You keep users around by allowing them to express their voice, by making sure they feel heard.

Sometimes you may not fix the bug, and still have satisfied users. And sometimes you may fix a bug and still fail at making them feel heard; better than not fixing the bug, but you can do better.

To make your users feel heard when they encounter bugs you need to make sure:

  1. They can report bugs with as little effort as possible.
  2. They hear back from you.
  3. If you choose to fix the bug, you can actually figure out the problem from their bug report.
  4. The bug fix actually gets delivered to them.

Let's go through these requirements one by one.

Bug reporting

Once your users notice a problem you want them to communicate it to you immediately. This will ensure they choose the path of voice and don't contemplate exiting to your shiny competitor at the domain next door.

Faster communication also makes it much more likely the bug report will be useful. If the bug occurred 10 seconds ago the user will probably remember what happened. If it's a few hours later you're going to hear something about how "there was a flying moose on the screen? or maybe a hyena?" Remember: users are human, just like you and me, and humans have a hard time remembering things (and sometimes we forget that.)

To ensure bugs get reported quickly (or at all) you want to make it as easy as possible for the user to report the problem. Each additional step, e.g. creating an account in an issue tracker, means more users dropping out of the reporting process.

In practice many applications are designed to make it as hard as possible to report bugs. You'll be visiting a website, when suddenly:

"An error has occurred, please refresh your browser."

And of course the page will give no indication of how or where you should report the problem if it reoccurs, and do the people who run this website even care about your problem?

Make sure that's not how you're treating your users when an error occurs.

Improving bug reporting

So let's say you include a link to the bug report page, and let's say the user doesn't have to jump through hoops and email verification to sign up and then fill out the 200 fields that JIRA or Bugzilla think are really important and would you like to tell us about your most common childhood nightmare from the following list? We'll assume that bug reporting is easy to find and easy to fill it out, as it should be.

But... you need some information from the user. Like, what version of the program they were using, the operating system and so on and so forth. And you do actually need that information.

But the user just wants to get this over with, and every additional piece of information you ask for is likely to make them give up and go away. What to do?

Here's one solution: include it for them.

I've been using the rather excellent Spacemacs editor, and as usual when I use new software I had to report a bug. So it goes.

Anyway, to report a bug in Spacemacs you just hit the bug reporting key combo. You get a bug reporting page. In your editor. And it's filled in the Spacemacs version and the Emacs version and all the configuration you made. And then you close the buffer and it pre-populates a new GitHub issue with all this information and you hit Submit and you're done.

This is pretty awesome. No need to find the right web page or copy down every single configuration and environmental parameter to report a bug: just run the error-reporting command.

Spacemacs screenshot

Also, notice that this is a lot better than automatic crash reporting. I got to participate and explain the bits that were important to me, it's not the "yeah we reported the crash info and let's be honest you don't really believe anyone looks at these do you?"-vibe that one gets from certain software.

You can do much the same thing with a command-line tool, or a web-based application. Every time an error occurs or the user is having issues (e.g. the user ran yourcommand --help):

  1. Ask for the user's feedback in the terminal, or within the context of the web page, and then file the bug report for them.
  2. Gather as much information as you can automatically and include it in the bug report, so the user only has to report the part they care about.

Responding to the bug report

The next thing you have to do is actually respond to the bug report. As I write this the Spacemacs issue I filed is sitting there all sad and lonely and unanswered, and it's a little annoying. I do understand, open source volunteer-run project and all. (This is where loyalty comes in.)

But for a commercial product I am paying for I want to know that my problem has been heard. And sometimes the answer might be "sorry, we're not going to fix that this quarter." And that's OK, at least I know where I stand. But silence is not OK.

Respond to your bug reports, and tell your users what you're doing about it. And no, an automated response is not good enough.

Diagnosing the bug

If you've decided to fix the bug you can now proceed to do so... if you can figure out what the actual problem is. If diagnosing problems is impossible, or even just too expensive, you're not going to do fix the problem. And that means your users will continue to be affected by the bug.

Let's look at a common bug report:

I clicked Save, and your program crashed and deleted a day's worth of my hopes and dreams.

This is pretty bad: an unhappy user, and as is often the case the user simply doesn't have the information you need to figure out what's going on.

Even if the bug report is useful it can often be hard to reproduce the problem. Testing happens in controlled environments, which makes investigation and reproduction easier. Real-world use happens out in the wild, so good luck reproducing that crash.

Remember how we talked about automatically including as much information as possible in the bug report? You did that, right? And included all the relevant logs?

If you didn't, now's the time to think about it: try to ensure that logs, core dumps, and so on and so forth will always be available for bug reports. And try to automate submitting them as much as possible, while still giving the user the ability to report their specific take on the problem.

(And don't forget to respect your user's privacy!)

Distributing the fix

So now you've diagnosed and fixed your bug: problem solved! Or rather, problem almost solved. If the user hasn't gotten the bug fix all this work has been a waste of time.

You need fast releases, and you need automated updates where possible. If it takes too long for fixes to reach your users then for for practical purposes your users are not being heard. (There are exceptions, as always: in some businesses your users will value stability over all else.)

A virtuous circle

If you've done this right:

  1. Your users will feel heard, and choose voice over exit.
  2. You will get the occasional useful bug report, allowing you to improve your product.
  3. All your users will quickly benefit from those improvements.

There is more to be done, of course: many bugs can and should be caught in advance. And many users will never tell you their problems unless you ask.

But it's a good start at making your users happy and loyal, even when they encounter the occasional bug.

10 Feb 2017 5:00am GMT

31 Jan 2017

feedPlanet Twisted

Jonathan Lange: Announcing haskell-cli-template

Last October, I announced servant-template, a cookiecutter template for creating production-ready Haskell web services.

Almost immediately after making it, I wished I had something for building command-line tools quickly. I know stack comes with a heap of them, but:

So I made haskell-cli-template. It's very simple, it just makes a Haskell command-line project with some tests, command-line parsing, and a CircleCI build.

I wanted to integrate logging-effect, but after a few months away from it my tired little brain wasn't able to figure it out. I like command-line tools with logging controls, so I suspect I'll add it again in the future.

Let me know if you use haskell-cli-template to make anything cool, and please feel free to fork and extend.

31 Jan 2017 12:00am GMT

30 Jan 2017

feedPlanet Twisted

Jonathan Lange: Announcing graphql-api: Haskell library for GraphQL

Late last year, my friend Tom tried to convince me that writing REST APIs was boring and repetitive and that I should give this thing called GraphQL a try.

I was initially sceptical. servant, the REST library that I'm most familiar with, is lovely. Its clever use of Haskell's type system means that all the boring boilerplate I'd have to write in other languages just goes away.

However, after watching Lee Byron's Strange Loop talk on GraphQL I began to see his point. Being able to get many resources with the same request is very useful, and as someone who writes & runs servers, I very much want clients to ask only for the data that they need.

The only problem is that there isn't really a way to write GraphQL servers in Haskell-until now.

Introducing graphql-api

Tom and I put together a proof-of-concept GraphQL server implementation called graphql-api, which we released to Hackage today.

It lets you take a GraphQL schema and translate it into a Haskell type that represents the schema. You can then write handlers that accept and return native Haskell types. graphql-api will take care of parsing and validating your user queries, and Haskell's type system will make sure that your handlers handle the right thing.

Using graphql-api

Say you have a simple GraphQL schema, like this:

type Hello {
  greeting(who: String!): String!
}

which defines a single top-level type Hello that contains a single field, greeting, that takes a single, required argument who.

A user would query it with something like this:

{
  greeting("World")
}

And expect to see an answer like:

{
  "data": {
    "greeting": "Hello World!"
  }
}

To do this in Haskell with GraphQL, first we'd define the type:

type Hello = Object "Hello" '[]
  '[ Argument "who" Text :> Field "greeting" Text ]

And then a handler for that type:

hello :: Handler IO Hello
hello = pure greeting
 where
   greeting who = pure ("Hello " <> who <> "!")

We can then run a query like so:

queryHello :: IO Response
queryHello = interpretAnonymousQuery @Hello hello "{ greeting(who: \"World\") }"

And get the output we expect.

There's a lot going on in this example, so I encourage you to check out our tutorial to get the full story.

graphql-api's future

Tom and I put graphql-api together over a couple of months in our spare time because we wanted to actually use it. However, as we dug deeper into the implementation, we found we really enjoyed it and want to make a library that's genuinely good and helps other people do cool stuff.

The only way to do that, however, is to release it and get feedback from our users, and that's what we've done. So please use graphql-api and tell us what you think. If you build something cool with it, let us know.

For our part, we want to improve the error messages, make sure our handling for recursive data types is spot on, and smooth down a few rough edges.

Thanks

Tom and I want to thank J. Daniel Navarro for his great GraphQL parser and encoder, which forms the basis for what we built here.

About the implementation

graphql-api is more-or-less a GraphQL compiler hooked up to type-based executing (aka "resolving") engine that's heavily inspired by Servant and uses various arcane type tricks from GHC 8.

We tried to stick to implementing the GraphQL specification. The spec is very well written, but implementing it has taught us that GraphQL is not at all as simple as it looks at first.

I can't speak very well to the type chicanery, except to point you at the code and at the Servant paper.

The compiler mostly lives in the GraphQL.Internal.Validation module. The main idea is that it takes the AST and translates it into a value that cannot possibly be wrong.

All the syntax stuff is from the original graphql-haskell, with a few fixes, and a tweak to guarantee that Name values are guaranteed to be correct.

30 Jan 2017 12:00am GMT

29 Jan 2017

feedPlanet Twisted

Itamar Turner-Trauring: Does your job contradict your beliefs?

As an employee your work is chosen by the owners and managers of the company: you choose the means, they choose the ends. Becoming an employee doesn't mean abdicating your moral responsibility, however. Even if someone else has chosen the goals you are still responsible for your own actions.

As an employee you need to ask yourself if the goals you're working for are worthwhile, and if the people you are working for deserve the power to direct your actions.

Let me tell you about the first time I was forced to ask myself this question.

The past is a different country

I grew up Israel, where as a Jewish citizen I was subject to the draft. The State of Israel is both Zionist and democratic, inherently contradictory ideals: a nation united by ethnicity, language and land versus the natural rights of individual human beings. Most Israeli citizens are indeed Jewish, but a large minority are Arab.

The contradiction is even more acute outside the borders of Israel, in the occupied territories of the West Bank and Gaza, where Israel's Jewish military rules millions of Palestinian Arabs. Whatever their opinion of the Occupation, most Israeli Jews are Zionists. They believe Israel must remain a Jewish state with a Jewish army, a bulwark against Palestinian terrorism and Israel's hostile Arab neighbors.

At age seventeen I had my first contact with the military: I was summoned to an initial screening at the Bakum, a military base that processes new recruits. The base was covered with asphalt, concrete, barbed wire and dirt, which turned to mud when it rained. The buildings were ugly concrete blocks and flimsy metal shacks that had seen better decades. Everything, inside and out, was painted in shades of khaki, gray and rust.

At the Bakum I was examined by a doctor. He gave me a physical profile of 97, the highest possible value; urban legend has it that the extra three points are taken off for circumcision. A profile of 97 meant I would be assigned combat duty, known in Israel as kravi.

As a kravi soldier I would have been required, as other Israeli soldiers have, to demolish homes, evict families, delay ambulances at checkpoints, threaten to shoot civilians for violating curfews. I didn't believe these actions protected Israel from Palestinian terrorists.

So I took the easy way out: I signed up for the Academic Reserve. I would go to college as a civilian, studying computer science, then serve in the army as a programmer for six years.

In the summer of 1999, after my first year in college, I went through a month of unstrenuous basic training for non-combatants. I hated every minute of it, the half-hearted brainwashing and the petty sadism. The following year, bored and unmotivated, I dropped out of college and started a software company.

Sooner or later the Academic Reserve office would notice I'd stopped taking classes, and I would be drafted into a kravi unit. My mother suggested I write a letter to the army - via a distant relative who worked for the Department of Defense - explaining my qualifications as a programmer and requesting my skills be put to use. Perhaps I could reach an accommodation allowing me to work on my company in my spare time.

I was summoned to a number of interviews in the following months, at office buildings, anonymous houses in residential areas, even a war monument behind a military base. None of them went well. I thoughtlessly, in retrospect perhaps deliberately, wrecked my chances at getting a security clearance, telling an interviewer my political views: I did not believe in the leadership of the country or the military. It was unlikely I would be trusted with classified material.

My final interview was different: I actually liked the soldiers I met. They were intelligent and sympathetic and I was sure I'd enjoy working with them, but I left the interview feeling miserable. I reached not so much a decision as a self-diagnosis: I could never be a soldier, I could not give up my right and duty to make my own decisions.

In the present

I did manage to get out of military service, but that's a longer story. But before that I spent literally years refusing to admit to myself that this was not a job I was willing to take, whatever the social expectations.

Being an employee is not quite the same as being a soldier. But these days when I'm looking for a job I try to be more aware of what I am willing to do. I won't even bother applying to companies that do anything related to "defense" or the military, for example.

You will have your own criteria, of course, but I would urge you to consider whether your current employment matches your beliefs.

29 Jan 2017 5:00am GMT

26 Jan 2017

feedPlanet Twisted

Itamar Turner-Trauring: Coding skills you won't learn in school: Object Ownership

There are many skills you need to acquire as a programmer, and some of them are not part of the standard software engineering curriculum. Instead you're expected to learn them by osmosis, or by working with someone more experienced. David MacIver covers one such skill: tracking which type a value has.

Another skill you need is an understanding object ownership in your code: knowing which part of your code owns a particular object in memory, and what its expectations are for access. Lacking this understanding you might write code that causes your program to crash or to suffer from subtle bugs. Even worse, some programming languages won't even provide you with facilities to help you in this task.

Learning by osmosis

Here's how I learned this skill. When I was in university I was once had to implement a Red-Black Tree in C. I'd basically skipped all the classes for the intro to C class, and still gotten a perfect grade, so as far as the school was concerned I knew how to code in C.

In reality I had no clue what I was doing. I couldn't write a working tree implementation: my code kept segfaulting, I couldn't keep the structure straight. Eventually I turned in a half-broken solution and squeaked by with a grade of 60 out of 100.

I spent the next 5 years writing Python code, and then got a job writing C and C++. I did make some mistakes, e.g. my first project crashed every night (you can hear that story by signing up for my newsletter), but in general I was able to write working code even though I hadn't really written any C or C++ for years.

What changed?

I believe the one of the key skills I learned was object ownership, as a result of all the concurrent Python I was writing, plus the fact that C++ has a better model than C for object ownership. Let's see what I learned over those years.

Object ownership for memory deallocation

Consider the following C function:

char* do_something(char* input);

Someone is going to have deallocate input and someone is going to have to deallocate the returned result of do_something(). But who? If two different functions try to deallocate the same allocation your program's memory will be corrupted. If no one deallocates the memory your program will suffer from memory leaks.

This is where object ownership comes in: you ensure each allocation has only one owner, and only that owner should deallocate it. The GNOME project's developer documentation explains how their codebase makes this work:

Each allocation has exactly one owner; this owner may change as the program runs, by transferring ownership to another piece of code. Each variable is owned or unowned, according to whether the scope containing it is always its owner. Each function parameter and return type either transfers ownership of the values passed to it, or it doesn't. ... By statically calculating which variables are owned, memory management becomes a simple task of unconditionally freeing the owned variables before they leave their scope, and not freeing the unowned variables.

GNOME has a whole set of libraries, conventions and rules for making this happen, because the C programming language doesn't have many built-in facilities to deal with ownership.

C++, on the other hand, has built a broad range of utilities for just this purpose. For example, you can wrap an allocation in a shared_ptr object. Every time it is copied it will increment a counter, every time it is deallocated it will decrement the counter. When the counter hits zero the wrapped allocation will be deallocated. That means you don't need to track ownership for purposes of deallocation: the shared_ptr is the owner, and will deallocate at the right time.

This can be simplified even further by using languages like Java or Python that provide garbage collection: the language runtime will do all the work for you. You never have to track ownership for purposes of deallocating memory.

Object access rights

Even when memory allocation is handled by the language runtime, there are still reasons to think about object ownership. In particular there is the question of mutation: modifying an object's contents. Memory deallocation is the ultimate form of mutation, but normal mutation can also break your program.

Consider the following Python program:

words = ["hello", "there", "another"]
counts = wordcount(words)
print(words)

What do you expect to be printed? Typically you'd expect to see ["hello", "there", "another"], but there is another option. You may also get [] printed if wordcount() was implemented as follows:

def wordcount(words):
    result = Counter()
    while words:
        word = words.pop()
        result[word] += 1
    return result

In this implementation wordcount() is mutating the list it is given. Reintroducing the concept of an object owner makes this clearer: each object is owned by a scope, and that scope might not want to grant write access to the object when it passes it to a function.

Unfortunately in Python, Java and friends there is no real way for a caller to tell whether a function will mutate an input, nor whether a parameter to a function can be mutated. So you need to learn a set of conventions and assumptions about when this will happen and when it's safe to do so: you build a mental model of object ownership and access rights. I suspect most Python programmers wouldn't expect wordcount() to mutate its inputs: it violates the mental model we have for object ownership and access.

The concept of private attributes (explicit in Java, by convention in Python) is one way access rights are controlled, but it doesn't solve the problem in all cases. When conventions don't help and you're uncertain you have to refer to API docs, or sometimes even the implementation, to know who can or might modify objects. This is similar to how C programmers deal with memory allocation.

Interestingly, C and C++ have a mechanism that can often solve this problem: const. You can define arguments as being const, i.e. unchangeable:

std::map<string,int> wordcount(const vector<string> &words);

If you try to mutate the argument the compiler will complain and prevent you from doing so.

Other approaches: Rust vs. functional languages

Outside of const the C++ features for object ownership management grew over time, as library code. The Rust programming language, in contrast, provides object ownership as a feature within the compiler itself. And this applies both to ownership for purposes of memory allocation but also for purposes of mutation. Rust attempts to provide this features while still providing the full power of C and C++, and in particular control over memory allocation.

Where Rust code requires the programmer to make explicit decisions about object ownership, functional languages take a different approach. In functional languages like Haskell or Clojure if you call a wordcount function you know the arguments you pass in won't be mutated, because objects are immutable and can't be changed. If objects can't be mutated it doesn't matter who owns them: they are the same everywhere.

The need to track object ownership in your head or in code for mutation control is obviated by making objects immutable. Couple this with garbage collection and you need spend much less time thinking about object ownership when writing purely functional code.

Summary: what you need to know

Depending on the programming language you're coding in you need to learn different models of thinking about object ownership:

Next time you're writing some code think about who owns each object, and what guarantees the owner expects when it passes an object to other code. Over time you'll build a better mental model of how ownership works.

And if you're writing in a non-functional language consider using immutable (sometimes known as "persistent") data structures. You can get much of the benefits of a functional language by simply reducing the scope of mutation, and therefore how much you need to track object ownership.

26 Jan 2017 5:00am GMT

19 Jan 2017

feedPlanet Twisted

Itamar Turner-Trauring: Specialist vs. Generalist: which is better for your career?

One of the decisions you'll need to make during the course of you career as a software developer is whether you should become:

  1. A specialist, an expert in a certain subject.
  2. A generalist, able to take on a wide variety of different work.

Miquel Beltran argues that specialization is the path to choose. At the end of his essay he suggests:

Stick to one platform, framework or language and your professional career will be better on the long run.

I think he's both right and wrong. Specialization is a great career move... but I don't think being a generalist is bad for your career either. In fact, you can be both and still have a good career, because there are two distinct areas in which this question plays out.

Getting hired is not the same as getting things done

Getting hired and getting things done are two different tasks, and you need different skills to do each.

When you're trying to get hired you are trying to show why you are the best candidate. That means dealing with the company's attitude towards employees, and the reason they are hiring, and the way they approach their work. It's not about how well you'll do or your job, or how good a programmer you are, or any of that: it's just about getting your foot in the door.

Once you're in, once you're an employee or a consultant, what matters is the results you deliver. It doesn't matter if you've only spent a few weeks previously writing iOS apps so long you do a good job writing an iOS app after you've been hired. And if you've spent years as an iOS developer and you fail to deliver, the fact you specialize in the iOS apps isn't going to help you.

Since getting hired and doing the work are separate tasks, that means you need to separate the decision to be a specialist or generalist into two questions: which will help you get hired, and which will make you better at actually doing your job?

Specialization is a marketing technique

If the question is how you should get hired then you are in the realm of marketing, not engineering. Specialization is a marketing technique: it's a way to demonstrate why you should be hired because you are an expert in your specialty.

Because specialization a marketing technique, specialization doesn't necessarily need to map to specialization on an engineering level. Let me give some examples from my career.

In 2001 I started contributing to an open source Python project, a networking framework called Twisted. I have used this experience in a variety of ways:

While all these specializations are related, they are not identical: each job I got involved being a specialist in a different area.

It's not what you can do, it's what you emphasize

Now, you could argue that the reasons I got hired enough are close enough that I am indeed a specialist: in networking or distributed systems. But consider that earlier in my career I did a number of years of web development. So back in 2004 I could have applied to web development jobs, highlighted that part of my resume, and relegated my open source networking work to a sentence at the end.

You likely have many engineering and "soft" skills available to you. Instead of focusing on one particular skillset ("I am an Android developer") you can focus on some other way you are special. E.g. If you're building a consulting pipeline then maybe it's a some business vertical you specialize in, to differentiate yourself from all the other Android developers.

But if you're marketing yourself on a one-off basis, which is certainly the case when you're applying for a job, you can choose a specialty that fits the occasion. Here's how my former colleague Adam Dangoor does it:

Pick one thing from what they talk about that you think is probably the least pitched-to aspect. E.g. if they're a Python shop everyone will say that they know Python well. But you can spot that e.g. they need help with growing a team and you have experience with that. It could very well be that 10 other candidates do too, but you just say that and you're the one candidate who can grow a team.

Specialist or Generalist?

So which should you chose, generalist or specialist?

When it comes to engineering skills, or just learning in general, my bias is towards being a generalist. When I went back to school to finish my degree I focused on the humanities and social science; I didn't take a single programming class. You may have different biases then I do.

But engineering skills are fundamentally different than how you market yourself. You can be a generalist in your engineering skills and market yourself as a specialist. In particular, when applying for jobs, you should try to be a specialist in what the company needs.

Sometimes a technical specialty is exactly what they want: you have some set of skills that are hard to find. But often there's a bit more to it than that. They might say they need an Android expert", but what they really need is someone to ship things fast.

They're looking for "an Android expert" because they don't want a learning curve. So if you emphasize the times you've delivered projects quickly and an on schedule you might get the job even, though another candidate had a couple more years of Android experience than you do..

In short, when it comes to engineering skills I tend towards being a generalist, but that may just be my personal bias. When marketing yourself, be a specialist... but there's nothing keeping you from being a different specialist every time you apply for a new job.

19 Jan 2017 5:00am GMT

18 Jan 2017

feedPlanet Twisted

Jack Moffitt: Servo Talk at LCA 2017

My talk from Linux.conf.au was just posted, and you can go watch it. In it I cover some of the features of Servo that make it unique and fast, including the constellation and WebRender.

Servo Architecture: Safety & Performance by Jack Moffitt, LCA 2017, Hobart, Australia.

18 Jan 2017 12:00am GMT

12 Jan 2017

feedPlanet Twisted

Jonathan Lange: Announcing grafanalib

Late last year, as part of my work at Weaveworks, I published grafanalib, a Python DSL for building Grafana dashboards.

We use it a lot, and it's made our dashboards much nicer to maintain. I've written a blog post about it that you can find it on the Weaveworks blog.

12 Jan 2017 12:00am GMT

11 Jan 2017

feedPlanet Twisted

Itamar Turner-Trauring: Your Job is Not Your Life: staying competitive as a developer

Are you worried about keeping your programming skills up-to-date so you can stay employable? Some programmers believe that to succeed you must spend all of your time learning, practicing and improving your craft. How do you fit all that in and still have a life?

In fact, it's quite possible to limit yourself to programming during work hours and still be employable and successful. If you do it right then staying competitive, if it's even necessary, won't require giving up your life for you job.

What does it mean to be "competitive?"

Before moving on to solutions it's worth understanding the problem a little more. The idea of "competitiveness" presumes that every programmer must continually justify their employment, or they will be replaced by some other more qualified developer.

There are shrinking industries where this is the case, but at the moment at least demand for programmers is quite high. Add on the fact that hiring new employees is always risky and worrying about "competitiveness" seems unnecessary. Yes, you need to do well at your job, but I doubt most programmers are at risk of being replaced a moment's notice.

Instead of worrying about "competitiveness" you should focus on the ability to easily find a new job. For example, there are other ways you improve your chances at finding a new job that have nothing to do with your engineering skills:

Moving on to engineering skills, the idea that you need to put in long hours outside of work is based both on the need to become an expert, and on the need to keep up with changing technology. Both can be done on the job.

Becoming an expert

You've probably heard the line about expertise requiring 10,000 hours of practice. The more hours you practice the better, then, right?

In fact many of the original studies were about number of years, not number of hours (10 years in particular). And the kind of practice matters. What you need is "deliberate practice":

... deliberate practice is a highly structured activity, the explicit goal of which is to improve performance. Specific tasks are invented to overcome weaknesses, and performance is carefully monitored to provide cues for ways to improve it further. We claim that deliberate practice requires effort and is not inherently enjoyable.

Putting aside knowledge of particular technologies, the kinds of things you want to become an expert at are problem solving, debugging, reading unknown code, etc.. And while you could practice them on your own time, the most realistic forms of practice will be at your job. What you need to do is utilize your work as a form of practice.

How should you practice? The key is to know your own weaknesses, and to get feedback on how you're doing so you can improve. Here are two ways to do that:

  1. Code reviews: a good code reviewer will point out holes in your design, in the ways you've tested your code, in the technology you're using. And doing code reviews will also improve your skills as you consider other people's approaches. A job at an organization with a good code review culture will be valuable to your skills and to your career.
  2. Self-critique: whenever you make a mistake, try to think about what you should have noticed, what mental model would have caught the problem, and how you could have chosen better. Notice the critique is not of the result. The key is to critique the process, so that you do better next time.

I write a weekly newsletter about my many mistakes, and while this is ostensibly for the benefit of the readers I've actually found it has helped me become a more insightful programmer. If you want to learn how to make self-critique useful than just an exercise in negativity I recommend the book The Power of Intuition by Gary Klein.

Learning technical skills

Beyond expertise you also need technical skills: programming languages, frameworks, and so on. You will never be able to keep up with all the changing technologies that are continuously being released. Instead, try the following:

Your job is not your life

All of the suggestions above shouldn't require much if any time outside of your job. If you enjoy programming and want to do it for fun, by all means do so. But your job shouldn't be the only thing you spend you life on.

If you would like to learn how to to get a job that doesn't overwhelm your life, join my free 6-part email course.

Join the course: Getting to a Sane Workweek

Don't let your job take over your life. Join over 800 other programmers on the journey to a saner workweek by taking this free 6-part email course. You'll learn how you can work reasonable hours and still succeed in your career a programmer.

Success! Now check your email to confirm your subscription.

There was an error submitting your subscription. Please try again.

I won't send you any spam. Unsubscribe at any time. Powered by ConvertKit

11 Jan 2017 5:00am GMT