19 Jan 2017

feedPlanet Twisted

Itamar Turner-Trauring: Specialist vs. Generalist: which is better for your career?

One of the decisions you'll need to make during the course of you career as a software developer is whether you should become:

  1. A specialist, an expert in a certain subject.
  2. A generalist, able to take on a wide variety of different work.

Miquel Beltran argues that specialization is the path to choose. At the end of his essay he suggests:

Stick to one platform, framework or language and your professional career will be better on the long run.

I think he's both right and wrong. Specialization is a great career move... but I don't think being a generalist is bad for your career either. In fact, you can be both and still have a good career, because there are two distinct areas in which this question plays out.

Getting hired is not the same as getting things done

Getting hired and getting things done are two different tasks, and you need different skills to do each.

When you're trying to get hired you are trying to show why you are the best candidate. That means dealing with the company's attitude towards employees, and the reason they are hiring, and the way they approach their work. It's not about how well you'll do or your job, or how good a programmer you are, or any of that: it's just about getting your foot in the door.

Once you're in, once you're an employee or a consultant, what matters is the results you deliver. It doesn't matter if you've only spent a few weeks previously writing iOS apps so long you do a good job writing an iOS app after you've been hired. And if you've spent years as an iOS developer and you fail to deliver, the fact you specialize in the iOS apps isn't going to help you.

Since getting hired and doing the work are separate tasks, that means you need to separate the decision to be a specialist or generalist into two questions: which will help you get hired, and which will make you better at actually doing your job?

Specialization is a marketing technique

If the question is how you should get hired then you are in the realm of marketing, not engineering. Specialization is a marketing technique: it's a way to demonstrate why you should be hired because you are an expert in your specialty.

Because specialization a marketing technique, specialization doesn't necessarily need to map to specialization on an engineering level. Let me give some examples from my career.

In 2001 I started contributing to an open source Python project, a networking framework called Twisted. I have used this experience in a variety of ways:

While all these specializations are related, they are not identical: each job I got involved being a specialist in a different area.

It's not what you can do, it's what you emphasize

Now, you could argue that the reasons I got hired enough are close enough that I am indeed a specialist: in networking or distributed systems. But consider that earlier in my career I did a number of years of web development. So back in 2004 I could have applied to web development jobs, highlighted that part of my resume, and relegated my open source networking work to a sentence at the end.

You likely have many engineering and "soft" skills available to you. Instead of focusing on one particular skillset ("I am an Android developer") you can focus on some other way you are special. E.g. If you're building a consulting pipeline then maybe it's a some business vertical you specialize in, to differentiate yourself from all the other Android developers.

But if you're marketing yourself on a one-off basis, which is certainly the case when you're applying for a job, you can choose a specialty that fits the occasion. Here's how my former colleague Adam Dangoor does it:

Pick one thing from what they talk about that you think is probably the least pitched-to aspect. E.g. if they're a Python shop everyone will say that they know Python well. But you can spot that e.g. they need help with growing a team and you have experience with that. It could very well be that 10 other candidates do too, but you just say that and you're the one candidate who can grow a team.

Specialist or Generalist?

So which should you chose, generalist or specialist?

When it comes to engineering skills, or just learning in general, my bias is towards being a generalist. When I went back to school to finish my degree I focused on the humanities and social science; I didn't take a single programming class. You may have different biases then I do.

But engineering skills are fundamentally different than how you market yourself. You can be a generalist in your engineering skills and market yourself as a specialist. In particular, when applying for jobs, you should try to be a specialist in what the company needs.

Sometimes a technical specialty is exactly what they want: you have some set of skills that are hard to find. But often there's a bit more to it than that. They might say they need an Android expert", but what they really need is someone to ship things fast.

They're looking for "an Android expert" because they don't want a learning curve. So if you emphasize the times you've delivered projects quickly and an on schedule you might get the job even, though another candidate had a couple more years of Android experience than you do..

In short, when it comes to engineering skills I tend towards being a generalist, but that may just be my personal bias. When marketing yourself, be a specialist... but there's nothing keeping you from being a different specialist every time you apply for a new job.

19 Jan 2017 5:00am GMT

18 Jan 2017

feedPlanet Twisted

Jack Moffitt: Servo Talk at LCA 2017

My talk from Linux.conf.au was just posted, and you can go watch it. In it I cover some of the features of Servo that make it unique and fast, including the constellation and WebRender.

Servo Architecture: Safety & Performance by Jack Moffitt, LCA 2017, Hobart, Australia.

18 Jan 2017 12:00am GMT

12 Jan 2017

feedPlanet Twisted

Jonathan Lange: Announcing grafanalib

Late last year, as part of my work at Weaveworks, I published grafanalib, a Python DSL for building Grafana dashboards.

We use it a lot, and it's made our dashboards much nicer to maintain. I've written a blog post about it that you can find it on the Weaveworks blog.

12 Jan 2017 12:00am GMT

11 Jan 2017

feedPlanet Twisted

Itamar Turner-Trauring: Your Job is Not Your Life: staying competitive as a developer

Are you worried about keeping your programming skills up-to-date so you can stay employable? Some programmers believe that to succeed you must spend all of your time learning, practicing and improving your craft. How do you fit all that in and still have a life?

In fact, it's quite possible to limit yourself to programming during work hours and still be employable and successful. If you do it right then staying competitive, if it's even necessary, won't require giving up your life for you job.

What does it mean to be "competitive?"

Before moving on to solutions it's worth understanding the problem a little more. The idea of "competitiveness" presumes that every programmer must continually justify their employment, or they will be replaced by some other more qualified developer.

There are shrinking industries where this is the case, but at the moment at least demand for programmers is quite high. Add on the fact that hiring new employees is always risky and worrying about "competitiveness" seems unnecessary. Yes, you need to do well at your job, but I doubt most programmers are at risk of being replaced a moment's notice.

Instead of worrying about "competitiveness" you should focus on the ability to easily find a new job. For example, there are other ways you improve your chances at finding a new job that have nothing to do with your engineering skills:

Moving on to engineering skills, the idea that you need to put in long hours outside of work is based both on the need to become an expert, and on the need to keep up with changing technology. Both can be done on the job.

Becoming an expert

You've probably heard the line about expertise requiring 10,000 hours of practice. The more hours you practice the better, then, right?

In fact many of the original studies were about number of years, not number of hours (10 years in particular). And the kind of practice matters. What you need is "deliberate practice":

... deliberate practice is a highly structured activity, the explicit goal of which is to improve performance. Specific tasks are invented to overcome weaknesses, and performance is carefully monitored to provide cues for ways to improve it further. We claim that deliberate practice requires effort and is not inherently enjoyable.

Putting aside knowledge of particular technologies, the kinds of things you want to become an expert at are problem solving, debugging, reading unknown code, etc.. And while you could practice them on your own time, the most realistic forms of practice will be at your job. What you need to do is utilize your work as a form of practice.

How should you practice? The key is to know your own weaknesses, and to get feedback on how you're doing so you can improve. Here are two ways to do that:

  1. Code reviews: a good code reviewer will point out holes in your design, in the ways you've tested your code, in the technology you're using. And doing code reviews will also improve your skills as you consider other people's approaches. A job at an organization with a good code review culture will be valuable to your skills and to your career.
  2. Self-critique: whenever you make a mistake, try to think about what you should have noticed, what mental model would have caught the problem, and how you could have chosen better. Notice the critique is not of the result. The key is to critique the process, so that you do better next time.

I write a weekly newsletter about my many mistakes, and while this is ostensibly for the benefit of the readers I've actually found it has helped me become a more insightful programmer. If you want to learn how to make self-critique useful than just an exercise in negativity I recommend the book The Power of Intuition by Gary Klein.

Learning technical skills

Beyond expertise you also need technical skills: programming languages, frameworks, and so on. You will never be able to keep up with all the changing technologies that are continuously being released. Instead, try the following:

Your job is not your life

All of the suggestions above shouldn't require much if any time outside of your job. If you enjoy programming and want to do it for fun, by all means do so. But your job shouldn't be the only thing you spend you life on.

If you would like to learn how to to get a job that doesn't overwhelm your life, join my free 6-part email course.

Join the course: Getting to a Sane Workweek

Don't let your job take over your life. Join over 720 other programmers on the journey to a saner workweek by taking this free 6-part email course. You'll learn how you can work reasonable hours and still succeed in your career a programmer.

Success! Now check your email to confirm your subscription.

There was an error submitting your subscription. Please try again.

I won't send you any spam. Unsubscribe at any time. Powered by ConvertKit

11 Jan 2017 5:00am GMT

06 Jan 2017

feedPlanet Twisted

Itamar Turner-Trauring: The fourfold path to software quality

How do you achieve software quality? How do you write software that actually works, software that isn't buggy, software that doesn't result in 4AM wake up calls when things break in production?

There are four different approaches you can take, four paths to the ultimate goal. Which path you choose to take will depend on your personality, skills and the circumstances of your work.

The path of the Yolo Programmer

The first path is that of the Yolo Programmer. As a follower of the famous slogan "You Only Live Once", the Yolo Programmer chooses not to think about software quality. Instead the Yolo Programmer enjoys the pure act of creation; writing code is a joy that would only be diminished by thoughts of maintenance or bugs.

It's easy to look down on the Yolo Programmer, to deride their approach a foolish attitude only suitable for children. As adults we suppress our playfulness because we worry about the future. But even though the future is important, the joy of creation is still a fundamental part of being human.

When you have the opportunity, when you're creating a prototype or some other code that doesn't need to be maintained, embrace the path of the Yolo Programmer. There's no shame in pure enjoyment.

The path of the Rational Optimizer

In contrast to the Yolo Programmer, the Rational Optimizer is well aware of the costs of bugs and mistakes. Software quality is best approached by counter-balancing two measurable costs: the cost of bugs to users and the business vs. the cost of finding and fixing the bugs.

Since bugs are more expensive the later you catch them, the Rational Optimizer invests in catching bugs as early as possible. And since human effort is expensive, the Rational Optimizer loves tools: software can be written once and used many times. Tools to find bugs are thus an eminently rational way to increase software quality.

David R. MacIver's post The Economics of Software Correctness is a great summary of this approach. And he's built some really wonderful tools: your company should hire him if you need to improve your software's quality.

The path of Mastery

The path of Mastery takes a different attitude, which you can see in the title Kate Thompson's book Zero Bugs and Program Faster (note that she sent me a free copy, so I may be biased).

Mastery is an attitude, a set of assumptions about how one should write code. It assumes that the code we create can be understood with enough effort. Or, if the code is not understandable, it can and should be simplified until we can understand it.

The path of Mastery is a fundamentally optimistic point of view: we can, if we choose, master our creations. If we can understand our code we can write quality code. We can do so by proving to ourselves that we've covered all the cases, and by learning to structure our code the right way. With the right knowledge, the right skills and the right attitude we can write code with very few bugs, perhaps even zero bugs.

To learn more about this path you should read Thompson's book; it's idiosyncratic, very personal, and full of useful advice. You'll become a better programmer by internalizing her lessons and attitude.

The path of the Software Clown

The final path is the path of the Software Clown. If Mastery is a 1980s movie training montage, the Software Clown is a tragicomedy: all software is broken, failure is inevitable, and nothing ever works right. There is always another banana peel to slip on, and that would be sad if it weren't so funny.

Since the Software Clown is always finding bugs, the Software Clown makes sure they get fixed, even when they're in someone else's software. Since software is always broken, the Software Clown plans for brokenness. For example, if bugs are inevitable then you should make sure users have an easy time reporting them.

Since banana peels are everywhere, the Software Clown learns how to avoid them. You can't avoid everything, and you won't avoid everything, but you can try to avoid as many as possible.

If you'd like to avoid the many mistakes I've made as a software engineer, sign up for my Software Clown newsletter. You'll get the story of one of my mistakes in your inbox every week and how you can avoid making it.

These are the four paths you can take, but remember: there is no one true answer, no one true path. Try to learn them all, and the skills and attitudes that go along with them; you'll become a better programmer and perhaps even a better person.

Avoid my programming mistakes!

Get a weekly email with one of my many software and career mistakes, and how you can avoid it. Here's what readers are saying:

"Are you reading @itamarst's "Software Clown" newsletter? If not, you should be. There's a gem in every issue." - @glyph

Success! Now check your email to confirm your subscription.

There was an error submitting your subscription. Please try again.

I won't share your email with anyone else. Unsubscribe at any time. Powered by ConvertKit

06 Jan 2017 5:00am GMT

02 Jan 2017

feedPlanet Twisted

Itamar Turner-Trauring: When software ecosystems die

How much can you rely on the frameworks, tools and libraries you build your software on? And what can you do to reduce the inherent risk of depending on someone else's software?

Years ago I watched a whole software ecosystem die.

Not the slow decline of a programming language that is losing its users, or a no longer maintained library that has a newer, incompatible replacement. This was perma-death: game over, no resurrection, no second chances.

Here's what happened, and what you can learn from it.

The story of mTropolis

Back in the 1990s the Next Big Thing was multimedia, and in particular multimedia CD-ROMs. The market leader was Macromedia Director, a rather problematic tool.

Macromedia Director started out as an animation tool, using a sequence of frames as its organizing metaphor, which meant using it for hypermedia involved a rather bizarre idiom. Your starting screen would be frame 1 on the timeline, with a redirect to itself on exit, an infinite busy loop. Remember this started as animation tool, so the default was to continue on to later frames automatically.

When you clicked on a button that took you to a new screen it worked by moving you to another frame, let's say frame 100. Frame 100 would have a "go to frame 100" on exit to made sure you didn't continue on to frame 101, and then 102, etc.

Then in 1995 mTropolis showed up, a newer, better competitor to Director. It was considered by many to be the superior alternative, even in its very first release. It had a much more suitable conceptual model, features that were good enough to be copied by Director, and a loyal fan base.

In 1997 mTropolis was bought by Quark, maker of the the QuarkXPress desktop publishing software. A year later in 1998 Quark decided to end development of mTropolis.

mTropolis' users were very upset, of course, so they tried to buy the rights off of Quark and continue development on their own.

The purchase failed. mTropolis died.

Market leader or scrappy competitor?

The story of mTropolis had a strong impression on me as a young programmer: I worked with Director, so I was not affected, but the developers who used mTropolis were dead in the water. All the code they'd built was useless as soon as a new OS release broke mTropolis in even the smallest of ways.

This isn't a unique story, either: spend some time reading Our Incredible Journey. Startups come and go, and software ecosystems die with them.

Professor Beekums has an excellent post about switching costs in software development. He argues that given the choice between equivalent market leader and smaller competitor you should choose the latter, so you don't suffer from monopoly pricing.

But what do you do when they're not equivalent, or it's hard to switch? You still need to pick. I would argue that if they're not equivalent, the market leader is much safer. Macromedia was eventually bought by Adobe, and so Director is now Adobe Director. Director was the market leader in 1998, and it's still being developed and still available for purchase, almost 20 years later.

mTropolis may have been better, but mTropolis wasn't the market leader. And mTropolis is dead, and has been for a very long time.

Making the choice

So which do you go for, when you have the choice?

If you're dealing with open source software, much of the problem goes away. Even if the company sponsoring the software shuts down, access to the source code gives you a way to switch off the software gradually.

With Software-as-a-Service you're back in the realm of choosing between monopoly pricing and chance of software disappearing. And at least with mTropolis the developers still could use their licensed copies; an online SaaS can shut down at any time.

Personally I'd err on the side of choosing the market leader, but it's hard to give a general answer. Just remember: the proprietary software you rely on today might be gone tomorrow. Be prepared.

02 Jan 2017 5:00am GMT

23 Dec 2016

feedPlanet Twisted

Ralph Meijer: Changes

For me, Christmas and Jabber/XMPP go together. I started being involved with the Jabber community around the end of 2000. One of the first things that I built was a bot that recorded the availability presence of my online friends, and show this on a Christmas tree. Every light in the tree represents one contact, and if the user is offline, the light is darkened.As we are nearing Christmas, I put the tree up on the frontpage again, as many years before.

Over the years, the tooltips gained insight in User Moods and Tunes, first over regular Publish-Subscribe, later enhanced with the Personal Eventing Protocol. A few years later, Jingle was born, and in 2009, stpeter wrote a great specification that solidifies the relationship between Christmas and Jabber/XMPP.

Many things have changed in those 16 years. I've changed jobs quite a few times, most recently switching from the Mailgun team at Rackspace, to an exciting new job at VimpelCom as Chat Expert last April, working on Veon (more on that later). The instant messaging landscape has changed quite a bit, too. While we, unfortunately, still have a lot of different incompatible systems, a lot of progress has been made as well.

XMPP's story is long from over, and as such I am happy and honored to serve as Chair of the XMPP Standards Foundation since last month. As every year, my current focus is making another success of the XMPP Summit and our presence with the Realtime Lounge and Devroom at FOSDEM in Brussels in February. This is always the highlight of the year, with many XMPP enthousiasts, as well as our friends of the wider Realtime Communications, showing and discussing everything they are working on, ranging from protocol discussions to WebRTC and IoT applications.

Like last year, one of the topics that really excite me is the specification known as Mediated Information eXchange (MIX). MIX takes the good parts of the Multi User Chat (MUC) protocol, that has been the basis of group chat in XMPP for quite a while, redesigned on top of XMPP Publish-Subscribe. Modern commercial messaging systems, for business use (e.g. Slack and HipChat), as well as for general use (e.g. WhatsApp, WeChat, Google's offerings), have tried various approaches on the ancient model of multi-part text exchange, adding multi-media and other information sources, e.g. using integrations, bots, and cards.

MIX is the community's attempt to provide a building block that goes beyond the tradional approach of a single stream of information (presence and messages) to a collection of orthogonal information streams in the same context. A room participant can select (manually or automatically by the user agent) which information streams are of interest at that time. E.g. for mobile use or with many participants, exchanging the presence information of all participants can be unneeded or even expensive (in terms of bandwidth or battery use). In MIX, presence is available as a separate stream of information that can be disabled.

Another example is Slack's integrations. You can add streams of information (Tweets, continuous integration build results, or pull requests) to any channel. However, all participants have no choice to receive the resulting messages, intermixed with discussion. The client's notification system doesn't make any distinction between the two, so you either suffer getting alerts for every build, or mute the channel and possibly miss interesting discussion. The way around it is to have separate channels for notifications and discussion, possibly muting the former.

Using MIX, however, a client can be smarter about this. It can offer the user different ways to consume these information streams. E.g. notifications on your builds could be in a side bar. Tweets can be disabled, or modeled as a ticker. And it can be different depending on which of the (concurrent) clients you are connected with. E.g. the desktop or browser-based client has more screen real-estate to show such orthogonal information streams at the same time, a mobile client might still show the discussion and notifications interleaved.

All-in-all MIX allows for much richer, multi-modal, and more scalable interactions. Some of the other improvements over MUC include persistent participation in channels (much like IRC bouncers, but integrated), better defined multi-device use (including individual addressing), reconnection, and message archiving. I expect the discussions at the XMPP Summit to tie the loose ends as a prelude to initial implementations.

I am sure that FOSDEM and the XMPP Summit will have many more exciting topics, so I hope to see you there. Until then, Jabber on!

23 Dec 2016 1:28pm GMT

20 Dec 2016

feedPlanet Twisted

Itamar Turner-Trauring: The one technology you need to learn in 2017

If you want to become a better programmer or find a better-paying job, you might wonder if there's a particular technology you should learn in the coming year. Once you learn this technology you will become far more productive, and provide far more value to current or future employers.

There is something you can learn, but it's not a new web framework, or a programming language, or an IDE. In fact, it's not a technology at all.

It's better than that.

Much better.

Why you need to learn this

Imagine you're a contractor for an online pet-supplies store, selling supplies for cats and dogs.

One Monday the owner of the store asks you to add a new section to the website, selling supplies for aardvarks. Being a skilled designer as well as a programmer you can update the website design, then change the database, write some new code... By the end of the week you've launched the new Aardvark section.

Next Monday the owner is back. This week you're adding badgers, so you add another section of the design, update the code to support another pet, and again you're done after a week. The next week is crabs, and so on and so forth until by the middle of the year you've added zebras.

Unfortunately by this point your website has 30+ sections, and almost all of them are for pets no one wants or owns. No one can find the cat and dog supplies amid the noise, sales are plummeting, and the owner of the store can't afford to pay you any more.

Knowing a better web framework or a more suitable programming language would have let you code the weekly animal's section faster. But coding the wrong thing faster is just as much a waste of time.

Even if you knew every programming language and web framework under the sun you'd still be missing a key skill.

The skill you need to learn

How could this problem have been prevented? Let's get in our time machine and go back to the beginning. Cars drive backwards, raindrops rush up to the clouds, flowers refold, the sun flies from west to east alternating with darkness...

One Monday the owner of the store asks you to add a new section to the website, selling supplies for aardvarks. And you think for a moment, and ask: "why do you want to do that? most people don't own aardvarks."

With a bit more probing you find out the reason the store owners wants to add aardvarks. Perhaps the store wants to sell supplies for all animals, in which case you can code generic animal support once, instead of adding a new one per week. Perhaps this is a misguided search engine optimization strategy, in which case you can suggest adding a blog.

If you ask and figure out the reason for a task, figure out the goal... that is an immensely useful skill. You can only succeed at a project if you know its goal.

Working towards goals

If you can figure out what your boss really needs, what your client's real problem is, you can actually solve their problem. Otherwise they're the ones who have to solve the problem; you're just a hired hand implementing someone else's solution.

If you can figure out what the real goal is you can make sure you're working towards it. You can avoid the irrelevant work, the nice-to-have work, the attractive but useless work: you can work just on what's needed. You won't be writing coding any faster, but you'll ship sooner and code far more effectively.

Technologies are useful and you need to learn them, but in the end they're just tools. Programming languages come and go, web frameworks come and go: what matters is what you choose to build with them.

Don't learn a technology, learn more fundamental skills. Figuring out root causes and why something needs to be done, discovering the real problem, not just the stated one: these skills will serve you this year, and next year and every year after that.

If you have these core skills, you'll be a far more valuable employee. If you also improve your bargaining position and negotiating skills you will be able to find a job on your own terms: a job with work hours you want, a job you love that doesn't own your life.

Join the course: Getting to a Sane Workweek

Don't let your job take over your life. Join over 720 other programmers on the journey to a saner workweek by taking this free 6-part email course. You'll learn how you can work reasonable hours and still succeed in your career a programmer.

Success! Now check your email to confirm your subscription.

There was an error submitting your subscription. Please try again.

I won't send you any spam. Unsubscribe at any time. Powered by ConvertKit

If you would like to learn the skills you need to get a job that doesn't overwhelm your life, join my free 6-part email course.

20 Dec 2016 5:00am GMT

19 Dec 2016

feedPlanet Twisted

Glyph Lefkowitz: Sourceforge Update

When I wrote my previous post about Sourceforge, things were looking pretty grim for the site; I (rightly, I think) slammed them for some pretty atrocious security practices.

I invited the SourceForge ops team to get in touch about it, and, to their credit, they did. Even better, they didn't ask for me to take down the article, or post some excuse; they said that they knew there were problems and they were working on a long-term plan to address them.

This week I received an update from said ops, saying:

We have converted many of our mirrors over to HTTPS and are actively working on the rest + gathering new ones. The converted ones happen to be our larger mirrors and are prioritized.

We have added support for HTTPS on the project web. New projects will automatically start using it. Old projects can switch over at their convenience as some of them may need to adjust it to properly work. More info here:


Coincidentally, right after I received this email, I installed a macOS update, which means I needed to go back to Sourceforge to grab an update to my boot manager. This time, I didn't have to do any weird tricks to authenticate my download: the HTTPS project page took me to an HTTPS download page, which redirected me to an HTTPS mirror. Success!

(It sounds like there might still be some non-HTTPS mirrors in rotation right now, but I haven't seen one yet in my testing; for now, keep an eye out for that, just in case.)

If you host a project on Sourceforge, please go push that big "Switch to HTTPS" button. And thanks very much to the ops team at Sourceforge for taking these problems seriously and doing the hard work of upgrading their users' security.

19 Dec 2016 1:19am GMT

15 Dec 2016

feedPlanet Twisted

Itamar Turner-Trauring: Experts, True Believers and Test-Driven Development: how expert advice becomes a religion

If you've encountered test-driven development (TDD), you may have encountered programmers who follow it with almost religious fervor. They will tell you that you must always write unit tests before you write code, no exceptions. If you don't, your code will be condemned to everlasting brokenness, tortured by evil edge cases for all eternity.

This is an example of a common problem in programming: good advice by experts that gets turned into a counter-productive religion. Test-driven development is useful and worth doing... some of the time, but not always. And the experts who came up with it in the first place will be the first to tell you that.

Let's see how expert advice turns into zealotry, and how you can prevent it from happening to you.

Expert advice becomes a religion

The problem with experts is that they suffer from Expert Blind Spot. Because experts understand the subject so well, they have trouble breaking it down into concepts and explaining it in ways that make sense to novices.

Thus the expert may say "always write unit tests before you write your code." What they actually meant is this:

  1. Write unit tests before you write your code.
  2. Unless it's code that isn't amenable to unit testing, e.g. because unit tests aren't informative for this particular kind of code.
  3. Or unless it's code that you're going to throw away immediately.
  4. And technically you can write the test after the code, and then break your code and check the test is failing. But this is annoying and a bit more error prone so as an expert I'm not going to mention that at all.

A True Believer in TDD might start arguing with these claims. But consider that even Extreme Programming, where TDD originates, discusses types of coding where unit tests are unnecessary.

In particular a "spike" in Extreme Programming is an architectural prototype, used to figure out the structure of the code you're going to write. Since it's going to be thrown away you don't need to write unit tests when you write a spike. You can see this visually in this overview of Extreme Programming; the Architectural Spike happens before Iterations, and unit tests are written as part of Iterations.

If you're certain all code is amenable to unit testing, consider these two examples: unit tests aren't very helpful in testing a cryptographically secure random number generator. And if the director of a movie has asked you write some code to 3D render a "really awesome-looking explosion" you won't benefit much from unit tests either, unless you can write a unit test to determine awesomeness.

So if experts know unit-testing-before code isn't always the right answer, why do they say "always write unit tests before you write your code"? Expert blind spot: they can't imagine anyone would write unit tests when they shouldn't.

To the expert, a prototype and code requiring tests are obviously very different things with different goals. But that isn't so obvious to the novice listener.

The novice listener takes the expert at their literal word, and comes to believe that they must always write unit tests before writing code. The novice is now a True Believer. They tell everyone they know "always wrote tests before you write code," and they try to do so under all circumstances.

Thus good advice turns into a counter-productive religion, with unit tests often being written when they needn't or shouldn't be.

Avoiding the trap

How can you keep this from happening to you?

If you're an expert, make sure you explain the qualifications to your statements. Don't say "always do X." Instead say "you should do X, because Y, under circumstance A but not circumstance B."

If you're not an expert things get a bit harder. Consider that both the expert and the True Believer are making the same statement: "always write tests before you write code." How can you tell whether the person telling you this is an expert or a True Believer?

You need to take every piece of advice and figure out when and where it does not apply. An expert's assumptions will often be implicit in the topic they're writing about, so that may give you some hints.

If you're having a two-way conversation try to get them to qualify their statement: ask them to come up with the boundary cases, where this advice is no longer applicable. Bring up the cases you've found where their advice seemingly won't work and see what they say.

An expert, given enough prodding, will be able to come up with cases where their advice isn't applicable, and explain the reason why. But a True Believer won't back down, won't agree to any compromise: they know the The Truth, and it is absolute and unchanging.

Programming is a broad field, and what makes good software depends on your circumstances, goals and tools. There is almost never one answer that applies everywhere. Learn from the experts, but never become a True Believer.

15 Dec 2016 5:00am GMT

Glyph Lefkowitz: Don’t Stop Tweeting On My Account

Shortly after my previous post, my good friend David Reid not-so-subtly subtweeted me for apparently yelling at everyone using a twitter thread to be quiet and stop expressing themselves. He pointed out:

Threads are being used to say things which might not otherwise be said.

If you only see threads from blowhards stop following blowhards.

- dreid (@dreid) December 14, 2016

This is the truth. There are, indeed, important, substantial essays being written on Twitter, in the form of threads. If I could direct your attention to one that's probably a better use of your time than what I have to say here, this is a great example:

Who radicalized Dylan Roof? And why aren't more people asking this question?

- Christiana A Mbakwe (@Christiana1987) December 13, 2016

Moreover, although the twitter character limit can inhibit the expression of nuance, just having a blog is not a get-out-of-jail-free card for clumsy, hot takes:

@glyph 🎶 isn't it ironic 🎶 that your blog post failed capture nuance 🎶

- dreid (@dreid) December 14, 2016

I screwed this one up. I'm sorry.

The point I was trying to primarily focus on in that post is that a twitter thread demands a lot of attention, and that publishers exploiting that aspect of the medium in order to direct more attention to themselves1 are leveraging a limited resource2 and thereby externalizing their marketing costs3. Further, this idiom was invented by4, and has extensively been used by people who don't really need any more attention than they already have.

If you're an activist trying to draw attention to an important cause, or a writer trying to find your voice, and social media (or twitter threads specifically) has helped you do that, I am not trying to scold you for growing an audience on - and deriving creative energy from - your platform of choice. If you're leveraging the focus-stealing power of twitter threads to draw attention to serious social issues, maybe you deserve that attention. Maybe in the face of such issues my convenience and comfort and focus are not paramount. And for people who really don't want that distraction, the 'unfollow' button is, obviously, only a click away.

That's not to say I think that relying on social media exclusively is a good idea for activists; far from it. I think recent political events have shown that a social media platform is often a knife that will turn in your hand. So I would encourage pretty much anyone trying to cultivate an audience to consider getting an independent web presence where you can host more durable and substantive collections of your thoughts, not because I don't want you to annoy me, but because it gives you a measure of independence, and avoids a potentially destructive monoculture of social media. Given the mechanics of the technology, this is true even if you use a big hosted service for your long-form stuff, like Medium or Blogger; it's not just about a big company having a hold on your stuff, but about how your work is presented based on the goals of the product presenting it.

However, the exact specifics of such a recommendation are an extremely complex set of topics, and not topics that I'm confident I've thought all the way through. There are dozens more problems with twitter threads for following long-form discussions and unintentionally misrepresenting complex points. Maybe they're really serious, maybe not.

As far as where the long-form stuff should go, there are very good reasons to want to self-host things, and very good reasons why self-hosting is incredibly dangerous, especially for high-profile activists and intellectuals. There are really good reasons to engage with social media platforms and really good reasons to withdraw.

This is why I didn't want to address this sort of usage of twitter threading; I didn't want to dive into the sociopolitical implications of the social media ecosystem. At some point, you can expect a far longer post from me about the dynamics of social media, but it is going to take a serious effort to do it justice.

A final thought before I hopefully stop social-media-ing about social media for a while:

One of the criticisms that I received during this conversation, from David as well as others who contacted me privately, is that I'm criticizing Twitter from a level of remove; implying that since I'm not fully engaged with the medium I don't really have the right (or perhaps the expertise) to be critical of it. I object to that.

In addition to my previously stated reasons for my reduced engagement - which mostly have to do with personal productivity and creative energy - I also have serious reservations about the political structure of social media. There's a lot that's good about it, but I think the incentive structures around it may mean that it is, ultimately, a fundamentally corrosive and corrupting force in society. At the very least, a social media platform is a tool which can be corrosive and corrupting and therefore needs to be used thoughtfully and intentionally to minimize the harm that it can do while retaining as many of its benefits as possible.

I don't have time to fully explore the problems that I'm alluding to now5 but at this point if I wrote something like "social media platforms are slowly destroying liberal democracy", I'm not even sure if I'd be exaggerating.

When I explain that I have these concerns, I'm often asked the obvious follow-up: if social media is so bad why don't I just stop using it entirely?

The problem is, social media companies effectively control access to an enormous audience, which is now difficult to reach without their intermediation. I have friends, as we all probably do, that are hard for me to contact via other channels. An individual cannot effectively boycott a communication tool, and I am not even sure yet that "stop using it" is the right way to combat its problems.

So, I'm not going to stop communicating with my friends because I have concerns about the medium they prefer, and I'm also not going to stop thinking or writing about how to articulate and address those concerns. I think I have as much a right as anyone to do that.

  1. ... even if they're not doing it on purpose ...

  2. the reader's attention

  3. interrupting the reader repeatedly to get them to pay attention rather than posting stuff as a finished work, allowing the reader to make efficient use of their attention

  4. I'm aware that many people outside of the white male tech nerd demographic - particularly women of color and the LGBTQ community - have made extensive use of the twitter thread for discussing substantive issues. But, as far as my limited research has shown (although the difficulty of doing such research is one of the problems with Twitter), Marc Andreessen was by far the earliest pioneer of the technique and by far its most prominent advocate. I'd be happy for a correction on this point, however.

  5. The draft in progress, which I've been working on for a month, is already one of the longest posts I've ever written and it's barely half done, if that.

15 Dec 2016 3:02am GMT

14 Dec 2016

feedPlanet Twisted

Glyph Lefkowitz: A Blowhard At Large

Update: I've written a brief follow-up to this post to clarify my feelings about other uses of the tweetstorm, or twitter thread, publishing idiom. This post is intended to be specifically critical of its usage as a self-promotional or commercial tool.

I don't like Tweetstorms™1, or, to turn to a neologism, "manthreading". They actively annoy me. Stop it. People who do this are almost always blowhards.

Blogs are free. Put your ideas on your blog.

As Eevee rightfully points out, however, if you're a massive blowhard in your Tweetstorms, you're likely a massive blowhard on your blog, too. So why care about the usage of Twitter threads vs. Medium posts vs. anything else for expressions of mediocre ideas?

Here's the difference, and here's why my problem with them does have something to do with the medium: if you put your dull, obvious thoughts in a blog2, it's a single entity. I can skim the introduction and then skip it if it's tedious, plodding, derivative nonsense.3

Tweetstorms™, as with all social media innovations, however, increase engagement. Affordances to read little bits of the storm abound. Ding. Ding. Ding. Little bits of an essay dribble in, interrupting me at suspiciously precisely calibrated 90-second intervals, reminding me that an Important Thought Leader has Something New To Say.

The conceit of a Tweetstorm™ is that they're in this format because they're spontaneous. The hottest of hot takes. The supposed reason that it's valid to interrupt me at 30-second intervals to keep me up to date on tweet 84 of 216 of some irrelevant commentator's opinion on the recent trend in chamfer widths on aluminum bezels is that they're thinking those thoughts in real time! It's an opportunity to engage with the conversation!

But of course, this is a pretense; transparently so, unless you imagine someone could divine the number after the slash without typing it out first.

The "storm" is scripted in advance, edited, and rehearsed like any other media release. It's interrupting people repeatedly merely to increase their chances of clicking on it, or reading it. And no Tweetstorm author is meaningfully going to "engage" with their readers; they just want to maximize their view metrics.

Even if I cared a tremendous amount about the geopolitics of aluminum chamfer calibration, this is a terrible format to consume those thoughts in. Twitter's UI is just atrocious for meaningful consideration of ideas. It's great for pointers to things (like a link to this post!) but actively interferes with trying to follow a thought to its conclusion.

There's a whole separate blog in there about just how gross pretty much all social-media UI is, and how much it goes out of its way to show you "what you might have missed", or things that are "relevant to you" or "people you should follow", instead of just giving you the actual content you requested from their platform. It's dark patterns all the way down, betraying the user's intent for those of the advertisers.

My tone here probably implies that I think everyone doing this is being cynically manipulative. That's possibly the worst part - I don't think they are. I think everyone involved is just being slightly thoughtless, trying to do the best that they can in their perceived role. Blowhards are blowing, social media is making you be more social and consume more media. All optimizing for our little niche in society. So unfortunately it's up to us, as readers, to refuse to consume this disrespectful trash, and pipe up about the destructive aspects of communicating that way.

Personally I'm not much affected by this, because I follow hardly anyone4, I don't have push enabled, and I would definitely unfollow (or possibly block) someone who managed to get retweeted at such great length into my feed. But a lot of people who are a lot worse than I am about managing the demands on their attention get sucked into the vortex that Tweetstorms™ (and related social-media communication habits) generate.

Attention is a precious resource; in many ways it is the only resource that matters for producing creative work.

But of course, there's a delicate balance - we must use up that same resource to consume those same works. I don't think anyone should stop talking. But they should mindfully speak in places and ways that are not abusive of their audience.

This post itself might be a waste of your time. Not everything I write is worth reading. Because I respect my readers, I want to give them the opportunity to ignore it.

And that's why I don't use Tweetstorms™5.

  1. Hi Ned.

  2. Like, for example, you can do with this blog!

  3. I subscribe to more RSS feeds than Twitter people by about an order of magnitude, and I heartily suggest you do the same.

14 Dec 2016 2:55am GMT

10 Dec 2016

feedPlanet Twisted

Moshe Zadka: On Raising Exceptions in Python

There is a lot of Python code in the wild which does something like:

raise SomeException("Could not fraz the buzz:"
                    "{} is less than {}".format(foo, quux)

This is, in general, a bad idea. Exceptions are not program panics. While they sometimes do terminate the program, or the execution thread with a traceback, they are different in that they can be caught. The code that catches the exception will sometimes have a way to recover: for example, maybe it's not that important for the application to fraz the buzz if foo is 0. In that case, the code would look like:

except SomeException as e:
    if ???

Oh, right. We do not have direct access to foo. If we formatted better, using repr, at least we could tell the difference between 0 and "0": but we still would have to start parsing the representation out of the string.

Because of this, in general, it is better to raise exceptions like this:

raise SomeException("Could not fraz the buzz: foo is too small", foo, quux)

This way exception handling has a lot of power: it can introspect foo, introspect quux and introspect the string. If by some reason the exception class is raised and we want to verify the reason, checking string equality, while not ideal, is still better than trying to match string parts or regular expression matching.

When the exception is presented to the user interface, in that case, it will not look as nice. Exceptions, in general, should reach the UI only in extreme circumstances: and in those cases, having something that has as much information is useful for root cause analysis.

10 Dec 2016 4:42am GMT

08 Dec 2016

feedPlanet Twisted

Itamar Turner-Trauring: Don't Get Stuck: 6 ways to get unstuck and code faster

A lot of my time as a programmer, and maybe yours as well, is spent being stuck. My day often goes like this:

  1. Write some code.
  2. Run the tests.
  3. "It failed."
  4. "Why did it fail?"
  5. "I don't know."
  6. "That makes no sense."
  7. "Seriously, what?"
  8. "That's impossible."
  9. "Lets add a print statement here."
  10. "And maybe try poking around with the debugger."
  11. "Oh! That's it!"
  12. "No wait, it isn't."
  13. "Ohhhhhhhh there we go."
  14. Run the tests.
  15. Tests pass.
  16. "Time for snacks!"

Given how much time you can end up wasting in this mode, Kaitlin Duck Sherwood points out that not getting stuck is one of the keys to programmer productivity. Imagine how much time I could save if I skipped steps 5 to 13 in the transcript above.

Not getting stuck will make you far more productive. Here are six ways to keep you from getting stuck:

Recognize when you're stuck

If you don't know you're stuck then you can't do anything about it, so the first step is having some way of measuring progress. Whenever you start a coding task you should have a time estimate in mind.

The time estimates should be short, no more than a few hours or a day, so a bigger project should be broken up into smaller tasks. The estimates don't have to particularly accurate, they just have to be in the right range: a task you estimate at a few hours should not require days of work.

Given an estimate of 4 hours, say, you can then recognize whether you're actually stuck:

Comparing actual time spent to the initial estimate tells you if you're making progress, and working in small chunks ensures you recognize problems quickly.

Ask for help

Now that you've recognized you're stuck, the next thing to do is find a solution. The easiest thing to do is talk to a colleague.

This is helpful in multiple ways:

In fact, this is so useful that sometimes you don't need a person, and talking to a rubber duck will do. I like debugging out loud for this reason, so occasionally I use a #rubberduck Slack channel so I don't distract my coworkers.

Eventually I shared my confusion, and someone noted expression relies on Double.equals(), and then went and found the Double.equals() API documentation. And indeed, the documentation notes that new Double(0.0).equals(new Double(-0.0)) is false even though in Java 0.0 == -0.0 is true, because reasons.

Use or copy an existing solution

If you or your colleague can't find a solution on your own, you can try using some one else's solution. This can range from the copy/paste-from-StackOverflow fallback (but be careful, sometimes StackOverflow code is wrong or insecure) to copying whole designs.

For example, I built a multicast data distribution protocol. This is not a trivial thing to do, so I found a research paper and copied their design and algorithm. Designing such an algorithm would have been far beyond my abilities, but I didn't have to: someone smarter and more knowledgeable had done the hard work.

Find a workaround

Sometimes you're faced with an important bug you can't fix. Working around it may suffice, however, as you can see in this story shared by reader James (Jason) Harrison:

Several years ago, I was working many late nights on a new Wii game that was going to have gesture recognition. The first part of the game activities went as smoothly as could be expected and then we came to a new level where the player was supposed to bring the Wiimote up and then down quickly. This must have tripped on a bug in the system because this gesture could not be reliably recognized.

Replaying recorded motions demonstrated that the problem wasn't "just" in the data form the Wiimote or in how the player made the motion but in the system. Instead of being deterministic, the system would work then not work. Looked for variables that were not being initialized, data buffers not being cleared, and all state that could possibly leak from previous inputs.

Unfortunately, all of the searching didn't find the problem in time. So it was decided to reset the recognition system between inputs. While wasteful, it was the only thing that did fix the system and let us ship the milestone to the publisher. Left a comment in to find the problem later. Never did find it. Game was shipped with this fix.

Drop the feature

If you're working on a feature and it's taking forever, maybe it's time to consider dropping it. Can it wait until the next release? Do you actually need it?

A feature usually lands on the requirements list for a reason, it's true. But a surprising number of features are in the nice-to-have category, especially when it's taking far too long to implement them. If other approaches have failed to get you unstuck, find the person in charge and give them the choice of shipping late or dropping the feature.

Redefine the problem

Finally, another approach you can take is redefining the problem. The way you're approaching the problem may be making it needlessly complicated to solve, so revisiting the problem statement can help get you unstuck.

You can redefine the problem by relaxing some of the constraints. For example, maybe you're having a hard time finding a date picker that matches the website design. If the problem statement is "add a usable, good looking, date picker that matches our website style" then you might spend a while looking and not finding any that are quite right.

Often you can redefine the problem, though, to "find a minimal date picker so we can demo the feature and see if anyone cares." With that more limited scope you can probably use one of the options you initially rejected.

You can also redefine the problem by figuring out the real problem lies elsewhere. Havoc Pennington has a wonderful story about the dangerous "UI team": they will feel their mandate is to build UIs. But software that doesn't have a UI and "just works" is an even better user experience, if you can manage it.

In short, to keep from getting stuck you should:

  1. Break all your work up into small chunks.
  2. Estimate each chunk in advance, and pay attention to your progress against the estimate.
  3. When you recognize you are stuck: ask for help, copy an existing solution, find a workaround, drop the feature or redefine the problem.

I've learned most of this the hard way, over the course of 20 years of being stuck while writing software. If you'd like to avoid the many mistakes I've made as a software engineer during that time, both coding and in my career, sign up for my Software Clown newsletter. You'll get the story of one of my mistakes in your inbox every week and how you can avoid making it.

Avoid my programming mistakes!

Get a weekly email with one of my many software and career mistakes, and how you can avoid it. Here's what readers are saying:

"Are you reading @itamarst's "Software Clown" newsletter? If not, you should be. There's a gem in every issue." - @glyph

"Definitely subscribe if you want to learn some things that Itamar learned the hard way." -- Victor Algaze

Success! Now check your email to confirm your subscription.

There was an error submitting your subscription. Please try again.

I won't share your email with anyone else. Unsubscribe at any time. Powered by ConvertKit

08 Dec 2016 5:00am GMT

Moshe Zadka: Moshe’z Messaging Preferences

The assumption here is that you have my phone number. If you don't have my phone number, and you think that's an oversight on my part, please send me an e-mail at zadka.moshe@gmail.com and ask for it. If you don't have my phone number because I don't know you, I am usually pretty responsive on e-mail.

In order of preference:

08 Dec 2016 3:03am GMT

05 Dec 2016

feedPlanet Twisted

Jp Calderone: Twisted Web in 60 Seconds: HTTP/2

Hello, hello. It's been a long time since the last entry in the "Twisted Web in 60 Seconds" series. If you're new to the series and you like this post, I recommend going back and reading the older posts as well.

In this entry, I'll show you how to enable HTTP/2 for your Twisted Web-based site. HTTP/2 is the latest entry in the HTTP family of protocols. It builds on work from Google and others to improve performance (and other) shortcomings of the older HTTP/1.x protocols in wide-spread use today.

Twisted implements HTTP/2 support by building on the general-purpose H2 Python library. In fact, all you have to do to have HTTP/2 for your Twisted Web-based site (starting in Twisted 16.3.0) is install the dependencies:

$ pip install twisted[http2]

Your TLS-based site is now available via HTTP/2! A future version of Twisted will likely extend this to non-TLS sites (which requires the Upgrade: h2c handshake) with no further effort on your part.

If you like this post or others in the Twisted Web in 60 Seconds series, let me know with a donation! I'll post another entry in the series when the counter hits zero. Topic suggestions welcome in the comment section.

05 Dec 2016 12:00pm GMT