18 May 2020

feedPlanet Twisted

Itamar Turner-Trauring: To get a better programming job, explain your problem-solving skills

When you're looking for a new programming job, how do you explain your value? The usual approach is a long list of technologies, but this leaves out a critical skill: your ability to solve problems.

If you can convey your level of skill at problem solving, you can get:

Often just a few extra words can make a big difference in demonstrating your skills. Let's see how you can do it.

Three levels of problem solving skills

As I discussed elsewhere in more detail, problem solving comes in three stages, each approximately corresponding to a particular career stage (these names are due to Randall Koutnik):

  1. Staff or principal software engineers are Finders: they find new problems.
  2. Senior software engineers are Solvers: they solve already-identified problems.
  3. Junior software engineers are Implementers: they implement already-identified solutions.

The earlier you are in the problem-solving process, the more productive you are, and therefore the more valuable as an employee.

As a result, you need to communicate how advanced your skill is across these three levels to demonstrate your productivity. Everything from your resume to the stories you tell in interviews should communicate your level of skill.

Explaining your skill level

Explaining your skill level involves telling stories that use the correct words and sufficient information to demonstrate your skill. I'm going to use resumes as an example here, but you should ensure you do this in interviews as well-if you're practicing with a friend, make sure they're checking for this, it's easy to leave the information out.

Consider the following entry from a resume:

Moved deployment from manually-managed hosts to a new Kubernetes cluster.

This experience entry uses an implementation-level verb: "moved". Similarly, "coded", "tested", "wrote", "fixed", "optimized"-these are all about implementation. And maybe it's implementation was tricky and difficult, and it's good to convey that, but if you also solved the problem or identified the problem it's impossible to tell from this phrasing.

Any task you write about in your resume and talk about in interviews was the result of someone identifying a problem, and someone coming up with the solution. If it was you, make sure you say that.

Let's say in our example above you were the one tasked with figuring out an alternative to manually-managed hosts. If so, you need to add additional context and verbs that convey that:

Investigated alternatives to manually-managed hosts, decided on Kubernetes, and moved the deployment to a new cluster.

Now we can clearly see you solved the problem.

If you were the one who identified the problem, again you need to make sure you explicitly call that out:

Identified manually-managed hosts as an operational problem, and got management buy-in to change to a better system. Subsequently investigated alternatives, decided on Kubernetes, and moved the deployment to a new cluster.

Now we can see all the value you provided.

If you only identified a problem, that's fine too, just say so:

Noticed a critical customer-facing bug that was impacting many users; after the team responsible for that area fixed it, they reported a $300,000 increase in revenue as a result.

Communicating value at different career stages

Now that you've seen how to phrase your skill level, let's see how this works at different career stages.

Implementers

When you're at the start of your career you will mostly be implementing other people's solutions. However, in one or two cases you might be starting to solve problems, or even find problems. Make sure your resume highlights those instances, however small.

Sometimes this will happen in non-programming contexts. For example, I knew one early stage software engineer who made very insightful suggestions about hiring. Mention it anyway.

If you had another career before switching to programming, you'll likely have plenty of examples. Make sure to highlight those even if they're unrelated to coding; those problem-finding and solving skills will at least partially transfer.

Solvers

If you can solve problems on your own, you want to both:

  1. Communicate this fact.
  2. Highlight the places where you did identify problems, even if it's happened in only a few cases.

Review your resume and make sure all the relevant entries explicitly talk about the ways in which you came up with the solution. If you already have a suitable job title, like senior software engineer, then getting the phrasing right isn't quite as important-but it's still worth doing.

If you still have a junior job title but your skills have progressed, it's doubly important to ensure you're highlighting your problems solving skills.

Once you've done that, try to expand on any places where you were involved in identifying problems.

Finders

Your goal as a Finder is to ensure you're not confused with a Solver. It's very easy to phrase things in a way that doesn't make clear you identified the problem-after all, identifying the problem may only have a taken a few minutes.

But those initial few minutes where you noticed something needs to be done are quite often the most value parts of the whole process, so make sure you explicitly talk about finding the problem. This is even more important if your current job title doesn't reflect your actual level of skill.

What to do next

Even if submitting resumes isn't the best way to find a job, you still need one, and writing it is a good way to rehearse for an interview.

So get your resume out, and for each experience entry make sure it's clear whether you implemented the solution, solved the problem, and/or found the problem. This will take you an hour, no more, and at the end of this process you'll have a much easier time communicating some of your most valuable non-technological skills.

And if you'd like to learn some more of those skills, check out my new book, The Secret Skills of Productive Programmers.



Tired of scrambling to get your job done?

If you were productive enough, you could take the afternoon off, confident you'd produced high value work. Not to mention having an easier time finding a new job when you need one.

Learn the secret skills of productive programmers.

18 May 2020 4:00am GMT

14 May 2020

feedPlanet Twisted

Itamar Turner-Trauring: How to prepare for losing your programming job

Another week has passed, and another 3 million people in the US have filed for unemployment. While the current situation hasn't impacted programming jobs quite as much, it's just a matter of time before the economic damage hits most everywhere. There will be layoffs, and plenty of them, and occasionally whole companies shutting down.

So even if your job is secure now, you might still lose it in the future. How can you prepare? What can you do to reduce your future risks?

The first thing you need to do is come up with a plan, which is what this article is all about. In particular, you will want to:

Let's go over these one-by-one.

Money in the bank

If you lose your job, you lose your paycheck-but you still have to pay your bills. And after the dot-com bust, the last big tech recession, it took years for all the jobs to come back

If you have at least six months of living expenses in cash, that's a good start. If not, it's best to think about how to get there.

There are two sides to this:

  1. If possible, you need to cut your expenses, which will both allow you to save and reduce how much money you need for each unsalaried month. See this more detailed article.
  2. Ensuring your financial assets, if you have any, aren't correlated with your job.
    1. If you own stock in your own company, you are making a double bet: if the company goes down, you will lose money and your job.
    2. If you work for a startup that needs to raise money soon, a crashing stock market will also greatly reduce the viability of your current job.
    3. More broadly, if you own stocks and to a lesser extent corporate bonds, how correlated are they with your ability to keep a job?
    4. Even more broadly, how much of your net worth is tied to the tech industry, or the economy as a whole?

In short, you want cash on hand, and plenty of it.

Making your future job hunt easier

Searching for a job will be much easier if you:

Let's cover those one by one.

Knowing lots of people

Applying for a job by sending in your resume is the hardest way to get hired. It's much easier if you know someone who can vouch for you, can get you past the initial screen, or can fill you in on what the hiring manager really wants.

So the more people you know, the better off you are. Elsewhere I have a guest post about (social) networking, but that can take time and is harder during a pandemic. But there are still a few easy things you can do in the short term:

Build useful skills

If you've been working at the same job for a while, it's easy for your technical skills to get a little stale. Unless you're working at the right place, hang out with the right people, or do the right things, you might not be aware of the latest technology, or you might be using out-of-date practices.

So you'll want to update your skills a little. As always, doing this extensively outside of your job may not be possible, so try to:

  1. Spend an hour a week, ideally during work hours, getting up-to-date on the latest technologies. The goal here is breadth, not depth: sign up for a newsletter for your technology stack (he's a partial list), skim the topics at a relevant conference, maybe watch a talk or two. I cover learning for breadth here, but the basic idea is that knowing a tool exists and what it does can take very little time, and is quite valuable on its own: both on the job, but also in interviews ("I haven't used it myself, but I believe tool X is how you would solve this").
  2. Try to learn more technologies on the job, because that is the best place to do so.

Create visible proof of skills

Having skills is one thing, proving you have them is another. It is therefore quite useful during a job hunt to have some visible, public proof you have these skills. For example:

Open source: When I moved to the US in my 20s, my work on an open source project made it much easier for me to get job interviews, and eventually job offers. It wasn't just that my resume said that I knew computer networking, I could point to a publicly available project used by real people and say "I worked on that".

Even if you share code that isn't widely used, it can still be useful as proof of skill.

Conference talks: Speaking publicly about a particular skill, technology, or project is a great way to get public proof of skills. With conferences moving online, speaking at conferences is now much easier. You don't have to travel or pay for you travel, and you don't have to get approval from your manager to lose work. If there's a topic where you know enough to help someone else, look for conferences on the topic and submit a proposal.

Blogging: Have something to share, or learning something new? Write it down and share it publicly. Writing well is an immensely useful skill in general, so this will also count as improving your skills. You can write for your own blog, or you can propose a blog post on your company's tech blog, if they have one.

Fallback plans

In an ideal world you would lose your job, start a job search, find a new job within a month, and everything will be fine. Sadly we don't always live in an ideal world.

So if you live in a country like the US that a shitty social net it's worth coming up with a series of fallback plans, if only for your own peace of mind.

For example, how can you make your money last longer?

  1. As soon as you lose your job, apply for unemployment.
  2. Cut additional costs.
  3. If time stretches on and you still don't have a job, figure out ways to reduce housing costs. Are you young and have the ability to move back in with your parents? Have more room than you need and the option of adding roommates? All in a pandemic-safe way, of course.
  4. Any ways you can make money some other way, if it's really taking too long?

If you can't find a job immediately, you will have probably have more time to upgrade your skills.

  1. Which skills are worth working on?
  2. What's the best way to improve them?

You'll also want to meet more people who can help you find a job.

  1. Can you go to online meetups?
  2. Find more places to interact with people online?

Write this all down, and when you're worried you'll at least have the comfort of knowing there will be some things you can do if and when your job goes away.

We're all in this together

As with most big problems, there is only so much you can do as an individual: to meaningfully improve the situations we need to work together, whether in mutual aid groups or via political organization. On the other hand, you need to ensure that you as an individual are doing OK; you can't help others if you're collapsing under your own troubles.

And since this can all be overwhelming, start with a few simple actions:

  1. Cut an expense or two.
  2. Get in touch with some old co-workers.
  3. Sign up for a newsletter.
  4. Start writing down your fallback plans.

And then, once you have things under control emotionally, when you have a plan and you know what you're doing next, start thinking about how you can help others, and work with other people to improve things for everyone.



Tired of scrambling to get your job done?

If you were productive enough, you could take the afternoon off, confident you'd produced high value work. Not to mention having an easier time finding a new job when you need one.

Learn the secret skills of productive programmers.

14 May 2020 4:00am GMT

04 May 2020

feedPlanet Twisted

Hynek Schlawack: Why You Should Document Your Tests

Some projects have the policy that all tests must have an explanatory comment - including all of mine. At first, I found that baffling. If that's you right now, this article is for you.

04 May 2020 12:00am GMT

27 Apr 2020

feedPlanet Twisted

Moshe Zadka: Numbers in Python

Numbers in Python come in all shapes and forms. The reason different kind of representations of numbers exist is because they all have different trade-offs. These trade-offs are often surprising!

Integers

The most surprising things about integers is how easily they stop being integers. Dividing two integers, for example, 4/3, gives a float, and (4/3)*3 is the float 4.0. Even if a program has no floating point numbers coming in, all that is needed for floating point numbers to exist somewhere is a division operation.

Floats

Floats do not behave like numbers. Numbers obey certain mathematical properties: subtraction is the inverse to addition, addition is associative, and more.

For example

>>> 1 + 2 - 2 - 1
0
>>> 0.1 + 0.2 - 0.2 - 0.1
2.7755575615628914e-17

adding two numbers, and then subtracting them one at a time, does not result in the same value.

They do not obey the associative law of addition, a + (b + c) = (a + b) + c:

>>> a = 2**-53
>>> (a + a) + 1 == a + (a + 1)
False

These show just two of the corner cases that floating point numbers exhibit, which can be surprising. A full treatise on the ways that floating point behavior can be surprising is too big to fit in the margin of this blog post.

Fractions

Many algorithms that look straightforward "explode" with exact fractions. Explosion usually starts as time explosion: the algorithm becomes "quadratic": the time it takes is proportional not to the input length, but to the scare of the input's length. In other words, doubling the input size quadruples the time it takes.

If enough time is spent, memory explosion is also possible: the space requirements increase, until all memory fills up.

One weird protection against memory explosion is that usually it will take too long to get it, and the program will be killed for "hanging".

One such "algorithm" is addition.

>>> print(set(type(p) for p in primes))
>>> one = fractions.Fraction(1)
>>> before = datetime.now()
>>> res = sum(one/p for p in primes[:10000])
>>> after = datetime.now()
>>> print("It took", after-before)
>>> print("Size of output", len(str(res)))
>>> print("Approximate value", float(res))
{<class 'int'>}
It took 0:01:16.033260
Size of output 90676
Approximate value 2.7092582487972945

This is just adding the inverses to some primes (I removed the first few from the list, and then chopped the list to be the next 10,000). On a nice laptop designed as a gaming rig, adding 10,000 numbers took over a minute, and resulted in an output that was over 90K!

In comparison, running the same algorithm with floats is much more efficient:

>>> print(set(type(p) for p in primes))
>>> before = datetime.now()
>>> res = sum(1/p for p in primes[:10000])
>>> after = datetime.now()
>>> print("It took", after-before)
>>> print("Size of output", len(str(res)))
>>> print("Approximate value", float(res))
{<class 'int'>}
It took 0:00:00.000480
Size of output 17
Approximate value 2.709258248797317

The time it took is less than a millisecond, and some of that is possibly measurement error from datetime. This is around 10,000 times faster. The output can be saved in 17 bytes: a mere 1000 reduction in space. However, the result is inaccurate:

Approximate value 2.7092582487972945
Approximate value 2.709258248797317
                    1234567891234

The results are off by less than 1e-14. This would be like getting the distance to the moon wrong by one millimeter. In cases that do not involve sending a rocket to the moon with less than a millimeter (one grain of sand) tolerance, floats give a result that is precise enough and several orders of magnitude more efficient.

A lot of the responses to this were along the lines of "fractions are slow because they are implemented in Python". Python can be responsible for a 10x slowdown, but not 10,000x. There is a third-party module, quicktions, which implements fractions using Cython.

Using quicktions was, indeed, quicker. It took the time down from a minute and sixteen seconds to to a minute and fifteen seconds on my laptop.

Fundamentally, the problem is that this is a quadratic algorithm. I chose the inputs carefully: the worst case behavior for fraction addition is on prime numbers. But unless you can predict the inputs to an algorithm, you cannot rely on anything but the worst-case behavior.

Decimals

Decimal numbers are useful when managing financial transactions. This is for the most boring reason possible: the laws governing finance are specified in decimals. However, all decimal point calculations in Python are governed by hidden global state: the context. The context determines precision, and is taken from the caricature of how action at a distance is problematic for APIs.

Quoting the documentation (for Python 3.8):

>>> getcontext().prec = 6
>>> Decimal(1) / Decimal(7)
Decimal('0.142857')
>>> getcontext().prec = 28
>>> Decimal(1) / Decimal(7)
Decimal('0.1428571428571428571428571429')

In practice, code might have hundreds of lines between setting the precision and doing a calculation. The calculation can be in another function, or even another file.

The only safe way to use decimal numbers in Python is with localcontext:

>>> getcontext().prec = 6
>>> # 6853 lines elided
... with localcontext() as ctx:
...     ctx.prec = 10
...     Decimal(1) / Decimal(7)
...
Decimal('0.1428571429')

As long as you are careful to use localcontext, decimals work pretty well. It is thread-safe and signal-safe.

Summary

Before you do things with numbers in your code, stop and think. What types should you use? What do you want to happen? What tolerances are important?

Not thinking means letting the corner cases in the code just happen.

(Thanks to Adi Stav, Aaron Hall, and Avy Faingezicht for their feedback on an earlier draft. All issues and mistakes that remain are my responsibility.)

27 Apr 2020 12:00am GMT

Moshe Zadka: My Little Pony -- DevOps is Magic

(This article is based on the one I originally published on OpenSource.com.)

In 2010, the My Little Pony franchise was rebooted with the animated show My Little Pony: Friendship is Magic. The combination of accessibility to children with the sophisticated themes the show tackled garnered a following that cut across ages. I was swept up in the wave and discovered there is a lot to learn about DevOps from the show.

The show begins with Twilight Sparkle reading obscure documentation, only to realize that Equestria, where the show is set, is due to suffer a calamity. Though Nightmare Moon has been imprisoned for a thousand years, there is a prophecy she will return.

Lesson 1: Technical debt matters.

Document technical debt. Pay attention to the signs of risk no matter how infrequently they occur. Have a plan to resolve it.

Twilight Sparkle goes to her manager with the news, only to be told that it is not a current priority. She is sent to Ponyville to prepare for the coming celebration, instead.

Lesson 2: Communication with management is key.

Twilight Sparkle communicated her priority but did not convince her management that it was more important than the celebration.

We all need to make clear what the business case is for resolving critical issues. It is also not straightforward to explain technical debt in business terms. If management does not agree on the severity, find new ways to communicate the risk, and team up with others who speak that language.

As the prophecy has foreseen, Nightmare Moon returns and declares eternal night. Twilight quickly understands that she cannot resolve the issue by herself, and she recruits the ponies who will become, with her, the "Mane Six." They each stand for a different element of harmony - Applejack stands for Honesty, Fluttershy for Kindness, Pinkie Pie for Laughter, Rarity for Generosity, Rainbow Dash for Loyalty, and Twilight Sparkle herself for Magic.

Lesson 3: Few are the issues that can be resolved by one person.

When facing an outage, reach out to other people with complementary skills who can help you. It is best if they are different than you: different backgrounds leads to differing perspectives, and that can lead to better problem-solving.

Lesson 4: When resolving an outage, honest communication is key.

Throughout the struggle against the eternal night, the Mane Six have to speak openly and honestly about what's not working. Their blameless communication is part of problem-solving.

Lesson 5: When resolving an outage, kindness to yourself and to others is crucial.

Though tempers flare hot in the land of Equestria, we all benefit from coming back to working together.

Lesson 6: Laughter is important.

Even when everything comes crashing down, remember to take a break, drink a glass of water, and take a deep breath. Stressing out does not help anything.

Lesson 7: Be generous.

Even if you are not on-call right now, if your help is needed to resolve a problem, help out as you hope your colleagues will do for you.

Lesson 8: Be loyal.

An outage is not a time to settle rivalries between teams. Focus on how to collaborate and resolve the outage as a team.

Lesson 9: Though people skills are important, you have to understand the technology on a deep level.

Keep your tech skills sharp. Expertise is not only the ability to learn; it is knowing when that information is needed. Part of being an expert is practice.

After the issue is resolved, Princess Celestia realizes that the Mane Six are crucial to the long-term survival of Equestria, and tells Twilight Sparkle to stay in Ponyville and keep researching the magic of friendship.

Lesson 10: After an outage is resolved, conduct a review, take concrete lessons, and act on them.

I could go on, episode by episode, detailing lessons relevant for DevOps, but I will wrap up with one of my favorite ones.

In the "Winter Wrap-Up" episode, all the ponies in Ponyville help in preparing for the spring. As per tradition, they do not use magic, leaving Twilight Sparkle to wonder how she can contribute. Eventually, she realizes that she can help by making a checklist to make sure everything is done in the right order.

Lesson 11: When automation is impossible or inadvisable, write a solid checklist, and follow it. Do not depend on your memory.

Twilight Sparkle and the Mane Six overcome great obstacles as a team, and now have a system to improve as a team. I hope you, too, can help bring a collaborative DevOps culture to your work.

27 Apr 2020 12:00am GMT

22 Apr 2020

feedPlanet Twisted

Moshe Zadka: Goodbye, John H. Conway

John H. Conway passed away ten days ago, and I think it's only now I can write a proper eulogy.

I was first introduced to his work, if not his name, when I was at the end of elementary school. I am sure everyone has heard about the Game of Life, but did you know it had a 1D version? The 1D version is significantly simpler, but has the advantage that on a grid paper, you can just play with yourself manually by putting a generation on each line.

This was 12 year old me's "fidget spinner", how I kept myself calm in classes. Starting with an initial configuration and letting it evolve.

Later on, when I went to college, I got to borrow his amazing book, "On Numbers and Games". Now, I am definitely the sort of person who reads math books for fun, but most of them are not fun. They are dry, poorly written, and make leaps all the time. "ONAG" was the exact opposite. It's a short, delightful book, that tries to get across the thinking, the intuition, the methods, and, yes, the joy.

Fast forward a decade or two, and again I found myself enamored with another one of his inventions: the Look-and-Say sequence. My old interview coding question was getting too popular on the interview-question-sites, and I was getting worried. Writing code for the look-and-say sequence is reasonably straightforward, but does require basic skills: looping while keeping a bunch of state variables.

Then I read about his work on the look-and-say sequence, and was utterly amazed and delighted by it. Atoms and decay and asymptotic growth!

Throughtout his career, I think what made his things special is that he embodies the truest mathematician spirit, which is also the truest geek spirit: starting out with something simple, and then nerding out about it until you have built a whole universe.

Whether it is a place where guns shoot spaceships at 3/8 the speed of light, an algebraic field so vast it includes all other ordered fields and also all infinities, or a concept of numbers atomically decaying, he was a master at whipping out mathematicially consistent fictional worlds.

Goodbye John H. Conway, you were taken from us too soon.

22 Apr 2020 12:00am GMT

20 Apr 2020

feedPlanet Twisted

Itamar Turner-Trauring: The secret skills of productive programmers

This article was written during abnormal circumstances, with much of the planet under lockdown due to the COVID-19 pandemic. Parents with children at home have far less time, and pretty much everyone is feeling stressed and distracted.

Under more normal circumstances there are only so many hours in the day to do your job; now it's even worse. And yet work needs to get done: code needs to get written, features need to be shipped, bugs need to be fixed.

Faced with an ever growing list of tasks, how do you get everything done?

The short answer is, you can't. You will never get everything done.

What you can do, though, is choose the right work, the most valuable work, the most useful work, the work with most leverage. Choose the right work and you can gets orders of magnitude improvement in your output.

Let's see how.

The goal: increased output

Your output as a programmer is based both on your productivity and on how much time you work:

Output = Productivity × Time Worked

The first thing to notice is that there is a hard limit on how much increasing your working hours can help. After all, there are only 168 hours in a week.

If you never slept, ate, or did anything but work-and this will literally kill you-you can work 4.2× as much as a 40-hour workweek, and that's it. And even with smaller increases in work hours, the gains quickly decline. As you work more hours you'll become fatigued and make more mistakes; beyond a certain point those extra work hours will decrease your productivity, canceling out any gains.

What is output for a programmer?

Since increasing working hours isn't really an option, the key to increasing your output is increasing your productivity. Productivity is the output you produce in each fixed unit of time, for example:

Productivity = Output per week

If you're going to improve your productivity, you need to understand how to measure output.

The obvious measure is how much code you write: the more code, the better. This measure is obvious, popular, and completely wrong.

All other things being equal, is it better to implement the same feature with 10 lines of code, or 10,000 lines of code? If we measure output by code produced, the latter solution is better, but in most cases a 10 line solution is preferable to 10,000 lines. More code means higher maintenance costs, not to mention more opportunities for defects.

Your job as a programmer is not writing code, your job is solving problems: software is a tool, a means to an end. Software becomes valuable because of the problems it solves.

As a rough measure, your output as a programmer can be measured by the problems you solve: the more significant the problems you can solve, the better.

If you work for a business, significance eventually translates directly or indirectly into monetary terms: money made or money saved. In other areas you can come up with domain-specific concrete measures of usefulness: number of people served, carbon emissions reduced, number of scientists using your software, and so on.

Note: If the problems you solve produce negative value you will become anti-productive: the better you are at your job, the more damage you will cause.

If making money hurts people or the environment, your work may be productive for your employer but anti-productive for society as a whole. So make sure you're carefully considering the ethical consequences of your actions as a worker.

How to increase productivity

Given the above, here's how you can increase your productivity:

  1. Find the most significant problem you can work on.
  2. Come up with the most efficient solution to that problem.
  3. Implement the solution with minimum wasted time.

Let's go through these steps one by one, and see why they're key to productivity.

1. Find the most significant problem

Let's consider our formula for productivity again:

Productivity = Significant problems solved / Week

There are many problems you could be working on, so first you have to choose one. If you could solve either of these problems, should you be working on:

  1. Implementing a particular missing feature; this will increase revenue by $50,000.
  2. Fixing a bug that was decreasing customer retention; this will increase revenue by $1,000,000.

All other things being equal, the second problem is obviously the one you should be focusing on. Even if it takes 10× as long to solve and implement that bug fix, it should still be the highest priority:

Productivity of #1 =    $50,000 /  1 Week  =  $50,000 / Week
Productivity of #2 = $1,000,000 / 10 Weeks = $100,000 / Week

Here's the issue: in order to fix that expensive bug and improve customer retention, you need to know the problem exists. If no one ever notices that customers are leaving, if no one ever finds that bug, if no one realizes the connection between the two-then that problem will never be solved.

And that's why finding problems is the first and most valuable step in increasing productivity.

2. Come up with an efficient solution

Once you've identified the most significant problem-or once your manager assigns you a problem they identified-you need to come up with a solution.

Which solution do you think is better?

  1. Takes 1000 lines of code and 4 weeks to implement.
  2. Takes 100 lines of code and 3 days to implement.

All other things being equal, the second solution is obviously better. But again, you need to find that solution.

If you only ever find that first solution, then no matter how efficiently you implement it, no matter how focused you are, no matter how much you manage to speed things up-you're still implementing a much less efficient solution.

And that's why identifying better solution is the second most valuable step in increasing productivity.

3. Implement the solution without wasting a time

Once you've identified a problem and chosen the solution, there is only so much leverage you have to improve productivity. You obviously want to avoid getting stuck and spinning your wheels, because wasted time reduces your productivity.

But given a particular solution, there's only so much waste you can reduce, only so fast you can go:

Wasted time → $50,000 / 2 weeks = $25,000 / week
No waste    → $50,000 / 1 week  = $50,000 / week

Efficient implementation is the last and least valuable way of increasing productivity.

Technological skills aren't enough

While you get the most increased productivity from identifying problems and the least from efficient implementation, your career as a programmer progresses in the opposite direction:

  1. Junior engineers implement solutions.
  2. Senior engineers find solutions and implement them.
  3. Principal or staff engineers identify problems, find solutions, and implement them.

So becoming more productive isn't just about helping your employer's bottom line, it's also about learning the skills that will give you more pay and more influence.

Critically, technological skills are necessary but not sufficient to increase your productivity:

Why these skills are "secret"

Most discussions of programming productivity tend to end up focusing purely on technology, coding, and design skills, and skip over these problem-solving skills. Of course, this isn't a conspiracy of silence, no one is deliberately hiding the existence of the skills.

My guess is that experienced programmers still have to learn new technologies, so they're more likely to realize the need to explain those particular skills. But having learned them once, they apply skills like timeboxing, or considering multiple different solutions to a problem, without even noticing they're doing it. And so they end up talking about problem-solving skills rather less, and about technological skills rather more.

How do you learn these skills?

This article is an excerpt from my book, The Secret Skills of Productive Programmers, covering the non-technical skills you need to get better at identifying problems, solving problems, and implementing them on schedule.

Elsewhere on this site you'll find many free articles on building up your skills.



Tired of scrambling to get your job done?

If you were productive enough, you could take the afternoon off, confident you'd produced high value work. Not to mention having an easier time finding a new job when you need one.

Learn the secret skills of productive programmers.

20 Apr 2020 4:00am GMT

14 Apr 2020

feedPlanet Twisted

Moshe Zadka: Using Twisted to Massively Parallelize Web Clients

The Twisted Requests (treq) package is an HTTP client built on the popular Twisted library that is used for web requests. Async libraries offer the ability to do large amounts of network requests in parallel with relatively little CPU impact. This can be useful in HTTP clients that need to make several requests before they have all the information they need.

This post shows an example of a problem like this, and how to solve it using treq.

I enjoy playing the real-time strategy game Clash Royale. Clash Royale is a mobile strategy player-vs-player game where players play cards in an arena to win. Each card has different strengths and weaknesses, and different players prefer different cards. Clash Royale remembers which card a player plays the most; this is their "favorite" card. Players come together in clans where they can help each other. Supercell, Clash Royale's developer, released an HTTP-based API where different statistics can be queried.

How can we write a program that will output the most popular favorite cards in a clan?

If you want to follow along, you will need to register an account. If you register an account, create an API token via the Clash Royale developer portal. Then choose "Create New Key" under your profile, and enter a name, description, and a valid IP address. (An exact address is required.) Since you should never save an API key in your code, keep it as a separate file in ~/.crtoken:

$ ls ~/.crtoken
/home/moshez/.crtoken

To make it easier to see what is going on, let's start with this introductory program that prints Hello world, and then we'll talk through what it does:

import collections, json, os, sys, urllib.parse
from twisted.internet import task, defer
import treq

with open(os.path.expanduser("~/.crtoken")) as fpin:
    token = fpin.read().strip()

def main(reactor):
    print("Hello world")
    return defer.succeed(None)

task.react(main, sys.argv[1:])

This imports many more modules than we need for the "Hello world" example. We will need these modules for the final version of the program, which will accomplish the more complex task of asynchronously querying an API. After the import, the program reads the token from the file and stores it in the variable token. (We are not going to do anything with the token right now, but it's good to see that code.) Next there is a main function that accepts a Twisted reactor. A reactor is sort of like an interface to the machinery of the Twisted package. In this case the function main is sent as a parameter to task.react, and, which will call main with the reactor and any arguments we give -- the command-line arguments, in this case.

The main function returns a defer.succeed(None). This is how it returns a value of the right type: a deferred value, but one that already has been "fired" or "called." Because of that, the program will exit immediately after printing Hello world, as we need.

Next, we will look at the concepts of async functions and ensureDeferred:

async def get_clan_details(clan):
     print("Hello world", clan)

def main(reactor, clan):
    return defer.ensureDeferred(get_clan_details(clan))

task.react(main, sys.argv[1:])

In this program, which should start with the same imports, we moved all the logic to the async function get_clan_details. Just like a regular function, an async function has an implicit return None at the end. However, async functions, sometimes called co-routines, are a different type than Deferred. In order to let Twisted, which has existed since Python 1.5.2, use this modern feature, we must adapt the co-routine using ensureDeferred.

While we could write all the logic without using co-routines, using the async syntax will allow us to write code that is easier to understand, and we will need to move a lot less of the code into embedded callbacks.

The next concept to introduce is that of await. Later, we will await a network call, but for simplicity, right now, we will await on a timer. Twisted has a special function, task.deferLater, which will call a function with given parameters after some time has passed.

The following program will take five seconds to complete:

async def get_clan_details(clan, reactor):
     out = await task.deferLater(
         reactor,
         5,
         lambda clan: f"Hello world {clan}",
         clan
     )
     print(out)

def main(reactor, clan):
    return defer.ensureDeferred(get_clan_details(clan, reactor))

task.react(main, sys.argv[1:])

A note about types: task.deferLater returns a Deferred, as do most Twisted functions that do not have the value already available. When running the Twisted event loop, we can await on both Deferred values and co-routines.

The function task.deferLater will wait five seconds and then call our lambda, calculating the string to print out.

Now we have all the Twisted building blocks needed to write an efficient clan-analysis program!

Since we will be using the global reactor, we no longer need to accept the reactor as a parameter in the function that calculates these statistics. The way to use the token is as a "bearer" token in the headers:

async def get_clan_details(clan):
    headers={b'Authorization': b'Bearer '+token.encode('ascii')}

We want clan tags to be sent, which will be strings. Clan tags begin with #, so they must be quoted before they're put in URLs. This is because # has the special meaning "URL fragment":

async def get_clan_details(clan):
     # ...
     clan = urllib.parse.quote(clan)

The first step is to get the details of the clan, including the clan members:

async def get_clan_details(clan):
     # ...
     res = await treq.get("https://api.clashroyale.com/v1/clans/" + clan,
                          headers=headers)

Notice that we have to await the treq.get call. We have to be explicit about when to wait and get information since it is an asynchronous network call. Just using the await syntax to call a Deferred function does not let us take full power of asynchronicity (we will see how to do it later).

Next, after getting the headers, we need to get the content. The treq library gives us a helper method that parses the JSON directly:

async def get_clan_details(clan):
     # ...
     content = await res.json()

The content includes some metadata about the clan, which is not interesting for our current purposes, and a memberList field that contains the clan members. Note that while it has some data about the players, the current favorite card is not part of it. It does include the unique "player tag" that we can use to retrieve further data.

We collect all player tags, and, since they also begin with #, we URL-quote them:

async def get_clan_details(clan):
     # ...
     player_tags = [urllib.parse.quote(player['tag'])
                    for player in content['memberList']]

Finally, we come to the real power of treq and Twisted: generating all requests for player data at once! That can really speed up tasks like this one, which queries an API over and over again. In cases of APIs with rate-limiting, this can be problematic.

There are times when we need to be considerate to our API owners and not run up against any rate limits. There are techniques to support rate-limiting explicitly in Twisted, but they are beyond the scope of this post. (One important tool is defer.DeferredSemaphore.)

async def get_clan_details(clan):
     # ...
     requests = [treq.get("https://api.clashroyale.com/v1/players/" + tag,
                          headers=headers)
                 for tag in player_tags]

Remember that requests do not return the JSON body directly. Earlier, we used await so that we did not have to worry about exactly what the requests return. They actually return a Deferred. A Deferred can have an attached callback that will modify the Deferred. If the callback returns a Deferred, the final value of the Deferred will be the value of the returned Deferred.

So, to each deferred, we attach a callback that will retrieve the JSON of the body:

async def get_clan_details(clan):
     # ...
     for request in requests:
         request.addCallback(lambda result: result.json())

Attaching callbacks to Deferreds is a more manual technique, which makes code that is harder to follow but uses the async features more efficiently. Specifically, because we are attaching all the callbacks at the same time, we do not need to wait for the network calls, which potentially can take a long time, to indicate how to post-process the result.

From Deferreds to values

We cannot calculate the most popular favorite cards until all results have been gathered. We have a list of Deferreds, but what we want is a Deferred that gets a list value. This inversion is exactly what the Twisted function defer.gatherResults does:

async def get_clan_details(clan):
     # ...
     all_players = await defer.gatherResults(requests)

This seemingly innocent call is where we use the full power of Twisted. The defer.gatherResults function immediately returns a deferred that will fire only when all the constituent Deferreds have fired and will fire with the result. It even gives us free error-handling: if any of the Deferreds error out, it will immediately return a failed deferred, which will cause the await to raise an exception.

Now that we have all the players' details, we need to munch some data. We get to use one of Python's coolest built-ins, collections.Counter. This class takes a list of things and counts how many times it has seen each thing, which is exactly what we need for vote counting or popularity contests:

async def get_clan_details(clan):
     # ...
     favorite_card = collections.Counter([player["currentFavouriteCard"]["name"]
                                          for player in all_players])

Finally, we print it:

async def get_clan_details(clan):
     # ...
     print(json.dumps(favorite_card.most_common(), indent=4))

So, putting it all together, we have:

import collections, json, os, sys, urllib.parse
from twisted.internet import task, defer
import treq

with open(os.path.expanduser("~/.crtoken")) as fpin:
    token = fpin.read().strip()


async def get_clan_details(clan):
     headers = headers={b'Authorization': b'Bearer '+token.encode('ascii')}
     clan = urllib.parse.quote(clan)
     res = await treq.get("https://api.clashroyale.com/v1/clans/" + clan,
                          headers=headers)
     content = await res.json()
     player_tags = [urllib.parse.quote(player['tag'])
                    for player in content['memberList']]
     requests = [treq.get("https://api.clashroyale.com/v1/players/" + tag,
                          headers=headers)
                 for tag in player_tags]
     for request in requests:
         request.addCallback(lambda result: result.json())
     all_players = await defer.gatherResults(requests)
     favorite_card = collections.Counter([player["currentFavouriteCard"]["name"]
                                          for player in all_players])
     print(json.dumps(favorite_card.most_common(), indent=4))

def main(reactor, clan):
    return defer.ensureDeferred(get_clan_details(clan))

task.react(main, sys.argv[1:])

Thanks to the efficiency and expressive syntax of Twisted and treq, this is all the code we need to make asynchronous calls to an API. If you were wondering about the outcome, my clan's list of favorite cards is Wizard, Mega Knight, Valkyrie, and Royal Giant, in descending order.

(This post is based on the article I wrote for opensource.com)

14 Apr 2020 3:00am GMT

05 Apr 2020

feedPlanet Twisted

Moshe Zadka: Comfort with Small Mistakes

It has been a long time since I learned how to program, and it is easy to forget some of the hard-won lessons in the beginning. Easy until I try to teach people to program. There is a lot of accidental and inherent complexity in programming, but I am ready for that: I remember to explain how carefully to follow the syntax, and the kind of syntactical gotchas that are easy to fail.

But there is one metalesson that is much easier to forget, and much harder to learn, and to teach. Humans are used to small mistakes having reasonably small consequences. But even in cases with catatrophic consequences, the consequences look related to the mistakes.

However, in programming, small mistakes can lead not just to big consequences, but to weird consequences. A missing comma might mean that things work fine in the testing environment, but in production, every third request gets a slightly wrong result.

This really stumps people. They copy code somewhat imperfectly from the board, or make a small mistake when they edit it to change from "Hello world" to "Goodbye world", and suddenly, a completely unrelated part of the program starts going haywire.

This happens to old hands too. The number of times I have edited code and ran the tests, only to discover a clearly unrelated test failing, is not small. The difference that comes with experience is that I take a deep breath, think "I've got this", and start down the troubleshooting path.

The troubleshooting can include any number of things: I might go in with a debugger, add print statements, do a bisect-diff to figure out what caused the problem, try random things to see what happens, or just trace the execution path carefully.

The troubleshooting process does not matter as much as what comes before it: the deep breath. This is my time to accept the problem has happened, and that I am in for a process which can take two hours, and at the end of which my entire productivity will be "added missing semicolon". Sometimes a breath is not enough, and I need to get up and get some tea. But the most important, and almost invisible step, at the beginning is to step back, remember that this is, weirdly, part of the job, and to become comfortable doing it.

If you want to be any kind of software developer, accept it now. Much of your life will be seeing weird consequences, and tracing it back to a small mistake. Eventually, like everyone, you will succeed. Flush with victory, make a note of your success somewhere: anywhere where the overhead of writing it is low, be it an e-mail to yourself or a note-keeping app.

If you ever decide to teach, or write a blog, this is an unending source of content.

(Thanks to Veronica Hanus for her feedback on an early draft. All issues and mistakes that remain are my responsibility.)

05 Apr 2020 5:20am GMT

23 Mar 2020

feedPlanet Twisted

Twisted Matrix Laboratories: Twisted Drops Python 2.7 Support

With the open-source Python community at large dropping Python 2.7 support in their projects, Twisted has decided to do the same. Twisted 20.3.0, the most recently released version, is the final release to offer Python 2.7 support.

Despite the break, the compatibility policy still applies. This means that if your code works with Twisted 20.3 on Python 2.7 and 3.5+, that updating your Twisted on Python 3 up to a theoretical 21.3 would not require changes that would make Python 2.7 + Twisted 20.3 stop working, despite a theoretical Twisted 21.3 not supporting 2.7. (This is, of course, in an ideal situation -- regressions and changes that are excepted from the policy such as security fixes do occur. Testing your applications on Twisted prereleases can help catch places where this happens, so, please do!)

- Amber (HawkOwl)

23 Mar 2020 7:42pm GMT

Twisted Matrix Laboratories: Twisted 20.3.0 Released

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 20.3! The highlights of this release are:

This is the final Twisted release to support Python 2.7.

You can find the downloads at <https://pypi.org/project/Twisted> (or alternatively <https://twistedmatrix.com/trac/wiki/Downloads>). The NEWS file is also available at <https://github.com/twisted/twisted/blob/twisted-20.3.0/NEWS.rst>.
Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!
- hawkowl


23 Mar 2020 7:28pm GMT

Hynek Schlawack: Hardening Your Web Server’s SSL Ciphers

There are many wordy articles on configuring your web server's TLS ciphers. This is not one of them. Instead I will share a configuration which is both compatible enough for today's needs and scores a straight "A" on Qualys's SSL Server Test.

23 Mar 2020 12:00am GMT

13 Mar 2020

feedPlanet Twisted

Moshe Zadka: Or else:

This was originally sent to my newsletter. I send one e-mail, always about Python, every other Sunday. If this blog post interests you, consider subscribing.

The underappreciated else keyword in Python has three distinct uses.

if/else

On an if statement, else will contain code that runs if the condition is false.

if anonymize:
    print("Hello world")
else:
    print("Hello, name")

This is probably the least surprising use.

loop/else

The easiest to explain is while/else: it works the same as if/else, and runs when the condition is false.

However, it does not run if the loop was broken out of using break or an exception: it serves as something that runs on normal loop termination.

for/else functions in the same way: it runs on normal loop termination, and not if the loop was broken out of using a break.

For example, searching for an odd element in a list:

for x in numbers:
    if x % 2 == 1:
        print("Found", x)
        break
else:
    print("No odd found")

This is a powerful way to avoid sentinel values.

try/except/else

When writing code that might raise an exception, we want to be able to catch it -- but we want to avoid catching unanticipated exceptions. This means we want to protect as little code with try as possible, but still have some code that runs only in the normal path.

try:
    before, after = things
except ValueError:
    part1 = things[0]
    part2 = 0
    after = 0
else:
    part1, part2 = before

This means that if things does not have two items, this is a valid case we can recover from. However, if it does have two items, the first one must also have two items. If this is not the case, this snippet will raise ValueError.

13 Mar 2020 2:00am GMT

09 Mar 2020

feedPlanet Twisted

Hynek Schlawack: Python in GitHub Actions

GitHub's own CI called GitHub Actions has been out of closed beta for a while and offers generous free quotas and a seamless integration with the rest of the site. Let's have a look on how to use it for an open source Python package.

09 Mar 2020 12:00am GMT

23 Feb 2020

feedPlanet Twisted

Hynek Schlawack: Python in Production

I'm missing a key part from the public Python discourse and I would like to help to change that.

23 Feb 2020 4:45pm GMT

Hynek Schlawack: Python Packaging Metadata

Since this topic keeps coming up, I'd like to briefly share my thoughts on Python package metadata because it's - as always - more complex than it seems.

23 Feb 2020 12:00am GMT