12 Apr 2019

feedPlanet Twisted

Itamar Turner-Trauring: Can software engineering be meaningful work?

The world is full of problems-from poverty to climate change-and it seems like software ought to be able to help. And yet your own programming job seems pointless, doing nothing to make things better. Far too many jobs are just about making some rich people just a little bit richer.

So how can you do something meaningful with your life?

There are no easy answers, but here's a starting point, at least.

Don't make things worse

Even beyond your moral obligations, working on something you actively find wrong is bad for you:

If you find yourself in this situation, you have the opportunity to try to make things a little better, by pushing your organization to change. But you can also just go look for another job elsewhere.

Some jobs are actually good

Of course, most software jobs aren't evil, but neither are they particularly meaningful. You can help an online store come up with a better recommendation engine, or optimize their marketing funnel, or build a web UI for support staff-but does it really matter that people buy at store A instead of store B?

So it's worth thinking in detail about what exactly it is you would find meaningful, and seeing if there's work that matches your criteria. There may not be a huge number of jobs that qualify, but chances are some exist.

If you care about climate change, for example, there are companies building alternative energy systems, working on public transportation planning, and more broadly just making computing more efficient.

Your job needn't be the center of your life

You may not be able to find such a job, or get such a job. So there's something to be said for not making your work the center of your life's existence.

As a programmer you are likely get paid well, and you can even negotiate a shorter workweek. Given enough free time and no worries about making a living, you have the ability to find meaning outside your work.

No easy answers

Unless you want to join a group that will tell you exactly what to think and precisely what to do-and there are certainly no lack of those-meaning is something you need to figure out for yourself.

It's unlikely that you'll solve it in one fell swoop, nor is it likely to be a fast process. The best you can do is just get started: a meaningful life isn't a destination, it's a journey.



Struggling with a 40-hour workweek? Too tired by the end of the day to do anything but collapse on the sofa and watch TV?

Learn how you can get a 3-day weekend, every single week.

12 Apr 2019 4:00am GMT

10 Apr 2019

feedPlanet Twisted

Twisted Matrix Laboratories: Twisted 19.2.0 Released

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 19.2! The highlights of this release are:

You can find the downloads at <https://pypi.python.org/pypi/Twisted> (or alternatively <http://twistedmatrix.com/trac/wiki/Downloads>). The NEWS file is also available at <https://github.com/twisted/twisted/blob/twisted-19.2.0/NEWS.rst>.

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,
Amber Brown (HawkOwl)

10 Apr 2019 12:35pm GMT

08 Apr 2019

feedPlanet Twisted

Moshe Zadka: Publishing a Book with Sphinx

A while ago, I decided I wanted to self-publish a book on improving your Python skills. It was supposed to be short, sweet, and fairly inexpensive.

The journey was a success, but had some interesting twists along the way.

From the beginning, I knew what technology I wanted to write the book with: Sphinx. This was because I knew that I can use Sphinx to create something reasonable: I have previously ported my "Calculus 101" book to Sphinx, and I have written other small things in it. Sphinx uses ReStructuredText, which I am most familiar with.

I decided I wanted to publish as PDF (for self-printers or others who find it convenient), as browser-ready HTML directory, and as an ePub.

The tox environments I created are: epub builds the ePub, html builds the browser-ready HTML, and pdf builds the PDF.

Initially, the epub environment created a "singlehtml", and I used Calibre command-line utility to transform it into an ePub. This made for a prettier ePub than the one sphinx creates: it had a much nicer cover, which is what most book reading applications use as an icon. However, that rendered poorly on Books.app (AKA iBooks).

One of the projects I still plan to tackle is how to improve the look of the rendered ePub, and add a custom cover image.

Finally, a script runs all the relevant tox environments, and then packs everything into a zip file. This is the zip file I upload to Gumroad, so that people can buy it.

I have tried to use other sellers, but Gumroad was the one with the easiest store creation. In order to test my store, even before the book was ready, I created a simple "Python cheat-sheet" poster, and put it on my store.

I then asked friends to buy it, as well as trying to do it myself. After it all worked, I refunded all the test-run purchases, of course!

Refunding on Gumroad is a pleasant process, which means that if people buy the book, and are unhappy with it, I am happy to refund their money.

(Thanks to Glyph Lefkowitz for his feedback on an earlier draft. All mistakes that remain are my responsibility.)

08 Apr 2019 7:00am GMT

03 Apr 2019

feedPlanet Twisted

Itamar Turner-Trauring: Setting boundaries at your job as a programmer

There's always another bug, another feature, another deadline. So it's easy to fall into the trap of taking on too much, saying "yes" one time too many, staying at the office a little later, answering a work email on the weekend…

If you're not careful you can end up setting unreasonable expectations, and ending up tethered to your work email and Slack. Your manager will expect you to work weekends, and your teammates will expect you to reply to bug reports in the middle of your vacation.

What you want is the opposite: when you're at home or on vacation, you should be spending your time however you want, merry and free.

You need to set boundaries, which is what I'll be discussing in the rest of this article.

Prepping for a new job

Imagine you're starting a new job in a week, you enjoy programming for fun, and you want to be productive as soon as possible. Personally, I wouldn't do any advance preparation for a new job: ongoing learning is part of a programmer's work, and employers ought to budget time for it. But you might choose differently.

If so, it's tempting to ask for some learning material so you can spend a few days beforehand getting up to speed. But you're failing to set boundaries if you do that, and they might give you company-specific material, in which case you're just doing work for free.

Learning general technologies is less of a problem-knowing more technologies is useful in your career in general, and maybe you enjoy programming for fun. So instead of asking for learning material, you can go on your own and learn the technologies you know they use, without telling them you're doing so.

Work email and Slack

Never set up work email or Slack on your phone or personal computer:

  1. It will tempt you to engage with work in your free time.
  2. When you do engage, you'll be setting expectations that you're available to answer questions 24/7.

While you're at work you'll always have your computer, so you don't need access on your phone. If you do need to set up work email on your phone for travel, remove the account when you're back home.

And if you want to have your work calendar on your phone, you can share it with your personal calendar account; that way you're sharing only your calendar, nothing else.

Vacations

When you're on vacation, you're on vacation: no work allowed. That means you're not taking your work laptop with you, or turning it on if you're at home.

A week or so in advance of your vacation, explain to your team that you won't be online, and that you won't have access to work files. Figure out what information they might need-documentation, in-progress work you want to hand off, and the like-and write it all down where they can find it.

If you must, give your personal phone number for emergencies: given you lack access to your work credentials and email, the chances of your being called for something unimportant are quite low.

You're paid for your normal work hours (and that's it)

A standard workweek in the US is 40 hours a week; elsewhere it can be a little less. Whatever it is, outside those hours you shouldn't be working, because you're not being paid for that work. Your evenings, your weekends, your holidays, your vacations-all of these belong to you, not your employer.

If you don't enforce that boundary between work and non-work, you are sending the message that your time doesn't belong to you. And if you have a bad manager, they're going to take advantage of that-or you might end up working long hours out of a misplaced sense of obligation.

So unless you're dealing with an emergency, you should forget your job exists when your workday ends-and unless you're on your on-call rotation, you should make sure you're inaccessible by normal work channels.



Struggling with a 40-hour workweek? Too tired by the end of the day to do anything but collapse on the sofa and watch TV?

Learn how you can get a 3-day weekend, every single week.

03 Apr 2019 4:00am GMT

30 Mar 2019

feedPlanet Twisted

Moshe Zadka: A Local LRU Cache

"It is a truth universally acknowledged, that a shared state in possession of mutability, must be in want of a bug." -- with apologies to Jane Austen

As Ms. Austen, and Henrik Eichenhardt, taught us, shared mutable state is the root of all evil.

Yet, the official documentation of functools tells us to write code like:

@lru_cache(maxsize=32)
def get_pep(num):
    'Retrieve text of a Python Enhancement Proposal'
    resource = 'http://www.python.org/dev/peps/pep-%04d/' % num
    try:
        with urllib.request.urlopen(resource) as s:
            return s.read()
    except urllib.error.HTTPError:
        return 'Not Found'

(This code is copied from the official documentation, verbatim.)

The decorator, @lru_cache(maxsize=32), is now... module-global mutable state. It doesn't get any more shared, in Python, than module-global: every import of the module will share the object!

We try and pretend like there is no "semantic" difference: the cache is "merely" an optimization. However, very quickly things start falling apart: after all, why would the documentation even tell us how to get back the original function (answer: .__wrapped__) if the cache is so benign?

No, decorating the function with lru_cache is anything but benign! For one, because it is shared-thread mutable state, we have introduced some thread locking, with all the resulting complexity, and occasional surprising performance issues.

Another example of non-benign-ness is that, in the get_pep example, sometimes a transient error, such as a 504, will linger on, making all subsequent requests "fail", until a cache eviction (because an unrelated code path went through several PEPs) causes a retry. These are exactly the kind of bugs which lead to warnings against shared mutable state!

If we want to cache, let us own it explicitly in the using code, and not have a global implementation dictate it. Fortunately, there is a way to properly use the LRU cache.

First, remove the decorator from the implementation:

def get_pep(num):
    'Retrieve text of a Python Enhancement Proposal'
    # Same code as an in official example

Then, in the using code, build a cache:

def analyze_peps():
    cached_get_pep = lru_cache(maxsize=32)(get_pep)
    all_peps, pep_by_type = analyze_index(cached_get_pep(0))
    words1 = get_words_in_peps(cached_get_pep, all_peps)
    words2 = get_words_in_informational(cached_get_pep,
                                        pep_by_type["Informational"])
    do_something(words1, words2)

Notice that in this example, the lifetime of the cache is relatively clear: we create it in the beginning of the function, passed it to called functions, and then it goes out of scope and is deleted. (Barring one of those functions sneakily keeping a reference, which would be a bad implementation, and visible when reviewing it.)

This means we do not have to worry about cached failures if the function is retried. If we retry analyze_peps, we know that it will retry retrieving any PEPs, even if those failed before.

If we wanted the cache to persist between invocations of the function, the right solution would be to move it one level up:

def analyze_peps(cached_get_peps):
    # ...

Then it is the caller's responsibility to maintain the cache: once again, we avoid shared mutable state by making the state management be explicit.

In this example, based on the official lru_cache documentation, we used a network-based function to show some of the issues with a global cache. Often, lru_cache is used for performance reasons. However, even there, it is easy to create issues: for example, one function using non-common inputs to the LRU-cached functions can cause massive cache evictions, with surprising performance impacts!

The lru_cache implementation is great: but using it as a decorator means making the cache global, with all the bad effects. Using it locally is a good use of a great implementation.

(Thanks to Adi Stav, Steve Holden, and James Abel for their feedback on early drafts. Any issues that remain are my responsibility.)

30 Mar 2019 4:30am GMT

29 Mar 2019

feedPlanet Twisted

Itamar Turner-Trauring: On learning new technologies: why breadth beats depth

As a programmer you face an ever-growing stream of new technologies: new frameworks, new libraries, new tools, new languages, new paradigms. Keeping up is daunting.

The answer, of course, is that you can't learn them all-in depth. What you can do is learn about the tools' existence, and learn just enough about them to know when it might be worth learning more.

Quite often, spending 5 minutes learning about a new technology will give you 80% of the benefit you'd get from spending 5 days on it.

In the rest of this article I'll cover:

  1. The cost of unnecessary in-depth learning.
  2. Why breadth of knowledge is so useful.
  3. Some easy ways to gain breadth of knowledge.

The cost of in-depth learning

Having a broad range of tools and techniques to reach for is a valuable skill both at your job and when looking for a new job. But there are different levels of knowledge you can have: you can be an expert, or you can have some basic understanding of what the tool does and why you might use it.

The problem with becoming an expert is that it's time consuming, and you don't want to put that level of effort into every new tool you encounter.

  1. Some new technologies will die just as quickly as they were born; there's no point wasting time on a dead end.
  2. Most technologies just aren't relevant to your current situation. GitHub keeps recommending I look at a library for analyzing pulsar data, and not being an astrophysicist I'm going to assume I can safely ignore it.
  3. Software changes over time: even if you end up using a new library in a year or two, by that point the API may have changed. Time spent learning the current API would be wasted.

If you try to spend a few days-or even hours-on every intriguing new technology you hear about, you're going to waste a lot of time.

The alternative: shallow breadth of knowledge

Most of the time you don't actually need to use new tools and techniques. As long as you know a tool exists you'll be able to learn more about it when you need to.

For example, there is a tool named Logstash that takes your logs and sends them to a central location. That's pretty much all you have to remember about it, and it took you just a few seconds to read that previous sentence.

Maybe you'll never use that information… or maybe one day you'll need to get logs from a cluster of machines to a centralized location. At that point you'll remember the name "Logstash", look it up, and have the motivation to actually go read the documentation and play around with it.

This is also true when it comes to finding a new job. I was once asked in a job interview about the difference between NoSQL and traditional databases. At the time I'd never used MongoDB or any other NoSQL database, but I knew enough to answer satisfactorily. Being able to answer that question told the interviewer I'd be able to use that tool, if necessary, even if I hadn't done it before.

Gaining breadth of knowledge

Learning about the existence of tools can be a fairly fast process. And since this knowledge will benefit your employer and you don't need to spend significant time on it, you can acquire it during working hours.

You're never actually working every single minute of your day, you always have some time when you're slacking off on the Internet. Perhaps you're doing so right now! You can use that time to expand your knowledge.

Here are a couple ways you can get pointers to new tools and techniques:

Newsletters

A great way to learn new tools and techniques are weekly email newsletters. There are newsletters on many languages and topics, from DevOps to PostgreSQL: here's one fairly detailed list of potential newsletters you can sign up for.

Conferences and Meetups (you don't have to go!)

Another good source of new tools and techniques are conferences and Meetups. Good conferences and Meetups will aim for a broad range of talks, on topics both new and classic.

But you don't have to go to the conference or Meetup to benefit, or even watch a recording of the talks to learn something. Just skimming talk topics will give you a sense of what the community is talking and thinking about-and if something sounds particularly relevant to your interests you can spend the extra time to learn more.

Of course, if you can convince your employer to send you to a conference that's even better: you'll learn more, and you'll do it on the company's dime and time.

Your time is valuable-use it accordingly

There are only so many hours in the day, so many days in a year. That means you need to work efficiently, spending your limited time in ways that have the most impact:

  1. Spend an hour a week learning about new tools, just enough to know when they might be useful.
  2. Keep a record of these tools so you can find them when you need to: star them on GitHub, or add them your bookmarks or note-taking system.
  3. Only spend the extra time and effort needed to gain more in-depth understanding once you actually need to use the tool. And when you do learn a new tool, do it at your job if you can.



Struggling with a 40-hour workweek? Too tired by the end of the day to do anything but collapse on the sofa and watch TV?

Learn how you can get a 3-day weekend, every single week.

29 Mar 2019 4:00am GMT

19 Mar 2019

feedPlanet Twisted

Itamar Turner-Trauring: You are not a commodity

Recently a reader wrote in with a question:

I'll be going to [a coding boot camp]. [After I graduate], my current view is to try hard to negotiate for whatever I can and then get better for my second job, but both of those steps are based on the assumption that I understand what an acceptable range for pay, benefits, etc are, and I feel like it's leaving money (or time) on the table.

I'm not even sure if entry level jobs should be negotiated since they seem to be such a commodity. Do you have any advice for someone standing on the edge of the industry, looking to jump in?

What I told him, and what I'd like to share with you as well, is this:

This is perhaps more obvious if you have lots of experience, but it's just as true for someone looking for their first job.

We all have different strengths, different weaknesses, different experiences and background. So when it comes to finding a job, you should be highlighting your strengths, instead of all the ways you're the same as everyone else.

In the rest of this article I'll show just a few of the ways this can be applied by someone who is switching careers into the tech industry; elsewhere I talk more about the more theoretical side of marketing yourself.

Negotiating as a bootcamp graduate

Since employment is a negotiated relationship, negotiation starts not when you're discussing your salary with a particular company, but long before that when you start looking for a job.

Here are just some of the ways you can improve your negotiating strength.

1. Highlight your previous experience

If you're going to a coding bootcamp chances are you've had previous job experience. Many of those job skills will translate to your new career as a software developer: writing software as an employee isn't just about writing code.

Whether you worked as a marketer or a carpenter, your resume and interviews should highlight the relevant skills you learned in your previous career. Those skills will make you far more competent than the average college graduate.

This might include people management, project management, writing experience, knowing when to cut corners and when not to, attention to detail, knowing how to manage your time, and so on.

And if you can apply to jobs where your business knowledge is relevant, even better: if you used to work in insurance, you'll have an easier time getting a programming job at an insurance company.

2. Do your research

Research salaries in advance. There are a number of online salary surveys-e.g. StackOverlow has one-which should give you some sense of what you should be getting.

Keep in mind that top companies like Google or some of the big name startups use stock and bonuses as a large part of total compensation, and salary doesn't reflect that. Glassdoor has per-company salary surveys but they often tend to be skewed and lower than actual salaries.

3. Get multiple job offers if you can

Imagine candidate A and candidate B: as far as the hiring manager is concerned they seem identical. However, if candidate B has another job offer already, that is evidence that someone elsewhere has decided they like them. So candidate B is no longer seen as a commodity.

Try to apply to multiple jobs at once, and not to say "yes" immediately to the first job offer you can get. If you can get two offers at the same time, chances you'll be able to get better terms from one or the other.

In fact, even just saying "I'm in the middle of interviewing elsewhere" can be helpful.

You are not a commodity, so don't act like one

Notice how all of the suggestions above aren't about that final conversation about salary. There's a reason: by the time you've reached that point, most of the decisions about your salary range have already been made. You still need to ask for more, but there's only a limited upside at that point.

So it's important to present your unique strengths and capabilities from the start: you're not a commodity, and you shouldn't market yourself like one.



Struggling with a 40-hour workweek? Too tired by the end of the day to do anything but collapse on the sofa and watch TV?

Learn how you can get a 3-day weekend, every single week.

19 Mar 2019 4:00am GMT

05 Mar 2019

feedPlanet Twisted

Itamar Turner-Trauring: You can't avoid negotiating—but you can make it easier

You're looking for a new job, or feel like it's time for a raise, or maybe you just want to set some boundaries with your boss. And that means negotiating, and you hate the whole idea: asking for things is hard, you don't want to be treated specially, the idea of having the necessary conversation just makes you super-uncomfortable.

And that's a problem, because you can't avoid negotiating: employment is a negotiated relationship. From the minute you start looking for a job until you leave for a new one, you are negotiating.

And maybe you didn't quite realize that, and maybe you didn't ever ask for what you want, but in that case you're still negotiating. You're just negotiating badly.

But once you internalize this idea, negotiation can get easier.

That awkward, scary conversation where you ask for what you want is really just a small fraction of the negotiation. Which means if you do it right, that final conversation can be shorter, more comfortable, and much more likely to succeed.

To see why, let's take the example of a job search, and see how the final conversation where you discuss your salary is just a small part of the real negotiation.

How your salary is determined

To simplify matters, we will specifically focus just on your salary as a programmer.

Companies tend to have different job titles based on experience, with corresponding ranges of salaries. Your salary is determined first by the prospective job title, and second by the specific salary within that title's range.

The process goes something like this:

  1. When you send in your resume the HR or recruiting person who reads it puts you into some sort of mental bucket: "this is a senior software engineer."
  2. The hiring manager reads your resume and refines that initial guess.
  3. The interview process then hardens that job title, and gives the company some sense of how much they want you and therefore where in that title's salary range to put you.
  4. Finally, you get an offer, and you can push back and try to get more.

That final step, the awkward conversation we tend to think of as the negotiation, is only the end of a long process. By the time you've reached it much of your scope for negotiation has been restricted: you'll have a harder time convincing the company you're a Software Engineer IV if they've decided you're a Software Engineer II.

Employment is a negotiated relationship

Negotiation isn't a one-time event, it's an ongoing part of how you interact with an employer You start negotiating for your salary, for example, from the day you start applying:

  1. You can choose companies to apply to where your enthusiasm will come across, or where you have highly relevant technical skills.
  2. You can get yourself introduced by a friend on the inside, instead of just sending in your resume.
  3. You can ensure you've demonstrated your correct level of problem-solving skills in your resume. If you can identify problems, it's very easy to give the impression you can only solve problems if you don't phrase things right ("I switched us over from VMs to Kubernetes" vs. "I identified hand-built VMs as a problem, investigated, chose Kubernetes, etc.").
  4. You can interview for multiple jobs at once, so you can use a job offer from company A as independent proof of your value to company B.
  5. You can do well on the technical interview, which correlates with higher salaries.
  6. You can avoid whiteboard puzzles if you tend not to do well on those sorts of interviews.

All of these-and more-are part of the negotiation, long before you get the offer and discuss your salary.

You still need to ask (and negotiation doesn't stop then)

Yes, you do need to ask for what you want at the end. And yes, that's going to be scary and awkward and no fun at all. But asking for things is something you can practice in many different contexts, not just job interviews.

But if you treated the whole job interview process as a negotiation, that final conversation will be much easier because the company will really want to hire you-and because they'll be worried you'll take that other job offer you mentioned.

You're not done negotiating when you've accepted the offer, though.

When your boss asks you to do something, you don't have to say yes. In fact, as a good employee it's your duty not to say yes, but to listen, dig a little deeper, and find the real problem.

Similarly, how many hours you work is not just up to your boss, is also about how you push back on unreasonable demands. And again, it's your duty as a good employee to push back, because work/life balance makes you a better software engineer.

All of which is to say:

  1. You can't avoid negotiating.
  2. Negotiation is far broader than just that awkward conversation where you make the ask.
  3. Being a good negotiator will make you a far more effective software engineer.



Struggling with a 40-hour workweek? Too tired by the end of the day to do anything but collapse on the sofa and watch TV?

Learn how you can get a 3-day weekend, every single week.

05 Mar 2019 5:00am GMT

14 Feb 2019

feedPlanet Twisted

Moshe Zadka: Don't Make It Callable

There is a lot of code that overloads the __call__ method. This is the method that "calling" an object activates: something(x, y, z) will call something.__call__(x, y, z) if something is a member of a Python-defined class.

At first, like every operator overload, this seems like a nifty idea. And then, like most operator overload cases, we need to ask: why? Why is this better than a named method?

The first use-case is easily done better with a named method, and more readably: accepting callbacks. Let's say that the function interesting_files will call the passed-in callback with names of interesting files.

We can, of course, use __call__:

class PrefixMatcher:

    def __init__(self, prefix):
        self.prefix = prefix
        self.matches = []

    def __call__(self, name):
        if name.startswith(self.prefix):
            self.matches.append(name)

    def random_match(self):
        return random.choice(self.matches)

matcher = PrefixMatcher("prefix")
interesting_files(matcher)
print(matcher.random_match())

But it is more readable, and obvious, if we...don't:

class PrefixMatcher:

    def __init__(self, prefix):
        self.prefix = prefix
        self.matches = []

    def get_name(self, name):
        if name.startswith(self.prefix):
            self.matches.append(name)

    def random_match(self):
        return random.choice(self.matches)

matcher = PrefixMatcher("prefix")
interesting_files(matcher.get_name)
print(matcher.random_match())

We can pass the matcher.get_name method, which is already callable directly to interesting_files: there is no need to make PrefixMatcher callable by overloading __call__.

If something really is nothing more than a function call with some extra arguments, then either a closure or a partial would be appropriate.

In the example above, the random_match method was added to make sure that the class PrefixMatcher is justified. If this was not there, either of these implementations would be more appropriate:

def prefix_matcher(prefix):
    matches = []
    def callback(name):
        if name.startswith(prefix):
            matches.append(name)
    return callback, matches

matcher, matches = prefix_matcher("prefix")
interesting_files(matcher)
print(random.choice(matches))

This uses the function closure to capture some variables and return a function.

def prefix_matcher(prefix, matches, name):
    if name.startswith(prefix):
        matches.append(name)

matches = []
matcher = functools.partial(prefix_matcher, "prefix", matches)
interesting_files(matcher)
print(random.choice(matches))

This uses the funcotools.partial functions to construct a function that has some of the arguments "prepared".

There is one important use case for __call__, but it is specialized: it is a powerful tool when constructing a Python-based DSL. Indeed, this is exactly the time when we want to trade away "doing exactly when the operator always does" in favor of "succint syntax dedicated to the task at hand."

A good example of such a DSL is stan, where the __call__ function is used to construct XML tags with attributes: div(style="color: blue").

In almost every other case, avoid the temptation to make your objects callable. They are not functions, and should not be pretending.

14 Feb 2019 6:30am GMT

31 Jan 2019

feedPlanet Twisted

Itamar Turner-Trauring: Do they have work/life balance? Investigating potential employers with GitHub

When you're searching for a new programming job you want to avoid companies with long work hours. You can ask about work/life balance during the interview (and unless you're desperate, you always should ask), but that means wasting time applying and interviewing at companies where you don't want to work.

What you really want is to only apply to companies that actually allow-or even better, encourage-good work/life balance.

Close reading of the job posting and company hiring pages can sometimes help you do that, but some good companies don't talk about it, and some bad companies will just lie.

So in this article I'll explain how you can use GitHub to empirically filter out at least some companies with bad work-life balance.

Let's look at GitHub profiles

If you go to the profile for a GitHub user (here's mine) you will see a chart showing contributions over time. The columns are weeks, and each row is a day of the week.

Each square shows contribution on a particular day: the darker the color, the more contributions.

What's most interesting here are the first and last rows, which are Sundays and Saturdays respectively. As you can see in the image above, this GitHub user doesn't tend to code much on the weekends: the weekend boxes are mostly blank.

You can also use this UI to see if the company uses GitHub at all. Given this many boxes, the user's employer probably does use GitHub, but you can also click on a particular box and see the specific contributions. In this case, you would see that on one particular weekday the user contributed to "private repositories", presumably those for their employer:

On the other hand, if you were to click on the weekend boxes you would see that all the weekend contributions are to open source projects. In short, this user doesn't code much on the weekend, and when they do it's on personal open source projects, not work projects.

Generalizing this process will allow you to filter out companies that don't have work/life balance for developers.

Investigating work/life balance using GitHub

There are two assumptions here:

Company size you can figure out from the company page or LinkedIn, usage of GitHub will be something we can figure out along the way.

This is not a perfect process, since users can disable showing private repository contributions, or it's possible the developer has personal private repositories. This is why you want to check as many profiles as possible.

Here's what you should do:

  1. Find a number of programmers who work for the company. You can do this via the company's website, and by the company's page on LinkedIn, which will list employees who have LinkedIn profiles. You can also check for members of the company's GitHub organization, if they have a public list, but don't rely on this exclusively since it will likely omit many employees.
  2. For each programmer, use your favorite search engine and search for "$NAME github" to find their GitHub profile. Try to do some sanity checks to make sure it's the right person, especially if they have a common name: location, organization membership, technologies used, etc..
  3. For each programmer, check if they contribute to private repositories during the week. You do can this by visually seeing if there are lots of green boxes, and by clicking on the Monday-Friday boxes in the timeline and reading the results below. If they mostly don't, the company might not use GitHub.
  4. If they do use GitHub at work, for each programmer, check if they code on the weekend. You can visually see if they have lots of green boxes on Sundays and Saturdays.
  5. If they do code on the weekend, check if those are work contributions. You can do this by clicking on the weekend boxes and seeing if those are contributions to private repositories.

Additional tips:

By the end of the process you will often have a decent sense if most of the programmers are coding on the weekend. Note that this method can only tell you if they don't have work/life balance-you still can't know if they do have work/life balance.

So even if you decide to apply to a company that passed this filter, you will still need to ask questions about work/life balance and listen closely during the interview process.

Always do your research

Before you apply for a job you should always do your research: read their website, reads all their job postings, talk to a friend there if you have one… and as I explained in this article, their GitHub profiles as well. You'll still need to look for warning signs during interviews, but research is still worth it.

The more you research, the more you'll know whether you want to work there. And if you do want to work there, the more you know there better your interview will go.



Struggling with a 40-hour workweek? Too tired by the end of the day to do anything but collapse on the sofa and watch TV?

Learn how you can get a 3-day weekend, every single week.

31 Jan 2019 5:00am GMT

25 Jan 2019

feedPlanet Twisted

Moshe Zadka: Staying Safe with Open Source

A couple of months ago, a successful attack against the Node ecosystem resulted in stealing an undisclosed amount of bitcoins from CoPay wallets.

The technical flow of the attack is well-summarized by the NPM blog post. Quick summary:

  1. nodemon, a popular way to run Node applications, depends on event-stream.
  2. The event-stream maintainer has not had time to maintain it.
  3. right9control asked event-stream maintainer for commit privileges to help maintain it.
  4. right9control added a dependency on a new library, flatmap-stream.
  5. flatmap-stream contained malicious code to steal wallets.

Obfuscation

A number of methods were done to disguise the attack.

The dependency was an added in a minor version, and a new version was immediately released. This meant that most projects, which pin to minor, would get the updates, while it stayed invisible on the main GitHub landing page, or the main npm landing page.

The malicious code was only in the minified version of the library that was uploaded to npm.org. The non-minified source code on both GitHub and npm.org, as well as the minified code on GitHub, did not contain the malicious code.

The malicious code was encrypted with a key that used the description of other packages in the dependency tree. That made it impossible to understand the attack without guessing which package decrypts it.

The combination of all those methods meant that the problem remained undetected for two months. It was only luck that detected it: the decryption code was using a deprecated function, and investigating the deprecation message led to the issue being figured out.

This bears thinking about: if the code had been written slightly better, the problem would have still be happening now, and nobody would be the wiser. We should not discount the possibility that currently, someone who followed the same playbook but managed to use AES correctly is still attacking some package, and we have no idea.

Solutions and Non-Solutions

I want to discuss some non-solutions in trying to understand how this problem came about.

Better Vetting of Maintainers

It is true, the person who made this commit had an obviously-auto-generated username (<word>-<digit>-<word>) and made few contributions before getting control. But short of meeting people in person, I do not think this would work.

Attackers adapt. Ask for better usernames, they will generate "<firstname>-<lastname>" names. Are you going to disallow my GitHub username, moshez? Ask for more contributions, you will get some trivial-code-that's-uploaded-to-npm, autogenerated a bit to disguise it. Ask for longer commit history, they'll send fixes to trivial issues.

Remember that this is a distributed problem, with each lead maintainer having to come up with a vetting procedure. Otherwise, you get usernames through the vetting process, and then you use those to spam maintainers, who now are sure they can trust those "vetted".

In short, this is one of the classical defenses that fails to take into considerations that attackers adapt.

Any Solution That Depends on JavaScript-specific Things

This attack could easily have been executed against PyPI or RubyGems. Any solution that relies on JavaScript's ability to have a least-access-based solution only helps make sure that these attacks go elsewhere.

It's not bad to do it. It just does not solve the root cause.

This also means that "stop relying on minified code" is a non-solution in the world where we encourage Python engineers to upload wheels.

Any Solution That Depends on "Audit Code"

A typical medium-sized JavaScript client app depends on some 2000 packages. Auditing each one, on each update, would make using third-packages untenable. This means that start-ups playing fast and loose with these rules would gain an advantage over those who do not. Few companies can afford that pay that much for security.

Hell, we knew this was a possibility a few months before the attack was initiated and still nobody did code auditing. Starting now would mostly mean availability bias, which means it would be over as soon as another couple of months go by without a documented attack.

Partial Solution -- Open Source Sustainability

If we could just pay maintainers, they would be slightly more comfortable maintaining packages and less desperate for help. This means that it would become inherently slightly harder to quickly become a new maintainer.

However, it is worthwhile to consider that this still would not solve the subtler "adding a new dependency" attack described earlier: just making a "good" library and getting other libraries to depend on it.

Summary

I do not know how to prevent the "next" attack. Hillel makes the point that a lot of "root causes" will only prevent almost-exact repeats, while failing to address trivial variations. Remember that one trivial variation, avoiding deprecation warnings, would have made this attack much more successful.

I am concerned that, as an industry, we are not discussing this attack a mere two months after discovery and mitigation. We are vulnerable. We will be attacked again. We need to be prepared.

25 Jan 2019 5:00am GMT

Itamar Turner-Trauring: A 4-day workweek for programmers, the easy way

You're dreaming of a programming job with 30 hours a week, a job where you'll have time for your own projects, your own hobbies. But this sort of job seems practically non-existent-almost no one advertises programming jobs with shorter workweeks.

How do you carve out a job like this, a job with a shorter workweek?

The ideal would be some company or employer where you just can ask for a shorter workweek, without having to apply or interview, and have a pretty good chance of getting a "yes".

In this post I'll talk about the easiest way to get what you want: negotiating at your current job.

The value of being an existing employee

as an existing employee you are much more valuable than an equally experienced programmer who doesn't work there.

During your time at your employer you have acquired a variety of organization-specific knowledge and skills. It can take quite a while for a new employee to acquire these, and during the ramp-up period they will also take up their colleagues' time.

Here are just a few of the things that you've likely learned during your time as an employee:

Not only do you have hard to-replace skills and knowledge, you also have your work record to build on: your manager knows what you can do. Once you've been working for a manager for a while they'll know your skills, and whether they can trust you.

A real-life example

In my own career, being an existing employee has benefited me on multiple occasions:

After a number of years working as a software engineer for one company, I got a bad case of RSI (repetitive strain injury). I could no longer type, which meant I could no longer work as a programmer. But I did stay on as an employee: one of the managers at the company, who I'd worked for in an earlier role, offered me a position as a product manager.

In part this was because the company was run by decent people, who for the most part took care of their employees. But it was also because I had a huge amount of hard-to-replace business and technical knowledge of the company's product, in a highly complex domain.

I worked as a product manager for a few years, but I never really enjoyed it. And with time my hands recovered, at least partially, potentially allowing me to take up programming again. After my daughter was born, my wife and I decided that I'd become a part-time consultant, and take care of our child the rest of the time, while she continued working full-time.

My manager was upset when I told him I was leaving. I felt bad-but really, if your boss is unhappy when you're leaving, that's a good thing.

In fact, my boss offered to help me find a less-than-full-time programming position within the company so I wouldn't have to leave. I'd already made up my mind to go, but under other circumstances I might have taken him up on the offer.

Notice how I was offered reduced hours, even though companies will never advertise such positions. That's the value of being an existing employee.

Asking for what you want

Unless you work for a really bad manager-or a really badly managed company-a reasonable manager would much prefer to have your experience for 4 days a week than have to find a train a replacement. That doesn't mean they'll be happy if you ask for a shorter workweek: you are likely to get some pushback.

But if you have a strong negotiating position-financial savings, valuable work, the willingness to find another job if necessary-there's a decent chance they will eventually say "yes".

Does negotiating seem too daunting, or not something you can do? Plenty of other programmers have done it, even those with no previous negotiation experience.

Much of this article was an excerpt from my book, Negotiate a 3-Day Weekend. It covers everything from negotiation exercises you can do on the job, to a specific script for talking to your boss, to negotiating at new employers if you can't do it at your current job.

With a little bit of practice, you can get the free time you need.



Struggling with a 40-hour workweek? Too tired by the end of the day to do anything but collapse on the sofa and watch TV?

Learn how you can get a 3-day weekend, every single week.

25 Jan 2019 5:00am GMT

18 Jan 2019

feedPlanet Twisted

Itamar Turner-Trauring: Negotiate your salary like a 6-year old

You're applying for jobs, you're hoping to get an offer soon-and when you do you'll have to face the scary issue of negotiation.

If you read a negotiation guide like Valerie Aurora's HOWTO (and you should!) you're told you need to negotiate: you need to ask for more. And that's good advice.

The problem is that asking for more is scary: it feels confrontational, your body will kick in with an unhelpful adrenaline surge, it's just not comfortable. And honestly, given you only negotiate your salary every few years, that adrenaline surge probably isn't going to go away.

But I think you can get past that and negotiate anyway-by embracing your inner 6-year old. In particular, a 6-year old negotiating for snacks on a playdate.

Snack time!

Let's set the scene.

A 6-year old named Emma is visiting her friend Jackson, and now it's time for a snack. Jackson's parent-Michael-now needs to feed Emma.

Michael would prefer the children not eat crackers, but he has a problem. Michael has some authority over Jackson since he's his parent, and some knowledge of what Jackson is willing to eat. So he can say "you're eating one of these mildly healthy snacks" and that's the end of it.

But Emma is just visiting: Michael has less authority, less knowledge, and a hungry 6-year old is Bad News. So Michael comes up with an acceptable range of snacks, and tries his best to steer towards the ones he considers healthier.

The conversation goes something like this:

Michael: "Would you like some fruit?"

Emma: blank stare.

Michael: "How about same cheese?"

Emma: shakes her head.

Michael: "Yogurt?"

Emma: shakes her head.

Michael: "Crackers?"

Emma and Jackson: "Yes!"

Michael has committed to feeding Emma something, he doesn't want her to go hungry-but he doesn't have the normal leverage he has as a parent. As a result, Emma can just stall until she gets what she wants. Particularly enterprising children will ask for candy (even when they would never get candy at home!), but stalling seems well within the capacity of most 6-year olds.

Salary time!

The dynamic of hiring a new employee is surprisingly similar.

Whoever is making the offer-HR, an internal recruiter, or the hiring manager-has already committed to hiring you. They've decided: they had interviews and meetings and they want to get it over with and just get you on board.

So they come up with an acceptable salary range, and offer you the low end of the range. If you accept that, great. And if you say "can you do better?"

Well, they've already decided on their acceptable salary range: they'll just move up within that range. They're not insulted, they're used to this. They're not horrified at a hole in their budget, this is still within their acceptable range.

You went from fruit to crackers, and they can live with that. All you have to do is ask.

Embrace your inner 6-year old

Much of what determines your salary happens before you get the offer, when the decision is made about what salary range to offer you.

You can influence this by the language on your resume, by how you present yourself, how you interview, and by noting you have competing offers. It may not feel like negotiation, but it's all part of the process-and while it's a set of skills you can master as an adult, that part is far beyond what your 6-year old self could do.

But the actual conversation about salary? Pretend you're 6, pretend it's a snack, and ask for more-chances are you'll get some delicious crackers.



Struggling with a 40-hour workweek? Too tired by the end of the day to do anything but collapse on the sofa and watch TV?

Learn how you can get a 3-day weekend, every single week.

18 Jan 2019 5:00am GMT

09 Jan 2019

feedPlanet Twisted

Moshe Zadka: Checking in JSON

JSON is a useful format. It might not be ideal for hand-editing, but it does have the benefit that it can be hand-edited, and it is easy enough to manipulate programmatically.

For this reason, it is likely that at some point or another, checking in a JSON file into your repository will seem like a good idea. Perhaps it is even beyond your control: some existing technology uses JSON as a configuration file, and the easiest thing is to go with it.

It is useful to still keep the benefit of programmatic manipulation. For example, if the JSON file encodes a list of numbers, and we want to add 1 to every even number, we can do:

with open("myfile.json") as fp:
    content = json.load(fp)
content = [x + (2 % i) for i, x in enumerate(content)]
with open("myfile.json", "w") as fp:
    json.dumps(fp, content)

However, this does cause a problem: presumably, before, the list was formatted in a visually-pleasing way. Having dumped it, now the diff is unreadable -- and hard to audit visually.

One solution is to enforce consistent formatting.

For example, using pytest, we can write the following test:

def test_formatting():
    with open("myfile.json") as fp:
        raw = fp.read()
    content = json.loads(raw)
    redumped = json.dumps(content, indent=4) + "\n"
    assert raw == redumped

Assuming we gate merges to the main branches on passing tests, it is impossible to check in something that breaks the formatting. Automated programs merely need to remember to give the right options to json.dumps. However, what happens when humans make mistakes?

It turns out that Python already has a command-line tool to reformat:

$ python -m json.tool myfile.json > myfile.json.formatted
$ mv myfile.json.formatted myfile.json

A nice test failure will remind the programmer of this trick, so that it is easy to do and check in.

09 Jan 2019 6:00am GMT

Itamar Turner-Trauring: Competing with a "Stanford grad just dying to work all nighters on Red Bull"

A reader of my newsletter wrote in, talking about the problem of finding a job with work/life balance in Silicon Valley:

It seems like us software engineers are in a tough spot: companies demand a lot of hard work and long hours, and due to the competitiveness here in Silicon Valley, you have to go along with it (or else there's some bright young Stanford grad just dying to work all nighters on Red Bull to take your place).

But they throw you aside once the company has become established and they no longer need the "creative" types.

In short, how do you get a job with work/life balance when you're competing against people willing to work long hours?

All nighters make for bad software

The starting point is realizing that working long hours makes you a much less productive employee, to the point that your total output will actually decrease (see Evan Robinson on crunch time). If you want to become an effective and productive worker, you're actually much better off working normal hours and having a personal life than working longer hours.

Since working shorter hours makes you more productive, that gives you a starting point for why you should be hired.

Instead of focusing on demonstrative effort by working long hours, you can focus on identifying and solving valuable problems, especially the bigger and longer term problems that require thought and planning to solve.

Picking the right company

Just because you're a valuable, productive programmer doesn't mean you're going to get hired, of course. So next you need to find the right company.

You can imagine three levels of managerial skill:

When you look for a job you will want to avoid Level 1 managers. However good your work, they will be happy to replace you with someone incompetent so long as they can get more hours out of them. So you'll be forced to work long hours and work on broken code.

Level 3 managers are ideal, and they do exist. So if you can find a job working for them then you're all set.

Level 2 managers are probably more common though, and you can get work/life balance working for them-if you set strong boundaries. Since they can recognize actual competence and skills, they won't judge you during your interview based on how many hours you're willing to work. You just need to convey your skill and value, and a reasonable amount of dedication to your job.

And once you've started work, if you can actually be productive (and if you work 40 hours/week you will be more productive!) they won't care if you come in at 9 and leave at 5, because they'll be happy with your work.

Unlike Level 3 managers, however, you need to be explicit about boundaries during the job interview, and even more so after you start. Elsewhere I wrote up some suggestions about how to convey your value, and how to say "no" to your boss.

Employment is a negotiated relationship

To put all this another way: employment is a negotiated relationship. Like it or not, you are negotiating from the moment you start interviewing, while you're on the job, and until the day you leave.

You are trying to trade valuable work for money, learning opportunities, and whatever else your goals are (you can, for example, negotiate for a shorter workweek). In this case, we're talking about negotiating for work/life balance:

  1. Level 1 managers you can't negotiate with, because what they want (long hours) directly conflicts with what you want.
  2. Level 2 managers you can negotiate with, by giving them one of the things they want: valuable work.
  3. Level 3 managers will give you what you want without your having to do anything, because they know it's in the best interest of everyone.

Of course, even for Level 3 managers you will still need to negotiate other things, like a higher salary.

So how do you get a job with work/life balance? Focus on providing and demonstrating valuable long-term work, avoid bad companies, and make sure you set boundaries from the very start.



Struggling with a 40-hour workweek? Too tired by the end of the day to do anything but collapse on the sofa and watch TV?

Learn how you can get a 3-day weekend, every single week.

09 Jan 2019 5:00am GMT

12 Dec 2018

feedPlanet Twisted

Itamar Turner-Trauring: Tests won't make your software correct

Automated tests are immensely useful. Once you've started writing tests and seen their value, the idea of writing software without them becomes unimaginable.

But as with any technique, you need to understand its limitations. When it comes to automated testing-unit tests, BDD, end-to-end tests-it's tempting to think that if your tests pass, your software is correct.

But tests don't, tests can't tell you that your software is correct. Let's see why.

How to write correct software

To implement a feature or bugfix, you go through multiple stages; they might be compressed or elided, but they are always necessary:

  1. Identification: Figure out what the problem is you're trying to solve.
  2. Solution: Come up with a solution.
  3. Specification: Define a specification, the details of how the solution will be implemented.
  4. Implementation: Implement the specification in code.

Your software might end up incorrect at any of these points:

  1. You might identify the wrong problem.
  2. You might choose the wrong solution.
  3. You might create a specification that doesn't match the solution.
  4. You might write code that doesn't match the specification.

Only human judgment can decide correctness

Automated tests are also a form of software, and are just as prone to error. The fact that your automated tests pass doesn't tell you that your software is correct: you may still have identified the wrong problem, or chosen the wrong solution, and so on.

Even when it comes to ensuring your implementation matches your specification, tests can't validate correctness on their own. Consider the following test:

def test_addition():
    assert add(2, 2) == 5

From the code's perspective-the perspective of an automaton with no understanding-the correct answer of 4 is the one that will cause it to fail. But merely by reading that you can tell it's wrong: you, the human, are key.

Correctness is something only a person can decide.

The value of testing: the process

While passing tests can't prove correctness, the process of writing tests and making them pass can help make your software correct. That's because writing the tests involves applying human judgment: What should this test assert? Does match the specification? Does this actually solve our problem?

When you go through the loop of writing tests, writing code, and checking if tests pass, you continuously apply your judgment: is the code wrong? is the test wrong? did I forget a requirement?

You write the test above, and then reread it, and then say "wait that's wrong, 2 + 2 = 4". You fix it, and then maybe you add to your one-off hardcoded tests some additional tests based on core arithmetic principles. Correctness comes from applying the process, not from the artifacts created by the process.

This may seem like pedantry: what does it matter whether the source of correctness is the tests themselves or the process of writing the tests? But it does matter. Understanding that human judgment is the key to correctness can keep you from thinking that passing tests are enough: you also need other forms of applied human judgment, like code review and manual testing.

(Formal methods augment human judgment with automated means… but that's another discussion.)

The value of tests: stability

So if correctness comes from writing the tests, not the tests themselves, why do we keep the tests around?

Because tests ensure stability. once we judge the software is correct, the tests can keep the software from changing, and thus reduce the chances of its becoming incorrect. The tests are never enough, because the world can change even if the software isn't, but stability has its value.

(Stability also has costs if you make the wrong abstraction layer stable…)

Tests are useful, but they're not sufficient

To recap:

  1. Write automated tests.
  2. Run those tests.
  3. Don't mistake passing tests for correctness: you will likely need additional processes and techniques to achieve that.



Struggling with a 40-hour workweek? Too tired by the end of the day to do anything but collapse on the sofa and watch TV?

Learn how you can get a 3-day weekend, every single week.

12 Dec 2018 5:00am GMT