16 Aug 2019

feedPlanet Twisted

Twisted Matrix Laboratories: Twisted 19.7.0 Released

On behalf of Twisted Matrix Laboratories and our long-suffering release manager Amber Brown, I am honored to announce1 the release of Twisted 19.7.0!


The highlights of this release include:
  • A full description on the PyPI page! Check it out here: https://pypi.org/project/Twisted/19.7.0/ (and compare to the slightly sad previous version, here: https://pypi.org/project/Twisted/19.2.1/)
  • twisted.test.proto_helpers has been renamed to "twisted.internet.testing"
    • This removes the gross special-case carve-out where it was the only "public" API in a test module, and now the rule is that all test modules are private once again.
  • Conch's SSH server now supports hmac-sha2-512.
  • The XMPP server in Twisted Words will now validate certificates!
  • A nasty data-corruption bug in the IOCP reactor was fixed. If you're doing high-volume I/O on Windows you'll want to upgrade!
  • Twisted Web no longer gives clients a traceback by default, both when you instantiate Site and when you use twist web on the command line. You can turn this behavior back on for local development with twist web --display-tracebacks.
  • Several bugfixes and documentation fixes resolving bytes/unicode type confusion in twisted.web.
  • Python 3.4 is no longer supported.
pip install -U twisted[tls] and enjoy all these enhancements today!

Thanks for using Twisted,

-glyph

1: somewhat belatedly: it came out 10 days ago. Oops!

16 Aug 2019 6:38am GMT

08 Aug 2019

feedPlanet Twisted

Moshe Zadka: Designing Interfaces

One of the items of feedback I got from the article about interface immutability is that it did not give any concrete feedback for how to design interfaces. Given that they are forever, it would be good to have some sort of guidance.

The first item is that you want something that uses the implementation, as well as several distinct implementations. However, this item is too obvious: in almost all cases I have seen in the wild of a bad interface, this guideline was followed.

It was also followed in all cases of a good interface.

I think this guideline is covered well enough that by the time anyone designs a real interface, they understand that. Why am I mentioning this guideline at all, then?

Because I think it is important for the context of the guideline that I do think actually distinguishes good interfaces from bad interfaces. It is almost identical to the non-criterion above!

The real guideline is: something that uses the implementation, as well as several distinct implementations that do not share a superclass (other than object or whatever is in the top of the hierarchy).

This simple addition, preventing the implementations from sharing a superclass, is surprisingly powerful. It means each implementation has to implement the "boring" parts by hand. This will immediately cause pressure to avoid "boring" parts, and instead put them in a wrapper, or in the interface user.

Otherwise, the most common failure mode is that the implementations are all basic variants on what is mostly the "big superclass".

In my experience, just the constraint on not having a "helper superclass" puts appropriate pressure on interfaces to be good.

(Thanks to Tom Most for his encouragement to write this, and the feedback on an earlier draft. Any mistakes that remain are my responsibility.)

08 Aug 2019 5:20am GMT

13 Jul 2019

feedPlanet Twisted

Moshe Zadka: Interfaces are forever

(The following talks about zope.interface interfaces, but applies equally well to Java interfaces, Go interfaces, and probably other similar constructs.)

When we write a function, we can sometimes change it in backwards-compatible ways. For example, we can loosen the type of a variable. We can restrict the type of the return value. We can add an optional argument.

We can even have a backwards compatible path to make an argument required. We add an optional argument, and encourage people to change it. Then, in the next version, we make the default value be one that causes a warning. In a version after that, we make the value required. At each point, someone could write a library that worked with at least two consecutive versions.

In a similar way, we can have a path to remove an argument. First make it optional. Then warn when it is passed in. Finally, remove it and make it an error to pass it in.

As long as we do not intend to support inheritance, making backwards compatible changes to classes also works. For example, to remove a method we first have a version that warns when you call it, and then remove it in a succeeding version.

However, what changes can we make to an interface?

Assume we have an interface like:

from zope.interface import Interface, implements

class IFancyFormat(Interface):

    def fancify_int(value: int) -> str:
        pass

It is a perfectly reasonable, if thin, interface. Implementing it seems like fun:

@implements(IFancyFormat)
@attr.s(auto_attribs=True)
class FancySuffixer:
    suffix: str

    def fancify_int(self, value: int) -> str:
        return str(value) + self.suffix

Using it also seems like fun:

def dashify_fancy_five(fancifier: IFancyFormat) -> str:
    return f"---{fancifier.fancify_int(5)}---"

These are very different kinds of fun, though! Probably the kind of fun that appeals to different people. The first implementation is in the superfancy open-source library. The second one is in the dash_five open-source library. Such is the beauty of open source: it takes all kinds of people.

We cannot add a method to IFancyFormat: the superfancy library has a unit test that uses verifyImplements, which will fail if we add a method. We cannot remove the method fancify_int, since this will break dash_five: the mypy check will fail, since IFancifySuffixer will not have that method.

Similarly, we cannot make the parameter optional without breaking superfancy, or loosen the return type without breaking dash_five. Once we have published IFancyFormat as an API, it cannot change.

The only way to recover from a bad interface is to create a new interface, IAwesomeFancyFormat. Then write conversion functions from and to IFancyFormat and IAwesomeFancyFormat. Then deprecate using the IFancyFormat interface. Finally, we can remove the interface. Then we can alias IFancyFormat == IAwesomeFancyFormat, and eventually, maybe even deprecate the name IAwesomeFancyFormat.

When publishing interfaces, one must be careful: to a first approximation, they are forever.

(Thanks to Glyph Lefkowitz for his helpful suggestions. Any mistakes or issues that are left are my responsibility.)

13 Jul 2019 5:00am GMT

14 Jun 2019

feedPlanet Twisted

Glyph Lefkowitz: Toward a “Kernel Python”

Prompted by Amber Brown's presentation at the Python Language Summit last month, Christian Heimes has followed up on his own earlier work on slimming down the Python standard library, and created a proper Python Enhancement Proposal PEP 594 for removing obviously obsolete and unmaintained detritus from the standard library.

PEP 594 is great news for Python, and in particular for the maintainers of its standard library, who can now address a reduced surface area. A brief trip through the PEP's rogues gallery of modules to deprecate or remove1 is illuminating. The python standard library contains plenty of useful modules, but it also hides a veritable necropolis of code, a towering monument to obsolescence, threatening to topple over on its maintainers at any point.

However, I believe the PEP may be approaching the problem from the wrong direction. Currently, the standard library is maintained in tandem with, and by the maintainers of, the CPython python runtime. Large portions of it are simply included in the hope that it might be useful to somebody. In the aforementioned PEP, you can see this logic at work in defense of the colorsys module: why not remove it? "The module is useful to convert CSS colors between coordinate systems. [It] does not impose maintenance overhead on core development."

There was a time when Internet access was scarce, and maybe it was helpful to pre-load Python with lots of stuff so it could be pre-packaged with the Python binaries on the CD-ROM when you first started learning.

Today, however, the modules you need to convert colors between coordinate systems are only a pip install away. The bigger core interpreter is just more to download before you can get started.

Why Didn't You Review My PR?

So let's examine that claim: does a tiny module like colorsys "impose maintenance overhead on core development"?

The core maintainers have enough going on just trying to maintain the huge and ancient C codebase that is CPython itself. As Mariatta put it in her North Bay Python keynote, the most common question that core developers get is "Why haven't you looked at my PR?" And the answer? It's easier to not look at PRs when you don't care about them. This from a talk about what it means to be a core developer!

One might ask, whether Twisted has the same problem. Twisted is a big collection of loosely-connected modules too; a sort of standard library for networking. Are clients and servers for SSH, IMAP, HTTP, TLS, et. al. all a bit much to try to cram into one package?

I'm compelled to reply: yes. Twisted is monolithic because it dates back to a similar historical period as CPython, where installing stuff was really complicated. So I am both sympathetic and empathetic towards CPython's plight.

At some point, each sub-project within Twisted should ideally become a separate project with its own repository, CI, website, and of course its own more focused maintainers. We've been slowly splitting out projects already, where we can find a natural boundary. Some things that started in Twisted like constantly and incremental have been split out; deferred and filepath are in the process of getting that treatment as well. Other projects absorbed into the org continue to live separately, like klein and treq. As we figure out how to reduce the overhead of setting up and maintaining the CI and release infrastructure for each of them, we'll do more of this.


But is our monolithic nature the most pressing problem, or even a serious problem, for the project? Let's quantify it.

As of this writing, Twisted has 5 outstanding un-reviewed pull requests in our review queue. The median time a ticket spends in review is roughly four and a half days.2 The oldest ticket in our queue dates from April 22, which means it's been less than 2 months since our oldest un-reviewed PR was submitted.

It's always a struggle to find enough maintainers and enough time to respond to pull requests. Subjectively, it does sometimes feel like "Why won't you review my pull request?" is a question we do still get all too often. We aren't always doing this well, but all in all, we're managing; the queue hovers between 0 at its lowest and 25 or so during a bad month.

By comparison to those numbers, how is core CPython doing?

Looking at CPython's keyword-based review queue queue, we can see that there are 429 tickets currently awaiting review. The oldest PR awaiting review hasn't been touched since February 2, 2018, which is almost 500 days old.

How many are interpreter issues and how many are stdlib issues? Clearly review latency is a problem, but would removing the stdlib even help?

For a quick and highly unscientific estimate, I scanned the first (oldest) page of PRs in the query above. By my subjective assessment, on this page of 25 PRs, 14 were about the standard library, 10 were about the core language or interpreter code; one was a minor documentation issue that didn't really apply to either. If I can hazard a very rough estimate based on this proportion, somewhere around half of the unreviewed PRs might be in standard library code.


So the first reason the CPython core team needs to stop maintaining the standard library because they literally don't have the capacity to maintain the standard library. Or to put it differently: they aren't maintaining it, and what remains is to admit that and start splitting it out.

It's true that none of the open PRs on CPython are in colorsys3. It does not, in fact, impose maintenance overhead on core development. Core development imposes maintenance overhead on it. If I wanted to update the colorsys module to be more modern - perhaps to have a Color object rather than a collection of free functions, perhaps to support integer color models - I'd likely have to wait 500 days, or more, for a review.

As a result, code in the standard library is harder to change, which means its users are less motivated to contribute to it. CPython's unusually infrequent releases also slow down the development of library code and decrease the usefulness of feedback from users. It's no accident that almost all of the modules in the standard library have actively maintained alternatives outside of it: it's not a failure on the part of the stdlib's maintainers. The whole process is set up to produce stagnation in all but the most frequently used parts of the stdlib, and that's exactly what it does.

New Environments, New Requirements

Perhaps even more importantly is that bundling together CPython with the definition of the standard library privileges CPython itself, and the use-cases that it supports, above every other implementation of the language.

Podcast after podcast after podcast after keynote tells us that in order to keep succeeding and expanding, Python needs to grow into new areas: particularly web frontends, but also mobile clients, embedded systems, and console games.

These environments require one or both of:

In all of these cases, determining which modules have been removed from the standard library is a sticking point. They have to be discovered by a process of trial and error; notably, a process completely different from the standard process for determining dependencies within a Python application. There's no install_requires declaration you can put in your setup.py that indicates that your library uses a stdlib module that your target Python runtime might leave out due to space constraints.

You can even have this problem even if all you ever use is the standard python on your Linux installation. Even server- and desktop-class Linux distributions have the same need for a more minimal core Python package, and so they already chop up the standard library somewhat arbitrarily. This can break the expectations of many python codebases, and result in bugs where even pip install won't work.

Take It All Out

How about the suggestion that we should do only a little a day? Although it sounds convincing, don't be fooled. The reason you never seem to finish is precisely because you tidy a little at a time. [...] The ultimate secret of success is this: If you tidy up in one shot, rather than little by little, you can dramatically change your mind-set.

- Kondō, Marie.
"The Life-Changing Magic of Tidying Up"
(p. 15-16)

While incremental slimming of the standard library is a step in the right direction, incremental change can only get us so far. As Marie Kondō says, when you really want to tidy up, the first step is to take everything out so that you can really see everything, and put back only what you need.

It's time to thank those modules which do not spark joy and send them on their way.

We need a "kernel" version of Python that contains only the most absolutely minimal library, so that all implementations can agree on a core baseline that gives you a "python", and applications, even those that want to run on web browsers or microcontrollers, can simply state their additional requirements in terms of requirements.txt.

Now, there are some business environments where adding things to your requirements.txt is a fraught, bureaucratic process, and in those places, a large standard library might seem appealing. But "standard library" is a purely arbitrary boundary that the procurement processes in such places have drawn, and an equally arbitrary line may be easily drawn around a binary distribution.

So it may indeed be useful for some CPython binary distributions - perhaps even the official ones - to still ship with a broader selection of modules from PyPI. Even for the average user, in order to use it for development, at the very least, you'd need enough stdlib stuff that pip can bootstrap itself, to install the other modules you need!

It's already the case, today, that pip is distributed with Python, but isn't maintained in the CPython repository. What the default Python binary installer ships with is already a separate question from what is developed in the CPython repo, or what ships in the individual source tarball for the interpreter.

In order to use Linux, you need bootable media with a huge array of additional programs. That doesn't mean the Linux kernel itself is in one giant repository, where the hundreds of applications you need for a functioning Linux server are all maintained by one team. The Linux kernel project is immensely valuable, but functioning operating systems which use it are built from the combination of the Linux kernel and a wide variety of separately maintained libraries and programs.

Conclusion

The "batteries included" philosophy was a great fit for the time when it was created: a booster rocket to sneak Python into the imagination of the programming public. As the open source and Python packaging ecosystems have matured, however, this strategy has not aged well, and like any booster, we must let it fall back to earth, lest it drag us back down with it.

New Python runtimes, new deployment targets, and new developer audiences all present tremendous opportunities for the Python community to soar ever higher.

But to do it, we need a newer, leaner, unburdened "kernel" Python. We need to dump the whole standard library out on the floor, adding back only the smallest bits that we need, so that we can tell what is truly necessary and what's just nice to have.

I hope I've convinced at least a few of you that we need a kernel Python.

Now: who wants to write the PEP?

🚀

Acknowledgments

Thanks to Jean-Paul Calderone, Donald Stufft, Alex Gaynor, Amber Brown, Ian Cordasco, Jonathan Lange, Augie Fackler, Hynek Schlawack, Pete Fein, Mark Williams, Tom Most, Jeremy Thurgood, and Aaron Gallagher for feedback and corrections on earlier drafts of this post. Any errors of course remain my own.


  1. sunau, xdrlib, and chunk are my personal favorites.

  2. Yeah, yeah, you got me, the mean is 102 days.

  3. Well, as it turns out, one is on colorsys, but it's a documentation fix that Alex Gaynor filed after reviewing a draft of this post so I don't think it really counts.

14 Jun 2019 4:51am GMT

06 Jun 2019

feedPlanet Twisted

Twisted Matrix Laboratories: Twisted 19.2.1 Released

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 19.2.1!

This is a security release, and contains the following changes:
  • All HTTP clients in twisted.web.client now raise a ValueError when called with a method and/or URL that contain invalid characters. This mitigates CVE-2019-12387. Thanks to Alex Brasetvik for reporting this vulnerability.
It is recommended you update to this release as soon as is practical.

Additional mitigation may be required if Twisted is not your only HTTP client library:

You can find the downloads at <https://pypi.python.org/pypi/Twisted> (or alternatively <http://twistedmatrix.com/trac/wiki/Downloads>). The NEWS file is also available at <https://github.com/twisted/twisted/blob/twisted-19.2.1/NEWS.rst>.

Twisted Regards,
Amber Brown (HawkOwl)

06 Jun 2019 2:49pm GMT

03 Jun 2019

feedPlanet Twisted

Hynek Schlawack: Python in Azure Pipelines, Step by Step

Since the acquisition of Travis CI, the future of their free offering is unclear. Azure Pipelines has a generous free tier, but the examples I found are discouragingly complex and take advantage of features like templating that most projects don't need. To close that gap, this article shows you how to move a Python project with simple CI needs from Travis CI to Azure Pipelines.

03 Jun 2019 9:14am GMT

28 May 2019

feedPlanet Twisted

Moshe Zadka: Analyzing the Stack Overflow Survey

The Stack Overflow Survey Results for 2019 are in! There is some official analysis, that mentioned some things that mattered to me, and some that did not. I decided to dig into the data and see if I can find some things that would potentially interest my readership.

import csv, collections, itertools
with open("survey_results_public.csv") as fpin:
    reader = csv.DictReader(fpin)
    responses = list(reader)
len(responses)
88883

Wow, almost 90K respondents! This is the sweet spots of "enough to make meaningful generalizations" while being able to analyze with rudimentary tools, not big-data-ware.

pythonistas = [x for x in responses if 'Python' in x['LanguageWorkedWith']]
len(pythonistas)/len(responses)
0.41001091322300104

About 40% of the respondents use Python in some capacity. That is pretty cool! This is one of the things where I wonder if there is bias in the source data. Are people who use Stack Overflow, or respond to surveys for SO, more likely to be the kind of person who uses Python? Or less?

In any case, I am excited! This means my favorite language, for all its issues, is doing well. This is also a good reminder that we need to think about the consequences of our decisions on a big swath of developers we will never ever meet.

opensource = collections.Counter(x['OpenSourcer'] for x in pythonistas)
sorted(opensource.items(), key=lambda x:x[1], reverse=True)
[('Never', 11310),
 ('Less than once per year', 10374),
 ('Less than once a month but more than once per year', 9572),
 ('Once a month or more often', 5187)]
opensource['Once a month or more often']/len(pythonistas)
0.1423318607139917

Python is open source. Almost all important libraries (Django, Pandas, PyTorch, requests) are open source. Many important tools (Jupyter) are open source. The number of people who contribute to them with any kind of regular cadence is less than 15%.

general_opensource = collections.Counter(x['OpenSourcer'] for x in responses)
sorted(general_opensource.items(), key=lambda x:x[1], reverse=True)
[('Never', 32295),
 ('Less than once per year', 24972),
 ('Less than once a month but more than once per year', 20561),
 ('Once a month or more often', 11055)]

The Python community does compare well to the general populace, though!

devtype = collections.Counter(itertools.chain.from_iterable(x["DevType"].split(";") for x in pythonistas))
devtype['DevOps specialist']/len(responses)
0.052282213696657406

About 5% of total respondents are my peers: using Python for DevOps. That is pretty exciting! My interest in that is not merely theoretical, my upcoming book targets that crowd.

general_devtype = collections.Counter(itertools.chain.from_iterable(x["DevType"].split(";") for x in responses))
general_devtype['DevOps specialist']/len(responses), devtype['DevOps specialist']/len(pythonistas)
(0.09970410539698255, 0.12751420025793705)

In general, DevOps specialists are 10% of respondents.

devtype['DevOps specialist']/general_devtype['DevOps specialist']
0.524373730534868

Over 50% of DevOps specialists use Python!

def safe_int(x):
    try:
        return int(x)
    except ValueError:
        return -1

intermediate = sum(1 for x in pythonistas if 1<=safe_int(x['YearsCode'])<=5)

My next hush-hush (for now!) project is going to be targeting intermediate Python developers. I wish I could slice by "number of years writing in Python, but this is the best I could do. (I treat "NA" responses as "not intermediate". This is OK, since I prefer to underestimate rather than overestimate.)

intermediate/len(responses)
0.11346376697456206

11%! Not bad.

general_intermediate = sum(1 for x in responses if 1<=safe_int(x['YearsCode'])<=5)
intermediate/len(pythonistas), general_intermediate/len(responses)
(0.27673352907279863, 0.2671264471271222)

Seems like using Python does not change much the chances of someone being intermediate.

Summary

  • 40% of respondents use Python. Python is kind of a big deal.
  • 5% of respondents use Python for DevOps. This is a lot! DevOps as a profession is less than 10 years old.
  • 11% of respondents are intermediate Python users. My previous book targets this crowd.

(Thanks to Robert Collins and Matthew Broberg for their comments on an earlier draft. Any remaining issues are purely my responsibility.)

28 May 2019 5:20am GMT

16 May 2019

feedPlanet Twisted

Moshe Zadka: Inbox Zero

I am the parent of two young kids. It is easy to sink into random stuff, and not follow up on goals. Strict time management and prioritization means I get to work on open source projects, write programming books and update my blog with a decent cadence. Since a lot of people were asking me how to do it, I wanted to share my methodology. The following is descriptive, not prescriptive.

One thing I am proud of is that the initial draft for the post was written a year ago. I have done my edits for clarity, but found that my description of the process, for the most part, has remained the same. This made me confident that it is time to publish: this process has existed in its current form for at least a year, and I believe almost two years. This is not some fad diet for me: this process has proved its worth.

Glyph has already written at length about how a full Inbox is a sign of misprioritized tasks. Saying "no" is one example (in other words, prioritizing away). But when saying "yes", it is a good idea to know when it can be done, when should you give up, and potentially apologize, and when should you give a heads-up that it is being delayed.

His description, being more high-level, is prescriptive. The follow-up is the process I use, shaped by those general ideas.

The tool I use is TODOist. The first time I tried it, I decided it lacked some necessary features. I still feel this way -- about the free version. The free version is completely unusable. The premium version is perfectly usable.

The salient features of TODOist, that the rest of the explanation depends on, are:

E-mail scan process

I read e-mail "when I get around to it". Usually several times a day. I do have notifications enabled on my phone, so I can easily see if the e-mail is urgent. Otherwise, I just ignore the notification.

When I do go through my e-mail, I follow the rules:

  • If it's obvious there is no task, archive
  • If it's something short, obvious and I have the time, do it and archive. However, if I find in the middle that I am wrong about it being short and obvious, I abort. Usually it is obvious if an e-mail will require a lengthy research project. The most common way of being wrong is when, while responding, I find myself getting too emotional. I have trained myself to consider this as a trigger for aborting.
  • Otherwise, I "Forward" and send it to the TODOist auto-task e-mail -- and then immediately archive. The forwarded message, having literally all the words in the original, is enough information to search for the original in my archive.

Browser

The only "permanently" open tabs in a browser should be "communication" tabs: FB messenger, whatsapp, slack, etc. If any other tab feels like it would be bad to close, create task from it. I verify that each tab is OK to close, or needs a task + close, by closing all non-communication tabs if the tabs become too small to read the titles (Chrome) or the tabs need scrolling (Firefox).

My usual research task takes several tabs (Python documentation, StackOverflow, GitHub pull requests, tickets and more), so tab accumulation happens naturally, thus triggering the garbage collection process.

Reviewing tasks

Clean triage

This is a daily task, to go to the filter "triage" and clean it out. The filter is defined as "not marked 'time permitting' and does not have a due date". Since tasks come in without marking or due date, this is a filter for tasks that come in. The task is "done" when the filter is empty. Any task that actually needs to get done will get Scheduled with a due date. Note that this due date is not a real "due": it is when I plan to do it. This will get determined based on the task, on my available time, and when other tasks got scheduled.

Otherwise, the task is marked "time permitting". This means, in real terms, that I will probably never get around to it. This is fine -- and it feels nicer than archiving or deleting the task. It allows me to be less FOMO when doing the triage.

Occasionally, an external trigger will rescue a task from the "time permitting" graveyard.

Rebalance

Rebalance means that I do not want to have an empty day, followed by an avalanche day: I'll be as carefree as the grasshopper that day, watch TV and frolic, and then drown in tasks the next.

I look ahead, and if I see a day with less than 5-6 tasks, I will move some tasks forward to get done sooner. I do not worry about the opposite. If there are too many tasks one day, they'll naturally get postponed.

Non-meta Tasks

I treat the due date as an "ETA". I try to do all tasks due a given day on that day. If there is an objective deadline, e.g. a CFP that closes on a date, that deadline will be in human readable form on the task.

If I am too tired, or cannot handle more load, I start rescheduling "today" tasks. This process will take into considersation the "objective" deadlines, if any. It will also take into account the subjective value of the task to me.

Any task that gets postponed "too many times" gets moved to "time permitting".

Dependencies

Humans are social creatures. Some tasks, I cannot do alone. For example, when publishing a blog post, I like to have some trusted people review it. This means that I need their feedback.

When I need something from someone, that's a task. The task is to use that thing. The due date is the date to poke them about the delivery of the thing. Because I try to build in a buffer, it allows me to be nice about it. I am endlessly patient, with e-mails asking "let me know how it is going".

Some people are also busy. If someone tells me "I'll give it to you in a week", I make a task to ask them about it in a week. If they deliver, they will never know: the task gets done when I get what I need. If not, I'll mention, gently, "hey, it's been a week, wondering if there's an update."

Some people, for good or bad reasons, do not deliver. Then I have the task of deciding what to do about it. Sometime I'll ask someone else for help. Sometime I'll do it myself. Sometime I'll drop it. Whatever it is, it was my explicit decision.

Spoon Management

If there are too many tasks, and I feel overwhelmed, I will start postponing any non-urgent tasks. Sometimes, this means I will postpone everything. If I lack the spoons, I lack the spoons. I do not feel guilt about it.

Summary

Inbox Zero is possible. Not only that. Inbox Zero, I have found, is easy. Doing everything I want to do is not easy. But the meta-process: deciding what I want to do, deciding what I am going to say "no" or flake on, that is easy.

This leads to less anxiety. I do what I can, and decide that this is enough. I am kind to myself. Be kind to yourself. Go Inbox Zero.

(Thanks to Shae Erisson for his feedback. Any issues that remain are my responsibility.)

16 May 2019 4:45am GMT

15 May 2019

feedPlanet Twisted

Hynek Schlawack: The Price of the Hallway Track

There are many good reasons to not go to every talk possible when attending conferences. However increasingly it became hip to boast with not going to talks at all - encouraging others to follow suit. As a speaker, that rubs me the wrong way and I'll try to explain why.

15 May 2019 6:00pm GMT

Itamar Turner-Trauring: Learning negotiation from Jane Austen

Looking for a job as a software developer can be scary, exhausting, or overwhelming. Where you apply and how you interview impacts whether you'll get a job offer, and how good it will be, so in some sense the whole job search is a form of negotiation.

So how do you learn to make a good impression, to convince people of your worth, to get picked by the job you want? There are many skills to learn, and in this article I'd like to cover one particular subset.

Let us travel to England, some 200 years in the past, and see what we can learn.

Jane Austen, Game Theorist

What does a novelist writing in the early 19th century have to do with getting a programming job?

In his book Jane Austen, Game Theorist, Michael Suk-Young Chwe argues quite convincingly that Austen's goal in writing her books is to teach strategic thinking: understanding what and why people do what they do, and how to interact with them accordingly, in order to achieve the outcomes you want.

Strategic thinking is a core skill in negotiation: you're trying to understand what the other side wants (even if they don't explicitly say it), and to find a way to use that to get what you want. The hiring manager might want someone who both understands their particular technical domain and can help a team grow, whereas you might want a higher salary, or a shorter workweek. Strategic thinking can help you use the one to achieve the other.

Strategic thinking is of course a useful skill for anyone, but why would Jane Austen in particular care about strategic thinking? To answer that we need a little historical context.

The worst job search ever

Imagine you could only get one job your whole life, that leaving your job was impossible, and that you'd be married to your boss. This is the "job search" that Austen faced in her own life, and is one the main topics covered in her books.

Austen's own family, and the people she writes about, were part of a very small and elite minority. Even the poorest of the families Austen writes about have at least one servant, for example.

While the men of the English upper classes, if they were not sufficiently wealthy, could and did work-as lawyers, doctors, officers-their wives and daughters for the most part could not. So if they weren't married and didn't have sufficient wealth of their own, upper-class women had very few choices-they could live off money from relations, or take on the social status loss of becoming a governess.

Marriage was therefore the presumed path to social status, economic security, and of course it determined who they would live with for the rest of their lives (divorce was basically impossible).

Finding the right husband was very important. And getting that husband-who had all the legal and social authority-to respect their wishes after marriage was just as important. And of course the women who didn't marry lived at the mercy of the family members who supported them.

And that's where strategic thinking comes in: it was a critical skill for women in Austen's class and circumstances.

Learning from Austen

If, as Michael Chwe argues, Austin's goal with her books is to teach strategic thinking, how can you use them to improve your negotiation skills?

All of Austen's books are worth reading-excepting the unfortunate Mansfield Park-but for educational purposes Northanger Abbey is a good starting point. Northanger Abbey is the story of Catherine, a naive young woman, and how she becomes less naive and more strategic.

Instead of just reading it as an entertaining novel, you can use it to actively practice your own strategic understanding:

  1. In every social interaction, Catherine has a theory about other people's motivations, why they're doing or saying certain things.
  2. Notice the assumptions underlying her theory, and then come up with your alternative theory or explanation for other characters' actions.
  3. Then, compare both theories as the plot unfolds and you learn more.

Other characters also offer a variety of opportunities to see strategic thinking-or lack of it-in action. Once you've gone through the book and experienced the growth of Catherine's strategic thinking, start practicing those skills in your life.

Why are your coworkers, family, and friends doing what they're doing? Do they have the same motivations, goals, and expectations that you do? The more you pay attention and compare your assumptions to reality, the more you'll learn-and the better you'll do at your next job interview.

Ready to get started? You can get a paper copy from the library, or download a free ebook from Project Gutenberg.



Struggling with a 40-hour workweek? Too tired by the end of the day to do anything but collapse on the sofa and watch TV?

Learn how you can get a 3-day weekend, every single week.

15 May 2019 4:00am GMT

09 May 2019

feedPlanet Twisted

Itamar Turner-Trauring: Part-time software developer jobs don't exist, right?

If you're tired of working long hours, a part-time-or even just 4 days a week-programming jobs seems appealing. You'll still get paid, you'll still hopefully enjoy your job-but you'll also have more time for other things in your life.

Hypothetically you could negotiate for more free time, but obviously no company would ever agree to a shorter workweek, right?

And indeed there are plenty of people-on Hacker News especially-who will explain to you in great detail why this can't be done, that no manager would ever agree to this, that it's a logical impossibility, a mirage, a delusion, not even worth considering.

But-

The fact is there are quite a few software developers who work less than full-time. And to help convince you, I figured I would share just a few of the examples I know of.

I've done it

Personally I've worked at three different software jobs at between 28 and 35 hours a week. And before that, when I left my last full-time job, my manager offered to help me find a part-time job there so that I would stay.

People who have read my book have done it

Since I appreciated having a shorter workweek so much, I ended up writing a book about negotiating a 3-day weekend, and a number of people who read my book have successfully done so.

I could share quotes from people who did it, and the sales page above includes just some of them, but you might feel that lacks a little credibility. So let's move on-

People I've interviewed have done it

I also interviewed a number of people for the book, including a guy by the name of Mike who has been working 4 days a week for 15 years now. You can read the full interview with Mike if you want to get his perspective.

But he's just one person, so let's move on to the final category: random people on the Internet.

Random people on the Internet have done it

Here's just a sample:

pushcx on lobste.rs: "I've worked part-time for about six years of my career."

Seitsebb on lobste.rs: "I work four days a week and can recommend it."

stsp on lobste.rs: "I was fortunate enough to be able to negotiate [Fridays off] while employed and it had a very positive impact on both my work and quality of life in general."

acflint on dev.to: "I negotiated a 4 day weekend so I could spend time on my side project … and enjoy life more."

autarch on Hacker News: "As part of my negotiations for my current job, I negotiated a 4-day (32 hour) work week. I take Fridays off and do my own projects and volunteer work."

Boycy on Hacker News: "I asked my then employer if I could drop to 4 days a week, pro-rata, and was surprised when the answer was yes!"

notacoward on Hacker News: "When I reduced my hours, I was amused to notice that everyone from the VP who approved it down to the person in HR who handled the paperwork said they wished they could do the same. I told them all that they could."

lubonay on Hacker News: "I worked on a 4-day week for about a year between 2017 and 2018 for a small consultancy company."

duckworthd on Hacker News: "I've been working a 4 day/week schedule for 1.5 years now."

I could go on, but no doubt this is getting repetitive.

You can do it too

Want to join us and get more time for yourself?

For most programmers, the easiest place to negotiate a 3-day weekend is at your current job.



Struggling with a 40-hour workweek? Too tired by the end of the day to do anything but collapse on the sofa and watch TV?

Learn how you can get a 3-day weekend, every single week.

09 May 2019 4:00am GMT

30 Apr 2019

feedPlanet Twisted

Itamar Turner-Trauring: The new parent's guide to surviving a programming job

Working as a programmer will keep you busy; parenting a baby is a massive amount of work. Doing both at once isn't easy!

While it does get better with time, there are ways you can make it calmer, simpler, and easier in the short term. Based on my experience as a new parent and programmer, in the rest of this article I'll cover:

  1. Mitigating sleep deprivation.
  2. Dealing with limited work hours.
  3. Other random tips.

Sleep deprivation is terrible

Sleep deprivation is awful. It makes you less focused, more irritable and cranky, and in many ways it's similar to being drunk. When done deliberately sleep deprivation is literally a form of torture.

If you're lucky your child will start sleeping through the night after a few months, but (from personal experience) not everyone is so lucky. So here are some ways to deal with lack of sleep.

Be kind

The irritability that results from sleep deprivation is going to impact all of your relationships-with your spouse/partner, your friends, and your coworkers. Keep in mind that you are going to get annoyed more easily, and try to compensate.

Remind yourself that the reason you're so annoyed by the code you're reviewing is probably nothing to do with your coworker's skill, and everything to do with being woken up at 1AM, 3AM, and finally at 5AM.

Compensate for cognitive impairment

Besides being irritable, you are also cognitively impaired-you're less smart than you usually are. You can compensate for this in a variety of ways:

Your time is limited

Even if you used to work longer hours (and you really shouldn't have), you really shouldn't be working long hours as a new parent. That means:

Other advice

Pumping milk at the office: Pumping milk multiple times a day at the office can be time consuming. If your baby is healthy, I'm told you can pump once in the morning, stick the equipment in the fridge without cleaning it, and then pump a second time later in day. This saves you one cleaning cycle at the office.

(I am not a medical professional, ask your pediatrician first before doing this.)

Working at home: Some babies, I'm told, will just lie there happily babbling to themselves while you work. If you have the other kind of baby, the kind that screams continuously if they're not held, you might be able to get a little work done at home by putting them in a baby carrier and using a standing desk.

A shorter workweek: Even if you're not working long hours, a full-time 40-hours-a-week job may still be too much as a new parent. You can often negotiate a shorter workweek at your existing job fairly easily.

What really matters to you?

However efficient you are, having a child is going to take up a whole lot of time. And that means you're going to have to make some choices about priorities: what things really matter you? Where do you really want to spend your time?

It's a personal choice-I am glad I got to work part-time and take care of my kid the rest of the time, but I would hate to take care of a baby full-time. Your preferences may well be different.

But whatever you decide, just remember you need to choose: you can't do everything.



Struggling with a 40-hour workweek? Too tired by the end of the day to do anything but collapse on the sofa and watch TV?

Learn how you can get a 3-day weekend, every single week.

30 Apr 2019 4:00am GMT

12 Apr 2019

feedPlanet Twisted

Itamar Turner-Trauring: Can software engineering be meaningful work?

The world is full of problems-from poverty to climate change-and it seems like software ought to be able to help. And yet your own programming job seems pointless, doing nothing to make things better. Far too many jobs are just about making some rich people just a little bit richer.

So how can you do something meaningful with your life?

There are no easy answers, but here's a starting point, at least.

Don't make things worse

Even beyond your moral obligations, working on something you actively find wrong is bad for you:

If you find yourself in this situation, you have the opportunity to try to make things a little better, by pushing your organization to change. But you can also just go look for another job elsewhere.

Some jobs are actually good

Of course, most software jobs aren't evil, but neither are they particularly meaningful. You can help an online store come up with a better recommendation engine, or optimize their marketing funnel, or build a web UI for support staff-but does it really matter that people buy at store A instead of store B?

So it's worth thinking in detail about what exactly it is you would find meaningful, and seeing if there's work that matches your criteria. There may not be a huge number of jobs that qualify, but chances are some exist.

If you care about climate change, for example, there are companies building alternative energy systems, working on public transportation planning, and more broadly just making computing more efficient.

Your job needn't be the center of your life

You may not be able to find such a job, or get such a job. So there's something to be said for not making your work the center of your life's existence.

As a programmer you are likely get paid well, and you can even negotiate a shorter workweek. Given enough free time and no worries about making a living, you have the ability to find meaning outside your work.

No easy answers

Unless you want to join a group that will tell you exactly what to think and precisely what to do-and there are certainly no lack of those-meaning is something you need to figure out for yourself.

It's unlikely that you'll solve it in one fell swoop, nor is it likely to be a fast process. The best you can do is just get started: a meaningful life isn't a destination, it's a journey.



Struggling with a 40-hour workweek? Too tired by the end of the day to do anything but collapse on the sofa and watch TV?

Learn how you can get a 3-day weekend, every single week.

12 Apr 2019 4:00am GMT

10 Apr 2019

feedPlanet Twisted

Twisted Matrix Laboratories: Twisted 19.2.0 Released

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 19.2! The highlights of this release are:

You can find the downloads at <https://pypi.python.org/pypi/Twisted> (or alternatively <http://twistedmatrix.com/trac/wiki/Downloads>). The NEWS file is also available at <https://github.com/twisted/twisted/blob/twisted-19.2.0/NEWS.rst>.

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,
Amber Brown (HawkOwl)

10 Apr 2019 12:35pm GMT

08 Apr 2019

feedPlanet Twisted

Moshe Zadka: Publishing a Book with Sphinx

A while ago, I decided I wanted to self-publish a book on improving your Python skills. It was supposed to be short, sweet, and fairly inexpensive.

The journey was a success, but had some interesting twists along the way.

From the beginning, I knew what technology I wanted to write the book with: Sphinx. This was because I knew that I can use Sphinx to create something reasonable: I have previously ported my "Calculus 101" book to Sphinx, and I have written other small things in it. Sphinx uses ReStructuredText, which I am most familiar with.

I decided I wanted to publish as PDF (for self-printers or others who find it convenient), as browser-ready HTML directory, and as an ePub.

The tox environments I created are: epub builds the ePub, html builds the browser-ready HTML, and pdf builds the PDF.

Initially, the epub environment created a "singlehtml", and I used Calibre command-line utility to transform it into an ePub. This made for a prettier ePub than the one sphinx creates: it had a much nicer cover, which is what most book reading applications use as an icon. However, that rendered poorly on Books.app (AKA iBooks).

One of the projects I still plan to tackle is how to improve the look of the rendered ePub, and add a custom cover image.

Finally, a script runs all the relevant tox environments, and then packs everything into a zip file. This is the zip file I upload to Gumroad, so that people can buy it.

I have tried to use other sellers, but Gumroad was the one with the easiest store creation. In order to test my store, even before the book was ready, I created a simple "Python cheat-sheet" poster, and put it on my store.

I then asked friends to buy it, as well as trying to do it myself. After it all worked, I refunded all the test-run purchases, of course!

Refunding on Gumroad is a pleasant process, which means that if people buy the book, and are unhappy with it, I am happy to refund their money.

(Thanks to Glyph Lefkowitz for his feedback on an earlier draft. All mistakes that remain are my responsibility.)

08 Apr 2019 7:00am GMT

03 Apr 2019

feedPlanet Twisted

Itamar Turner-Trauring: Setting boundaries at your job as a programmer

There's always another bug, another feature, another deadline. So it's easy to fall into the trap of taking on too much, saying "yes" one time too many, staying at the office a little later, answering a work email on the weekend…

If you're not careful you can end up setting unreasonable expectations, and ending up tethered to your work email and Slack. Your manager will expect you to work weekends, and your teammates will expect you to reply to bug reports in the middle of your vacation.

What you want is the opposite: when you're at home or on vacation, you should be spending your time however you want, merry and free.

You need to set boundaries, which is what I'll be discussing in the rest of this article.

Prepping for a new job

Imagine you're starting a new job in a week, you enjoy programming for fun, and you want to be productive as soon as possible. Personally, I wouldn't do any advance preparation for a new job: ongoing learning is part of a programmer's work, and employers ought to budget time for it. But you might choose differently.

If so, it's tempting to ask for some learning material so you can spend a few days beforehand getting up to speed. But you're failing to set boundaries if you do that, and they might give you company-specific material, in which case you're just doing work for free.

Learning general technologies is less of a problem-knowing more technologies is useful in your career in general, and maybe you enjoy programming for fun. So instead of asking for learning material, you can go on your own and learn the technologies you know they use, without telling them you're doing so.

Work email and Slack

Never set up work email or Slack on your phone or personal computer:

  1. It will tempt you to engage with work in your free time.
  2. When you do engage, you'll be setting expectations that you're available to answer questions 24/7.

While you're at work you'll always have your computer, so you don't need access on your phone. If you do need to set up work email on your phone for travel, remove the account when you're back home.

And if you want to have your work calendar on your phone, you can share it with your personal calendar account; that way you're sharing only your calendar, nothing else.

Vacations

When you're on vacation, you're on vacation: no work allowed. That means you're not taking your work laptop with you, or turning it on if you're at home.

A week or so in advance of your vacation, explain to your team that you won't be online, and that you won't have access to work files. Figure out what information they might need-documentation, in-progress work you want to hand off, and the like-and write it all down where they can find it.

If you must, give your personal phone number for emergencies: given you lack access to your work credentials and email, the chances of your being called for something unimportant are quite low.

You're paid for your normal work hours (and that's it)

A standard workweek in the US is 40 hours a week; elsewhere it can be a little less. Whatever it is, outside those hours you shouldn't be working, because you're not being paid for that work. Your evenings, your weekends, your holidays, your vacations-all of these belong to you, not your employer.

If you don't enforce that boundary between work and non-work, you are sending the message that your time doesn't belong to you. And if you have a bad manager, they're going to take advantage of that-or you might end up working long hours out of a misplaced sense of obligation.

So unless you're dealing with an emergency, you should forget your job exists when your workday ends-and unless you're on your on-call rotation, you should make sure you're inaccessible by normal work channels.



Struggling with a 40-hour workweek? Too tired by the end of the day to do anything but collapse on the sofa and watch TV?

Learn how you can get a 3-day weekend, every single week.

03 Apr 2019 4:00am GMT