One of my first TIL entries was about how you can imitate Node's node_modules semantics in Python on UNIX-like operating systems. A lot has happened since then (to the better!) and it's time for an update. direnv still rocks, though.

02 Apr 2024 12:00am GMT

The Cost of Coffee

One of the ideas that James Hoffmann - probably the most influential… influencer in the coffee industry - works hard to popularize is that "coffee needs to be more expensive".

The coffee industry is famously exploitative. Despite relatively thin margins for independent café owners¹, there are no shortage of horrific stories about labor exploitation and even slavery in the coffee supply chain.

To summarize a point that Mr. Hoffman has made over a quite long series of videos and interviews², some of this can be fixed by regulatory efforts. Enforcement of supply chain policies both by manufacturers and governments can help spot and avoid this type of exploitation. Some of it can be fixed by discernment on the part of consumers. You can try to buy fair-trade coffee, avoid brands that you know have problematic supply-chain histories.

Ultimately, though, even if there is perfect, universal, zero-cost enforcement of supply chain integrity… consumers still have to be willing to, you know, pay more for the coffee. It costs more to pay wages than to have slaves.

The Price of Software

The problem with the coffee supply chain deserves your attention in its own right. I don't mean to claim that the problems of open source maintainers are as severe as those of literal child slaves. But the principle is the same.

Every tech company uses huge amounts of open source software, which they get for free.

I do not want to argue that this is straightforwardly exploitation. There is a complex bargain here for the open source maintainers: if you create open source software, you can get a job more easily. If you create open source infrastructure, you can make choices about the architecture of your projects which are more long-term sustainable from a technology perspective, but would be harder to justify on a shorter-term commercial development schedule. You can collaborate with a wider group across the industry. You can build your personal brand.

But, in light of the recent xz Utils / SSH backdoor scandal, it is clear that while the bargain may not be entirely one-sided, it is not symmetrical, and significant bad consequences may result, both for the maintainers themselves and for society.

To fix this problem, open source software needs to get more expensive.

A big part of the appeal of open source is its implicit permission structure, which derives both from its zero up-front cost and its zero marginal cost.

The zero up-front cost means that you can just get it to try it out. In many companies, individual software developers do not have the authority to write a purchase order, or even a corporate credit card for small expenses.

If you are a software engineer and you need a new development tool or a new library that you want to purchase for work, it can be a maze of bureaucratic confusion in order to get that approved. It might be possible, but you are likely to get strange looks, and someone, probably your manager, is quite likely to say "isn't there a free option for this?" At worst, it might just be impossible.

This makes sense. Dealing with purchase orders and reimbursement requests is annoying, and it only feels worth the overhead if you're dealing with a large enough block of functionality that it is worth it for an entire team, or better yet an org, to adopt. This means that most of the purchasing is done by management types or "architects", who are empowered to make decisions for larger groups.

When individual engineers need to solve a problem, they look at open source libraries and tools specifically because it's quick and easy to incorporate them in a pull request, where a proprietary solution might be tedious and expensive.

That's assuming that a proprietary solution to your problem even exists. In the infrastructure sector of the software economy, free options from your operating system provider (Apple, Microsoft, maybe Amazon if you're in the cloud) and open source developers, small commercial options have been marginalized or outright destroyed by zero-cost options, for this reason.

If the zero up-front cost is a paperwork-reduction benefit, then the zero marginal cost is almost a requirement. One of the perennial complaints of open source maintainers is that companies take our stuff, build it into a product, and then make a zillion dollars and give us nothing. It seems fair that they'd give us some kind of royalty, right? Some tiny fraction of that windfall? But once you realize that individual developers don't have the authority to put $50 on a corporate card to buy a tool, they super don't have the authority to make a technical decision that encumbers the intellectual property of their entire core product to give some fraction of the company's revenue away to a third party. Structurally, there's no way that this will ever happen.

Despite these impediments, keeping those dependencies maintained does cost money.

Some Solutions Already Exist

There are various official channels developing to help support the maintenance of critical infrastructure. If you work at a big company, you should probably have a corporate Tidelift subscription. Maybe ask your employer about that.

But, as they will readily admit there are a LOT of projects that even Tidelift cannot cover, with no official commercial support, and no practical way to offer it in the short term. Individual maintainers, like yours truly, trying to figure out how to maintain their projects, either by making a living from them or incorporating them into our jobs somehow. People with a Ko-Fi or a Patreon, or maybe just an Amazon wish-list to let you say "thanks" for occasional maintenance work.

Most importantly, there's no path for them to transition to actually making a living from their maintenance work. For most maintainers, Tidelift pays a sub-hobbyist amount of money, and even setting it up (and GitHub Sponsors, etc) is a huge hassle. So even making the transition from "no income" to "a little bit of side-hustle income" may be prohibitively bureaucratic.

Let's take myself as an example. If you're a developer who is nominally profiting from my infrastructure work in your own career, there is a very strong chance that you are also a contributor to the open source commons, and perhaps you've even contributed more to that commons than I have, contributed more to my own career success than I have to yours. I can ask you to pay me³, but really you shouldn't be paying me, your employer should.

What To Do Now: Make It Easy To Just Pay Money

So if we just need to give open source maintainers more money, and it's really the employers who ought to be giving it, then what can we do?

Let's not make it complicated. Employers should just give maintainers money. Let's call it the "JGMM" benefit.

Specifically, every employer of software engineers should immediately institute the following benefits program: each software engineer should have a monthly discretionary budget of $50 to distribute to whatever open source dependency developers they want, in whatever way they see fit. Venmo, Patreon, PayPal, Kickstarter, GitHub Sponsors, whatever, it doesn't matter. Put it on a corp card, put the project name on the line item, and forget about it. It's only for open source maintenance, but it's a small enough amount that you don't need intense levels of approval-gating process. You can do it on the honor system.

This preserves zero up-front cost. To start using a dependency, you still just use it⁴. It also preserves zero marginal cost: your developers choose which dependencies to support based on perceived need and popularity. It's a fixed overhead which doesn't scale with revenue or with profit, just headcount.

Because the whole point here is to match the easy, implicit, no-process, no-controls way in which dependencies can be added in most companies. It should be easier to pay these small tips than it is to use the software in the first place.

This sub-1% overhead to your staffing costs will massively de-risk the open source projects you use. By leaving the discretion up to your engineers, you will end up supporting those projects which are really struggling and which your executives won't even hear about until they end up on the news. Some of it will go to projects that you don't use, things that your engineers find fascinating and want to use one day but don't yet depend upon, but that's fine too. Consider it an extremely cheap, efficient R&D expense.

A lot of the options for developers to support open source infrastructure are already tax-deductible, so if they contribute to something like one of the PSF's fiscal sponsorees, it's probably even more tax-advantaged than a regular business expense.

I also strongly suspect that if you're one of the first employers to do this, you can get a round of really positive PR out of the tech press, and attract engineers, so, the race is on. I don't really count as the "tech press" but nevertheless drop me a line to let me know if your company ends up doing this so I can shout you out.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor! I am also available for consulting work if you think your organization could benefit from expertise on topics such as "How do I figure out which open source projects to give money to?".

30 Mar 2024 11:00pm GMT

This year I will be going to two conferences: PyCon 2024, and North Bay Python 2024.

At PyCon, I will be promoting my open source work and my writing on this blog. As I'm not giving a talk this year, I am looking forward to organizing and participating in some open spaces about topics like software architecture, open source sustainability, framework interoperability, and event-driven programming.

At North Bay Python, though, I will either be:

1. doing a lot more of that, or
2. looking for new full-time employment, pausing the patreon, and winding down this experiment.

If you'd like to see me doing the former, now would be a great time to sign up to my Patreon to support the continuation of my independent open source work and writing.

The Bad News

It has been nearly a year since I first mentioned that I have a Patreon on this blog. That year has been a busy one, with consulting work and personal stuff consuming more than enough time that I have never been full time on independent coding & blogging. As such, I've focused more on small infrastructure projects and less on user-facing apps than I'd like, but I have spent the plurality of my time on it.

For that to continue, let alone increase, this work needs to-at the very least-pay for health insurance. At my current consulting rate, a conservative estimate based on some time tracking is that I am currently doing this work at something like a 98.5% discount. I do love doing it! I would be happy to continue doing it at a significant discount! But "significant" and "catastrophic" are different thresholds.

As I have said previously, this is an appeal to support my independent work; not to support me. I will be fine; what you will be voting for with your wallet is not my well-being but a choice about where I spend my time.

Hiding The Hat

When people ask me what I do these days, I sometimes struggle to explain. It is confusing. I might say I have a Patreon, I do a combination of independent work and consulting, or if I'm feeling particularly sarcastic I might say I'm an ✨influencer✨. But recently I saw the very inspiring Matt Ricardo describing the way he thinks about his own Patreon, and I realized what I am actually trying to do, which is software development busking.

Previously, when I've been thinking about making this "okay, it's been a year of Patreon, let's get serious now" post, I've been thinking about adding more reward products to my Patreon, trying to give people better value for their money before asking for anything more, trying to finish more projects to make a better sales pitch, maybe making merch available for sale, and so on. So aside from irregular weekly posts on Friday and acknowledgments sections at the bottom of blog posts, I've avoided mentioning this while I think about adding more private rewards.

But busking is a public performance, and if you want to support my work then it is that public aspect that you presumably want to support. And thus, an important part of the busking process is to actually pass the hat at the end. The people who don't chip in still get to see the performance, but everyone else need to know that they can contribute if they liked it.¹

I'm going to try to stop hiding the metaphorical hat. I still don't want to overdo it, but I will trust that you'll tell me if these reminders get annoying. For my part today, in addition to this post, I'm opening up a new $10 tier on Patreon for people who want to provide a higher level of support, and officially acknowledging the rewards that I already provide.

What's The Deal?

So, what would you be supporting?

What You Give (The Public Part)

1. I have tended to focus on my software, and there has been a lot of it! You'd be supporting me writing libraries and applications and build infrastructure to help others do the same with Python, as well as maintaining existing libraries (like the Twisted ecosystem libraries) sometimes. If I can get enough support together to more than bare support for myself, I'd really like to be able to do things like pay people to others to help with aspects of applications that I would struggle to complete myself, like accessibility or security audits.
2. I also do quite a bit of writing though, about software and about other things. To make it easier to see what sort of writing I'm talking about, I've collected just the stuff that I've written during the period where I have had some patrons, under the supported tag.
3. Per my earlier sarcastic comment about being an "influencer", I also do quite a bit of posting on Mastodon about software and the tech industry.

What You Get (Just For Patrons)

As I said above, I will be keeping member benefits somewhat minimal.

1. I will add you to SponCom so that your name will be embedded in commit messages like this one on a cadence appropriate to your support level.
2. You will get access to my private Patreon posts where I discuss what I've been working on. As one of my existing patrons put it:

   I figure I'm getting pretty good return on it, getting not only targeted project tracking, but also a peek inside your head when it comes to Sores Business Development. Maybe some of that stuff will rub off on me :)

3. This is a somewhat vague and noncommittal benefit, but it might be the best one: if you are interested in the various different projects that I am doing, you can help me prioritize! I have a lot of things going on. What would you prefer that I focus on? You can make suggestions in the comments of Patreon posts, which I pay a lot more attention to than other feedback channels.
4. In the near future² I am also planning to start doing some "office hours" type live-streaming, where I will take audience questions and discuss software design topics, or maybe do some live development to showcase my process and the tools I use. When I figure out the mechanics of this, I plan to add some rewards to the existing tiers to select topics or problems for me to work on there.

If that sounds like a good deal to you, please sign up now. If you're already supporting me, sharing this and giving a brief testimonial of something good I've done would be really helpful. Github is not an algorithmic platform like YouTube, despite my occasional jokey "remember to like and subscribe", nobody is getting recommended DBXS, or Fritter, or Twisted, or Automat, or this blog unless someone goes out and shares it.

29 Mar 2024 11:56pm GMT

New Release

Yesterday I published a new release of DBXS for you all. It's still ZeroVer, but it has graduated from double-ZeroVer as this is the first nonzero minor version.

More to the point though, the meaning of that version increment this version introduces some critical features that I think most people would need to give it a spin on a hobby project.

What's New

• It has support for MySQL and PostgreSQL using native asyncio drivers, which means you don't need to take a Twisted dependency in production.

• While Twisted is still used for some of the testing internals, Deferred is no longer exposed anywhere in the public API, either; your tests can happily pretend that they're doing asyncio, as long as they can run against SQLite.

• There is a new repository convenience function that automatically wires together multiple accessors and transaction discipline. Have a look at the docstring for a sense of how to use it.

• Several papercuts, like confusing error messages when messing up query result handling, and lack of proper handling of default arguments in access protocols, are now addressed.

It's A Good Time To Contribute!

If you've been looking for an open source project to try your hand at contributing to, DBXS might be a great opportunity, for a few reasons:

1. The team is quite small (just me, right now!), so it's easy to get involved.

2. It's quite generally useful, so there's a potential for an audience, but right now it doesn't really have any production users; there's still time to change things without a lot of ceremony.

3. Unlike many other small starter projects, it's got a test suite with 100% coverage, so you can contribute with confidence that you're not breaking anything.

4. There's not that much code (a bit over 2 thousand SLOC), so it's not hard to get your head around.

5. There are a few obvious next steps for improvement, which I've filed as issues if you want to pick one up.

Share and enjoy, and please let me know if you do something fun with it.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor! I am also available for consulting work if you think your organization could benefit from expertise on topics such as "How do I shot SQL?".

29 Mar 2024 10:19pm GMT

Contrary to popular belief, it's possible to ship portable executables of Python applications without sending your users to Python packaging hell.

19 Mar 2024 12:00am GMT

I do consulting¹ on software architecture, network protocol development, python software infrastructure, streamlined cloud deployment, and open source strategy, among other nerdy things. I enjoy solving challenging, complex technical problems or contributing to the open source commons. On the best jobs, I get to do both.

Today I would like to share with you a secret of the software technology consulting trade.

I should note that this secret is not specific to me. I have several colleagues who have also done software consulting and have reflected versions of this experience back to me.²

We'll get to the secret itself in a moment, but first, some background.

Companies do not go looking for consulting when things are going great. This is particularly true when looking for high-level consulting on things like system architecture or strategy. Almost by definition, there's a problem that I have been brought in to solve. Ideally, that problem is a technical challenge.

In the software industry, your team probably already has some software professionals with a variety of technical skills, and thus they know what to do with technical challenges. Which means that, as often as not, the problem is to do with people rather than technology, even it appears otherwise.

When you hire a staff-level professional like myself to address your software team's general problems, that consultant will need to gather some information. If I am that consultant and I start to suspect that the purported technology problem that you've got is in fact a people problem, here is the secret technique that I am going to use:

I am going to go get a pen and a pad of paper, then schedule a 90-minute meeting with the most senior IC³ engineer that you have on your team. I will bring that pen and paper to the meeting. I will then ask one question:

What is fucked up about this place?

I will then write down their response in as much detail as I can manage. If I have begun to suspect that this meeting is necessary, 90 minutes is typically not enough time, and I will struggle to keep up. Even so, I will usually manage to capture the highlights.

One week later, I will schedule a meeting with executive leadership, and during that meeting, I will read back a very lightly edited⁴ version of the transcript of the previous meeting. This is then routinely praised as a keen strategic insight.

I should pause here to explicitly note that - obviously, I hope - this is not an oblique reference to any current or even recent client; if I'd had this meeting recently it would be pretty awkward to answer that "so, I read your blog…" email.⁵ But talking about clients in this way, no matter how obfuscated and vague the description, is always a bit professionally risky. So why risk it?

The thing is, I'm not a people manager. While I can do this kind of work, and I do not begrudge doing it if it is the thing that needs doing, I find it stressful and unfulfilling. I am a technology guy, not a people person. This is generally true of people who elect to go into technology consulting; we know where the management track is, and we didn't pick it.

If you are going to hire me for my highly specialized technical expertise, I want you to get the maximum value out of it. I know my value; my rates are not low, and I do not want clients to come away with the sense that I only had a couple of "obvious" meetings.

So the intended audience for this piece is potential clients, leaders of teams (or organizations, or companies) who have a general technology problem and are wondering if they need a consultant with my skill-set to help them fix it. Before you decide that your issue is the need to implement a complex distributed system consensus algorithm, check if that is really what's at issue. Talk to your ICs, and - taking care to make sure they understand that you want honest feedback and that they are safe to offer it - ask them what problems your organization has.

During this meeting it is important to only listen. Especially if you're at a small company and you are regularly involved in the day-to-day operations, you might feel immediately defensive. Sit with that feeling, and process it later. Don't unload your emotional state on an employee you have power over.⁶

"Only listening" doesn't exclusively mean "don't push back". You also shouldn't be committing to fixing anything. While the information you are gathering in these meetings is extremely valuable, and you should probably act on more of it than you will initially want to, your ICs won't have the full picture. They really may not understand why certain priorities are set the way they are. You'll need to take that as feedback for improving internal comms rather than "fixing" the perceived problem, and you certainly don't want to make empty promises.

If you have these conversations directly, you can get something from it that no consultant can offer you: credibility. If you can actively listen, the conversation alone can improve morale. People like having their concerns heard. If, better still, you manage to make meaningful changes to address the concerns you've heard about, you can inspire true respect.

As a consultant, I'm going to be seen as some random guy wasting their time with a meeting. Even if you make the changes I recommend, it won't resonate the same way as someone remembering that they personally told you what was wrong, and you took it seriously and fixed it.

Once you know what the problems are with your organization, and you've got solid technical understanding that you really do need that event-driven distributed systems consensus algorithm implemented using Twisted, I'm absolutely your guy. Feel free to get in touch.

05 Feb 2024 10:36pm GMT

Today is the 40th anniversary of the announcement of the Macintosh. Others have articulated compelling emotional narratives that easily eclipse my own similar childhood memories of the Macintosh family of computers. So instead, I will ask a question:

What is the Macintosh?

As this is the anniversary of the beginning, that is where I will begin. The original Macintosh, the classic MacOS, the original "System Software" are a shining example of "fake it till you make it". The original mac operating system was fake.

Don't get me wrong, it was an impressive technical achievement to fake something like this, but what Steve Jobs did was to see a demo of a Smalltalk-76 system, an object-oriented programming environment with 1-to-1 correspondences between graphical objects on screen and runtime-introspectable data structures, a self-hosting high level programming language, memory safety, message passing, garbage collection, and many other advanced facilities that would not be popularized for decades, and make a fake version of it which ran on hardware that consumers could actually afford, by throwing out most of what made the programming environment interesting and replacing it with a much more memory-efficient illusion implemented in 68000 assembler and Pascal.

The machine's RAM didn't have room for a kernel. Whatever application was running was in control of the whole system. No protected memory, no preemptive multitasking. It was a house of cards that was destined to collapse. And collapse it did, both in the short term and the long. In the short term, the system was buggy and unstable, and application crashes resulted in system halts and reboots.

In the longer term, the company based on the Macintosh effectively went out of business and was reverse-acquired by NeXT, but they kept the better-known branding of the older company. The old operating system was gradually disposed of, quickly replaced at its core with a significantly more mature generation of operating system technology based on BSD UNIX and Mach. With the removal of Carbon compatibility 4 years ago, the last vestigial traces of it were removed. But even as early as 2004 the Mac was no longer really the Macintosh.

What NeXT had built was much closer to the Smalltalk system that Jobs was originally attempting to emulate. Its programming language, "Objective C" explicitly called back to Smalltalk's message-passing, right down to the syntax. Objects on the screen now did correspond to "objects" you could send messages to. The development environment understood this too; that was a major selling point.

The NeXSTEP operating system and Objective C runtime did not have garbage collection, but it provided a similar developer experience by providing reference-counting throughout its object model. The original vision was finally achieved, for real, and that's what we have on our desks and in our backpacks today (and in our pockets, in the form of the iPhone, which is in some sense a tiny next-generation NeXT computer itself).

The one detail I will relate from my own childhood is this: my first computer was not a Mac. My first computer, as a child, was an Amiga. When I was 5, I had a computer with 4096 colors, real multitasking, 3D graphics, and a paint program that could draw hard real-time animations with palette tricks. Then the writing was on the wall for Commodore and I got a computer which had 256 colors, a bunch of old software that was still black and white, an operating system that would freeze if you held down the mouse button on the menu bar and couldn't even play animations smoothly. Many will relay their first encounter with the Mac as a kind of magic, but mine was a feeling of loss and disappointment. Unlike almost everyone at the time, I knew what a computer really could be, and despite many pleasant and formative experiences with the Macintosh in the meanwhile, it would be a decade before I saw a real one again.

But this is not to deride the faking. The faking was necessary. Xerox was not going to put an Alto running Smalltalk on anyone's desk. People have always grumbled that Apple products are expensive, but in 2024 dollars, one of these Xerox computers cost roughly $55,000.

The Amiga was, in its own way, a similar sort of fake. It managed its own miracles by putting performance-critical functions into dedicated hardware which rapidly became obsolete as software technology evolved much more rapidly.

Jobs is celebrated as a genius of product design, and he certainly wasn't bad at it, but I had the rare privilege of seeing the homework he was cribbing from in that subject, and in my estimation he was a B student at best. Where he got an A was bringing a vision to life by creating an organization, both inside and outside of his companies.

If you want a culture-defining technological artifact, everybody in the culture has to be able to get their hands on one. This doesn't just mean that the builder has to be able to build it. The buyer also has to be able to afford it, obviously. Developers have to be able to develop for it. The buyer has to actually want it; the much-derided "marketing" is a necessary part of the process of making a product what it is. Everyone needs to be able to move together in the direction of the same technological future.

This is why it was so fitting that Tim Cook was made Jobs's successor. The supply chain was the hard part.

The crowning, final achievement of Jobs's career was the fact that not only did he fake it - the fakes were flying fast and thick at that time in history, even if they mostly weren't as good - it was that he faked it and then he built the real version and then he bridged the transitions to get to the real thing.

I began here by saying that the Mac isn't really the Mac, and speaking in terms of a point in time analysis that is true. Its technology today has practically nothing in common with its technology in 1984. This is not merely an artifact of the length of time here: the technology at the core of various UNIXes in 1984 bears a lot of resemblance of UNIX-like operating systems today¹. But looking across its whole history from 1984 to 2024, there is undeniably a continuity to the conceptual "Macintosh".

Not just as a user, but as a developer moving through time rather than looking at just a few points: the "Macintosh", such as it is, has transitioned from the Motorola 68000 to the PowerPC to Intel 32-bit to Intel 64-bit to ARM. From obscurely proprietary to enthusiastically embracing open source and then, sadly, much of the way back again. It moved from black and white to color, from desktop to laptop, from Carbon to Cocoa, from Display PostScript to Display PDF, all the while preserving instantly recognizable iconic features like the apple menu and the cursor pointer, while providing developers documentation and SDKs and training sessions that helped them transition their apps through multiple near-complete rewrites as a result of all of these changes.

To paraphrase Abigail Thorne's first video about Identity, identity is what survives. The Macintosh is an interesting case study in the survival of the idea of a platform, as distinct from the platform itself. It is the Computer of Theseus, a thought experiment successfully brought to life and sustained over time.

If there is a personal lesson to be learned here, I'd say it's that one's own efforts need not be perfect. In fact, a significantly flawed vision that you can achieve right now is often much, much better than a perfect version that might take just a little bit longer, if you don't have the resources to actually sustain going that much longer². You have to be bad at things before you can be good at them. Real artists, as Jobs famously put it, ship.

So my contribution to the 40th anniversary reflections is to say: the Macintosh is dead. Long live the Mac.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!

25 Jan 2024 6:31am GMT

I am going to tell you why I don't think you should sign your Git commits, even though doing so with SSH keys is now easier than ever. But first, to contextualize my objection, I have a brief hypothetical for you, and then a bit of history from the evolution of security on the web.

It seems like these days, everybody's signing all different kinds of papers.

Bank forms, permission slips, power of attorney; it seems like if you want to securely validate a document, you've gotta sign it.

So I have invented a machine that automatically signs every document on your desk, just in case it needs your signature. Signing is good for security, so you should probably get one, and turn it on, just in case something needs your signature on it.

We also want to make sure that verifying your signature is easy, so we will have them all notarized and duplicates stored permanently and publicly for future reference.

No? Not interested?

Hopefully, that sounded like a silly idea to you.

Most adults in modern civilization have learned that signing your name to a document has an effect. It is not merely decorative; the words in the document being signed have some specific meaning and can be enforced against you.

In some ways the metaphor of "signing" in cryptography is bad. One does not "sign" things with "keys" in real life. But here, it is spot on: a cryptographic signature can have an effect.

It should be an input to some software, one that is acted upon. Software does a thing differently depending on the presence or absence of a signature. If it doesn't, the signature probably shouldn't be there.

Consider the most venerable example of encryption and signing that we all deal with every day: HTTPS. Many years ago, browsers would happily display unencrypted web pages. The browser would also encrypt the connection, if the server operator had paid for an expensive certificate and correctly configured their server. If that operator messed up the encryption, it would pop up a helpful dialog box that would tell the user "This website did something wrong that you cannot possibly understand. Would you like to ignore this and keep working?" with buttons that said "Yes" and "No".

Of course, these are not the precise words that were written. The words, as written, said things about "information you exchange" and "security certificate" and "certifying authorities" but "Yes" and "No" were the words that most users read. Predictably, most users just clicked "Yes".

In the usual case, where users ignored these warnings, it meant that no user ever got meaningful security from HTTPS. It was a component of the web stack that did nothing but funnel money into the pockets of certificate authorities and occasionally present annoying interruptions to users.

In the case where the user carefully read and honored these warnings in the spirit they were intended, adding any sort of transport security to your website was a potential liability. If you got everything perfectly correct, nothing happened except the browser would display a picture of a small green purse. If you made any small mistake, it would scare users off and thereby directly harm your business. You would only want to do it if you were doing something that put a big enough target on your site that you became unusually interesting to attackers, or were required to do so by some contractual obligation like credit card companies.

Keep in mind that the second case here is the best case.

In 2016, the browser makers noticed this problem and started taking some pretty aggressive steps towards actually enforcing the security that HTTPS was supposed to provide, by fixing the user interface to do the right thing. If your site didn't have security, it would be shown as "Not Secure", a subtle warning that would gradually escalate in intensity as time went on, correctly incentivizing site operators to adopt transport security certificates. On the user interface side, certificate errors would be significantly harder to disregard, making it so that users who didn't understand what they were seeing would actually be stopped from doing the dangerous thing.

Nothing fundamental¹ changed about the technical aspects of the cryptographic primitives or constructions being used by HTTPS in this time period, but socially, the meaning of an HTTP server signing and encrypting its requests changed a lot.

Now, let's consider signing Git commits.

You may have heard that in some abstract sense you "should" be signing your commits. GitHub puts a little green "verified" badge next to commits that are signed, which is neat, I guess. They provide "security". 1Password provides a nice UI for setting it up. If you're not a 1Password user, GitHub itself recommends you put in just a few lines of configuration to do it with either a GPG, SSH, or even an S/MIME key.

But while GitHub's documentation quite lucidly tells you how to sign your commits, its explanation of why is somewhat less clear. Their purse is the word "Verified"; it's still green. If you enable "vigilant mode", you can make the blank "no verification status" option say "Unverified", but not much else changes.

This is like the old-style HTTPS verification "Yes"/"No" dialog, except that there is not even an interruption to your workflow. They might put the "Unverified" status on there, but they've gone ahead and clicked "Yes" for you.

It is tempting to think that the "HTTPS" metaphor will map neatly onto Git commit signatures. It was bad when the web wasn't using HTTPS, and the next step in that process was for Let's Encrypt to come along and for the browsers to fix their implementations. Getting your certificates properly set up in the meanwhile and becoming familiar with the tools for properly doing HTTPS was unambiguously a good thing for an engineer to do. I did, and I'm quite glad I did so!

However, there is a significant difference: signing and encrypting an HTTPS request is ephemeral; signing a Git commit is functionally permanent.

This ephemeral nature meant that errors in the early HTTPS landscape were easily fixable. Earlier I mentioned that there was a time where you might not want to set up HTTPS on your production web servers, because any small screw-up would break your site and thereby your business. But if you were really skilled and you could see the future coming, you could set up monitoring, avoid these mistakes, and rapidly recover. These mistakes didn't need to badly break your site.

We can extend the analogy to HTTPS, but we have to take a detour into one of the more unpleasant mistakes in HTTPS's history: HTTP Public Key Pinning, or "HPKP". The idea with HPKP was that you could publish a record in an HTTP header where your site commits² to using certain certificate authorities for a period of time, where that period of time could be "forever". Attackers gonna attack, and attack they did. Even without getting attacked, a site could easily commit "HPKP Suicide" where they would pin the wrong certificate authority with a long timeline, and their site was effectively gone for every browser that had ever seen those pins. As a result, after a few years, HPKP was completely removed from all browsers.

Git commit signing is even worse. With HPKP, you could easily make terrible mistakes with permanent consequences even though you knew the exact meaning of the data you were putting into the system at the time you were doing it. With signed commits, you are saying something permanently, but you don't really know what it is that you're saying.

Today, what is the benefit of signing a Git commit? GitHub might present it as "Verified". It's worth noting that only GitHub will do this, since they are the root of trust for this signing scheme. So, by signing commits and registering your keys with GitHub, you are, at best, helping to lock in GitHub as a permanent piece of infrastructure that is even harder to dislodge because they are not only where your code is stored, but also the arbiters of whether or not it is trustworthy.

In the future, what is the possible security benefit? If we all collectively decide we want Git to be more secure, then we will need to meaningfully treat signed commits differently from unsigned ones.

There's a long tail of unsigned commits several billion entries long. And those are in the permanent record as much as the signed ones are, so future tooling will have to be able to deal with them. If, as stewards of Git, we wish to move towards a more secure Git, as the stewards of the web moved towards a more secure web, we do not have the option that the web did. In the browser, the meaning of a plain-text HTTP or incorrectly-signed HTTPS site changed, in order to encourage the site's operator to change the site to be HTTPS.

In contrast, the meaning of an unsigned commit cannot change, because there are zillions of unsigned commits lying around in critical infrastructure and we need them to remain there. Commits cannot meaningfully be changed to become signed retroactively. Unlike an online website, they are part of a historical record, not an operating program. So we cannot establish the difference in treatment by changing how unsigned commits are treated.

That means that tooling maintainers will need to provide some difference in behavior that provides some incentive. With HTTPS where the binary choice was clear: don't present sites with incorrect, potentially compromised configurations to users. The question was just how to achieve that. With Git commits, the difference in treatment of a "trusted" commit is far less clear.

If you will forgive me a slight straw-man here, one possible naive interpretation is that a "trusted" signed commit is that it's OK to run in CI. Conveniently, it's not simply "trusted" in a general sense. If you signed it, it's trusted to be from you, specifically. Surely it's fine if we bill the CI costs for validating the PR that includes that signed commit to your GitHub account?

Now, someone can piggy-back off a 1-line typo fix that you made on top of an unsigned commit to some large repo, making you implicitly responsible for transitively signing all unsigned parent commits, even though you haven't looked at any of the code.

Remember, also, that the only central authority that is practically trustable at this point is your GitHub account. That means that if you are using a third-party CI system, even if you're using a third-party Git host, you can only run "trusted" code if GitHub is online and responding to requests for its "get me the trusted signing keys for this user" API. This also adds a lot of value to a GitHub credential breach, strongly motivating attackers to sneakily attach their own keys to your account so that their commits in unrelated repos can be "Verified" by you.

Let's review the pros and cons of turning on commit signing now, before you know what it is going to be used for:

I feel like the "Con" column is coming out ahead.

That probably seemed like increasingly unhinged hyperbole, and it was.

In reality, the consequences are unlikely to be nearly so dramatic. The status quo has a very high amount of inertia, and probably the "Verified" badge will remain the only visible difference, except for a few repo-specific esoteric workflows, like pushing trust verification into offline or sandboxed build systems. I do still think that there is some potential for nefariousness around the "unknown and unlimited" dimension of any future plans that might rely on verifying signed commits, but any flaws are likely to be subtle attack chains and not anything flashy and obvious.

But I think that one of the biggest problems in information security is a lack of threat modeling. We encrypt things, we sign things, we institute rotation policies and elaborate useless rules for passwords, because we are looking for a "best practice" that is going to save us from having to think about what our actual security problems are.

I think the actual harm of signing git commits is to perpetuate an engineering culture of unquestioningly cargo-culting sophisticated and complex tools like cryptographic signatures into new contexts where they have no use.

Just from a baseline utilitarian philosophical perspective, for a given action A, all else being equal, it's always better not to do A, because taking an action always has some non-zero opportunity cost even if it is just the time taken to do it. Epsilon cost and zero benefit is still a net harm. This is even more true in the context of a complex system. Any action taken in response to a rule in a system is going to interact with all the other rules in that system. You have to pay complexity-rent on every new rule. So an apparently-useless embellishment like signing commits can have potentially far-reaching consequences in the future.

Git commit signing itself is not particularly consequential. I have probably spent more time writing this blog post than the sum total of all the time wasted by all programmers configuring their git clients to add useless signatures; even the relatively modest readership of this blog will likely transfer more data reading this post than all those signatures will take to transmit to the various git clients that will read them. If I just convince you not to sign your commits, I don't think I'm coming out ahead in the felicific calculus here.

What I am actually trying to point out here is that it is useful to carefully consider how to avoid adding junk complexity to your systems. One area where junk tends to leak in to designs and to cultures particularly easily is in intimidating subjects like trust and safety, where it is easy to get anxious and convince ourselves that piling on more stuff is safer than leaving things simple.

If I can help you avoid adding even a little bit of unnecessary complexity, I think it will have been well worth the cost of the writing, and the reading.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor! I am also available for consulting work if you think your organization could benefit from expertise on topics such as "What else should I not apply a cryptographic signature to?".

25 Jan 2024 12:29am GMT

In 2015, I wrote one of my more popular blog posts, "Your Text Editor Is Malware", about the sorry state of security in text editors in general, but particularly in Emacs and Vim.

It's nearly been a decade now, so I thought I'd take a moment to survey the world of editor plugins and see where we are today. Mostly, this is to allay fears, since (in today's landscape) that post is unreasonably alarmist and inaccurate, but people are still reading it.

The big takeaway here is that the main issue of there being no security whatsoever on Emacs and Vim package installation and update has been fully corrected.

Where To Go Next?

Since I believe that post was fairly influential, in particular in getting MELPA to tighten up its security, let me take another big swing at a call to action here.

More modern editors have made greater strides towards security. VSCode, for example, has enabled the Chromium sandbox and added some level of process separation. Emacs has not done much here yet, but over the years it has consistently surprised me with its ability to catch up to its more modern competitors, so I hope it will surprise me here as well.

Even for VSCode, though, this sandbox still seems pretty permissive - plugins still seem to execute with the full trust of the editor itself - but it's a big step in the right direction. This is a much bigger task than just turning on HTTPS, but I really hope that editors start taking the threat of rogue editor packages seriously before attackers do, and finding ways to sandbox and limit the potential damage from third-party plugins, maybe taking a cue from other tools.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!

23 Jan 2024 2:05am GMT

Today I saw a short YouTube video about "cozy games" and started writing a comment, then realized that this was somehow prompting me to write the most succinct summary of my own personal views on politics and economics that I have ever managed. So, here goes.

Apparently all I needed to trim down 50,000 words on my annoyance at how the term "capitalism" is frustratingly both a nexus for useful critque and also reductive thought-terminating clichés was to realize that Animal Crossing: New Horizons is closer to my views on political economy than anything Adam Smith or Karl Marx ever wrote.

Cozy games illustrate that the core mechanics of capitalism are fun and motivating, in a laboratory environment. It's fun to gather resources, to improve one's skills, to engage in mutually beneficial exchanges, to collect things, to decorate. It's tremendously motivating. Even merely pretending to do those things can captivate huge amounts of our time and attention.

In real life, people need to be motivated to do stuff. Not because of some moral deficiency, but because in a large complex civilization it's hard to tell what needs doing. By the time it's widely visible to a population-level democratic consensus of non-experts that there is an unmet need - for example, trash piling up on the street everywhere indicating a need for garbage collection - that doesn't mean "time to pick up some trash", it means "the sanitation system has collapsed, you're probably going to get cholera". We need a system that can identify utility signals more granularly and quickly, towards the edges of the social graph. To allow person A to earn "value credits" of some kind for doing work that others find valuable, then trade those in to person B for labor which they find valuable, even if it is not clearly obvious to anyone else why person A wants that thing. Hence: money.

So, a market can provide an incentive structure that productively steers people towards needs, by aggregating small price signals in a distributed way, via the communication technology of "money". Authoritarian communist states are famously bad at this, overproducing "necessary" goods in ways that can hold their own with the worst excesses of capitalists, while under-producing "luxury" goods that are politically seen as frivolous.

This is the kernel of truth around which the hardcore capitalist bootstrap grindset ideologues build their fabulist cinematic universe of cruelty. Markets are motivating, they reason, therefore we must worship the market as a god and obey its every whim. Markets can optimize some targets, therefore we must allow markets to optimize every target. Markets efficiently allocate resources, and people need resources to live, therefore anyone unable to secure resources in a market is undeserving of life. Thus we begin at "market economies provide some beneficial efficiencies" and after just a bit of hand-waving over some inconvenient details, we get to "thus, we must make the poor into a blood-sacrifice to Moloch, otherwise nobody will ever work, and we will all die, drowning in our own laziness". "The cruelty is the point" is a convenient phrase, but among those with this worldview, the prosperity is the point; they just think the cruelty is the only engine that can possibly drive it.

Cozy games are therefore a centrist¹ critique of capitalism. They present a world with the prosperity, but without the cruelty. More importantly though, by virtue of the fact that people actually play them in large numbers, they demonstrate that the cruelty is actually unnecessary.

You don't need to play a cozy game. Tom Nook is not going to evict you from your real-life house if you don't give him enough bells when it's time to make rent. In fact, quite the opposite: you have to take time away from your real-life responsibilities and work, in order to make time for such a game. That is how motivating it is to engage with a market system in the abstract, with almost exclusively positive reinforcement.

What cozy games are showing us is that a world with tons of "free stuff" - universal basic income, universal health care, free education, free housing - will not result in a breakdown of our society because "no one wants to work". People love to work.

If we can turn the market into a cozy game, with low stakes and a generous safety net, more people will engage with it, not fewer. People are not lazy; laziness does not exist. The motivation that people need from a market economy is not a constant looming threat of homelessness, starvation and death for themselves and their children, but a fun opportunity to get a five-star island rating.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!

22 Jan 2024 5:41pm GMT

Codecov's unreliability breaking CI on my open source projects has been a constant source of frustration for me for years. I have found a way to enforce coverage over a whole GitHub Actions build matrix that doesn't rely on third-party services.

02 Jan 2024 12:00am GMT

When contributing to other users' repositories, always start a new branch in your fork.

21 Dec 2023 5:00am GMT

PEP 0593 added the ability to add arbitrary user-defined metadata to type annotations in Python.

At type-check time, such annotations are… inert. They don't do anything. Annotated[int, X] just means int to the type-checker, regardless of the value of X. So the entire purpose of Annotated is to provide a run-time API to consume metadata, which integrates with the type checker syntactically, but does not otherwise disturb it.

Yet, the documentation for this central purpose seems, while not exactly absent, oddly incomplete.

The PEP itself simply says:

A tool or library encountering an Annotated type can scan through the annotations to determine if they are of interest (e.g., using isinstance()).

But it's not clear where "the annotations" are, given that the PEP's entire "consuming annotations" section does not even mention the __metadata__ attribute where the annotation's arguments go, which was only even added to CPython's documentation. Its list of examples just show the repr() of the relevant type.

There's also a bit of an open question of what, exactly, we are supposed to isinstance()-ing here. If we want to find arguments to Annotated, presumably we need to be able to detect if an annotation is an Annotated. But isinstance(Annotated[int, "hello"], Annotated) is both False at runtime, and also a type-checking error, that looks like this:

1

Argument 2 to "isinstance" has incompatible type "<typing special form>"; expected "_ClassInfo"

The actual type of these objects, typing._AnnotatedAlias, does not seem to have a publicly available or documented alias, so that seems like the wrong route too.

Now, it certainly works to escape-hatch your way out of all of this with an Any, build some version-specific special-case hacks to dig around in the relevant namespaces, access __metadata__ and call it a day. But this solution is … unsatisfying.

What are you looking for?

Upon encountering these quirks, it is understandable to want to simply ask the question "is this annotation that I'm looking at an Annotated?" and to be frustrated that it seems so obscure to straightforwardly get an answer to that question without disabling all type-checking in your meta-programming code.

However, I think that this is a slight misframing of the problem. Code that is inspecting parameters for an annotation is going to do something with that annotation, which means that it must necessarily be looking for a specific set of annotations. Therefore the thing we want to pass to isinstance is not some obscure part of the annotations' internals, but the actual interesting annotation type from your framework or application.

When consuming an Annotated parameter, there are 3 things you probably want to know:

1. What was the parameter itself? (type: The type you passed in.)
2. What was the name of the annotated object (i.e.: the parameter name, the attribute name) being passed the parameter? (type: str)
3. What was the actual type being annotated? (type: type)

And the things that we have are the type of the Annotated we're querying for, and the object with annotations we are interrogating. So that gives us this function signature:

1
2
3
4
5

def annotated_by(
    annotated: object,
    kind: type[T],
) -> Iterable[tuple[str, T, type]]:
    ...

To extract this information, all we need are get_args and get_type_hints; no need for __metadata__ or get_origin or any other metaprogramming. Here's a recipe:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

def annotated_by(
    annotated: object,
    kind: type[T],
) -> Iterable[tuple[str, T, type]]:
    for k, v in get_type_hints(annotated, include_extras=True).items():
        all_args = get_args(v)
        if not all_args:
            continue
        actual, *rest = all_args
        for arg in rest:
            if isinstance(arg, kind):
                yield k, arg, actual

It might seem a little odd to be blindly assuming that get_args(...)[0] will always be the relevant type, when that is not true of unions or generics. Note, however, that we are only yielding results when we have found the instance type in the argument list; our arbitrary user-defined instance isn't valid as a type annotation argument in any other context. It can't be part of a Union or a Generic, so we can rely on it to be an Annotated argument, and from there, we can make that assumption about the format of get_args(...).

This can give us back the annotations that we're looking for in a handy format that's easy to consume. Here's a quick example of how you might use it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

@dataclass
class AnAnnotation:
    name: str

def a_function(
    a: str,
    b: Annotated[int, AnAnnotation("b")],
    c: Annotated[float, AnAnnotation("c")],
) -> None:
    ...

print(list(annotated_by(a_function, AnAnnotation)))

# [('b', AnAnnotation(name='b'), <class 'int'>),
#  ('c', AnAnnotation(name='c'), <class 'float'>)]

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor! I am also available for consulting work if you think your organization could benefit from expertise on topics like "how do I do Python metaprogramming, but, like, not super janky".

08 Dec 2023 1:56am GMT

Facebook - and by extension, most of Silicon Valley - rightly gets a lot of shit for its old motto, "Move Fast and Break Things".

As a general principle for living your life, it is obviously terrible advice, and it leads to a lot of the horrific outcomes of Facebook's business.

I don't want to be an apologist for Facebook. I also do not want to excuse the worldview that leads to those kinds of outcomes. However, I do want to try to help laypeople understand how software engineers-particularly those situated at the point in history where this motto became popular-actually meant by it. I would like more people in the general public to understand why, to engineers, it was supposed to mean roughly the same thing as Facebook's newer, goofier-sounding "Move fast with stable infrastructure".

Move Slow

In the bad old days, circa 2005, two worlds within the software industry were colliding.

The old world was the world of integrated hardware/software companies, like IBM and Apple, and shrink-wrapped software companies like Microsoft and WordPerfect. The new world was software-as-a-service companies like Google, and, yes, Facebook.

In the old world, you delivered software in a physical, shrink-wrapped box, on a yearly release cycle. If you were really aggressive you might ship updates as often as quarterly, but faster than that and your physical shipping infrastructure would not be able to keep pace with new versions. As such, development could proceed in long phases based on those schedules.

In practice what this meant was that in the old world, when development began on a new version, programmers would go absolutely wild adding incredibly buggy, experimental code to see what sorts of things might be possible in a new version, then slowly transition to less coding and more testing, eventually settling into a testing and bug-fixing mode in the last few months before the release.

This is where the idea of "alpha" (development testing) and "beta" (user testing) versions came from. Software in that initial surge of unstable development was extremely likely to malfunction or even crash. Everyone understood that. How could it be otherwise? In an alpha test, the engineers hadn't even started bug-fixing yet!

In the new world, the idea of a 6-month-long "beta test" was incoherent. If your software was a website, you shipped it to users every time they hit "refresh". The software was running 24/7, on hardware that you controlled. You could be adding features at every minute of every day. And, now that this was possible, you needed to be adding those features, or your users would get bored and leave for your competitors, who would do it.

But this came along with a new attitude towards quality and reliability. If you needed to ship a feature within 24 hours, you couldn't write a buggy version that crashed all the time, see how your carefully-selected group of users used it, collect crash reports, fix all the bugs, have a feature-freeze and do nothing but fix bugs for a few months. You needed to be able to ship a stable version of your software on Monday and then have another stable version on Tuesday.

To support this novel sort of development workflow, the industry developed new technologies. I am tempted to tell you about them all. Unit testing, continuous integration servers, error telemetry, system monitoring dashboards, feature flags... this is where a lot of my personal expertise lies. I was very much on the front lines of the "new world" in this conflict, trying to move companies to shorter and shorter development cycles, and move away from the legacy worldview of Big Release Day engineering.

Old habits die hard, though. Most engineers at this point were trained in a world where they had months of continuous quality assurance processes after writing their first rough draft. Such engineers feel understandably nervous about being required to ship their probably-buggy code to paying customers every day. So they would try to slow things down.

Of course, when one is deploying all the time, all other things being equal, it's easy to ship a show-stopping bug to customers. Organizations would do this, and they'd get burned. And when they'd get burned, they would introduce Processes to slow things down. Some of these would look like:

1. Let's keep a special version of our code set aside for testing, and then we'll test that for a few weeks before sending it to users.
2. The heads of every department need to sign-off on every deployed version, so everyone needs to spend a day writing up an explanation of their changes.
3. QA should sign off too, so let's have an extensive sign-off process where each individual tester does a fills out a sign-off form.

Then there's my favorite version of this pattern, where management decides that deploys are inherently dangerous, and everyone should probably just stop doing them. It typically proceeds in stages:

1. Let's have a deploy freeze, and not deploy on Fridays; don't want to mess up the weekend debugging an outage.
2. Actually, let's extend that freeze for all of December, we don't want to mess up the holiday shopping season.
3. Actually why not have the freeze extend into the end of November? Don't want to mess with Thanksgiving and the Black Friday weekend.
4. Some of our customers are in India, and Diwali's also a big deal. Why not extend the freeze from the end of October?
5. But, come to think of it, we do a fair amount of seasonal sales for Halloween too. How about no deployments from October 10 onward?
6. You know what, sometimes people like to use our shop for Valentine's day too. Let's just never deploy again.

This same anti-pattern can repeat itself with an endlessly proliferating list of "environments", whose main role ends up being to ensure that no code ever makes it to actual users.

… and break things anyway

As you may have begun to suspect, there are a few problems with this style of software development.

Even back in the bad old days of the 90s when you had to ship disks in boxes, this methodology contained within itself the seeds of its own destruction. As Joel Spolsky memorably put it, Microsoft discovered that this idea that you could introduce a ton of bugs and then just fix them later came along with some massive disadvantages:

The very first version of Microsoft Word for Windows was considered a "death march" project. It took forever. It kept slipping. The whole team was working ridiculous hours, the project was delayed again, and again, and again, and the stress was incredible. [...] The story goes that one programmer, who had to write the code to calculate the height of a line of text, simply wrote "return 12;" and waited for the bug report to come in [...]. The schedule was merely a checklist of features waiting to be turned into bugs. In the post-mortem, this was referred to as "infinite defects methodology".

Which lead them to what is perhaps the most ironclad law of software engineering:

In general, the longer you wait before fixing a bug, the costlier (in time and money) it is to fix.

A corollary to this is that the longer you wait to discover a bug, the costlier it is to fix.

Some bugs can be found by code review. So you should do code review. Some bugs can be found by automated tests. So you should do automated testing. Some bugs will be found by monitoring dashboards, so you should have monitoring dashboards.

So why not move fast?

But here is where Facebook's old motto comes in to play. All of those principles above are true, but here are two more things that are true:

1. No matter how much code review, automated testing, and monitoring you have some bugs can only be found by users interacting with your software.
2. No bugs can be found merely by slowing down and putting the deploy off another day.

Once you have made the process of releasing software to users sufficiently safe that the potential damage of any given deployment can be reliably limited, it is always best to release your changes to users as quickly as possible.

More importantly, as an engineer, you will naturally have an inherent fear of breaking things. If you make no changes, you cannot be blamed for whatever goes wrong. Particularly if you grew up in the Old World, there is an ever-present temptation to slow down, to avoid shipping, to hold back your changes, just in case.

You will want to move slow, to avoid breaking things. Better to do nothing, to be useless, than to do harm.

For all its faults as an organization, Facebook did, and does, have some excellent infrastructure to avoid breaking their software systems in response to features being deployed to production. In that sense, they'd already done the work to avoid the "harm" of an individual engineer's changes. If future work needed to be performed to increase safety, then that work should be done by the infrastructure team to make things safer, not by every other engineer slowing down.

The problem is that slowing down is not actually value neutral. To quote myself here:

If you can't ship a feature, you can't fix a bug.

When you slow down just for the sake of slowing down, you create more problems.

The first problem that you create is smashing together far too many changes at once.

You've got a development team. Every engineer on that team is adding features at some rate. You want them to be doing that work. Necessarily, they're all integrating them into the codebase to be deployed whenever the next deployment happens.

If a problem occurs with one of those changes, and you want to quickly know which change caused that problem, ideally you want to compare two versions of the software with the smallest number of changes possible between them. Ideally, every individual change would be released on its own, so you can see differences in behavior between versions which contain one change each, not a gigantic avalanche of changes where any one of hundred different features might be the culprit.

If you slow down for the sake of slowing down, you also create a process that cannot respond to failures of the existing code.

I've been writing thus far as if a system in a steady state is inherently fine, and each change carries the possibility of benefit but also the risk of failure. This is not always true. Changes don't just occur in your software. They can happen in the world as well, and your software needs to be able to respond to them.

Back to that holiday shopping season example from earlier: if your deploy freeze prevents all deployments during the holiday season to prevent breakages, what happens when your small but growing e-commerce site encounters a catastrophic bug that has always been there, but only occurs when you have more than 10,000 concurrent users. The breakage is coming from new, never before seen levels of traffic. The breakage is coming from your success, not your code. You'd better be able to ship a fix for that bug real fast, because your only other option to a fast turn-around bug-fix is shutting down the site entirely.

And if you see this failure for the first time on Black Friday, that is not the moment where you want to suddenly develop a new process for deploying on Friday. The only way to ensure that shipping that fix is easy is to ensure that shipping any fix is easy. That it's a thing your whole team does quickly, all the time.

The motto "Move Fast And Break Things" caught on with a lot of the rest of Silicon Valley because we are all familiar with this toxic, paralyzing fear.

After we have the safety mechanisms in place to make changes as safe as they can be, we just need to push through it, and accept that things might break, but that's OK.

Some Important Words are Missing

The motto has an implicit preamble, "Once you have done the work to make broken things safe enough, then you should move fast and break things".

When you are in a conflict about whether to "go fast" or "go slow", the motto is not supposed to be telling you that the answer is an unqualified "GOTTA GO FAST". Rather, it is an exhortation to take a beat and to go through a process of interrogating your motivation for slowing down. There are three possible things that a person saying "slow down" could mean about making a change:

1. It is broken in a way you already understand. If this is the problem, then you should not make the change, because you know it's not ready. If you already know it's broken, then the change simply isn't done. Finish the work, and ship it to users when it's finished.
2. It is risky in a way that you don't have a way to defend against. As far as you know, the change works, but there's a risk embedded in it that you don't have any safety tools to deal with. If this is the issue, then what you should do is pause working on this change, and build the safety first.
3. It is making you nervous in a way you can't articulate. If you can't describe an known defect as in point 1, and you can't outline an improved safety control as in step 2, then this is the time to let go, accept that you might break something, and move fast.

The implied context for "move fast and break things" is only in that third condition. If you've already built all the infrastructure that you can think of to build, and you've already fixed all the bugs in the change that you need to fix, any further delay will not serve you, do not have any further delays.

Unfortunately, as you probably already know,

This motto did a lot of good in its appropriate context, at its appropriate time. It's still a useful heuristic for engineers, if the appropriate context is generally understood within the conversation where it is used.

However, it has clearly been taken to mean a lot of significantly more damaging things.

Purely from an engineering perspective, it has been reasonably successful. It's less and less common to see people in the industry pushing back against tight deployment cycles. It's also less common to see the basic safety mechanisms (version control, continuous integration, unit testing) get ignored. And many ex-Facebook engineers have used this motto very clearly under the understanding I've described here.

Even in the narrow domain of software engineering it is misused. I've seen it used to argue a project didn't need tests; that a deploy could be forced through a safety process; that users did not need to be informed of a change that could potentially impact them personally.

Outside that domain, it's far worse. It's generally understood to mean that no safety mechanisms are required at all, that any change a software company wants to make is inherently justified because it's OK to "move fast". You can see this interpretation in the way that it has leaked out of Facebook's engineering culture and suffused its entire management strategy, blundering through market after market and issue after issue, making catastrophic mistakes, making a perfunctory apology and moving on to the next massive harm.

In the decade since it has been retired as Facebook's official motto, it has been used to defend some truly horrific abuses within the tech industry. You only need to visit the orange website to see it still being used this way.

Even at its best, "move fast and break things" is an engineering heuristic, it is not an ethical principle. Even within the context I've described, it's only okay to move fast and break things. It is never okay to move fast and harm people.

So, while I do think that it is broadly misunderstood by the public, it's still not a thing I'd ever say again. Instead, I propose this:

Make it safer, don't make it later.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor! I am also available for consulting work if you think your organization could benefit from expertise on topics like "how do I make changes to my codebase, but, like, good ones".

06 Dec 2023 8:01pm GMT

First Cabal code contribution

introduction

A few weeks ago Shae Erisson and I took a look at Cabal to try to find a small contribution we could make. And we managed it! Here's the story.

First, an introduction. Cabal is a system for building and packaging Haskell libraries and programs. A few years ago I started learning Haskell in earnest. After a little time spent with stack, a similar tool, I've more recently switched to using Cabal and have gotten a great deal of value from it.

I did contribute a trivial documentation improvement earlier this year but before the experience related here I hadn't looked at the code.

Shae and I had a hunch that we might find some good beginner tasks in the Cabal issue tracker so we started there.

I had previously noticed that the Cabal project uses a label for supposedly "easy" issues. Shae also pointed out a group of labels related to "new comers". We found what appeared to be a tiered system with issues group by the anticipated size of the change. For example, there is a label for "one line change" issues and another for "small feature" and a couple more in between. We looked at the "one line change" issues first but the correct one line change was not obvious to us. These issues have a lot of discussion on them and we didn't want to try to figure out how to resolve the open questions. We decided to look at a more substantial issue and hope that there would be less uncertainty around them.

The candidate we found, Use ProjectFlags in CmdClean, was labeled with "one file change" and looked promising. It didn't have a description (presumably the reporter felt the summary made the task sufficiently obvious) but there was a promising hint from another contributor and a link to a very similar change made in another part of the codebase.

dev env

While we were looking around, Shae was getting a development environment set up. The Cabal project has a nice contributor guideline linked from the end of their README that we found without too much trouble. Since Shae and I both use Nix, Shae took the Nix-oriented route and got a development environment using:

nix develop github:yvan-sraka/cabal.nix

(I'll replicate the disclaimer from the contributor that this is "supported solely by the community".)

This took around 30 minutes to download and build (and Shae has pretty fast hardware). A nice improvement here might be to set up a Nix build cache for this development shell for everyone who wants to do Nix-based development on Cabal to share. I expect this would take the first-run setup time from 30 minutes to just one or two at most (depending mostly on download bandwidth available).

By the time we felt like we had a handle on what code changes to make and had found some of the pieces in a checkout of the project the development environment was (almost) ready. The change itself was quite straightforward.

the task

Our objective was to remove some implementation duplication in support of the --project-dir and --project-file options. Several cabal commands accept these options and some of them re-implement them where they could instead be sharing code.

We took the existing data type representing the options for the cabal clean command:

data CleanFlags = CleanFlags
  { cleanSaveConfig :: Flag Bool
  , cleanVerbosity :: Flag Verbosity
  , cleanDistDir :: Flag FilePath
  , cleanProjectDir :: Flag FilePath
  , cleanProjectFile :: Flag FilePath
  }
  deriving (Eq)

And removed the final two fields which exactly overlap with the fields defined by ProjectFlags. Then we changed the type of cleanCommand. It was:

cleanCommand :: CommandUI CleanFlags

And now it is:

cleanCommand :: CommandUI (ProjectFlags, CleanFlags)

Finally, we updated nearby code to reflect the type change. liftOptionL made this straightforward since it let us lift the existing operations on CleanFlags or ProjectFlags into the tuple.

One minor wrinkle we encountered is that ProjectFlags isn't a perfect fit for the clean command. Specifically, the former includes support for requesting that a cabal.project file be ignored but this option doesn't make sense for the clean command. This was easy to handle because someone had also already noticed and written a helper to remove the extra option where it is not wanted. However, this might suggest that the factoring of the types could be further improved so that it's easier to add only the appropriate options.

At this point we thought we were done. Cabal CI proved we were mistaken. The details of the failure were a little tricky to track down (as is often the case for many projects, alas). Eventually we dug the following error out of the failing job's logs:

Error: cabal: option `--project-dir' is ambiguous; could be one of:
  --project-dir=DIR Set the path of the project directory
  --project-dir=DIR Set the path of the project directory

Then we dug the test command out of the GitHub Actions configuration to try to reproduce the failure locally. Unfortunately, when we ran the command we got far more test failures than CI had. We decided to build cabal and test the case manually. This went smoothly enough and we were quickly able to find the problem and manually confirm the fix (which was simply to remove the field for --project-dir in a place we missed). GitHub Actions rewarded us for this change with a green checkmark.

the rest

At this point we waited for a code review on the PR. We quickly received two and addressed the minor point made (we had used NamedFieldPuns, of which I am a fan, but the module already used RecordWildCards and it's a bit silly to use both of these so close together).

Afterwards, the PR was approved and we added the "squash+merge me" label to trigger some automation to merge the changes (which it did after a short delay to give other contributors a chance to chime in if they wished).

In the end the change was +41 / -42 lines. For a refactoring that reduced duplication this isn't quite as much of a reduction as I was hoping for. The explanation ends up being that importing the necessary types and helper functions almost exactly balanced out the deleted duplicate fields. Considering the very small scope of this refactoring this ends up not being very surprising.

Thanks to Shae for working on this with me, to the Cabal maintainers who reviewed our PR, to all the Cabal contributors who made such a great tool, and to the "Cabal and Hackage" room on matrix.org for answering random questions.

20 Nov 2023 12:00am GMT

Good News Everyone

I'm turning my blog back on! I know you're so excited.

I'm going to try to gather up my older writing and host it here. Also, I promise to try to consider making this a little prettier.

Meanwhile, check out my new post about contributing to Cabal!

19 Nov 2023 12:00am GMT