11 Feb 2016

feedPlanet Twisted

Moshe Zadka: Learn Public Speaking: The Heart-Attack Way

I know there are all kinds of organizations, and classes, that teach public speaking. But if you want to learn how to do public speaking the way I did, it's pretty easy. Get a bunch of friends together, who are all interested in improving their skills. Meet once a week.

The rules are this: one person gets to give a talk with no more props than a marker and a whiteboard. If it's boring, or they hesitate, another person can butt in, say "boring" and give a talk about something else. Talks are to be prepared somewhat in advance, and to be about five minutes long.

This is scary. You stand there, at the board, never knowing when someone will decide you're boring. You try to entertain the audience. You try to keep their interest. You lose your point for 10 seconds…and you're out. It's the fight club of public speaking. After a few weeks of this, you'll give flowing talks, and nothing will ever scare you again about public speaking - you'll laugh as you realize that nobody is going to kick you out.

Yeah, Israel was a good training ground for speaking.

11 Feb 2016 4:52pm GMT

10 Feb 2016

feedPlanet Twisted

Glyph Lefkowitz

This is an experiment with a subtly different format.

Right now when I want to say something quickly, I pop open the Twitter app and just type it. But I realized that the only reason I'm doing this rather than publishing on my own site is a UI affordance: Twitter lets me hit two keys to start composing, ⌘N and then two keys to finish, ⌘Return. Also, to tweet something, I don't need to come up with a title.

So I added an Emacs minor-mode that lets me hit a comparable number of keys; translated into the Emacs keyboard shortcut idiom it is of course Meta-Shift-Control-Hyper-C Backflip-LeftPedal-N to create a post and Bucky-Reverse-Erase-x Omega-Shift-Epsilon-j to publish it. Such posts will be distinguished by the presence of the "microblog" tag and the empty title.

(Also, the sky's the limit in terms of character-count.)

Feel free to let me know if you think the format works or not.

10 Feb 2016 4:29am GMT

Hynek Schlawack: hasattr() – A Dangerous Misnomer

Don't use Python's hasattr() unless you're writing Python 3-only code and understand how it works.

10 Feb 2016 12:00am GMT

05 Feb 2016

feedPlanet Twisted

Twisted Matrix Laboratories: Jan '16 SFC Sponsored Development (HawkOwl)

Hi everyone!

The Twisted Fellowship, for this time, has come to an end, and as such, here is my final report.

Tickets reviewed/merged:


Tickets triaged:

Tickets worked on:

Even though we didn't get the Git migration done in time for the end of the fellowship, I am happy to report that it is in a much closer and much better known state than before. If you would like to assist in getting some of the work done above reviewed and merged, drop by https://github.com/twisted-infra/braid/pulls !

- Amber

05 Feb 2016 4:37pm GMT

02 Feb 2016

feedPlanet Twisted

Twisted Matrix Laboratories: January 2016 - SFC Sponsored Development

This is my report for the work done in January 2016 as part of the Twisted Maintainer Fellowship program.
It is my last report of the Twisted Maintainer Fellowship 2015 program.

With this fellowship the review queue size was reduced and the review round-trips were done much quicker.
This fellowship has produced the Git/GitHub migration plan but has failed to finalize its execution.

Tickets reviewed and merged

* #7671 - It is way too hard to specify a trust root combining multiple certificates, especially to HTTP
* #7993 - Port twisted.web.wsgi to Python 3
* #8140 - twisted.web.test.requesthelper.DummyRequest is out of sync with the real Request
* #8148 - Deprecate twisted.protocols.mice
* #8173 - Twisted does not have a code of conduct
* #8180 - Conch integration tests fail because DSA is deprecated in OpenSSH 7.
* #8187 - Use a less ancient OpenSSL method in twisted.test.test_sslverify

Tickets reviewed and not merged yet

* #7889 - replace win32api.OpenProcess and win32api.FormatMessage with cffi
* #8150 - twisted.internet.ssl.KeyPair should provide loadPEM
* #8159 - twisted.internet._win32serialport incompatible with pyserial 3.x
* #8169 - t.w.static.addSlash does not work on Python 3
* #8188 - Advertise H2 via ALPN/NPN when available.

Thanks to the Software Freedom Conservancy and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.

02 Feb 2016 1:18pm GMT

01 Feb 2016

feedPlanet Twisted

Moshe Zadka: Docker: Are we there yet?

Obviously, we know the answer. This blog is intended to allow me to have an easy place to point people when they ask me "so what's wrong with Docker"?

[To clarify, I use Docker myself, and it is pretty neat. All the more reason missing features annoy me.]

Docker itself:

Tooling:

01 Feb 2016 5:24am GMT

27 Jan 2016

feedPlanet Twisted

Moshe Zadka: Learning Python: The ecosystem

When first learning Python, the tutorial is a pretty good resource to get acquainted with the language and, to some extent, the standard library. I have written before about how to release open source libraries - but it is quite possible that one's first foray into Python will not be to write a reusable library, but an application to accomplish something - maybe a web application with Django or a tool to send commands to your TV. Much of what I said there will not apply - no need for a README.rst if you are writing a Django app for your personal website!

However, it probably is useful to learn a few tools that the Python eco-system engineered to make life more pleasant. In a perfect world, those would be built-in to Python: the "cargo" to Python's "Rust". However, in the world we live in, we must cobble together a good tool-chain from various open source projects. So strap in, and let's begin!

The first three are cribbed from my "open source" link above, because good code hygiene is always important.

Testing

There are several reasonably good test runners. If there is no clear reason to choose one, py.test is a good default. "Using Twisted" is a good reason to choose trial. Using coverage is a no-brainer. It is good to run some functional tests too. Test runners should be able to help with this too, but even writing a Python program that fails if things are not working can be useful.

Static checking

There are a lot of tools for static checking of Python programs - pylint, flake8 and more. Use at least one. Using more is not completely free (more ways to have to say "ignore this, this is ok") but can be useful to catch more style static issue. At worst, if there are local conventions that are not easily plugged into these checkers, write a Python program that will check for them and fail if those are violated.

Meta testing

Use tox. Put tox.ini at the root of your project, and make sure that "tox" (with no arguments) works and runs your entire test-suite. All unit tests, functional tests and static checks should be run using tox.

Set tox to put all build artifacts in a build/ top-level directory.

Pex

A tox test-environment of "pex" should result in a Python EXectuable created and put somewhere under "build/". Running your Python application to actually serve web pages should be as simple as taking that pex and running it without arguments. BoredBot shows an example of how to create such a pex that includes a web application, a Twisted application and a simple loop/sleep-based application. This pex build can take a requirements.txt file with exact dependencies, though it if it is built by tox, you can inline those dependencies directly in the tox file.

Collaboration

If you do collaborate with others on the project, whether it is open source or not, it is best if the collaboration instructions are as easy as possible. Ideally, collaboration instructions should be no more complicated than "clone this, make changes, run tox, possibly do whatever manual verification using './build/my-thing.pex' and submit a pull request".

If they are, consider investing some effort into changing the code to be more self-sufficient and make less assumptions about its environment. For example, default to a local SQLite-based database if no "-database" option is specified, and initialize it with whatever your code needs. This will also make it easier to practices the "infinite environment" methodology, since if one file is all it takes to "bring up" an environment, it should be easy enough to run it on a cloud server and allow people to look at it.

27 Jan 2016 5:45am GMT

17 Jan 2016

feedPlanet Twisted

Glyph Lefkowitz: Stop Working So Hard

Recently, I saw this tweet where John Carmack posted to a thread on Hacker News about working hours. As this post propagated a good many bad ideas about working hours, particularly in the software industry, I of course had to reply. After some further back-and-forth on Twitter, Carmack followed up.

First off, thanks to Mr. Carmack for writing such a thorough reply in good faith. I suppose internet arguments have made me a bit cynical in that I didn't expect that. I still definitely don't agree, but I think there's a legitimate analysis of the available evidence there now, at least.

When trying to post this reply to HN, I was told that the comment was too long, and I suppose it is a bit long for a comment. So, without further ado, here are my further thoughts on working hours.

... if only the workers in Greece would ease up a bit, they would get the productivity of Germany. Would you make that statement?

Not as such, no. This is a hugely complex situation mixing together finance, culture, management, international politics, monetary policy, and a bunch of other things. That study, and most of the others I linked to, is interesting in that it confirms the general model of ability-to-work (i.e. "concentration" or "willpower") as a finite resource that you exhaust throughout the day; not in that "reduction in working hours" is a panacea solution. Average productivity-per-hour-worked would definitely go up.

However, I do believe (and now we are firmly off into interpretation-of-results territory, I have nothing empirical to offer you here) that if the average Greek worker were less stressed to the degree of the average German one, combining issues like both overwork and the presence of a constant catastrophic financial crisis in the news, yes; they'd achieve equivalent productivity.

Total net productivity per worker, discounting for any increases in errors and negative side effects, continues increasing well past 40 hours per week. ... Only when you are so broken down that even when you come back the following day your productivity per hour is significantly impaired, do you open up the possibility of actually reducing your net output.

The trouble here is that you really cannot discount for errors and negative side effects, especially in the long term.

First of all, the effects of overwork (and attendant problems, like sleep deprivation) are cumulative. While productivity on a given day increases past 40 hours per week, if you continue to work more, you productivity will continue to degrade. So, the case where "you come back the following day ... impaired" is pretty common... eventually.

Since none of this epidemiological work tracks individual performance longitudinally there are few conclusive demonstrations of this fact, but lots of compelling indications; in the past, I've collected quantitative data on myself (and my reports, back when I used to be a manager) that strongly corroborates this hypothesis. So encouraging someone to work one sixty-hour week might be a completely reasonable trade-off to address a deadline; but building a culture where asking someone to work nights and weekends as a matter of course is inherently destructive. Once you get into the area where people are losing sleep (and for people with other responsibilities, it's not hard to get to that point) overwork starts impacting stuff like the ability to form long-term memories, which means that not only do you do less work, you also consistently improve less.

Furthermore, errors and negative side effects can have a disproportionate impact.

Let me narrow the field here to two professions I know a bit about and are germane to this discussion; one, health care, which the original article here starts off by referencing, and two, software development, with which we are both familiar (since you already raised the Mythical Man Month).

In medicine, you can do a lot of valuable life-saving work in a continuous 100-hour shift. And in fact residents are often required to do so as a sort of professional hazing ritual. However, you can also make catastrophic mistakes that would cost a person their life; this happens routinely. Study after study confirms this, and minor reforms happen, but residents are still routinely abused and made to work in inhumane conditions that have catastrophic outcomes for their patients.

In software, defects can be extremely expensive to fix. Not only are they hard to fix, they can also be hard to detect. The phenomenon of the Net Negative Producing Programmer also indicates that not only can productivity drop to zero, it can drop below zero. On the anecdotal side, anyone who has had the unfortunate experience of cleaning up after a burnt-out co-worker can attest to this.

There are a great many tasks where inefficiency grows significantly with additional workers involved; the Mythical Man Month problem is real. In cases like these, you are better off with a smaller team of harder working people, even if their productivity-per-hour is somewhat lower.

The specific observation from the Mythical Man Month was that the number of communication links on a fully connected graph of employees increases geometrically whereas additional productivity (in the form of additional workers) increases linearly. If you have a well-designed organization, you can add people without requiring that your communication graph be fully connected.

But of course, you can't always do that. And specifically you can't do that when a project is already late: you already figured out how the work is going to be divided. Brooks' Law is formulated as: "Adding manpower to a late software project makes it later." This is indubitable. But one of the other famous quotes from this book is "The bearing of a child takes nine months, no matter how many women are assigned."

The bearing of a child also takes nine months no matter how many hours a day the woman is assigned to work on it. So "in cases like these" my contention is that you are not "better off with ... harder working people": you're just screwed. Some projects are impossible and you are better off acknowledging the fact that you made unrealistic estimates and you are going to fail.

You called my post "so wrong, and so potentially destructive", which leads me to believe that you hold an ideological position that the world would be better if people didn't work as long. I don't actually have a particularly strong position there; my point is purely about the effective output of an individual.

I do, in fact, hold such an ideological position, but I'd like to think that said position is strongly justified by the data available to me.

But, I suppose calling it "so potentially destructive" might have seemed glib, if you are really just looking at the microcosm of what one individual might do on one given week at work, and not at the broader cultural implications of this commentary. After all, as this discussion shows, if you are really restricting your commentary to a single person on a single work-week, the case is substantially more ambiguous. So let me explain why I believe it's harmful, as opposed to merely being incorrect.

First of all, the problem is that you can't actually ignore the broader cultural implications. This is Hacker News, and you are John Carmack; you are practically a cultural institution yourself, and by using this site you are posting directly into the broader cultural implications of the software industry.

Software development culture, especially in the USA, suffers from a long-standing culture of chronic overwork. Startup developers in their metaphorical (and sometimes literal) garages are lionized and then eventually mythologized for spending so many hours on their programs. Anywhere that it is celebrated, this mythology rapidly metastasizes into a severe problem; the Death March

Note that although the term "death march" is technically general to any project management, it applies "originally and especially in software development", because this problem is worse in the software industry (although it has been improving in recent years) than almost anywhere else.

So when John Carmack says on Hacker News that "the effective output of an individual" will tend to increase with hours worked, that sends a message to many young and impressionable software developers. This is the exact same phenomenon that makes pop-sci writing terrible: your statement may be, in some limited context, and under some tight constraints, empirically correct, but it doesn't matter because when you expand the parameters to the full spectrum of these people's careers, it's both totally false and also a reinforcement of an existing cognitive bias and cultural trope.

I can't remember the name of this cognitive bias (and my Google-fu is failing me), but I know it exists. Let me call it the "I'm fine" bias. I know it exists because I have a friend who had the opportunity to go on a flight with NASA (on the Vomit Comet), and one of the more memorable parts of this experience that he related to me was the hypoxia test. The test involved basic math and spatial reasoning skills, but that test wasn't the point: the real test was that they had to notice and indicate when the oxygen levels were dropping and indicate that to the proctor. Concentrating on the test, many people failed the first few times, because the "I'm fine" bias makes it very hard to notice that you are impaired.

This is true of people who are drunk, or people who are sleep deprived, too. Their abilities are quantifiably impaired, but they have to reach a pretty severe level of impairment before they notice.

So people who are overworked might feel generally bad but they don't notice their productivity dropping until they're way over the red line.

Combine this with the fact that most people, especially those already employed as developers, are actually quite hard-working and earnest (laziness is much more common as a rhetorical device than as an actual personality flaw) and you end up in a scenario where a good software development manager is responsible much more for telling people to slow down, to take breaks, and to be more realistic in their estimates, than to speed up, work harder, and put in more hours.

The trouble is this goes against the manager's instincts as well. When you're a manager you tend to think of things in terms of resources: hours worked, money to hire people, and so on. So there's a constant nagging sensation for a manager to encourage people to work more hours in a day, so you can get more output (hours worked) out of your input (hiring budget). The problem here is that while all hours are equal, some hours are more equal than others. Managers have to fight against their own sense that a few more worked hours will be fine, and their employees' tendency to overwork because they're not noticing their own burnout, and upper management's tendency to demand more.

It is into this roiling stew of the relentless impulse to "work, work, work" that we are throwing our commentary about whether it's a good idea or not to work more hours in the week. The scales are weighted very heavily on one side already - which happens to be the wrong side in the first place - and while we've come back from the unethical and illegal brink we were at as an industry in the days of ea_spouse, software developers still generally work far too much.

If we were fighting an existential threat, say an asteroid that would hit the earth in a year, would you really tell everyone involved in the project that they should go home after 35 hours a week, because they are harming the project if they work longer?

Going back to my earlier explanation in this post about the cumulative impact of stress and sleep deprivation - if we were really fighting an existential threat, the equation changes somewhat. Specifically, the part of the equation where people can have meaningful downtime.

In such a situation, I would still want to make sure that people are as well-rested and as reasonably able to focus as they possibly can be. As you've already acknowledged, there are "increases in errors" when people are working too much, and we REALLY don't want the asteroid-targeting program that is going to blow apart the asteroid that will wipe out all life on earth to have "increased errors".

But there's also the problem that, faced with such an existential crisis, nobody is really going to be able to go home and enjoy a fine craft beer and spend some time playing with their kids and come back refreshed at 100% the next morning. They're going to be freaking out constantly about the comet, they're going to be losing sleep over that whether they're working or not. So, in such a situation, people should have the option to go home and relax if they're psychologically capable of doing so, but if the option for spending their time that makes them feel the most sane is working constantly and sleeping under their desk, well, that's the best one can do in that situation.

This metaphor is itself also misleading and out of place, though. There is also a strong cultural trend in software, especially in the startup ecosystem, to over-inflate the importance of what the company is doing - it is not "changing the world" to create a website for people to order room-service for their dogs - and thereby to catastrophize any threat to that goal. The vast majority of the time, it is inappropriate to either to sacrifice -- or to ask someone else to sacrifice -- health and well-being for short-term gains. Remember, given the cumulative effects of overwork, that's all you even can get: short-term gains. This sacrifice often has a huge opportunity cost in other areas, as you can't focus on more important things that might come along later.

In other words, while the overtime situation is complex and delicate in the case of an impending asteroid impact, there's also the question of whether, at the beginning of Project Blow Up The Asteroid, I want everyone to be burnt out and overworked from their pet-hotel startup website. And in that case, I can say, unequivocally, no. I want them bright-eyed and bushy-tailed for what is sure to be a grueling project, no matter what the overtime policy is, that absolutely needs to happen. I want to make sure they didn't waste their youth and health on somebody else's stock valuation.

17 Jan 2016 4:06am GMT

09 Jan 2016

feedPlanet Twisted

Twisted Matrix Laboratories: December 2015 second half - SFC Sponsored Development

This is my report for the work done in the second half of December as part of the 2015 Twisted Maintainer Fellowship program.

Important changes made in these weeks

* The Git migration plan was approved.

Git Migration tasks

* Update and add new tickets in twisted-infra/braid for the Git migration plan
* twisted-infra/braid #164- Plan SVN account migration

Tickets reviewed and merged

* #8129 - twisted.web.http_headers.Headers should work on bytes and Unicode
* #8138 - twisted.conch.endpoints._NewConnectionHelper leaks agent connection
* #8145 - Remove the deprecated gtkreactor
* #8154 - Update doc references to new logger

Tickets reviewed and not merged yet

* #7671 - it is way too hard to specify a trust root combining multiple certificates, especially to HTTP
* #7993 - 'twisted.web.wsgi' should be ported to Python 3
* #8025 - Make Trial work on Windows+Python3
* #8143- HTTP/2 Part 1: New IRequest interface
* #8160 - twisted.conch.ssh.agent is missing some request and response types

Thanks to the Software Freedom Conservancy and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.

09 Jan 2016 12:44pm GMT

06 Jan 2016

feedPlanet Twisted

Hynek Schlawack: Storing Passwords in a Highly Parallelized World

Why "Use bcrypt." is not the best recommendation (anymore).

06 Jan 2016 12:00am GMT

05 Jan 2016

feedPlanet Twisted

Glyph Lefkowitz: Taking Issue With Paul Graham’s Premises

Paul Graham has recently penned an essay on income inequality. Holly Wood wrote a pretty good critique of this piece, but it is addressing a huge amount of pre-existing context, as well as ignoring large chunks of the essay that nominally agree.

Eevee has already addressed how the "simplified version" didn't substantively change anything that the longer one was saying, so I'm not going to touch on that much. However, it's worth noting that the reason Paul Graham says he wrote that is that he thinks that "adventurous interpretations" are to blame for criticism of his "controversial" writing; in other words, that people are misinterpreting his argument because the conclusion is politically unacceptable.

Personally, I am deeply ambivalent about the political implications of his writing. I believe, strongly, in the power of markets to solve social problems that planning cannot. I don't think "capitalist" is a slur. But, neither do I believe that markets are inherently good; capitalist economic theory assumes an environment of equal initial opportunity which demonstrably does not exist. I am, personally, very open to ideas like the counter-intuitive suggestion that economic inequality might not be such a bad thing, if the case were well-made. I say this because I want to be clear that what bothers me about Paul Graham's writing is not its "controversial" content.

What bothers me about Paul Graham's writing is that the reasoning is desperately sloppy. I sometimes mentor students on their writing, and if this mess were handed to me by one of my mentees, I would tell them to rewrite it from scratch. Although the "thanks" section at the end of each post on his blog implies that he gets editing feedback, it must be so uncritical of his assumptions as to be useless.

Initially, my entirely subjective impression is that Paul Graham is not a credible authority on the topic of income inequality. He doesn't demonstrate any grasp of its causes, or indeed the substance of any proposed remedy. I would say that he is attacking a straw-man, but he doesn't even bother to assemble the straw-man first, saying only:

... the thing that strikes me most about the conversations I overhear ...

What are these "conversations" he "overhears"? What remedies are they proposing which would target income inequality by eliminating any possible reward for entrepreneurship? Nothing he's arguing against sounds like anything I've ever heard someone propose, and I spend a lot of time in the sort of conversation that he imagines overhearing.

His claim to credentials in this area doesn't logically follow, either:

I've become an expert on how to increase economic inequality, and I've spent the past decade working hard to do it. ... In the real world you can create wealth as well as taking it from others.

This whole passage is intended to read logically as: "I increase economic inequality, which you might assume is bad, but it's not so bad, because it creates wealth in the process!".

Hopefully what PG has actually been trying to become an expert in is creating wealth (a net positive), not in increasing economic inequality (a negative, or, at best, neutral by-product of that process). If he is focused on creating wealth, as so much of the essay purports he is, then it does not necessarily follow that the startup founders will be getting richer than their customers.

Of course, many goods and services provide purely subjective utility to their consumers. But in a properly functioning market, the whole point of of engaging in transactions is to improve efficiency.

To borrow from PG's woodworker metaphor:

A woodworker creates wealth. He makes a chair, and you willingly give him money in return for it.

I might be buying that chair to simply appreciate its chair-ness and bask in the sublime beauty of its potential for being sat-in. But equally likely, I'm buying that chair for my office, where I will sit in it, and produce some value of my own while thusly seated. If the woodworker hadn't created that chair for me, I'd have to do it myself, and it (presumably) would have been more expensive in terms of time and resources. Therefore, by producing the chair more efficiently, the woodworker would have increased my wealth as well as his own, by increasing the delta between my expenses (which include the chair) and my revenue (generated by tripping the light pythonic or whatever).

Note that "more efficient" doesn't necessarily mean "lower cost". Sitting in a chair is a substantial portion of my professional activity. A higher-quality chair that costs the same amount might improve the quality of my sitting experience, which might improve my own productivity at writing code, allowing me to make more income myself.

Even if the value of the chair is purely subjective, it is still an expense, and making it more efficient to make chairs would still increase my net worth.

Therefore, if startups really generated wealth so reliably, rather than simply providing a vehicle for transferring it, we would expect to see decreases in economic inequality, as everyone was able to make the most efficient use of their own time and resources, and was able to make commensurately more money.

... variation in productivity is accelerating ...

Counterpoint: no it isn't. It's not even clear that it's increasing, let alone that its derivative is increasing. This doesn't appear to be something that much data is collected on, and in the absence of any citation, I have to assume that it is a restatement of the not only false, but harmful, frequently debunked 10x programmer myth.

Most people who get rich tend to be fairly driven.

This sounds obvious: of course, if you "get" rich, you have to be doing something to "get" that way. First of all, this ignores many people who simply are rich, who get their wealth from inheritance or rent-seeking, which I think is discounting a pretty substantial number of rich people.

But it is implicitly making a bolder claim: that people who get rich are more driven than other people; i.e. those who don't get rich.

In my personal experience, the opposite is true. People who get rich do work hard, and are determined, but really poor people work a lot harder and are a lot more determined. A startup founder who is eating rice and beans to try to keep their burn rate low and their runway long may indeed be making sacrifices and working hard. They may be experiencing emotional turmoil. But implicitly, such a person always has the safety net of high-value skills they can use to go find another job if their attempt doesn't work out.

But don't take my word for it; think about it for yourself. Consider a single mother working three minimum-wage jobs and eating rice and beans because that's the only way she can feed her children. Would you imagine she is less determined and will work less hard to keep her children alive than our earlier hypothetical startup founder would work to keep their valuation high?

One of the most important principles in Silicon Valley is that "you make what you measure." It means that if you pick some number to focus on, it will tend to improve, but that you have to choose the right number, because only the one you choose will improve

A closely-related principle from outside of Silicon Valley is Goodhart's Law. It states, "When a measure becomes a target, it ceases to be a good measure". If you pick some number to focus on, the number as measured will improve, but since it's often cheaper to subvert the mechanisms for measuring than to actually make progress, the improvement will often be meaningless. It is a dire mistake to assume that as long as you select the right metric in a political process that you can really improve it.

The Silicon Valley version - assuming the number will genuinely increase, and all you have to do is choose the right one - really only works when the things producing the numbers are computers, and the people collecting them have clearly circumscribed reasons not to want to cheat. This is why people tend to select numbers like income inequality to optimize: it gives people a reason to want to avoid cheating.

It's still possible to get rich by buying politicians (though even that is harder than it was in 1880)

The sunlight foundation published a report in 2014, indicating that the return on investment of political spending is approximately 76,000%. While the sunlight foundation didn't exist in 1880, a similar report in 2009 suggested this number was 22,000% a few years ago, suggesting this number is going up, not down; i.e. over time, it is getting easier, not harder, to get rich by buying politicians.

Meanwhile, the ROI of venture capital, while highly variable, is, on average, at least two orders of magnitude lower than that. While outright "buying" a politican is a silly straw-man, manipulating goverment remains a far more reliable and lucrative source of income than doing anything productive, with technology or otherwise.

The rate at which individuals can create wealth depends on the technology available to them, and that grows exponentially.

In what sense does technology grow "exponentially"? Let's look at a concrete example of increasing economic output that's easy to quantify: wheat yield per acre. What does the report have to say about it?

Winter wheat yields have trended higher since 1960. We find that a linear trend is the best fit to actual average yields over that period and that yields have increased at a rate of 0.4 bushel per acre per year...

(emphasis mine)

In other words, when I go looking for actual, quantifiable evidence of the benefit of improving technology, it is solidly linear, not exponential.

What have we learned?

Paul Graham frequently writes essays in which he makes quantifiable, falsifiable claims (technology growth is "exponential", an "an exponential curve that has been operating for thousands of years", "there are also a significant number who get rich by creating wealth") but rarely, if ever, provides any data to back up those claims. When I look for specific examples to test his claims, as with the crop yield examples above, it often seems to me that his claims are exaggerated, entirely imagined, or, worse yet, completely backwards from the truth of the matter.

Graham frequently uses the language of rationality, data, science, empiricism, and mathematics. This is a bad habit shared by many others immersed in Silicon Valley culture. However, simply adopting an unemotional tone and co-opting words like "exponential" and "factor", or almost-quantifiable weasel words like "most" and "significant", is no substitute for actually doing the research, assembling the numbers, fitting the curves, and trying to understand if the claims are valid.

This continues to strike me as a real shame, because PG's CV clearly shows he is an intelligent and determined fellow, and he certainly has a fair amount of money, status, and power at this point. More importantly, his other writings clearly indicate he cares a lot about things like "niceness" and fairness. If he took the trouble to more humbly approach socioeconomic problems like income inequality and poverty, really familiarize himself with existing work in the field, he could put his mind to a solution. He might be able to make some real change. Instead, he continues to use misleading language and rhetorical flourishes to justify decisions he's already made. In doing so, he remains, regrettably, a blowhard.

05 Jan 2016 4:29pm GMT

30 Dec 2015

feedPlanet Twisted

Twisted Matrix Laboratories: SFC Sponsored Development, Oct-Dec by Amber

Hi everyone,

Along with Adi, I'm supposed to be publishing these too :)

Thanks to our generous sponsors, the Software Freedom Conservancy (of which our part is colloquially known as the Twisted Software Foundation) was able to financially support me as one of the Twisted Fellows. My job is to review tickets, and assist with the Git migration -- this report is mostly about the former, but does have some promising news about the latter.

This groups the tickets under reviewed and triaged -- you can get the full details of my work here, from Trac (which includes both SFC funded ticket reviewing, and non-SFC funded work on code).

Tickets reviewed:

- #5388 (merged): IRI implementation
- #5911 (merged): Support the 'HttpOnly' flag when setting cookies
- #6833: Port twisted.protocols.amp to Python 3
- #7037 (merged): Write test for cftp._cbPutTargetAttrs or remove untested / unused code.
- #7993: Port twisted.web.wsgi to Python 3
- #8082: Document deprecation policy and how to implement a deprecation
- #8107: Add a custom pyflakes runner to ignore Twisted excepted files
- #8116: Deprecate LoopingCall.deferred
- #8121: Port twisted.protocols.htb to Python 3
- #8122 (merged): Final pyflakes cleanup
- #8131 (merged): Port twisted.python.roots to Python 3
- #8132: Port twisted.web.vhost to Python 3
- #8138 (merged): twisted.conch.endpoints._NewConnectionHelper leaks agent connection

Tickets triaged:

- #1436 (invalid): add getCookie/setCookie to web2.http_headers.Headers
- #1920 (invalid): dav.resource.py: RedirectResponse takes a full href, not a uri
- #2342 (invalid): scgi client doesn't handle content-length
- #2343 (invalid): createCGIEnvironment should refuse if path segments have a /
- #2544 (invalid): twisted.web2.channel.[s]cgi contain multiple uses of a function that doesn't exist.
- #3329 (invalid): HTTP's #(...) syntax allows null contents
- #4081 (wontfix): tap2rpm should allow setting RPM dependency information
- #4082 (wontfix): tap2rpm should read configuration from the .tac file it packages.
- #4230 (invalid): Correct exception text in twisted.web2.http_headers:HeaderHandler
- #4426 (wontfix): tap2deb should generate LSB headers for DependencyBasedBoot
- #5875 (wontfix): tap2rpm assumes $PREFIX is /usr
- #6081 (wontfix): Reject unicode in http_headers
- #6168 (wontfix): Enforce type (bytes) of header name and values in twisted.web.http_headers
- #6178 (closed): Port twisted.web.util to Python 3
- #6920 (wontfix): twisted.scripts.test.test_tap2deb.TestTap2DEB.test_basicOperation fails (on Gentoo?)
- #7497 (closed): twistd produces confusing behavior under Python3
- #8007 (closed): API docs try to load urchin.js over http
- #8036: Document how the new logging system should be used and tested in new code

Other
- With others, assembled and proposed a Git Migration Plan, which has now been approved.

Expect to see more in the coming weeks about the Git migration!

- Amber Brown
TSF Fellow & Twisted Release Manager

30 Dec 2015 12:47pm GMT

Twisted Matrix Laboratories: Twisted 15.5 Released

(This was queued but not actually posted, sorry everybody!)

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 15.5!

The sixth (!!) release in 2015 has quite a few goodies in it -- incrementalism is the name of the game here, and everything is just a little better than it was before. Some of the highlights of this release are:

For more information, check the NEWS file (link provided below).

You can find the downloads at on PyPI (or alternatively our website) .
The NEWS file is also available at https://github.com/twisted/twisted/blob/twisted-15.5.0/NEWS.

Also worth noting is the two Twisted fellows -- Adi Roiban and myself -- who have been able to dedicate time to reviewing tickets and generally pushing things along in the process. We're funded by the Software Freedom Conservancy (through what is known as the "Twisted Software Foundation"), which is, in turn, funded by donators and sponsors -- potentially like you! If you would like to know how you can assist in the continued funding of the Fellowship program, see our website: https://twistedmatrix.com/trac/wiki/TwistedSoftwareFoundation#BenefitsofSponsorship

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,

Amber "Hawkie" Brown
Twisted Release Manager, Twisted Fellow

30 Dec 2015 12:47pm GMT

Twisted Matrix Laboratories: December 2015 first half - SFC Sponsored Development


This is my report for the work done in the first half of Decemeber as part of the 2015 Twisted Maintainer Fellowship program.

Important changes made in these weeks:


* The Git migration plan was sent for approval
* Port of the AMP protocol to Python 3.

Tickets reviewed and merged


* #4811 - @unittest.expectedFailure decorator breaks trial
* #5704 - mkdir -p for FilePath
* #6833 - Port twisted.protocols.amp to Python 3
* #7998 - twisted.conch.ssh.keys should be ported to Python 3
* #8115 - Some Twisted tests use the deprecated "assertEquals" and similar
* #8119 - Remove gadfly support / tests from ADBAPI
* #8125 - LoopingCall.withCount with interval 0
* #8136 - Remove the deprecated twisted.web.http.Request.headers/request_headers
* #8139 - None of the ADBAPI tests actually run

Tickets reviewed and not merged yet


* #6757 - SSL server endpoints don't provide a way to verify client certificates
* #7460 - Add HTTP 2.0 support to twisted.web
* #7993 - 'twisted.web.wsgi' should be ported to Python 3
* #8107 - Add a custom pyflakes runner to ignore Twisted excepted files
* #8124 - Update @deprecated to work with properties and __init__
* #8129 - twisted.web.http_headers.Headers should work on bytes and Unicode
* #8137 - twisted.web server breaks with non-IP addresses
* #8138 - twisted.conch.endpoints._NewConnectionHelper leaks agent connection


Thanks to the Software Freedom Conservancy and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.

30 Dec 2015 10:59am GMT

Twisted Matrix Laboratories: November 2015 week 3 and 4 - SFC Sponsored Development


This is my report for the work done in the last weeks of November as part of the 2015 Twisted Maintainer Fellowship program.

We continue to have a small waiting line for the review queue. In general the initial review response time was below 24 hours, and delayed mainly due to time zones.

Important changes made in these weeks:

* Release of Twisted 15.5.0
* Serial port support was ported to Python 3.
* SSH client and server now support SHA2 HMAC.

Tickets reviewed and merged

#6082 - Make test_updateWithKeywords work on Python 3
#6842 - Dead link in documentation of t.i._dumbwin32proc
#6978 - rst markup bugs in docs
#7668 - Links to Dive into Python in the testing documentation are dead
#7881 - Misleading example in Deferred Intro
#7944 - Typo in docs/core/howto/logger.rst
#7791 - Error in options howto doc
#8050 - Release Twisted 15.5
#8070 - Twisted web client IPV6 DNS lookup failed
#8099 - Port twisted.internet.serial to Python 3.
#8101 - Malformed HTTP Header causes exception in twisted.web.http server
#8102 - Malformed HTTP method causes exception in twisted.web.http server/Resource.render
#8104 - Fix documentation of default thread pool size
#8106 - Add Python 3.5 to our tox config.
#8108 - conch: support hmac-sha2-256 and hmac-sha2-512

Tickets reviewed and not merged yet

#5789 - Remove usage of UserDict
#6757 - SSL server endpoints don't provide a way to verify client certificates
#7050 - Remove long deprecated _bwHack parameter
#7460 - HTTP2 inital patch
#7879 - documentation incorrectly says that Agent does not do certificate validation in Agent
#7926 - TLSMemoryBIOProtocol.loseConnection does not work properly
#7998 - twisted.conch.ssh.keys should be ported to Python 3
#8009 - twisted.web.twcgi should be ported to Python 3
#8077 - Session cookie under python3 never found
#8115 - Some Twisted tests use the deprecated "assertEquals" and similar
#8125 - LoopingCall.withCount with interval 0
#8126 - Improvements to Twisted's SMTP server
#8127 - Support a user defined facility when using syslog
#8128 - ESMTP class can't be usefully subclassed to add extensions
#8129 - twisted.web.http_headers.Headers should work on bytes and Unicode

Thanks to the Software Freedom Conservancy and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.

30 Dec 2015 10:59am GMT

Twisted Matrix Laboratories: November 2015 week 1 and 2 - SFC Sponsored Development

This is my report for the work done in the first 2 weeks of November as part of the 2015 Twisted Maintainer Fellowship program.
Important changes made in these 2 weeks:

Tickets reviewed and merged:

Tickets reviewed and not merged yet:

Thanks to theSoftware Freedom Conservancy and all of the sponsors who made this possible, as well as to all the other Twisted developers who helped out by writing or reviewing code.

30 Dec 2015 10:59am GMT