30 Jul 2014
|Andrew Sorensen live-coding at OSCON 2014|
Shortly after Andrew Sorensen began the performance segment of his keynote at OSCON 2014, the #oscon Twitter topic began erupting with posts about the live coding session. Comments, retweets, and additional links persisted for that day and the next. In short, Andrew was a hit :-)
My first encounter with Andrew's work was a few years ago when I was getting back into Lisp. I was playing with generative music with Overtone (and then, a bit later, experimenting with SuperCollider, Hy, and Twisted) and came across his piece A Study in Keith. You might want to take a break from reading this port and watch that now ...
When Andrew started up his presentation, I didn't immediately recognize him. In fact, when the code was displayed on the big screens, I assumed it was Clojure until I looked closely and saw he was using (define ...) and not (defun ...). This seemed very familiar, and then I remembered Impromptu, which ultimately lead to my discovery of Extempore (see more links below) and the realization that this is what Andrew was using to live code.
At the end of the performance a bunch of us jumped up and gave a standing ovation. (In fact, you can hear me yell out "YEAH" at the end of his presentation when he says "And there we go."). It was quite a show. It seemed the OSCON 2014 had been given a theme song. The next step was getting the source code ...
|Andrew's gist (Dark Github Theme)|
Sharing the Code
Andrew gave a presentation on Extempore in the ballroom right after the keynote. This too was fantastic and resulted in much tweeting.
Afterwards a bunch of us went up front and chatted with him, enthusing about his work, the recent presentation, the keynote, and his previously published pieces.
I had Andrew's ear for a moment, and asked him if he was interested in sharing his keynote source -- there had been several requests for it on Twitter (that also got retweeted and/or favourited). Without hesitation, he gave an enthusiastic "yes" and we were off and running for the lounge where we could sit down to create a gist (and grab a cappuccino!). The availability of the source was announced immediately, to the delight of many.
Setting Up Extempore
|Sublime Text 3 connected to Extempore|
Later that night in my hotel room, I had time to download and run Extempore ... and discovered that I couldn't actually play the keynote code, since there was some implicit setup I was missing. However, after some digging around on the docs site and the mail list, music was pouring forth from my laptop -- to my great joy :-D
To ensure anyone else who is not familiar with Extempore can also have this pleasure, I've put together the all the prerequisites and setup necessary in a forked gist, in multiple parts. I will go through those in this blog post. Also: all of my testing and live coding was done using Ben Swift's Extempore Sublime Text plugin.
The first step is getting all the dependencies. You'll want to start the downloads right away, since they are large (the sample files are compressed .wavs). While that's going on, you can install Extempore using Homebrew (this worked for me on Mac OS X with no additional tweaking/configuration necessary):
With Extempore running, let's do some setup. We're going to need to:
- load some libraries (this takes a while for them to compile),
- define some samples, and then
- define some musical note aliases for convenience (and visual clarity).
At this point, we're ready to play!
Playing the Music
To get started on the music, open up the fourth file from the clone of the gist and ^x^x the root, scale, and left-hand-notes-* constants.
Here is the evolution of the left hand part:
Go ahead and start that first one playing (^x^x the definition as well as the call). Wait for a bit, and then execute the next one, etc. Once you've started playing the final left hand form, you can switch to the wider range of notes defined/updated at the bottom.
Next, you'll want to bring in the right hand ... then bassline ... then the higher fmsynth sparkles for the right hand:
Then you'll increase the energy with the drum section:
Finally, you'll bring it to the climax, and then start the gentle fade out:
A slightly modified code listing for the final keynote form is here:
Variation on a Theme
I have recorded a variation of Andrew's keynote based on the code above, for your listening pleasure :-) You can listen to it in your browser or download it.
This version plays part of the left hand piano an octave lower. There's a tiny bit of clipping in places, and I accidentally jazzed it up (and for too long!) with a hi-hat change in the middle. There are also some awkward transitions and volume oddities. However, these should be inspiration for you to make your own variation of the OSCON 2014 Theme Song :-)
The "script" used for the recording can found here.
Links of Note
Some of these were mentioned above, some haven't been. All relate to Extempore :-)
- A Study in Keith.
- The Disklavier Sessions - 2013
- Andrew on Github: https://github.com/digego
- Twitter: https://twitter.com/digego
30 Jul 2014 6:14am GMT
29 Jul 2014
- An Overview
- Themes at OSCON 2014
- Adopting the Functional Paradigm?
- Retrospective on Paradigms
- The Rise of Polyglotism
- Preparing for the Future
Why [have some large companies] not embraced functional programming to the extent that other large ones have?
- U.S. Companies
- Ranked by gross revenue (after adjustments for excise taxes).
- Apple (6) - Swift, Clojure, Scala
- AT&T (11) - Haskell
- HP (15) - F#, Scala
- Verizon Communications (16) - Scala
- IBM (20) - Scala
- Microsoft (35) - F#, F*
- Comcast (46) - Scala
- Amazon (49) - Haskell, Scala, Erlang
- Dell (51) - Erlang, Scala
- Intel (54) - Haskell, SML, PLT Scheme
- Google (55) - Haskell 
- Cisco (60) - Scala
- Ingram Micro (76) - ?
- Oracle (80) - Scala
- Avnet (117) - ?
- Tech Data (119) - ?
- Emerson Electric (123) - ?
- Xerox (131) - Scala
- EMC (133) - Scala
- Arrow Electronics (141) - ?
- Century Link (150) - ?
- Computer Sciences Corp. (176) - ?
- eBay (196) - Scala
- TI (218) - ?
- Western Digital (222) - ?
Rather than seeing the problem of comprehending FP as a physical limitation of the individual, I see instructional failure as the obstacle to overcome. If we start with the proposition that certain brains are deficient, we are essentially abandoning education. It is the responsibility of the instructor to engage creatively with each student's learning style. When adhering to the idea that certain brains are limited, one discards creative engagement; one doesn't even consider working with the students and their learning styles. This is a view that, however implicitly, can be used to shun diversity and dismiss potential.
If both "ideal use case" and "brain fit" are given as arguments against adopting FP (or any other new paradigm) in large organisations, and neither are considered logically or philosophically valid, what's at the root of the resistance?
Up next: Retrospective on Programming Paradigms
Previously: Themes at OSCON 2014
 If anyone has additional information as to which FP languages are used by these top 25 companies, please let me know, and I will include that information. Bonus points for knowing of business-critical applications.
 Type Physicality is a form of reductive materialism, also known as the Mind-Brain Identity Theory that does not allow for mental states to be realized in organisms or computational systems that do not have a brain. See "Criticisms of Type Physicality" at http://en.wikipedia.org/wiki/Identity_theory_of_mind#Multiple_realizability.
29 Jul 2014 3:53am GMT
28 Jul 2014
- An Overview
- Themes at OSCON 2014
- Adopting the Functional Paradigm?
- Retrospective on Paradigms
- The Rise of Polyglotism
- Preparing for the Future
A Qualitative OSCON Debrief
As you might have noticed from the OSCON Twitter-storm this year, the conference was a blast. Even if you weren't physically present, given the 17 tracks, you can imagine that the presentations -- and subsequent conversations -- were deeply varied.
This was the second OSCON I'd attended; the first was was in 2008 as a guest of Michael Bernstein, a friend who was speaking there. OSCON 2008 was a zoo - I'm not sure of the actual body count, but I've heard that attendees + vendors + miscellaneous topped 12,000 people over the course of the week (I would love to hear if someone has hard data on that -- googling didn't reveal much). OSCON 2008 was dominated by Big Data, Hadoop, endless buzzword bingo, and business posturing by all sorts. The most interesting bits of that conference were the outlines that formed around the conversations people weren't having. In fact, over the following 6 months, that's what I spent my spare time pondering: what people didn't say at OSCON.
This year's conference seemed like a completely different animal. It felt like easily 1/2 to 1/3rd the number of attendees in 2008. Where that one had all the anonymizing feel of rush-hour in a major metropolitan hub, OSCON 2014 had a distinctly small-town vibe to it -- I was completely charmed. Conversations (overheard as well as participated in) were not littered with examples from the latest bizspeak, but rather focused on essence. The interactions were not continually distracted, but rather steadily focused, allowing people to form, express, and dispute complete thoughts with their peers.
So what were people talking about this year? Here are some of the topics I heard covered during lunches, in hallways, and at podiums; at pubs, in restaurants and at parks :
- What communities are thriving?
- Which [projects, organisations, companies, etc.] are treating their people right?
- What successful processes are being followed at [project, organisation, etc.]?
- Who is hiring and why should someone want to work there?
- Where can I go to learn X? Who is teaching X? Who shares the most about X?
- Which [projects, organisations] support X?
- Why don't more [people, projects, organisations] care about [possible future X]?
- Why don't more [people, projects, organisations] spend more time investigating the history of X for "lessons learned"?
- There was so much more X in computing during the 60s and 70s -- what happened? 
- Why are we reinventing X?
- When is X going to be invented, and who's going to do it?
- Everything is changing! I can't keep up anymore.
- I want to keep up, but how?
- Why can't we stop making so many X?
- Nobody cares about Y anymore; we're all doing X now.
- Full stack developers!
- Fault-tolerant systems!
After lots of reflection, here's how I classified most of the conversations I heard:
- Developing communities,
- Developing careers and/or personal/professional qualities, and
- Developing software,
along lines such as:
- Effective maintenance, maturity, and health,
- Focusing on the "art", eventual mastery, and investments of time,
- Tempering bare pragmatism with something resembling science or academic excellence,
- Learning the new to bolster the old,
- Inspiring innovation from a place of contemplation and analysis,
- Mining the past for great ideas, and
- Figuring out how to better share and spread the adoption of good ideas.
Generalized to such a degree, this could have been pretty much any congregation of interested, engaged minds since the dawn of civilization. So what does it look like if we don't normalize quite so much? Weighing these with what may well be my own bias (and the bias of like-minded peers), I submit to your review these themes:
- A very strong interest in programming (thinking and creating) vs. integration (assessing and consuming).
- An express desire to become better at abstraction (higher-order functions, composition, and types) to better deal with growing systems complexities.
- An interest in building even more complicated systems.
- A fear of reimplementing past mistakes or of letting dust gather on past intellectual achievements.
As you might have guessed, these number very highly among the reasons why the conference was such an unexpected pleasure for me. But it should also not come as a surprise that these themes are present:
- We have had several years of companies such as Google and Amazon (AWS) building and deploying some of the most sophisticated examples of logic-made-manifest in human history. This has created perceived value in our industry and many wish to emulate it. Similarly, we have single purpose distributed systems being purchased for nearly 20 billion USD -- a different kind of complexity, with a different kind of perceived reward.
- In the 70s and 80s, OOP adoption brought with it the ability to create large software systems in ways that people had not dared dream or were impractical to realize. Today's growing adoption of the Functional paradigm is giving early signs of allowing us to better integrate complex systems with more predictability and fewer errors.
- Case studies of improvements in productivity or the capacity to handle highly complex or previously intractable problems with better abstractions, has ignited the passions of many. Not wanting to limit their scope of knowledge or sources of inspiration, people are not simply limiting themselves to the exploration of such things as Category Theory -- they are opening the vaults of computer science with such projects as Papers We Love.
There's a brave new world in the making. It's a world for programmers and thinkers, for philosophers and makers. There's a lot to learn, but it's really not so different from older worlds: the same passions drive us, the same idealism burns brightly. And it's nice to see that these themes arise not only in small, highly specialized venues such as university doctoral programs and StrangeLoop (or LambdaJam), but also in larger intersections of the industry like OSCON (or more general-audience ones like Meetups).
Up next: Adopting the Functional Paradigm?
Previously: An Overview
 It goes without saying that any one attendee couldn't possibly be exposed to enough conversations to form a perfectly accurate sense of the total distribution of conversation topics. No claim to the contrary is being made here :-)
here in the section titled "Why did all these ideas happen during this particular time period?"
28 Jul 2014 5:25am GMT
|Art by Philip Straub|
There's a new series of blog posts coming, inspired by on-going conversations with peers, continuous inspection of the development landscape, habitual navel-gazing, and participation at the catalytic OSCON 2014. As you might have inferred, these will be on the topic of "The Future of Programming."
Not to be confused with Bret Victor's excellent talk last year at DBX, these posts will be less about individual technologies or developer user experience, and more about historic trends and viewing the present (and near future) through such a lense.
In this mini-series, the aim is to present posts on following topics:
- An Overview
- Themes at OSCON 2014
- Adopting the Functional Paradigm?
- Retrospective on Paradigms
- The Rise of Polyglotism
- Preparing for the Future
I did a similar set of posts, conceived in late 2008 and published in 2009 on the future of cloud computing entitled After the Cloud. It was a very successful series and the cloud industry seems to be heading towards some of the predictions made in it -- ZeroVM and Docker are an incremental step towards the future of distributed processes/functions outlined in To Atomic Computation and Beyond.
- Alan Kay, 1998: A crucial key to growing large systems is effective communications between components.
- Joe Armstrong, 2004: To effectively model and solve problems in a distributed manner, we need concurrency... this is made easier when we isolate processes and do not share data.
In the decade since these statements were made, we have seen individuals, projects, and companies take that vision to heart -- and succeeding as a result. But as an industry, we continue to struggle with the definition of our art; we still are tormented by change -- both from within and externally -- and do not seem to adapt to it well.
28 Jul 2014 5:17am GMT
18 Jul 2014
I have started playing, recently, the online game "Clash of Clans". Clash of Clans has the option of "donating" troops to other clan members. Troops donated stay in a separate area (so they do not take up area for one's own troops), and also participate in base defense (as opposed to one's own troops, which are offensive only). In short, being donated-to makes both one's offense and one's defense stronger. However, donating troops means paying the cost of training them and reaping no benefit from it.
Does we have the pay-off matrix: assume paying the cost of troops is -1, and the benefit is +2 (it has to be bigger than the cost, since players do build troops for themselves with even less benefits):
Don't donate/Don't donate: 0/0
Donate/Don't donate: -1/2
Don't donate/Donate: 2/-1
What I love about this is this is a massive experiment with prisoners' dilemma on more-or-less average people. As expected, one sees both equilibrium: clans where hardly anyone donates, and clans where everyone donates. In the clans where hardly anyone donates, there is still some reciprocity: (some) people will try to donate "back" to people who donated to them. Since in practice the cost-benefit analysis turns that way, it is sometimes worthwhile to donate even for a .2 chance of a reciprocation, which allows the loop to be fed.
There are reputation effects (the number of troops one donated and one had donated to appears where everyone can see it), which would be assumed to improve the donation rate. However, from my anecdotal experience, being in two clans, the most distinguishing feature is "do you know these people in real life?" Clans made up of people whose reputation effects extend beyond the game are in the good equilibrium. Since I've joined the Facebook-employee-only clan, my life has been much better.
I started thinking that Facebook is like that too. In situations where you can help a colleague, where the benefit to the colleague would be disproportional to one's self, we encourage that kind of thinking. Facebook is in the good equilibrium of the dilemma!
[Plug: We're hiring engineers and eng. managers. Please contact me on FB with further questions or resumes.]
18 Jul 2014 4:10pm GMT
This post should have a subtitle: "Using Team Analysis and Levenshtein Distance to Reveal said Structure." It's the first part of that subtitle that is the secret, though: being able to correctly analyze and classify individual teams. Without that, using something clever like Levenshtein distance isn't going to be very much help.
But that's coming in towards the end of the story. Let's start at the beginning.
What You're Going to See
This post is a bit long. Here are the sections I've divided it into:
- What You're Going to See
- Introducing ACME
- Categorizing Teams
- Category Example
- Calculating the Levenshtein Distance of Teams
- Sorting and Interpretation
However, you don't need to read the whole thing to obtain the main benefits. You can get the Cliff Notes version by reading the Premise, Categorizing Teams, Interpretation, and the Conclusion.
Companies grow. Teams expand. If you're well-placed in your industry and providing in-demand services or products, this is happening to you. Individuals and small teams tend to deal with this sort of change pretty well. At an organizational level, however, this sort of change tends to have an impact that can bring a group down, or rocket it up to the next level.
Of the many issues faced by growing companies (or rapidly growing orgs within large companies), the structuring one can be most problematic: "Our old structures, though comfortable, won't scale well with all these new teams and all the new hires joining our existing teams. How do we reorganize? Where do we put folks? Are there natural lines along which we can provide better management (and vision!) structure?"
The answer, of course, is "yes" -- but! It requires careful analysis and a deep understanding every team in your org.
The remainder of this post will set up a scenario and then figure out how to do a re-org. I use a software engineering org as an example, but that's just because I have a long and intimate knowledge of them and understand the ways in which one can classify such teams. These same methods could be applied a Sales group, Marketing groups, etc., as long as you know the characteristics that define the teams of which these orgs are comprised.
ACME Corporation is the leading producer of some of the most innovative products of the 20th century. The CTO had previously tasked you, the VP of Software Development to bring this product line into the digital age -- and you did! Your great ideas for the updated suite are the new hottness that everyone is clamouring for. Subsequently, the growth of your teams has been fast, and dare we say, exponential.
More details on the scenario: your Software Development Group has several teams of engineers, all working on different products or services, each of which supports ACME Corporation in different ways. In the past 2 years, you've built up your org by an order of magnitude in size. You've started promoting and hiring more managers and directors to help organize these teams into sensible encapsulating structures. These larger groups, once identified, would comprise the whole Development Group.
Ideally, the new groups would represent some aspect of the company, software development, engineering, and product vision -- in other words, some sensible clustering of teams doing related work. How would you group the teams in the most natural way?
Simply dividing along language or platform lines may seem like the obvious answer, but is it the best choice? There are some questions that can help guide you in figuring this out:
- How do these teams interact with other parts of the company?
- Who are the stakeholders in feature development?
- Which sorts of customers does each team primarily serve?
ACME Software Development has grown the following teams, some of which focus on products, some on infrastructure, some on services, etc.:
- Digital Anvil Product Team
- Giant Rubber Band App Team
- Digital Iron Carrot Team
- Jet Propelled Unicycle Service Team
- Jet Propelled Pogo Stick Service Team
- Ultimatum Dispatcher API Team
- Virtual Rocket Powered Roller Skates Team
- Operations (release management, deployments, production maintenance)
- QA (testing infrastructure, CI/CD)
- Community Team (documentation, examples, community engagement, meetups, etc.)
Keep in mind that each category should be chosen such that it would make sense for teams categorized similarly to be grouped together. A counter example might be "Team Size"; you don't necessarily want all large teams together in one group, and all small teams in a different group. As such, "Team Size" is probably a poor category choice.
- Implementation Focus
- Supported OS
- Deployment Type
- License Type
- Industry Segment
- Customer Type
- Corporate Priority
Category Example(Things are going to get a bit more technical at this point; for those who care more about the outcomes than the methods used, feel free to skip to the section at the end: Sorting and Interpretation.)
In all cases, we will encode these values as binary digits -- this allows us to very easily compare teams using Levenshtein distance, since the total of all characteristics we are filtering on can be represented as a string value. An example should illustrate this well.
(The Levenshtein distance between two words is the minimum number of single-character edits -- such as insertions, deletions or substitutions -- required to change one word into the other. It is named after Vladimir Levenshtein, who defined this "distance" in 1965 when exploring the possibility of correcting deletions, insertions, and reversals in binary codes.)
- LFE - #b0000000001
- Erlang - #b0000000010
- Elixir - #b0000000100
- Ruby - #b0000001000
- Python - #b0000010000
- Hy - #b0000100000
- Clojure - #b0001000000
- Java - #b0010000000
- CoffeeScript - #b1000000000
We could then compare this to a team that used just Hy and Clojure (#b0001100000), which has a Levenshtein distance of 1 with the previous language category value. A team that used Ruby and Elixir (#b0000001100) would have a Levenshtein distance of 5 with the LFE/Hy/Clojure team (which makes sense: a total of 5 languages between the two teams with no languages shared in common).
Calculating the Levenshtein Distance of Teams
In the third table on that sheet, all of the values for each column are combined into a single binary string. This (or a slight modification of this) is what will be the input to your calculations. Needless to say, as a complete fan of LFE, you will be writing some Lisp code :-)
|Partial view of the spreadsheet's first page.|
Let's create a couple of data structures that will allow us to more easily work with the data you collected about your teams in the spreadsheet:
We can use a quick copy and paste into the LFE REPL for two of those numbers to do a sanity check on the distance between the Community Team and the Digital Iron Carrot Team:
That result doesn't seem unreasonable, given that at a quick glance we can see both of these strings have many differences in their respective character positions.
It looks like we're on solid ground, then, so let's define some utility functions to more easily work with our data structures:
Now we're ready to roll; let's try sorting the data based on a comparison with a one of the teams:
It may not be obvious at first glance, but what the levenshtein-sort function did for us is compare our "control" string to every other string in our data set, providing both the distance and the string that the control was compared to. The first entry in the results is the our control string, and we see what we would expect: the Levenshtein distance with itself is 0 :-)
The result above is not very easily read by most humans ... so let's define a custom sorter that will take human-readable text and then output the same, after doing a sort on the binary strings:
(If any of that doesn't make sense, please stop in and say "hello" on the LFE mail list -- ask us your questions! We're a friendly group that loves to chat about LFE and how to translate from Erlang, Common Lisp, or Clojure to LFE :-) )
- it has stakeholders in Sales, Engineering, and Senior Leadership;
- it has a "Corporate Priority" of "Business critical"; and
- it has both internal and external customers.
Here we're making the request "Show me the sorted results of each team's binary string compared to the binary string of the DICT." Here are the human-readable results:
For a better visual on this, take a look at the second tab of the shared spreadsheet. The results have been applied to the collected data there, and then colored by major groupings. The first group shares these things in common:
- Lisp- and Python-heavy
- Middleware running on BSD boxen
- Mostly proprietary
- Externally facing
- Focus on apps and APIs
As one might expect, most of the teams remain grouped in the same way ... the notable exception being the split-up of the Anvil and Rubber Band teams. Mostly no surprises, though -- the same groupings persist in this model.
|A sort (and thus grouping) by comparison to highly-connected team.|
As a last example, let's throw caution and good sense to the wind and get crazy. You know, like the many times you've seen bizarre, anti-intuitive re-orgs done: let's do a sort that compares a team of middling importance and a relatively low corporate impact with the rest of the teams. What do we see then?
This ruins everything. Well, almost everything: the only group that doesn't get split up is the middleware product line (Jet Propelled and Iron Carrot). Everything else suffers from a bad re-org.
|A sort (and thus grouping) by comparison to a non-critical team.|
If you were to do this because a genuine change in priority had occurred, where the Giant Rubber Band App Team was now the corporate leader/darling, then you'd need to recompute the bit values and do re-sorts. Failing that, you'd just be falling into a trap that has beguiled many before you.
ConclusionIf there's one thing that this exercise should show you, it's this: applying tools and analyses from one field to fresh data in another -- completely unrelated -- field can provide pretty amazing results that turn mystery and guesswork into science and planning.
If we can get two things from this, the other might be: knowing the parts of the system may not necessarily reveal the whole (c.f. Complex Systems), but it may provide you with the data that lets you better predict emergent behaviours and identify patterns and structure where you didn't see them (or even think to look!) before.
18 Jul 2014 6:09am GMT
A few weeks back -- the week of the PyCon sprints, in fact -- was the San Francisco Erlang conference. This was a small conference (I haven't been to one so small since PyCon was at GW in the early 2000s), and absolutely charming as a result. There were some really nifty talks and a lot of fantastic hallway and ballroom conversations... not to mention Robert Virding's very sweet Raspberry Pi Erlang-powered wall-sensing Lego robot.
My first Erlang Factory, the event lasted for two fun-filled days and culminated with a stroll in the evening sun of San Francisco down to the Rackspace office where we held a Meetup mini-conference (beer, food, and three more talks). Conversations lasted until well after 10pm with the remaining die-hards making a trek through the nighttime streets SOMA and the Financial District back to their respective abodes.
Before the close of the conference, however, we managed to sneak a ride (4 of us in a Mustang) to Scoble's studio and conduct an interview with Joe Armstrong and Robert Virding. We covered some of the basics in order to provide a gentle overview for folks who may not have been exposed to Erlang yet and are curious about what it has to offer our growing multi-core world. This wend up on the Rackspace blog as well as the Building 43 site (also on YouTube). We've got a couple of teams using Erlang in Rackspace; if you're interested, be sure to email Steve Pestorich and ask him what's available!
18 Jul 2014 5:49am GMT
16 Jul 2014
It's two weeks shy of a year since the last morituri release. It's been a pretty crazy year for me, getting married and moving to New York, and I haven't had much time throughout the year to do any morituri hacking at all. I miss it, and it was time to do something about it, especially since there's been quite a bit of activity on github since I migrated the repository to it.
I wanted to get this release out to combine all of the bug fixes since the last release before I tackle one of the number one asked for issues - not ripping the hidden track one audio if it's digital silence. There are patches floating around that hopefully will be good enough so I can quickly do another release with that feature, and there are a lot of minor issues that should be easy to fix still floating around.
But the best way to get back into the spirit of hacking and to remove that feeling of it's-been-so-long-since-a-release-so-now-it's-even-harder-to-do-one is to just Get It Done.
I look forward to my next hacking stretch!
Happy ripping everybody.
16 Jul 2014 4:01am GMT
11 Jul 2014
When in need to get two software processes to exchange datas, some sort of protocol is necessary to define how to encode/decode the datas to be transported. A large number of serialization formats are available (json, xml, ASN.1…) so as to tackle the cross process/cross programming language datas encoding problem and Python provides a large number of libraries to leverage them…
What distinguishes the solution provided by protocol buffers or thrift is the need to describe the datas to be exchanged in a central schema file written using an easy to read idl language. Such schema file is then compiled so as to provide data representation for a certain programming language. Not all problems will benefit from this approach, but when developping services that needs to be accessed by clients written in different programming languages ( eg Objective C, Java…) we have found that relying on a well defined schema file allows to save a lot of time.
The need for benchmarking
We write a large part of our server side code using Python and some of the projects we support are accessed mainly by native clients over TCP or UDP. Over the years we moved gradually from our home grown custom serialization solution built on top of Python struct module to protocol buffer…
The move to protocol buffer allowed us to cut down development time required to support new types of client to a minimum. We also realized how the reliance of a central schema was valuable in that everybody can vizualize what the datas are.
One thing however we are regretting from the previous home grown solution are the performances, and at the server performances matter tremendously even more if you are using an asynchronous networking framework like twisted.
For a long while we reinssured ourselves observing that a google supported extension module was available, and that deploying it, would allow us to accelerate serialization/deserialization by a factor 10 at least. Deploying such extension module was delayed till reaching stagging development phase, because it is quite cumbersome to do so. You need to build things from source and manage some environment variables in your server processes, to force the use of the implementation it provides.
Once we activated the protobuf extension module on our stagging server we started to observe random crashes of the server processes. It took us time to understand that those crashes were related to the use of such extension module. Well, we should not have underestimated the fact that Google was labelling this extension module as experimental, but here again we assumed that Google playing in a different category than the rest of us they were probably referring to some pretty advanced usecases :(
After all those hurdles, we realized that selecting the proper serialization technology for your projects is a decision that shall not be taken lightly. Thrift provides an obvious alternative to google protocol buffer, but how does its Python implementation performs ? They exist extensive performance benchmarks of java serialization frameworks, but we found nothing similar for Python.
We have published on GitHub, what we consider to be a good basis to compare the various serialization frameworks which one may want to leverage. The repository for the project can be reached here.
The benchmark for now compares the performances of protobuf and thrift serializations for messages defined in the StuffTotest schema . We welcome suggestions to extend such reference schema so as to explore performances variations more in the details or external help so as to cover more serialization frameworks…
We have published on GitHub a result run obtained on a low end development machine. If we consider performance to be the average in between serialization and deserialization time for a certain message of the schema, Thrift :
- outperforms protocol buffers in 75% of the cases.
- is stable.
- is much easier to deploy (pip install thrift and you are done…)
So there is currently a clear winner to this benchmark. We will be happy to rerun it so as to validate that things have changed.
11 Jul 2014 9:00pm GMT
10 Jul 2014
One of the great pleasures of advancing in one's craft is the thrill of solving newer, more difficult problems. In the course of my work, I frequently encounter many bugs which appear at first glance to be impossible. Not just impossible to solve, but intrinsically impossible. It's as if being at the very frontiers of technology exposes one to strange new logics that the human mind is not fit to grasp.
Each of these bugs is, of course, unique, and it would be folly to suggest that there could be any sort of systematic method that could be applied at the cutting edge of technology, as if one could push back the boundaries of ignorance in much the same way as one fills in a tax return. Nevertheless, the following principles have served me pretty well, and perhaps deserve a wider audience.
- You're editing the wrong file
- You're editing the right file, but on the wrong machine
- You're editing the right file, but you forgot to save it
- You've edited the right file, but you forgot to recompile
- The thing you thought you turned on, you actually turned off
- The thing you thought you turned off, you actually turned on
- You're in a meeting, and you should be paying attention. They wouldn't have eighteen people talking about it unless it was really important, would they?
- You're running the wrong version
- You're running the right version, but on the wrong machine
- You've fixed the problem, but forgot to push
- You've fixed the problem and remembered to push, but you forgot to commit
- You've fixed the problem, committed, and pushed. However, many users were relying on the previous, broken behaviour and so now you must roll back.
I offer up these principles humbly: no single approach can suffice at the very fringes of the known. I earnestly hope that by meditating on them we can achieve things as yet undreamt of.
10 Jul 2014 11:00pm GMT
07 Jul 2014
A while ago my friend teh recommended that I use conda for Python package management, because of its intrinsic superiority to pip & virtualenv. I'm no fan of either, and teh generally has pretty good taste, so I thought I'd check it out.
I haven't formed an overall opinion yet, but I can say without reservation that the overall experience for someone who has become used to pip & virtualenv is very frustrating.
What I found myself wanting was a one pager explaining conda & binstar to someone familiar with pip & virtualenv, which are unquestionably vastly more popular. Alas, there's none to be found.
There are some guidelines here and there on the net about how to install a package that's available on PyPI into a conda system, but they don't always work.
For example, I'd like to play around with pyrsistent. The recommended approach is to do the following:
$ conda skeleton pypi pyrsistent $ conda build pyrsistent $ binstar upload //anaconda/conda-bld/osx-64/pyrsistent-0.3.1-py27_0.tar.bz2 $ conda install pyrsistent
Everything works fine up until the
build step, but the upload fails because someone has already uploaded the package. Fine. So
conda install pyrsistent should work, right?
$ conda install pyrsistent Fetching package metadata: ... Error: No packages found matching: pyrsistent
Searching binstar finds pyristent. Apparently I have to add some kind of channel (a concept that's not well explained, and that does not seem to appear at all on the binstar UI).
I try this way:
$ conda config --add channels https://binstar.org/auto
But apparently that's not a channel:
$ conda install pyrsistent Fetching package metadata: ....Error: HTTPError: 404 Client Error: NOT FOUND: https://binstar.org/auto/osx-64/
So I try this way:
$ conda config --add channels https://binstar.org/auto
And that is a channel, but pyrsistent it doth not find:
$ conda install pyrsistent Error: No packages found matching: pyrsistent
I take a look at the files page for pyrsistent and it looks like the only file is
linux-64/pyrsistent-0.1.0-py27_0.tar.bz2. I'm running OS X (for my sins) and my guess is that the "No packages found matching" error refers to a lack of binary builds.
So, what am I supposed to do? At this point, the documentation and the error messages from the tools leave me with no obvious way forward. Perhaps there's some underlying concept that would illuminate all for me, if only I could grasp it. Perhaps it's simply a bug.
In any case, I'm not abandoning conda yet (in part because I have no idea how to extricate it from my system), but it has some way to go before it is properly better than pip & virtualenv.
07 Jul 2014 11:00pm GMT
05 Jul 2014
A while ago, I got sick of Blogger, so I switched to Octopress.
No comments yet, and still quite a few things that ought to be done. If you notice something broken or missing, please file a bug.
I'm afraid I have neither the energy nor the inclination to do a full write-up of the migration process. However, I'll provide the following notes in case they help anyone who wants to do the same thing.
- Handle unicode authors
- Basic blogger import
- Comment support for blogger
- Post-processing of extracted HTML
Make the blog
$ pelican-quickstart Welcome to pelican-quickstart v3.3.1.dev. This script will help you create a new Pelican-based website. Please answer the following questions so this script can generate the files needed by Pelican. > Where do you want to create your new web site? [.] code-blog > What will be the title of this web site? Mere Code > Who will be the author of this web site? Jonathan M. Lange > What will be the default language of this web site? [en] > Do you want to specify a URL prefix? e.g., http://example.com (Y/n) Y > What is your URL prefix? (see above example; no trailing slash) http://code.mumak.net > Do you want to enable article pagination? (Y/n) Y > How many articles per page do you want?  20 > Do you want to generate a Fabfile/Makefile to automate generation and publishing? (Y/n) Y > Do you want an auto-reload & simpleHTTP script to assist with theme and site development? (Y/n) Y > Do you want to upload your website using FTP? (y/N) n > Do you want to upload your website using SSH? (y/N) > Do you want to upload your website using Dropbox? (y/N) > Do you want to upload your website using S3? (y/N) > Do you want to upload your website using Rackspace Cloud Files? (y/N) > Do you want to upload your website using GitHub Pages? (y/N) Y > Is this your personal page (username.github.io)? (y/N) N
Extract the things
$ pelican-import --blogger -m markdown \ -o output-directory/ --wp-custpost --dir-page \ blog-08-29-2013.xml
Transfer the things
$ cp output-directory/* blog-directory/content/ $ rm -rf blog-directory/content/settings blog-directory/content/templates/ $ mv blog-directory/content/comments blog-directory
Tweak the blog
Only basic tweaks. Note that the
pelican_comment_system plugin is used in order to nicely render the imported blogger comments.
I had to manually tweak the theme to support the comment system.
05 Jul 2014 2:21pm GMT
30 Jun 2014
Warrants (and similar things, like indictments at preliminary hearings) rely on the concept of "probable cause". The definition of probable cause varies, but in general requires for something to be more probable than not - or in terms of Bayesian probability (which applies probability to states of mind, not just to repeatable experiments), higher than 0.5 probability. Do our warrants actually obey this ruling? This is a fairly simple test - at least 50% of warrants handed should, eventually, result in evidence leading to conviction. Do they? Hell, in less than 50% of cases does it lead to an indictment!
Since warrant judges hand out a lot of warrants, we can decide to actually measure and automatically disbar judges whose warrant rates fall significantly below 50% rates (e.g., using p values of 0.005 would correspond to having a pretty high Bayesian prior that judges are "good"). The criteria could simply be - "of all warrants handed out where the case has ended, how many convictions have there been" (so any cases still in trial would be eliminated from consideration). After doing this for a while, we can update the Bayesian prior, periodically, to how many judges actually get disbarred by this.
In a perfect world, we would also apply this to convictions - however, since convictions are handled by one-offs (jurors), it is hard to place blame for overturning convictions. However, at least for warrants and non-grand-jury-indictments, this allows a process of automatic continuous improvement of our standards, avoiding the "judges and police co-operate to come up with lots of warrants" problem.
 Technically, since conviction is "beyond reasonable doubt", this is a lighter standard than the true standard of actual guilt. However, based on the "better a hundred guilty people go free…" (Benjamin Franklin) standard, reasonable doubt being >=1% probability, the effect should be minor.
30 Jun 2014 4:29pm GMT
I will be visiting San Francisco this August and staying until November, spending my days working on a TLS v1.2 implementation in Python, using existing crypto math, thanks to Stripe's Open Source Retreat. Keep an eye out for a blog post with more details on that.
I just wrapped up work at HippyVM, and have gathered numerous stories looking at the PHP source code every day for the past six months. I've also just graduated, and it feels great to be able to spend the next few months working full-time on open source software.
What after, you ask? Some of you reading this right now have intimated that you'd like to offer me a job, if I were available. Well, now's your chance. Get in touch, and let's talk.
To my various acquaintances in the bay area, let's get together and do something fun. To others, if grabbing a coffee with me is an interesting idea to you, please drop me a line. I would love to hear your story about how PHP ruined your summer, or how Twisted changed your life, or how you had to stay up all night to fix an OpenSSL vulnerability. Really, I love stories; if you've got an interesting one to share, I'd be happy to lend an ear.
30 Jun 2014 12:53pm GMT
29 Jun 2014
It's been very long since I last posted something. Getting married, moving across the Atlantic, enjoying the city, it's all taken its time. And the longer you don't do something, the harder it is to get back into.
So I thought I'd start simple - I updated mach to support Fedora 19 and 20, and started rebuilding some packages.
Get the source, update from my repository, or wait until updates hit the Fedora repository.
29 Jun 2014 9:09pm GMT
22 Jun 2014
The Clojure community (in particular, James Reeves) has produced one of the best APIs I've seen for creating RESTful services, or any application using path dispatching (routing), really: Compojure.
I've been really missing this in LFE. Despite the fact that creating RESTful apps in LFE with YAWS is so simple it doesn't even need a framework, the routes can be a bit difficult to read, due to some of the destructuring that's done (e.g., URL paths).
Take the following route declaration from the yaws-rest-starter project:
Note that this example delegates additional routing to other functions, since all the routing in one function was pretty difficult to read. This, however, detracts from the overall readability in a different way: there's not one place to look to see what functions map to the complete URL-space for this service.
I wanted to be able to do something like what you can do in Clojure:
LFE is a Lisp with macros, so why not? Also, nox or mheise (I forget who) on the #erlang-lisp channel had previously noted all the lfe* projects (and these too), and suggested that someone create the inevitable "lfest" repo on Github.
Enter lfest. After a weekend of hacking and debugging some tiny macros, surprisingly little code now supports routes definitions like the following in LFE+YAWS web apps:
If you're curious to see what this actually expands to before getting compiled in a .beam file, you can view it here.
Bonus for LFE webdevs: this project also has a module of HTTP status codes, for easy re-use in other projects.
The recent work I've been doing with macros has really helped shape the section I've been planning to write in the LFE User Guide. I haven't wanted it to be yet another dry description of macros in a Lisp-2. Instead, my hope is to help jump-start people into how to think about macros, how to start writing them immediately (without years of study first), and how to debug them.
The trick, as with all complicated subjects, is how to remove the barrier to entry and get folks that necessary hands-on experience without dumbing-down the material. It's almost ready...
Also, it's Bay to Breakers today, and Ocean Beach is bumpin. The synchronicity is just eery. May you party just as hard with routes in LFE.
22 Jun 2014 3:16am GMT