12 Aug 2015

feedPlanet Twisted

Moshe Zadka: Kushiel’s Legacy (the Phedre Trilogy) — Sort of a book review

Below spoilers for the Kushiel's Legacy books (first trilogy). If you have not read them, I highly recommend reading before looking below - they are pretty awesome. Technically, there are also spoilers for Lord of the Rings, though if you have not read it by now, I don't know what else to do.

I want to start by talking about Lord of the Rings. As everyone knows, it tells the story of the failed attempt of the dwarves to restore their ancient homeland in a retrospective. I think they had a side-plot about a couple of short guys throwing a ring into a mountain or something? Anyway, back to the main point of LotR: the dwarves. The story of a people who everyone thinks of as just being greedy, master of goldsmithing and gem cutting, but with the desire to return to a homeland - but who wake an ancient evil upon returning there. Though Tolkien maintained he was not into allegories, this one is a pretty thin facade for talking about the struggles of the Jewish people. The reason for this long introduction to LotR is to explain how I read books: parochially, maybe, if they have Jews in them, the Jewish plot lines stir my heart much more than the story the author tried to write.

Well, I have to say, Jacqueline Carey knows how to write Jews. Admittedly, it took me until the end of Kushiel's Dart to figure out that the Yeshuites were not supposed to be Jesuits. They are really an interesting take on Jews for Jesus. In a world that lacked Christianity, since Elua and his followers took on the role of taking down the Roman ("Tiberian") empire, there are no evangelical religions. The Jews end up accepting Jesus as the "Mashiach" (not "Christ", since the Romans aren't there to take over the terms), and keep being…pretty Jewish. They are discriminated against in a heart-touching manner in Venice (ok, "La Serenissima") and, like every good Jews, have fierce arguments about restoring their homeland. Near as I can tell, judging from estimated geography, they decide to make their homeland in Switzerland? Or maybe Lichtenstein? Belgium? Netherlands? In any case, the more devout, as always, pray for salvation while the younger and more practically minded take up the sword, and head up north to make a place where they will be free from discrimination.

All the while, the Jews in the Kushielverse keep studying the Talmud, and the more mystical minded of them take up something suspiciously like Qabbalah and seek the true name of God. They speak "Habiru" among each other, and have stories of the lost tribes. That's, of course, the most awesome parts - the ten tribes have a "Judaism without Jesus" (what we would call, well, Judaism) since they have not heard of him having been sequestered in…Uganda. I, well, I certainly appreciate the subtle humor there.

[Please note that as far as I know, the author really intended the Yeshuites to add color to the religions in the Kushielverse, and actually fairly little of the plotline involves them - it is really the story of a masochist prostitute in a country where not being polyamorous is a blasphemy…]

12 Aug 2015 3:16am GMT

04 Aug 2015

feedPlanet Twisted

Twisted Matrix Laboratories: Twisted 15.3 Released

On behalf of Twisted Matrix Labs, I am honoured to announce the release of Twisted 15.3.

We're marching confidently into the future, and this release demonstrates this:

You can find the downloads at https://pypi.python.org/pypi/Twisted (or alternatively http://twistedmatrix.com/trac/wiki/Downloads). The NEWS file can be found at https://github.com/twisted/twisted/blob/trunk/NEWS.

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,

Amber "Hawkie" Brown

04 Aug 2015 7:07am GMT

17 Jul 2015

feedPlanet Twisted

Glyph Lefkowitz: Sometimes, You Just Want A Button

I like making computers do stuff. I find that getting a computer to do what I want it to produces a tremendously empowering feeling. I think Python is a great language to use to tell computers what to do.

In my experience, the most empowering feeling (albeit often, admittedly, not the most useful or practical one) is making one's own computer do something cool. But due to various historical accidents, the practical bent of the Python community mainly means that we get "server-side" or "cloud" computers to do things: in other words, "other people's computers".

If you, like me, are a metric standard tech industry hipster, "your own computer" means "Mac OS X". Many of us have written little command-line tools to automate things, and thank goodness OS X is enough of a UNIX that that works nicely - and it's often good enough. But sometimes, you just want to press a button and have a thing happen; sometimes a character grid isn't an expressive enough output device. Sometimes you want to be able to take that same logical core, or some useful open source code, that you would use in the cloud, and instead, put it into an app. Sometimes you want to be able to learn about writing desktop apps while leveraging your existing portfolio of skills. I have done some of this, and intend to do more of it at work, and so I'd like to share with you some things I've learned about how to do it.

Back when I was a Linux user, this was a fairly simple proposition. All the world was GTK+ back then, in the blissful interlude between the ancient inconsistent mess when everything was Xt or Motif or Tk or Swing or XForms, and the modern inconsistent mess where everything is Qt or WxWidgets or Swing or some WebKit container with its own pile of gross stylesheet fruit salad.

If you had a little script that you wanted to put a GUI on back then, the process for getting started was apt-get install python-gtk and then just to do something like

1
2
3
4
5
import gtk
w = gtk.Window("My Window")
b = gtk.Button("My Button")
w.add(b)
gtk.main()

and you were pretty much off to the races. If you wanted to, you could load a UI file that you made with glade, if you had some complicated fancy stuff to display, but that was optional and reasonably straightforward to use. I used to do that all the time, for various personal computing tasks. Of course, if I did that today, I'm sure they'd all laugh at me.

So today I'd like to show you how to do the same sort of thing with OS X.

To be clear, this is not a tutorial in Objective C, or a deep dive into Mac application programming. I'm going to make some assumptions about your skills: you're a relatively experienced Python programmer, you've done some introductory Xcode material like the Temperature Converter tutorial, and you either already know or don't mind figuring out the PyObjC bridged method naming conventions on your own.

If you are starting from scratch, and you don't have any code yet, and you want to work in Objective C or Swift, the modern Mac development experience is even smoother than what I just described. You don't need to install any "packages", just Xcode, and you have a running app before you've written a single line of code, because it will give you a working-by-default template.

The problem is, if you're a Python developer just trying to make a little utility for your own use, once you've got this lovely Objective C or Swift application ready for you to populate with interesting stuff, it's a really confusing challenge to get your Python code in there. You can drag the Python.framework from homebrew into Xcode and then start trying to call some C functions like PyRun_SimpleString to get your program bootstrapped, but then you start running into tons of weird build issues. How do you copy your scripts to the app bundle? How do you invoke distutils? What about shared libraries that you've linked against? It's hard enough to try to jam anything like setuptools or pip into the Xcode build system that you might as well give up and rewrite the logic; it'll be less effort.

If you're a Python developer you probably expect tools like virtualenv and pip to "just work". You expect to be able to put a file into a folder and then import it without necessarily writing a whole build toolchain first. And if you haven't worked with it a lot, you probably find Xcode's UI bewildering. Not to mention the fact that you probably already have a text editor that you like already, and don't want to spend a bunch of time coming up to speed on a new one just to display a button.

Luckily, of course, there's pyobjc, which lets you write your whole application in Python, skipping the whole Xcode / Objective C / Swift side-show and just add a little UI logic. The problem I'm trying to address here is that "just adding a little UI logic" (rather than designing your program from the ground up as an App) in the Cocoa / ObjC universe is famously and maddeningly obscure. gtk.Window() is a reasonably straightforward first function to call; [[NSWindow alloc] initWithContentRect:styleMask:backing:defer:] not so much.

This is not even to mention the fact that all the documentation, all the tutorials, and all the community resources pretty much expect you to be working with .xibs or storyboards inside the Interface Builder portion of Xcode, and you're really on your own if you are trying to do everything outside of that. py2app can help with some of the build issues, but it has its own problems you probably don't want to be tackling if you're just getting started; it'll sap all your energy for actually coding.

I should mention before we finally dive in here that if you really just want to display a button, a text area, maybe a few fields, Toga is probably a better option for you than the route that I'm describing in this post; it's certainly a lot less work. What I am assuming here is that you want to be able to present a button at first, but gradually go on to mess around with arbitrary other OS X native APIs to experiment with the things you can do with a desktop that you can't do in the cloud, like displaying local notifications, tracking your location, recording the screen, controlling other applications, and so on.

So, what is an enterprising developer - who is still lazy enough for my rhetorical purposes here - to do? Surprisingly enough, I have a suggestion!

First, you want to make a new, empty Xcode project. So fire up Xcode, go to File, New, Project, and then:

Just-A-Button-1

Just-A-Button-2

Go ahead and give it a Git repository:

Just-A-Button-3

Now that you've got a shiny new blank project, you'll need to create two resources in it: one, a user interface document, and the other, a Python file. So select File, New, and then choose OS X, User Interface, Empty:

Just-A-Button-4

I'm going to call this "MyUI", but you can call it whatever:

Just-A-Button-5

As you can see, this creates a MyUI.xib file in your project, with nothing in it.

Just-A-Button-6

We want to put a window and a button into it, so, let's start with that. Search for "window" in the bottom right, and drag the "Window" item onto the canvas in the middle:

Just-A-Button-7

Just-A-Button-8

Now we've got a UI file which contains a window. How can we display it? Normally, Xcode sets all this up for you, creating an application bundle, a build step to compile the .xib into a .nib, to copy it into the appropriate location, and code to load it for you. But, as we've discussed above, we're too lazy for all that. Instead, we're going to create a Python script that compiles the .xib automatically and then loads it.

You can do this with your favorite text editor. The relevant program looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
import os
os.system("ibtool MyUI.xib --compile MyUI.nib")

from Foundation import NSData
from AppKit import NSNib

nib_data = NSData.dataWithContentsOfFile_(u"MyUI.nib")
(NSNib.alloc().initWithNibData_bundle_(nib_data, None)
 .instantiateWithOwner_topLevelObjects_(None, None))

from PyObjCTools.AppHelper import runEventLoop
runEventLoop()

Breaking this down one step at a time, what it's doing is:

1
2
import os
os.system("ibtool MyUI.xib --compile MyUI.nib")

We run Interface Builder Tool, or ibtool, to convert the xib, which is a version-control friendly, XML document that Interface Builder can load, into a nib, which is a binary blob that AppKit can load at runtime. We don't want to add a manual build step, so for now we can just have this script do its own building as soon as it runs.

Next, we need to load the data we just compiled:

1
nib_data = NSData.dataWithContentsOfFile_(u"MyUI.nib")

This needs to be loaded into an NSData because that's how AppKit itself wants it prepared.

Finally, it's time to load up that window we just drew:

1
2
(NSNib.alloc().initWithNibData_bundle_(nib_data, None)
 .instantiateWithOwner_topLevelObjects_(None, None))

This loads an NSNib (the "ib" in its name also refers to "Interface Builder") with the init... method, and then creates all the objects inside it - in this case, just our Window object - with the instantiate... method. (We don't care about the bundle to use, or the owner of the file, or the top level objects in the file yet, so we are just leaving those all as None intentionally.)

Finally, runEventLoop() just runs the event loop, allowing things to be displayed.

Now, in a terminal, you can run this program by creating a virtualenv, and doing pip install pyobjc-framework-Cocoa, and then python button.py. You should see your window pop up - although it will not take focus. Congratulations, you've made a window pop up!

One minor annoyance: you're probably used to interrupting programs with ^C on the command line. In this case, the PyObjC helpers catch that signal, so instead you will need to use ^\ to hard-kill it until we can hook up some kind of "quit" functionality. This may cause crash dialogs to pop up if you don't use virtualenv; you can just ignore them.

Of course, now that we've got a window, we probably want to do something with it, and this is where things get tricky. First, let's just create a button; drag the button onto your window, and save the .xib:

Just-A-Button-9

And now, the moment of truth: how do we make clicking that button do anything? If you've ever done any Xcode tutorials or written ObjC code, you know that this is where things get tricky: you need to control-drag an action to a selector. It figures out which selectors are available by magically knowing things about your code, and you can't just click on a thing in the interface-creation component and say "trust me, this method exists". Luckily, although the level of documentation for it these days makes it on par with an easter egg, Xcode has support for doing this with Python classes just as it does with Objective C or Swift classes.1

First though, we'll need to tell Xcode our Python file exists. Go to the "File" menu, and "Add files to...", and then select your Python file:

Just-A-Button-10

Here's the best part: you don't need to use Xcode's editor at all; Xcode will watch that file for changes. So keep using your favorite text editor and change button.py to look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
import os
os.system("ibtool MyUI.xib --compile MyUI.nib")

from objc import IBAction
from Foundation import NSObject, NSData
from AppKit import NSNib

class Clicker(NSObject):
    @IBAction
    def clickMe_(self, sender):
        print("Clicked!")

the_clicker = Clicker.alloc().init()

nib_data = NSData.dataWithContentsOfFile_(u"MyUI.nib")
(NSNib.alloc().initWithNibData_bundle_(nib_data, None)
 .instantiateWithOwner_topLevelObjects_(the_clicker, None))

from PyObjCTools.AppHelper import runEventLoop
runEventLoop()

In other words, add a Clicker subclass of NSObject, give it a clickMe_ method decorated by objc.IBAction, taking one argument, and then make it do something you can see, like print something. Then, make a global instance of it, and pass it as the owner parameter to NSNib.

At this point it would probably be good to explain a little about what the "file's owner" is and how loading nibs works.

When you instantiate a Nib in AppKit, you are creating a collection of graphical objects, connected to an object in your program that you construct. In other words, if you had a program that displayed a Person, you'd have a Person.nib and a Person class, and each time you wanted to show a Person you'd instantiate the Nib again with the new Person as the owner of that Nib. In the interface builder, this is represented by the "file's owner" placeholder.

I am explaining this because if you're interested in reading this article, you've probably been interested enough in Mac programming to do something like the aforementioned Temperature Converter tutorial, but such tutorials almost universally just use the single default "main" nib that gets loaded when the application launches, so although they show you how to do many different things with UI elements, it's not clear how the elements got there. This, here, is how you make new UI elements get there in the first place.

Back to our clicker example: now that we have a class with a method, we need to tell Xcode that the clicking the button should call that method. So what we're going to tell Xcode is that we expect this Nib to be owned by an instance of Clicker. To do this, go to MyUI.xib and select the "File's Owner" (the thing that looks like a transparent cube), to to the "Identity Inspector" (the tiny icon that looks like a driver's license on the right) and type "Clicker" in the "Class" field at the top.

If you've properly added button.py to your project and declared that class (as an NSObject), it should automatically complete as you start typing the name:

Just-A-Button-11

Now you need to connect your clickMe action to the button you already created. If you've properly declared your method as an IBAction, you should see it in the list of "received actions" in the "Connections Inspector" of the File's Owner (the tiny icon that looks like a right-pointing arrow in a circle):

Just-A-Button-12

Drag from the circle to the right of the clickMe: method there to the button you've created, and you should see the connection get formed:

Just-A-Button-13

If you save your xib at this point and re-run your python file, you should be able to click on the button and see something happen.

Finally, we want to be able to not just get inputs from the GUI, but also produce outputs. To do this, we want to populate an outlet on our Clicker class with a pointer to an object in the Nib. We can do this by declaring a variable as an objc.IBOutlet(); simply add a from objc import IBOutlet, and change Clicker to read:

1
2
3
4
5
class Clicker(NSObject):
    label = IBOutlet()
    @IBAction
    def clickMe_(self, sender):
        self.label.setStringValue_(u"\N{CHECK MARK}")

In case you're wondering where setStringValue_ comes from, it's a method on NSTextField, since labels are NSTextFields.

Then we can place a label into our xib; and we can see it is in fact an NSTextField in the Identity Inspector:

Just-A-Button-14

I've pre-filled mine out with a unicode "BALLOT X" character, for style points.

Then, we just need to make sure that the label attribute of Clicker actually points at this value; once again, select the File's Owner, the Connections Inspector, and (if you declared your IBOutlet correctly), you should see a new "label" outlet. Drag the little circle to the right of that outlet, to the newly-created label object, and you should see the linkage in the same way the action was linked:

Just-A-Button-15

And there you have it! Now you have an application that can open a window, take input, and display output.

This is, as should be clear by now, not really the preferred way of making an application for OS X. There's no app bundle, so there's nothing to code-sign. You'll get weird behavior in certain cases; for example, as you've probably already noticed, the window doesn't come to the front when you launch the app like you might expect it to. But, if you're a Python programmer, this should provide you with a quick scratch pad where you can test ideas, learn about how interface builder works, and glue a UI to existing Python code quickly before wrestling with complex integration and distribution issues.

Speaking of interfacing with existing Python code, of course you wouldn't come to this blog and expect to get technical content without just a little bit of Twisted in it. So here's how you hook up Twisted to an OS X GUI: instead of runEventLoop, you need to run your application like this:

1
2
3
4
5
6
7
# Before importing anything else...
from PyObjCTools.AppHelper import runEventLoop
from twisted.internet.cfreactor import install
reactor = install(runner=runEventLoop)

# ... then later, when you want to run it ...
reactor.run()

In your virtualenv, you'll want to pip install twisted[osx_platform] to get all the goodies for OS X, including the GUI integration. Since all Twisted callbacks run on the main UI thread, you don't need to know anything special to do stuff; you can call methods on whatever UI objects you have handy to make changes to them.

Finally, although I definitely don't have room in this post to talk about all the edge cases here, I will address two particularly annoying ones; often if you're writing a little app like this, you really want it to take keyboard focus, and by default this window will come up in the background. To fix that, you can do this right before starting the main loop:

1
2
3
4
from AppKit import NSApplication, NSApplicationActivationPolicyAccessory
app = NSApplication.sharedApplication()
app.setActivationPolicy_(NSApplicationActivationPolicyAccessory)
app.activateIgnoringOtherApps_(True)

And also, closing the window won't quit the application, which might be pretty annoying if you want to get back to using your terminal, so a quick fix for that is:

1
2
3
4
class QuitWhenClosed(NSObject):
    def applicationShouldTerminateAfterLastWindowClosed_(self, app):
        return True
app.setDelegate_(QuitWhenClosed.alloc().init().retain())

(the "retain" is necessary because ObjC is not a garbage collected language, and app.delegate is a weak reference, so the QuitWhenClosed would be immediately freed (and there would be a later crash) if you didn't hold it in a global variable or call retain() on it.)

You might have other problems with this technique, and this post definitely can't fit solutions to all of them, but now that you can load a nib, create classes that interface builder can understand, and run a Twisted main loop, you should be able to use the instructions in other tutorials and resources relatively straightforwardly.

Happy hacking!


(Thanks to everyone who helped with this post. Of course my errors are entirely my own, but thanks especially to Ronald Oussoren for his tireless work on pyObjC and py2app over the years, as well as to Mark Eichin and Amber Brown for some proofreading and feedback before this was posted.)


  1. If it didn't, we could get Xcode to produce the appropriate XML by simply writing appropriately-shaped classes in Objective C - or possibly Swift - and then never bothering to build them, since all Xcode cares about in this part of the process is that it can see the source. My understanding is that's how it worked when PyObjC was first developed, and it might always become necessary again if Xcode ever stops supporting Python. I have no idea if that's likely, but unfortunately it seems clear Python isn't a very popular language for mac apps.

17 Jul 2015 7:52am GMT

10 Jul 2015

feedPlanet Twisted

Duncan McGreggor: Mastering matplotlib: Acknowledgments

The Book

Well, after nine months of hard work, the book is finally out! It's available both on Packt's site and Amazon.com. Getting up early every morning to write takes a lot of discipline, it takes even more to say "no" to enticing rabbit holes or herds of Yak with luxurious coats ripe for shaving ... (truth be told, I still did a bit of that).

The team I worked with at Packt was just amazing. Highly professional and deeply supportive, they were a complete pleasure with which to collaborate. It was the best experience I could have hoped for. Thanks, guys!

The technical reviewers for the book were just fantastic. I've stated elsewhere that my one regret was that the process with the reviewers did not have a tighter feedback loop. I would have really enjoyed collaborating with them from the beginning so that some of their really good ideas could have been integrated into the book. Regardless, their feedback as I got it later in the process helped make this book more approachable by readers, more consistent, and more accurate. The reviewers have bios at the beginning of the book -- read them, and look them up! These folks are all amazing!

The one thing that slipped in the final crunch was the acknowledgements, and I hope to make up for that here, as well as through various emails to everyone who provided their support, either directly or indirectly.

Acknowledgments

The first two folks I reached out to when starting the book were both physics professors who had published very nice matplotlib problems -- one set for undergraduate students and another from work at the National Radio Astronomy Observatory. I asked for their permission to adapt these problems to the API chapter, and they graciously granted it. What followed were some very nice conversations about matplotlib, programming, physics, education, and publishing. Thanks to Professor Alan DeWeerd, University of Redlands and Professor Jonathan W. Keohane, Hampden Sydney College. Note that Dr. Keohane has a book coming out in the fall from Yale University Press entitled Classical Electrodynamics -- it will contain examples in matplotlib.

Other examples adapted for use in the API chapter included one by Professor David Bailey, University of Toronto. Though his example didn't make it into the book, it gets full coverage in the Chapter 3 IPython notebook.

For one of the EM examples I needed to derive a particular equation for an electromagnetic field in two wires traveling in opposite directions. It's been nearly 20 years since my post-Army college physics, so I was very grateful for the existence and excellence of SymPy which enabled me to check my work with its symbolic computations. A special thanks to the SymPy creators and maintainers.

Please note that if there are errors in the equations, they are my fault! Not that of the esteemed professors or of SymPy :-)

Many of the examples throughout the book were derived from work done by the matplotlib and Seaborn contributors. The work they have done on the documentation in the past 10 years has been amazing -- the community is truly lucky to have such resources at their fingertips.

In particular, Benjamin Root is an astounding community supporter on the matplotlib mail list, helping users of every level with all of their needs. Benjamin and I had several very nice email exchanges during the writing of this book, and he provided some excellent pointers, as he was finishing his own title for Packt: Interactive Applications Using Matplotlib. It was geophysicist and matplotlib savant Joe Kington who originally put us in touch, and I'd like to thank Joe -- on everyone's behalf -- for his amazing answers to matplotlib and related questions on StackOverflow. Joe inspired many changes and adjustments in the sample code for this book. In fact, I had originally intended to feature his work in the chapter on advanced customization (but ran out of space), since Joe has one of the best examples out there for matplotlib transforms. If you don't believe me, check out his work on stereonets. There are many of us who hope that Joe will be authoring his own matplotlib book in the future ...

Olga Botvinnik, a contributor to Seaborn and PhD candidate at UC San Diego (and BioEng/Math double major at MIT), provided fantastic support for my Seaborn questions. Her knowledge, skills, and spirit of open source will help build the community around Seaborn in the years to come. Thanks, Olga!

While on the topic of matplotlib contributors, I'd like to give a special thanks to John Hunter for his inspiration, hard work, and passionate contributions which made matplotlib a reality. My deepest condolences to his family and friends for their tremendous loss.

Quite possibly the tool that had the single-greatest impact on the authoring of this book was IPython and its notebook feature. This brought back all the best memories from using Mathematica in school. Combined with the Python programming language, I can't imagine a better platform for collaborating on math-related problems or producing teaching materials for the same. These compliments are not limited to the user experience, either: the new architecture using ZeroMQ is a work of art. Nicely done, IPython community! The IPython notebook index for the book is available in the book's Github org here.

In Chapters 7 and 8 I encountered a bit of a crisis when trying to work with Python 3 in cloud environments. What was almost a disaster ended up being rescued by the work that Barry Warsaw and the rest of the Ubuntu team did in Ubuntu 15.04, getting Python 3.4.2 into the release and available on Amazon EC2. You guys saved my bacon!

Chapter 7's fictional case study examining the Landsat 8 data for part of Greenland was based on one of Milos Miljkovic's tutorials from PyData 2014, "Analyzing Satellite Images With Python Scientific Stack". I hope readers have just as much fun working with satellite data as I did. Huge thanks to NASA, USGS, the Landsat 8 teams, and the EROS facility in Sioux Falls, SD.

My favourite section in Chapter 8 was the one on HDF5. This was greatly inspired by Yves Hilpisch's presentation "Out-of-Memory Data Analytics with Python". Many thanks to Yves for putting that together and sharing with the world. We should all be doing more with HDF5.

Finally, and this almost goes without saying, the work that the Python community has done to create Python 3 has been just phenomenal. Guido's vision for the evolution of the language, combined with the efforts of the community, have made something great. I had more fun working on Python 3 than I have had in many years.

10 Jul 2015 9:35pm GMT

27 Jun 2015

feedPlanet Twisted

Moshe Zadka: ncolony 0.0.2 released

Fixed:

Mostly internal cleanup release: running scripts is now nicer (all through "python -m ncolony <command>"), added Code of Conduct, releasing based on versioneer, cleaned up tox.ini, added HTTP healthchecker.

Available via PyPI and GitHub!

27 Jun 2015 4:38am GMT

26 Jun 2015

feedPlanet Twisted

Moshe Zadka: mainland: the main of your Python

I don't like console-scripts. Among what I dislike is their magicality and the actual code produced. I mean, the autogenerated Python looks like:

#!/home/moshez/src/mainland/build/toxenv/bin/python2

# -*- coding: utf-8 -*-
import re
import sys

from tox import cmdline

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(cmdline())

Notice the encoding header for an ASCII only file, and the weird windows compatibility maybe generated along with a unix-style header that also contains the name of the path? Truly, a file that only its generator could love.

Then again, I don't much like what we have in Twisted, with the preamble:

#!/home/moshez/src/ncolony/build/env/bin/python2
# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.
import os, sys

try:
    import _preamble
except ImportError:
    sys.exc_clear()

sys.path.insert(0, os.path.abspath(os.getcwd()))

from twisted.scripts.twistd import run
run()

This time, it's not auto-generated, it is artisanal. What it lacks in codegen ugliness it makes up in importing an under-module at top-level, and who doesn't like a little bit of sys.path games?

I will note that furthermore, neither of these options would play nice with something like pex. Worst of all, all of these options, regardless of their internal beauty (or horrible lack thereof), must be named veeeeery caaaaarefully to avoid collision with existing UNIX, Mac or Windows command. Think this is easy? My Ubuntu has two commands called "chromium-browser" and "chromium-bsu" because Google didn't check what popular games there are on Ubuntu before naming their browser.

Enter the best thing about Python, "-m" which allows executing modules. What "-m" gains in awesomeness, it loses in the horribleness of the two-named-module, where a module is executed twice, once resulting in sys.modules['__main__'] and once in sys.modules[real_name], with hilarious consequences for class hierarchies, exceptions and other things relying on identity of defined things.

Luckily, packages will automatically execute "package.__main__" if "python -m package" is called. Ideally, nobody would try to import __main__, but this means packages can contain only one command. Enter 'mainland', which defines a main which will, carefully and accurately, dispatch to the right module's main without any code generation. It has not been released yet, but is available from GitHub. Pull requests and issues are happily accepted!

Edit: Released! 0.0.1 is up on PyPI and on GitHub

26 Jun 2015 3:13am GMT

23 Jun 2015

feedPlanet Twisted

Moshe Zadka: On Codes of Conducts and “Protection”

Related: In Favor of Niceness, Community, and Civilization

I've seen elsewhere (thankfully, not my project, and no, I won't link to it) that people want the code of conduct to "protect contributors from 3rd parties as well as 3rd parties from contributors. I would like to first suggest the idea that this is…I don't even know. The US government has literal nuclear bombs, as well as aircraft carriers, and it cannot enforce its law everywhere. The idea that an underfunded OSS project can do literally anything to 3rd parties is ludicrous. The only enforcement mechanism any project can have is "you can't contribute here" (which, by necessity, only applies to those who wanted to contribute in the first place.)

So why would a bunch of people take on a code of conduct that will only limit them?

Because, to quote the related article above, "the death cry of liberalism is not 'death to the unbelievers', it is 'if you're nice, you can join our cuddle pile." Perhaps describing open source projects as "cuddle piles" is somewhat of an exaggeration, but the point remains. A code of conduct defines what "nice enough is". The cuddle piles? Those conquer the world. WebKit is powering both Android and iPhone browsers, for example, making open source be, literally, the way people see the online world.

Adopting a code of conduct that says that we do not harass, we do not retaliate and we accept all people is a powerful statement. This is how we keep our own garden clear of the pests of prejudice, of hatred. Untended gardens end up wilderness. Well-tended gardens grow until we need to keep a fence around the wild and call it a "preserve".

When I first saw the Rust code of conduct I thought "cool, but why does an open source project need a code of conduct"? Now I wonder how any open source project will survive without one. It will have to survive without me - in three years, I commit to not contribute to any project without a code of conduct, and I hope others will follow. Any project that does not have a CoC, in the interim, better behave as though it has one.

Twisted is working hard on adopting a code of conduct, and I will check one into NColony soon (a simpler one, appropriate to what is, so far, a one-person show).

23 Jun 2015 4:22am GMT

18 Jun 2015

feedPlanet Twisted

Moshe Zadka: Everything but the code

(Or, "So you want to build a new open-source Python project")

This is not going to be a "here is a list of options, but you do what is right for you" pandering post. This is going to be a "this is 2015, there are right ways to do things, and here they are" post. This is going to be an opinionated, in-your-face, THIS IS BEST post. That said, if you think that anything here does not represent the best practices in 2015, please do leave a comment. Also, if you have specific opinions on most of these, you're already ahead of the curve - the main target audience are people new to creating open-source Python projects, who could use a clear guide.

This post will also emphasize the new project. It is not worth it, necessarily, to switch to these things in an existing project - certainly not as a "whole-sale, stop the world and let's change" thing. But when starting a new project with zero legacy, there is no reason to do things wrong.

tl:dr; Use GitHub, put MIT license in "LICENSE", README.rst with badges, use py.test and coverage, flake8 for static checking, tox to run tests, package with setuptools, document with sphinx on RTD, Travis CI/Coveralls for continuous integration, SemVer and versioneer for versioning, support Py2+Py3+PyPy, avoid C extensions, avoid requirements.txt.

Where

When publishing kids' photos, you do it on Facebook, because that's where everybody is. LinkedIn is where you connect with colleagues and lead your professional life. When publishing projects, put them on GitHub, for exactly the same reason. Have GitHub pull requests be the way contributors propose changes. (There is no need to use the "merge" button on GitHub - it is fine to merge changes via git and push. But the official "how do I submit a change" should be "pull request", because that's what people know).

License

Put the license in a file called "LICENSE" at the root. If you do not have a specific reason to choose otherwise, MIT is reasonably permissive and compatible. Otherwise, use something like the license chooser and remember the three most important rules:

At the end of the license file, you can have a list of the contributors. This is an easy place to credit them. It is a good idea to ask people who send in pull requests to add themselves to the contributor list in their first one (this allows them to spell their name and e-mail exactly the way they want to).

Note that if you use the GPL or LGPL, they will recommend putting it in a file called "COPYING". Put it in "LICENSE" (the licenses explicitly allow it as an option, and it makes it easier for people to find the license if it always has the same name).

README

The GitHub default is README.md, but README.rst (restructured text) is perfectly supported via Sphinx, and is a better place to put Python-related documentation, because ReST plays better with Pythonic toolchains. It is highly encouraged to put badges on top of the document to link to CI status (usually Travis), ReadTheDocs and PyPI.

Testing

There are several reasonably good test runners. If there is no clear reason to choose one, py.test is a good default. "Using Twisted" is a good reason to choose trial. Using the built-in unittest runner is not a good option - there is a reason the cottage industry of "test runner" evolved. Using coverage is a no-brainer. It is good to run some functional tests too. Test runners should be able to help with this too, but even writing a Python program that fails if things are not working can be useful.

Distribute your tests alongside your code, by putting them under a subpackage called "tests" of the main package. This allows people who "pip install …" to run the tests, which means sending you bug reports is a lot easier.

Static checking

There are a lot of tools for static checking of Python programs - pylint, flake8 and more. Use at least one. Using more is not completely free (more ways to have to say "ignore this, this is ok") but can be useful to catch more style static issue. At worst, if there are local conventions that are not easily plugged into these checkers, write a Python program that will check for them and fail if those are violated.

Meta testing

Use tox. Put tox.ini at the root of your project, and make sure that "tox" (with no arguments) works and runs your entire test-suite. All unit tests, functional tests and static checks should be run using tox. It is not a bad idea to write a tox clause that builds and tests an installed wheel. This will require including all test code in the deployed package, which is a good idea.

Set tox to put all build artifacts in a build/ top-level directory.

Packaging

Have a setup.py file that uses setuptools. Tox will need it anyway to work correctly.

Structure

It is unlikely that you have a good reason to take more than one top-level name in the package namespace. Barring completely unavoidable name conflicts, your PyPI package name should be the same as your Python package name should be the same as your GitHub project. Your Python package should live at the top-level, not under "src/" or "py/".

Documentation

Use sphinx for prose documentation. Put it in doc/ with a relevant conf.py. Use either pydoctor or sphinx autodoc for API docs. "Pydoctor" has the potential for nicer docs, sphinx is well integrated with ReadTheDocs. Configure ReadTheDocs to auto-build the documentation every check-in.

Continuous Integration

If you enjoy owning your own machines, or platform diversity in testing really matters, use buildbot. Otherwise, take advantage for free Travis CI and configure your project with a .travis.yml that breaks your tox tests into one test per Travis clause. Integrate with coveralls to have coverage monitored.

Version management

Use SemVer. Take advantage of versioneer to help you manage it.

Release

A full run of "tox" should leave in its wake tested .zip and .whl files. A successful, post-tag run of tox, combined with versioneer, should leave behind tested .zip and .whl. The release script could be as simple as "tox && git tag $1 && (tox || (git tag -d $1;exit 1) && cp …whl and zip locations… dist/"

GPG sign dist/ files, and then use "twine" to upload them to PyPI. Make sure to upload to TestPyPI first, and verify the upload, before uploading to PyPI. Twine is a great tool, but badly documented - among other things, it is hard to find information about .pypirc. ".pypirc" is an ini file, which needs to have the following sections:

.gitignore

Python versions

If all your dependencies support Python 2 and 3, support Python 2 and 3. That will almost certainly require using "six" (or one of its competitors, like "future"). Run your unit tests under both Python 2 and 3. Make sure to run your unit tests under PyPy, as well.

C extensions

Avoid, if possible. Certainly do not use C extensions for performance improvements before (1) making sure they're needed (2) making sure they're helpful (3) trying other performance improvements. Ideally structure your C extensions to be optional, and fall back to a slow(er) Python implementation if they are unavailable. If they speed up something more general than your specific needs, consider breaking them out into a C-only project which your Python will depend on.

If using C extensions, regardless of whether to improve performance or integrate with 3rd party libraries, use CFFI.

If C extensions have successfully been avoided, and Python 3 compatibility kept, build universal wheels.

requirements.txt

The only good "requirements.txt" file is a non-existing one. The "setup.py" file should have the dependencies (ideally as weak-versioned as possible, usually just a ">=" for a library that tends not to break backwards compatibility a lot). Tox will maintain the virtualenv needed based on the things in the tox file, and if needing to test against specific versions, this is where specific versions belong. The "requirements.txt" file belongs in Salt-style (Chef, Fab, …) configurations, Docker-style (Vagrant-, etc.) configurations or Pants-style (Buck-, etc.) build scripts when building a Pex. This is the "deployment configuration", and needs to be decided by a deployer.

If your package has dependencies, they belong in a setup.py. Use extended_dependencies for test-only dependencies. Lists of test dependencies, and reproducible tests, can be configured in tox.ini. Tox will take care of pip-installing relevant packages in a virtualenv when running the tests.

Thanks

Thanks to John A. Booth, Dan Callahan, Robert Collins, Jack Diedrich, Steve Holden for their reviews and insightful comments. Any mistakes that remain are mine!

18 Jun 2015 2:31am GMT

09 Jun 2015

feedPlanet Twisted

Glyph Lefkowitz: Sorry I Unfollowed You

Since Alex Gaynor wrote his seminal thinkpiece on the subject, "I Hope Twitter Goes Away", I've been wrestling to define my relationship to this often problematic product.

On the one hand, Twitter has provided me with delightful interactions with human beings who I would not otherwise have had the opportunity to meet or interact with. If you are the sort of person who likes following people, four suggestions I'd make on that front are Melissa 🔔, Gary Bernhardt, Eevee and Matt Blaze, all of whom have blogs but none of whom I would have discovered without Twitter.

Twitter has also allowed me to reach a larger audience with my writing than I otherwise would have been able to. Lots of people click on links to this blog from Twitter either from following me directly or from a retweet. (Thank you, retweeters, one and all.)

On the other hand, the effect of using Twitter on my productivity is like having a constant, low-grade headache. While Twitter has never been a particularly bad distraction as measured by hours spent on it (I keep metrics on that, and it's rarely even in the top 10), I feel like consulting Twitter is something I do when I am stuck, or having to think about something hard. "I'll just check Twitter" is an easy way to "take a break" right at the moment that I ought to be thinking harder, eliminating distractions, mustering my will to focus.

This has been particularly stark for me as I've been trying to get some real writing done over the last couple of weeks and have been consistently drawing a blank. Given that I have a deadline coming up on Wednesday and another next Monday, something had to give.

Or, as Joss Whedon put it, when he quit Twitter:

If I'm going to start writing again, I have to go to the quiet place, and this is the least quiet place I've ever been in my life.

I'm an introvert, and using Twitter is more like being at a gigantic, awkward party all the time than any other online space I've ever been in.

There's an irony here. Mostly what people like that I put on Twitter (and yes, I've checked) are announcements that link to other things, accomplishments in other areas, like a blog post, or a feature in Twisted, but using Twitter itself is inimical to completing those things.

I'm loath to abandon the positive aspects of Twitter. Some people also use Twitter as a replacement for RSS, and I don't want to break the way they choose to pay attention to the stuff that I do. And a few of my friends communicate exclusively through direct messages.

The really "good" thing about Twitter is discovery. It enables you to discover people, content, and, eugh, "brands" that appeal to you. I have discovered things that I enjoy many times. The fundamental problem I am facing, which is a little bit hard to admit to oneself, is that I have discovered enough. I have enough games to play, enough books and articles to read, enough podcasts to listen to, enough movies to watch, enough code to write, enough open source libraries to investigate, that I will be busy for years based on what I already know.

For me, using Twitter's timeline at this point to "discover" more things is like being at a delicious buffet, being so full I'm nauseous, and stuffing my pockets with shrimp "just in case" I'm hungry "when I get home" - and then, of course, not going home.

Even disregarding my desire to produce useful content, if I just want to enjoy consuming content more deeply, I have to take the time to engage with it properly.

So here's what I'm doing:

  1. I am turning on the "anyone can direct message me" feature. We'll see how that goes; I may have to turn it off again later. As always, I'd prefer you send email (or text me, if it's time-critical).
  2. I am unfollowing literally everyone, and will not follow people in the future. Checking my timeline was the main information junk-food I want to avoid.
  3. Since my timeline, rather than mentions and replies, was my main source of distraction, I'll continue paying attention to mentions and replies (at least for now; I'll have to see if that becomes a problem in the absence of a timeline).
  4. In order to avoid producing such information junk-food myself, I'm going to try to directly tweet less, and put more things into brief blog posts so I have enough room to express them. I won't say "not at all", but most of the things that I put on Twitter would really be better as longer, more thoughtful articles.

Please note that there's nothing prescriptive here. I'm outlining what I'm doing in the hopes that others might recognize similar problems with themselves - if everyone used Twitter this way, there would hardly be a point to the site.

Also, if I've unfollowed you, that doesn't mean I'm not interested in what you have to say. I already have a way of keeping in touch with people's more fully-formed ideas: I use Blogtrottr to deliver relevant blog articles to my email. If I previously followed you and you think I might not be reading your blog already (in most cases I believe I already am), please feel free to drop me a line with an RSS link.

09 Jun 2015 12:41am GMT

07 Jun 2015

feedPlanet Twisted

Twisted Matrix Laboratories: Twisted Fellowship 2015: Call for proposals

On behalf of the Software Freedom Conservancy and the Twisted project I'm happy to announce that we're looking for a paid maintainer for the Twisted project.

Funding a software developer to work as a maintainer will help Twisted grow as a project, and enable Twisted's development community to increase their output of innovative code for the public's benefit. Twisted has strict coding, testing, documentation, and review standards, which ensures excellent code quality, continually improving documentation and code test coverage, and minimal regressions. Code reviews are historically a bottleneck for getting new code merged. A funded maintainer will help alleviate this bottleneck, and speed Twisted's development.

You can read more about the 2015 fellowship at https://twistedmatrix.com/trac/wiki/Fellowship2015

07 Jun 2015 11:12pm GMT

06 Jun 2015

feedPlanet Twisted

Moshe Zadka: (Somewhat confused) Thoughts on Languages, Eco-systems and Development

There are a few interesting new languages that would be fun to play with: Rust, Julia, LFE, Go and D all have interesting takes on some domain. But ultimately, a language is only as interesting as the things it can do. It used to be that "things it can do" referred to the standard library. I am old enough to remember when "batteries included" was one of the most interesting benefits Python had - the language had dared doing such avant-garde things in '99 as having a *built-in* url fetching library. That behaved, pretty much, the same on all platform. Out of the box. (It's hard to convey how amazing this was in '99.)

This is no longer the case. Now what distinguishes Python as "can do lots of things" is its built-in package management. Python, pip and virtualenv together give the ability to work on multiple Python projects, that need different versions of libraries, without any interference. They are, to a first approximation, 100% reliable. With pip 7.0 supporting caching wheels, virtualenvs are even more disposable (I am starting to think pip uninstall is completely unnecessary). In fact, except for virtualenv itself, it is rare nowadays to install any Python module globally. There are some rusty corners, of course:

I'll note npm, for example, clearly does better on these three (while doing worse on some things that Python does better). Of the languages mentioned above, it is nice to see that most have out-of-the-box built-in tools for ecosystem creation. Julia's hilariously piggybacks on GitHub's. Go's…well…uses URLs as the equivalent PyPI-level-names. Rust has a pretty decent system in cargo and crates.io (the .toml file is kind of weird, but I've seen worse). While there is justifiable excitement about Rust's approach to memory and thread-safety, it might be that it will win in the end based on having a better internal package management system than the low-level alternatives.

Note: OS package mgmt (yum, apt-get, brew and whatever Windows uses nowadays) is not a good fit for what developers need from a language-level package manager. Locality, quick package refreshes - these matter more to developers than to OS end-users.

06 Jun 2015 7:55am GMT

05 Jun 2015

feedPlanet Twisted

Duncan McGreggor: Scientific Computing and the Joy of Language Interop

The scientific computing platform for Erlang/LFE has just been announced on the LFE blog. Though written in the Erlang Lisp syntax of LFE, it's fully usable from pure Erlang. It wraps the new py library for Erlang/LFE, as well as the ErlPort project. More importantly, though, it wraps Python 3 libs (e.g., math, cmath, statistics, and more to come) and the ever-eminent NumPy and SciPy projects (those are in-progress, with matplotlib and others to follow).

(That LFE blog post is actually a tutorial on how to use lsci for performing polynomial curve-fitting and linear regression, adapted from the previous post on Hy doing the same.)

With the release of lsci, one can now start to easily and efficiently perform computationally intensive calculations in Erlang/LFE (and any other Erlang Core-compatible language, e.g., Elixir, Joxa, etc.) That's super-cool, but it's not quite the point ...

While working on lsci, I found myself experiencing a great deal of joy. It wasn't just the fact that supervision trees in a programming language are insanely great. Nor just the fact that scientific computing in Python is one of the best in any language. It wasn't only being able to use two syntaxes that I love (LFE and Python) cohesively, in the same project. And it wasn't the sum of these either -- you probably see where I'm going with this ;-) The joy of these and many other fantastic aspects of inter-operation between multiple powerful computing systems is truly greater than the sum of its parts.

I've done a bunch of Julia lately and am a huge fan of this language as well. One of the things that Julia provides is explicit interop with Python. Julia is targeted at the world of scientific computing, aiming to be a compelling alternative to Fortran (hurray!), so their recognition of the enormous contribution the Python scientific computing community has made to the industry is quite wonderful to see.

A year or so ago I did some work with Clojure and LFE using Erlang's JInterface. Around the same time I was using LFE on top of Erjang, calling directly into Java without JInterface. This is the same sort of Joy that users of Jython have, and there are many more examples of languages and tools working to take advantage of the massive resources available in the computing community.

Obviously, language inter-op is not new. Various FFIs have existed for quite some time (I'm a big fan of the Common Lisp CFFI), but what is new (relatively, that is ... as I age, anything in the past 10 years is new) is that we are seeing this not just for programs reaching down into C/C++, but reaching across, to other higher-level languages, taking advantage of their great achievements -- without having to reinvent so many wheels.

When this level of cooperation, credit, etc., is done in the spirit of openness, peer-review, code-reuse, and standing on the shoulders of giants (or enough people to make giants!), we get joy. Beautiful, wonderful coding joy.

And it's so much greater than the sum of the parts :-)


05 Jun 2015 2:53pm GMT

24 May 2015

feedPlanet Twisted

Twisted Matrix Laboratories: Twisted 15.2.1 Released

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 15.2.1.

This is a bugfix release for the 15.2 series that fixes a regression in the new logging framework.

You can find the downloads on PyPI (or alternatively on the Twisted Matrix Labs website).

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,
HawkOwl

24 May 2015 12:22pm GMT

23 May 2015

feedPlanet Twisted

Moshe Zadka: Unicode, UTF-8 and you

Unicode is not a panacea. Some people's names can't even be written in unicode. However, as far as universal encodings go, it is the best we have got - warts and all. It is the only reasonable way to represent text inside programs, except for very very specialized needs (no, you don't qualify).

Now, programs are made of libraries, and often there are several layers of abstraction between the library and the program. Sometimes, some weird abstraction layer in the middle will make it hard to convey user configuration into the library's guts. Code should figure things out itself, most of the time.

So, there are several ways to make dealing with unicode not-horrible.

Unicode internally

I've already mentioned it, but it bears repeating. Internal representation should use the language's built-in type (str in Python 3, String in Java, unicode in Python 2). All formatting, templating, etc. should be, internally, represented as taking unicode parameters and returning unicode results.

Standards

Obviously, when interacting with an external protocol that allows the other side to specify encoding, follow the encoding it specifies. Your program should support, at least, UTF-8, UTF-16, UTF-32 and Latin-1 through Latin-9. When choosing output encoding, choose UTF-8 by default. If there is some way for the user to specify an encoding, allow choosing between that and UTF-16. Anything else should be under "Advanced…" or, possibly, not at all.

Non-standards

When reading input that is not marked with an encoding, attempt to decode as UTF-8, then as UTF-16 (most UTF-16 decoders will auto-detect endianity, but it is pretty easy to hand-hack if people put in the BOM. UTF-8/16 are unlikely to have false positives, so if either succeeds, it's likely correct. Otherwise, as-ASCII-and-ignore-high-order is often the best that can be done. If it is reasonable, allow user-intervention in specifying the encoding.

When writing output, the default should be UTF-8. If it is non-trivial to allow user specification of the encoding, that is fine. If it is possible, UTF-16 should be offered (and BOM should be prepended to start-of-output). Other encodings are not recommended if there is no way to specify them: the reader will have to guess correctly. At the least, giving the user such options should be hidden behind an "Advanced…" option.

The most popular I/O that does not have explicit encoding, or any way to specify one, is file names on UNIX systems. UTF-8 should be assumed, and reasonably recovered from when it proves false. No other encoding is reasonable (UTF-16 is uniquely unsuitable since UNIX filenames cannot have NULs, and other encodings cannot encode some characters).

23 May 2015 4:04pm GMT

19 May 2015

feedPlanet Twisted

Twisted Matrix Laboratories: Twisted 15.2.0 Released

On behalf of Twisted Matrix Labs, I'm honoured to announce the release of Twisted 15.2.

Bringing not only headlining features but also a lot of incremental improvements, this release has got plenty to like:


You can find the downloads on PyPI (or alternatively the Twisted website).

Many thanks to everyone who had a part in this release - the supporters of the Twisted Software Foundation, the developers who contributed code as well as documentation, and all the people building great things with Twisted!

Twisted Regards,
HawkOwl

19 May 2015 6:51am GMT

09 May 2015

feedPlanet Twisted

Glyph Lefkowitz: Separate your Fakes and your Inspectors

When you are writing unit tests, you will commonly need to write duplicate implementations of your dependencies to test against systems which do external communication or otherwise manipulate state that you can't inspect. In other words, test fakes. However, a "test fake" is just one half of the component that you're building: you're also generally building a test inspector.

As an example, let's consider the case of this record-writing interface that we may need to interact with.

1
2
3
4
5
6
class RecordWriter(object):
    def write_record(self, record):
        "..."

    def close(self):
        "..."

This is a pretty simple interface; it can write out a record, and it can be closed.

Faking it out is similarly easy:

1
2
3
4
5
class FakeRecordWriter(object):
    def write_record(self, record):
        pass
    def close(self):
        pass

But this fake record writer isn't very useful. It's a simple stub; if our application writes any interesting records out, we won't know about it. If it closes the record writer, we won't know.

The conventional way to correct this problem, of course, is to start tracking some state, so we can assert about it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
class FakeRecordWriter(object):
    def __init__(self):
        self.records = []
        self.closed = False

    def write_record(self, record):
        if self.closed:
            raise IOError("cannot write; writer is closed")
        self.records.append(record)

    def close(self):
        if self.closed:
            raise IOError("cannot close; writer is closed")
        self.closed = True

This is a very common pattern in test code. However, it's an antipattern.

We have exposed 2 additional, apparently public attributes to application code: .records and .closed. Our original RecordWriter interface didn't have either of those. Since these attributes are public, someone working on the application code could easily, inadvertently access them. Although it's unlikely that an application author would think that they could read records from a record writer by accessing .records, it's plausible that they might add a check of .closed before calling .close(), to make sure they won't get an exception. Such a mistake might happen because their IDE auto-suggested the completion, for example.

The resolution for this antipattern is to have a separate "fake" object, exposing only the public attributes that are also on the object being faked, and an "inspector" object, which exposes only the functionality useful to the test.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
class WriterState(object):
    def __init__(self):
        self.records = []
        self.closed = False

    def raise_if_closed(self):
        if self.closed:
            raise ValueError("already closed")


class _FakeRecordWriter(object):
    def __init__(self, writer_state):
        self._state = writer_state

    def write_record(self, record):
        self._state.raise_if_closed()
        self._state.records.append(record)

    def close(self):
        self._state.raise_if_closed()
        self._state.closed = True


def create_fake_writer():
    state = WriterState()
    return state, _FakeRecordWriter(state)

In this refactored example, we now have a top-level entry point of create_fake_writer, which always creates a pair of WriterState and thing-which-is-like-a-RecordWriter. The type of _FakeRecordWriter can now be private, because it's no longer interesting on its own; it exposes nothing beyond the methods it's trying to fake.

Whenever you're writing test fakes, consider writing them like this, to ensure that you can hand application code the application-facing half of your fake, and test code the test-facing half of the fake, and not get them mixed up.

09 May 2015 6:52am GMT