fidzu : Twisted

In part 1 of this humble attempt to document the interfaces and contracts that unittest actually cares about, we talked about TestSuite and TestCase, how they both implement a common interface that's used for running tests, ITest and how they each implement their own interfaces, ITestSuite and ITestCase.

Now we're moving on to a much more complicated object, TestResult, to see how we can pick apart the ways it interacts with the rest of the system.

TestResult

A TestResult object is all about dealing with the results of tests, as you might expect. However, it doesn't generally represent a single test result. You could say it represents the results of a number of tests, but I don't think that's terribly helpful.

Better to think of a TestResult object as an event handler. A TestResult object receives events from a test run and then does something with them.

Just as TestCase has a two-faced nature, presenting one interface to the testing framework and another to test authors, so to TestResult can be thought of has having many interfaces:

1. Its interface to a TestCase. This can be thought of as the test event handling interface
2. A result querying interface, normally used by a test runner
3. An interface for events that come from the test runner, the runner event handling interface.
4. An execution control interface.

Note that the result querying interface and the runner event handling interface together make up the interface between the TestResult and test runner.

Let's start with the test event handling interface. The methods below are the interface between TestCase.run() and TestResult. (I guess TestCase.debug too, but no one cares about it).

startTest(test)

Called when test commences running. Although not enforced, it's impolite to provide any results for test before calling this.

stopTest(test)

Called when test is completely finished. Although not enforced, it's impolite to provide any more results for test after calling this, unless you call startTest(test) again first.

addSuccess(test)

Called when test has been shown to be successful. The default implementation does nothing.

addError(test, err)

Called when test raises an unexpected error. err is a tuple such as you might get from sys.exc_info(). Calling this method for the first time must change the result of wasSuccessful().

addFailure(test, err)

Called when test has failed one of its assertions. err is a tuple such as you might get from sys.exc_info().

The above interface is tightly coupled to the implementation of TestCase.run(). In particular, if you wish to add more kinds of results to your testing framework ("skip" results are a fairly common addition), then you must change both TestCase.run() and the TestResult interface.

If you do something like that, I recommend making sure that your modified TestCase can handle TestResult objects that do not provide the extensions to the interface that you need. One common way of doing this is to have the TestCase fall back to the primitive result types, e.g. "skip" might become "success" for a TestResult that doesn't know what skipping means.

Importantly, the interface between TestCase and TestResult has been fattened in Python 2.7.

addSkip(test, reason)

Called when test is skipped. reason is a string explaining why the test was skipped.

addExpectedFailure(test, err)

Called when test failed in a way that was expected. err is a tuple such as the one returned by sys.exc_info().

addUnexpectedSuccess(test)

Called when test was expected to fail, but didn't.

The following interface is a way of learning about test results after they have happened, the result querying interface, and is part of the contract between the test runner and the TestResult.

wasSuccessful()

If there have been no errors and no failures, return True. Return False otherwise.

testsRun

An integer that is the number of tests that have been run.

errors

A list of tuples of (test, error_message) for all of the tests with unexpected errors, where test is an ITestCase and error_message is a string suitable for display to humans, generally containing a traceback.

failures

A list of tuples of (test, error_message) for all of the failing tests, where test is an ITestCase and error_message is a string suitable for display to humans, generally containing a traceback.

And of course, Python 2.7 fattens this interface again to have the following:

skipped

A list of tuples of (test, reason) for all of the skipped tests, where test is an ITestCase and reason is a string suitable for display to humans, generally containing a traceback.

expectedFailures

A list of tuples of (test, error_message) for all of the tests that were expected to fail, where test is an ITestCase and error_message is a string suitable for display to humans, generally containing a traceback.

unexpectedSuccesses

A list of all of the tests that unexpectedly succeeded. Members of the list are ITestCases.

In Python 2.7, TestResult also extended its interface to the test runner beyond simple result querying and into allowing the test runner itself to send two very important events to the TestResult, behold the runner event handling interface:

startTestRun()

Called before any tests have been run. It is impolite to provide any test results before calling this.

stopTestRun()

Called after all the tests have finished running. It is impolite to provide any test results after calling this. A TestResult object is generally not expected to handle any events at all after this method has been called.

Some test runners rely on TestResults to use those events to display the results to the user. These runners frequently do not use the result querying part of the interface.

There is one more interface that TestResult implements: the execution control interface:

stop()

Signal that the execution of further tests should stop now. Sets shouldStop to True.

shouldStop

If True, then test execution should stop. TestSuite.run() should monitor this value and stop execution if ever it is True.

This interface is mostly used as a way of handling KeyboardInterrupts cleanly.

Summary

If you want your TestResult object to work with standard Python TestCase objects, or any TestCase objects that try to stick close to the standard, then you must provide the test event handling interface described above. If you are writing your own test framework or test runner, you care about this, because you want to run everyone's unit tests.

If you want your TestResult object to work with the standard Python test runner before Python 2.7, then you must provide the result querying interface. If you are using the standard Python test runner, you care about this. For Trial or testtools, you must provide the runner event handling interface. For anything else, I'm afraid you are on your own.

Always provide the execution control interface.

Comments

In this documentation, I've been trying to describe the various interfaces without inserting too much of my own opinion about their design. However, I think some commentary might actually help to make things easier to understand.

By providing a querying interface for TestResult to be used by a test runner, the original designers of unittest practically insisted that responsibility for displaying the results of a test run be split between two different classes. The TestResult takes care of displaying incremental feedback from the running tests and the test runner takes care of displaying the summary. You can see evidence of this design in Python 2.6's unittest.py, where there's a hidden _TextTestResult subclass which has extra methods that are called only by a special TextTestRunner.

The addition of startTestRun() and stopTestRun() mean that now a TestResult object can be fully in charge of displaying its results. As such, providing a query interface and exposing details like the list of test failures somewhat vestigial.

I'm less happy with this post than the previous one. As such your critique is even more welcome.

Still to come: the interface for test authors and just what is a test runner anyway?

30 Jul 2010 3:37pm GMT

It's a little known fact, but unittest actually has an API.

This isn't the API that you deal with when you write tests, but rather an API that unittest itself uses when running tests. You could think of it as two interfaces: one for test frameworks and one for test authors. Both APIs are real, but both are poorly documented and often misunderstood or abused.

TestCase

An instance of TestCase represents a single test. What you think of as a single test is up to you, but most of the time it's a unit test.

A TestCase object must provide the following methods.

This first list of methods can be thought of as a single interface, which these blog posts will call ITest given the lack of any better name.

countTestCases()

A method that returns the number of test cases this represents. It should always return 1.

run(result=None)

Calling this method actually runs the test. result is a TestResult object. run must call result.startTest(self) when it commences running the test and result.stopTest(self) when it is finished. Between these calls it must call a method on result to signal the result of the test. run must never raise an exception, and its return value is ignored. If result is not provided, the TestCase is obliged to make one.

__call__(result)

Identical to run(result), provided for backwards compatibility.

debug()

Calling this method runs the test without collecting its results. It may raise exceptions. This method is rarely called by test frameworks.

The following methods are specific to individual test case objects. We call this interface ITestCase.

id()

Should return a string that uniquely identifies the test. For Python tests, the fully-qualified Python name works well. The uniqueness of the id is not enforced.

shortDescription()

Should return a string that describes the test. Many test frameworks use this value to display test results.

__str__

Should return a string that describes the test. Frequently the same as either shortDescription() or id(). Many test frameworks use this value to display test results.

There is also a second interface, one that matters to code that subclasses TestCase. We'll deal with that in a later post.

TestSuite

A TestSuite represents nothing more or less than a bunch of tests.

A TestSuite must provide the ITest interface described above, with the differences that you would expect from something that represents many tests: countTestCases returns the number of tests in the suite; run runs many tests and thus calls result.startTest and kin many times over; debug is the same and can explode anywhere.

One difference is that TestSuite.run must stop running tests as soon as it detects that result.shouldStop is true.

In addition, TestSuite implements the following interface, which I'm giving the completely arbitrary non-existent name of ITestSuite.

addTest(test)

Takes an ITest and adds it to the suite.

addTests(tests)

Takes an iterable of ITests and adds them to the suite. Normally equivalent to [suite.addTest(test) for test in tests].

__iter__

All test suites must be iterable. Iterating over a test suite yields ITests. These may differ from the ITests provided to addTest and addTests.

In later posts, I hope to document TestResult, the subclassing interface of TestCase and tell you exactly what I think about test loaders, test runners and the like.

I'm blogging this partly because I don't know where else to write this up, but mostly because I need your help to make sure that I'm being clear and correct. Please comment with questions and corrections, and let me know if you find this at all helpful.

29 Jul 2010 4:56pm GMT

(Friends have shown me a stand-up routine about "Ironic" which prompted this.)

Picture this: you're in your literature class. The teacher gives the definition of "irony", and proceeds to rattle off a few examples of irony. I'm dating myself this, but back when I was in this situation (highschool lit) the song "Ironic" did not achieve cult-status yet, at least around my part. So my teacher did this, we copied this down to our notebooks, and eventually got tested on this.

The scene as detailed above has two interesting things about it: (a) there is absolutely nothing ironic about it and (b) it could never happen, as described, today.

I am imagining this scene happening today: the teacher says "today, we will discuss irony." Without fail, one of the brighter lights in the class says, "oh, it's like rain on your wedding day." The teacher visibly sighs, rolls her eyes, and explains how there is absolutely nothing ironic about rain on your wedding day. I am pretty sure this is more or less how it goes in every classroom around the world: everyone thinks they know what irony is, because of the song, and they're all horribly, dreadfully mistaken.

This is ironic. The ur-example, the immediate connection in our shared knowledge, of irony is a list of examples of things which are not ironic. I think that even if I tried, hard, I could not manage to associate something more ironic with the word ironic.

Thank you, Alannis, for creating the most delicious irony of all, and ensuring that irony can never be mentioned without causing irony. The world is perfect once again.

28 Jul 2010 9:21pm GMT

Six years ago, we did the first large scale Ogg Theora stream from the 2004 GUADEC conference.

It was a dime on its side to get things ready for this year. I purposely removed myself from the organization, because for various reasons I'm not going to GUADEC this year, but I was hoping the rest of the company would do their part to get this working, and I just provided the necessary prodding along the way. I've been told one of the organisers in charge of this got ill at some point and communication went a bit south during that period, so I had some complaints from our support guys that they had to do last-minute rushing.

But the streams are live today, and a few developers here are giddily running around looking at the stream, the image, working on some typical bugs you get when you're doing stuff like this for the first time (the artifacts on keyframes the encoder seems to have remind me a lot of the Theora bugs we had to squash back in the day, and obviously they are worse on still images, like, say, an empty conference room…)

Go check out the stream and make sure you have a WebM-enabled browser, like the Firefox 4.0 beta or latest Opera.

Congratulations to our intrepid hackers like Zaheer and Andoni for their hard work a few weeks ago on WebM, and I've been told Marc-André actually went to Holland just to deliver the encoders :)

28 Jul 2010 2:46pm GMT

I would be much more sympathetic to the whole Python 3 endeavour if they had made a serious effort to keep the major 2.x releases mutually compatible.

26 Jul 2010 11:40pm GMT

Many people dismiss evolution as "survival of the fittest is a tautology - the fittest are the ones who survive."

This is wrong for many reasons, but one of the problems with that is something I have yet to see among the writings of evolution. As is standard in defenses of evolution, I will begin my thesis with analogy to gravity. But before that, I would like to analyze what we mean by "tautology". A "tautology" is something that is true by definition, or cannot be logically anything other than true. An example is "x=x". X must always be equal to itself.

Now, open a math journal. Any math journal. Choose a paper. Any paper. I will do it, now, by choosing a paper from arxiv.org (it's not strictly a math journal, but close enough for what I have in mind): I tried to be "random" about it by going to "new" and choosing the first one that I could half-way understand: All automorphisms of all Calkin algebras. As an aside, much like most readers of this, I'm not really qualified to judge it. But just reading the abstract, the author claims: "The Proper Forcing Axiom implies all automorphisms of every Calkin algebra associated with an infinite-dimensional complex Hilbert space and the ideal of compact operators are inner." Assuming his proof is correct, this means that this thing that is fairly non-trivial to understand is a tautology. I am willing to bet that even if you specialize in the field, the above was not obvious to you. In fact, this is true of any math paper: it is completely filled with tautologies, and is completely and utterly non-obvious.

Let me give you another example of a tautology: given that the gravitational force is GMm/r**2, a "solar system" (one very heavy body and a bunch of significantly lighter bodies) will move in close approximations to ellipses (that is not an exact phrasing, but a proper estimate of the approximation is outside the real of this post). That is a tautology, and is also fairly non-trivial.

Now, here is another: assuming that genes are passed in a mostly-faithful (small number of mutations) to the next generation, and that genes correlate to reproductive success, we expect to see "evolution". This is a mathematical theorem, in that the axiom can be modeled mathematically and "evolution" (meaning many of the phenomena under "evolutionary biology" such as exaptation, adaptation, complexity and variety) be proven. Note that the initial axiom is not a tautology: there is no a-priory logical reason to assume changes in an individual's life-style won't change their genetic code. However, the initial axiom, despite not being a tautology, meaning logically true, is factually true: many experiments have been done on this. By the way, it's as true as GMm/r**2: it's a useful approximation, not the complete truth: there are epigenetic pressures on heredity. Just like in gravitation theory, we use a simple model to derive predictions, and the correlation of the predictions with the model gives us evidence that the model is true.

Summary: a significant part of evolutionary biology is a tautology. But tautology is not a bad word - it means that a significant part of biology is mathematics. This is good. Mathematical modeling is the best modeling.

26 Jul 2010 9:02pm GMT

Someone at Canonical (I don't know who) wrote something cool once to get Pyflakes (the best Python linter) to run on doctests.

It has recently been deleted from the Launchpad tree, but since it's so useful I thought I'd make it available.

Download pyflakes-doctest whenever you'd like. If you can get it into Pyflakes trunk, then you'll become even more wonderfully, deliciously fabulous and creamy than you undoubtedly are already. Get rid of the Launchpad-specific stuff though.

20 Jul 2010 1:08pm GMT

The current phishing attempts are ludicrously easy to mount, and they are this ludicrous because the current web's security model is fundamentally broken.

If Joe Q. Random discovers a new site (maybe a forum site dedicated to discussing cats - Mr. Random is a big cat afficionado), he needs to create an account. He creates a username, and a password, and will be asked to supply an e-mail address to confirm the registration. Random does so.

Now, all someone needs to do to get Random's credentials is to fake a Realistic Mail from SillyCatForum.com saying "someone has responded to your message. Click <here> to read the reply." Random clicks, and is presented with the login screen. Random enters his credentials. What R. didn't notice was that the link actually went to hackersrus.com. But two seconds later, he's on the real SillyCatForum.com, mildly bemused by there not being a response to any of the messages. "Maybe someone deleted it, or something", thinks Random, without a thought. Random's cat-forum credentials are now in the hands of Hackers'R'Us. Security - failed. Faking the realistic e-mail is a child's play - all someone from Hackers'R'Us needs to do is to create two accounts, reply from one to the other, and presto!, copy'n'paste the e-mail.

"Why is Random clicking on things sent to him in the e-mail?" cry all those holier-than-thou security experts. Because, they seemed to have forgot, the web has trained him to do it. E-mailed links is the normal notification mechanism on the web.

How can we solve this? Simple, do away with passwords.

All sent links should contain "credential information" which will auto-log the user. If the user needs to log in, he can click for "send me credentials", which will send him a credentialed link. E-mail providers can ask for a cellphone number, to which they will send a one-use (ONE USE) code, which will log the user in. They can also, on creation, ask for an address they can snail-mail a one-use code, in case the user loses his cellphone. (Naturally, you might have to pay for that - but on the off-chance that this happen, to your primary e-mail address, paying 10$ to restore it should be within the means of most people.)

Of course, most sites should not even need that much. Just offer the chances of a Facebook Connect/Open ID log-in, and don't even try asking for a "credentials". Let someone else handle credentials for you.

15 Jul 2010 10:09pm GMT

# coding: utf-8

# Python's email API is simple and easy to use!!!!!!!!!!!!!!

# Requirements:
# * UTF-8 headers
# * UTF-8 body
# * prefer quoted-printable to base64 transfer-encoding.
# * Don't escape "From" at the beginning of a line in the message - it's not
#   the 1800s any more

from cStringIO import StringIO
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.header import Header
from email import Charset
from email.generator import Generator


subject = u'Hello あ'
recipient = u'Bあb '
from_address = u'Bかb '

html = u'<html><body>Hey böb!\nFrom Jack, I got enhanced pills!</body></html>'
text = u'Hey böb!\nFrom Jack, I got enhanced pills!'

# Override python's weird assumption that utf-8 text should be encoded with
# base64, and instead use quoted-printable (for both subject and body).  I
# can't figure out a way to specify QP (quoted-printable) instead of base64 in
# a way that doesn't modify global state. :-(
Charset.add_charset('utf-8', Charset.QP, Charset.QP, 'utf-8')


# This example is of an email with text and html alternatives.
multipart = MIMEMultipart('alternative')

# We need to use Header objects here instead of just assigning the strings in
# order to get our headers properly encoded (with QP).
# You may want to avoid this if your headers are already ASCII, just so people
# can read the raw message without getting a headache.
multipart['Subject'] = Header(subject.encode('utf-8'), 'UTF-8').encode()
multipart['To'] = Header(recipient.encode('utf-8'), 'UTF-8').encode()
multipart['From'] = Header(from_address.encode('utf-8'), 'UTF-8').encode()

# Attach the parts with the given encodings.
htmlpart = MIMEText(html.encode('utf-8'), 'html', 'UTF-8')
multipart.attach(htmlpart)
textpart = MIMEText(text.encode('utf-8'), 'plain', 'UTF-8')
multipart.attach(textpart)

# And here we have to instantiate a Generator object to convert the multipart
# object to a string (can't use multipart.as_string, because that escapes
# "From" lines).

io = StringIO()
g = Generator(io, False) # second argument means "should I mangle From?"
g.flatten(multipart)

# Pass the result of this to your SMTP library of choice.
print io.getvalue()

edited: The last part in a multipart message is the preferred one, so I moved the HTML part to the bottom.

edited AGAIN: I found out that in order to avoid ridiculous "From" quoting, I needed to use a Generator object instead of multipart.as_string().

15 Jul 2010 4:16pm GMT

Hello readers,

12 Jul 2010 7:47pm GMT

Documenting your code is not important. Well-documented code is not a thing to be desired nor a goal to be sought. A lack of documentation is not a problem to be solved. Everyone who tells you otherwise is lying.

What is important is that other programmers can easily understand your code and the intent behind it, and that you can understand the code and the intent six months later.

Documenting your code is merely a way of achieving that. It might even be necessary for it. But do not confuse interface with implementation and do not confuse what you want with how to get it.

Solve the problem of confusing, opaque code. Aim for code that can be easily understood. Document if necessary.

(it's pretty much always necessary)

06 Jul 2010 1:14pm GMT

Although I love TDD, I don't always use it for everything. I wrote a Python script the other day to clean up my music collection, and I didn't write any tests for it.

Someone asked me, what's the trade-off? At what point do you start writing tests?

For me, it's at one of a few points:

• Somewhere between 100-200 lines of code I start feeling the need for tests
• When I start feeling unclear in my head, I reach to tests to make them clear
• If someone wants to work with me, tests instantly become mandatory
• When I come back to some code after a while
• When I notice my first bug

05 Jul 2010 2:17pm GMT

Twisted 10.1.0 is finally out. It's a month late, but it's finally out there and free from the shackles of British rule.

Highlights include:

• Deferreds now support cancellation
• A new "endpoint" interface which can abstractly describe stream transport endpoints such as TCP and SSL
• inotify support for Linux, which allows monitoring of file system events.
• AMP supports transferring timestamps

Note also that this is the last release in which we will support Windows for Python 2.4.

For more information, see the NEWS file.

It's stable, backwards compatible, well tested and in every way an improvement. Download it the tarball, the Windows installer for Python 2.5 or the Windows installer for Python 2.6.

Update: We now have executable Windows installers for Python 2.5 and Python 2.6 to complement the MSIs.

Many thanks to Glyph Lefkowitz, who helped prepare the release, and the PyCon 2010 sprinters, who did so much of the work for this release.

05 Jul 2010 1:19pm GMT

I'm in the middle of releasing Twisted 10.1, following the procedure document that I wrote when I released Twisted 10.0. Having everything written down has been a wonderful aide so far, but doing the release twice has made me think about what it takes to breathe life into old procedure.

The very first step has already been taken, figure out what the process is and write it down. There are still some bits that are unknown and hazy, but I expect they'll be clear by the time we're done. Anyway, writing things down is only the beginning, after that, there are two things that I think we ought to do roughly concurrently.

The first is automate the existing procedure for which here are already many tickets filed, and the second is simplify the process itself. Are all of the steps in the process really needed? Why do we have so many tarballs? Why upload the tarballs to a server that is only periodically mirrored by the actual official download location? Why generate a PDF?

I don't want to start a discussion on the details here, but rather raise the need for Twisted to begin considering this simplification, and for myself to begin articulating some of my own thoughts about process in general.

The final, on-going step in revitalizing procedure is to delegate the task, either to another human being or better yet a machine. I wonder if it would be possible to have the Twisted release done by a monthly cronjob?

In summary, to revive an existing process:

• Figure out what the process is and write it down
• Simplify the process itself, reducing the number of steps
• Automate as many of the steps as possible, thus combining them
• Delegate the execution of the process

04 Jul 2010 1:45pm GMT

Similarities:

• Both are fruit
• Both have about the same caloric value

Orange advantages:

• Significantly more vitamin C

Apple advantages:

• Does not require peeling
• Bakes well

Warning: if I ever see you saying "it's like comparing apples and oranges", I will link you to this post.

02 Jul 2010 6:38pm GMT

Dear intarweb,

I've been searching all over for data on what these fancy Dell R815 servers (which can house 4 12-core Opteron CPU's for a total of 48 cores in 2U) draw in terms of power.

This handy capacity planner from Dell doesn't have this machine listed yet.

Anyone out there know where I can find this info, or have one of these babies actually running ?

30 Jun 2010 4:43pm GMT