10 Nov 2011

feedPlanet Twisted

Itamar Shtull-Trauring: The Euro crisis, in musical form

DOOM.

10 Nov 2011 4:03am GMT

09 Nov 2011

feedPlanet Twisted

Thomas Vander Stichele: Mach 1.0.0 “Madera” released

Another November, another Fedora. 16 came out, so it was time to update mach again.

And today I thought, is there any reason mach isn't 1.0 yet ? Am I going to do anything more to this piece of code before I want to call it that ?

And the answer is, no. It's the first Python application I've written, and I'm not particularly proud of the code, but I'm happy I've made good use of it for so long, and that it helped push packaging approaches forward and sparked ideas for the Fedora build system.

Since I didn't like the original code for mach2 (there was a version 1 which was Makefile-based), I started a rewrite with unit tests, better code layout, decent classes for abstracting distro-specific stuff, and so on.

The experience of how mock was created based off mach2 was a slightly sour one however, so I wasn't really motivated to finish the mach3 rewrite. Sometimes that's the drawback of open source - sure, forking is specifically allowed, so don't whine about it when it happens. But when it's done gratuitously, with no serious attempt at collaborating, it doesn't feel like it's in the spirit of open source.

Anyway, that was a long time ago. mach2 as it is today, is done. It really only needs updating for newer versions. As long as it works for me, it's unlikely I will continue mach3, but who knows?

Enjoy the release!

09 Nov 2011 10:56pm GMT

05 Nov 2011

feedPlanet Twisted

Glyph Lefkowitz: Blocking vs. Running

I've heard tell of some confusion lately around what the term "non-blocking" means. This isn't the first time I've tried to explain it, and it certainly won't be the last, but blogging is easier than the job Sisyphus got, so I can't complain.

A thread is blocking when it is performing an input or output operation that may take an unknown amount of time. Crucially, a blocking thread is doing no useful work. It is stuck, consuming resources - in particular, its thread stack, and its process table entry. It is sucking up resources and getting nothing done. These are resources that one can most definitely run out of, and are in fact artificially limited on most operating systems, because if one has too many of them, the system bogs down and becomes unusable.

A thread may also be "stuck" doing some computationally intensive work; performing a complex computation, and sucking up CPU cycles. There is a very important distinction here, though. If that thread is burning up CPU, it is getting work done. It is computing. This is why we have computers: to compute things.

It is of course possible for a program to have a bug where a program goes into an infinite loop, or otherwise performs work on the CPU without actually getting anything useful to the user done, but if that's happening then the program is just buggy, or inefficient. But such a program is not blocking: it might be "thrashing" or "stuck" or "broken", but "blocking" means something more specific: that the program is sitting around, doing nothing, while it is waiting for some other thing to get work done, and not doing any of its own.

A program written in an event-driven style may be busy as long as it needs to be, but that does not mean it is blocking. Hence, event-driven and non-blocking are synonyms.

Furthermore, non-blocking doesn't necessarily mean single-process. Twisted is non-blocking, for example, but it has a sophisticated facility for starting, controlling and stopping other processes. Information about changes to those processes is represented as plain old events, making it reasonably easy to fold the results of computation in another process back into the main one.

If you need to perform a lengthy computation in an event-driven program, that does not mean you need to stop the world in order to do it. It doesn't mean that you need to give up on the relatively simple execution model of an event loop for a mess of threads, either. Just ask another process to do the work, and handle the result of that work as just another event.

05 Nov 2011 1:16am GMT

04 Nov 2011

feedPlanet Twisted

Jonathan Lange: How to feel better (or, some tips on refactoring)

A few months back I gave a lightning talk at the Launchpad Thunderdome about how I do refactoring. It's very opinionated, and mostly applies to big, old code bases, but worth writing up anyway.


The core idea here is that very few things make me feel as good as deleting code. I love cleaning up code and the clean code base that results, and I'm sure that many others feel like me. As such, this is a guide on how to feel better.

1. Know your enemy

"Functionality is an asset, code is a liability". Truer words were never spoken. Every line of code is a potential source of bugs and a barrier to understanding, and thus carries a maintenance cost.

Maintain an awareness of things that need refactoring. Here's a quick and incomplete list:

2. Keep a "yak stack"

Yak shaving is "any seemingly pointless activity which is actually necessary to solve a problem which solves a problem which, several levels of recursion later, solves the real problem you're working on." (Jargon File)

A few of us have extended the concept beyond the "actually necessary" to include anything that's making the task at hand more difficult and less fun but is not worth fixing right now. Hence the yak stack. Here's how it works:

Whenever you come across something in your code base that is difficult to understand or that slows you down: make a note of it. When you've finished the task at hand, fix the problem. If you encounter other things that slow you down, write them down. Work through the list.

3. Every option is wrong

In a big, old code base, there are probably many, many areas that need refactoring. Don't worry about which is the "best" place to start - there is no best place.

4. Start from green, stay green

Never, ever refactor while your tests are failing. Refactoring is about changing the shape of code while preserving its behaviour. It's much harder to be sure you're keeping the behaviour if you are comparing one set of thirty tracebacks with another set of thirty tracebacks. Better to compare a passing ("green") test run with another passing test run.

Run tests frequently. More often then you think you should. Commit often - think of it like quick save in a tough level of a video game. It frees you up to experiment more and means you have less in your head at any one time.

5. Do not nest

Don't begin a refactoring while you are in the middle of another refactoring. If you find you must, use tools like 'bzr shelve' to store your current diff and then work from the clean head of your branch.

6. Keep moving, leave a trail

Don't get bogged down in details, otherwise you'll never finish. Literally. Someone will come along and distract you and before you know it, three months will pass and your refactoring branch will be full of conflicts. If you see something you are unsure of, mark it with a XXX or a FIXME or a TODO or whatever works for you and then continue with what you are doing.

Tools like 'bzr todo' can make it really easy to check to see if you've added any XXX comments.

7. Translate tests with care

As said above, refactoring is about changing the shape of code while preserving its behaviour. When you update tests, you risk changing your definition of the system's behaviour - so be careful.

8. Confront uncertainty with destruction

If you see some code and you are not sure if it's needed, delete it. Doesn't matter if it's a whole function or just an odd line. If you have a test suite, and it was important, that will catch the failure. If you have version control, and it was important, one of your collaborators will notice and revert the change.

If it was important and neither of these happened, then your whole project has learned something new about itself, and that's probably worth the hassle. (Oh, add tests & better docs after this happens.)

9. Good grep tools

Remember that symbols aren't only referenced from files that match *.py. In big code bases there are often other sorts of files that refer to symbols in the main code. In Launchpad, for example, we have ZCML and doctest files that refer to symbols. When you want to know how something is used or you want to rename something, make sure you use a grep command that actually finds everything.

Ideally, you should be able to run this command faster than you can think about wanting to do it.

Personally, I use 'bzr grep' a lot. Others recommend 'ack'.

10. There will be failures

Mentally prepare yourself for the fact that the first two or three full test runs after your refactoring will fail. This is especially important for code bases that have multi-hour test run times.

If you think this way, then you won't be as discouraged when it actually happens.

11. Finish the job

Busy people refactoring a big code base are often tempted to apply a refactoring to only a single part. For example, some useful function is extracted from repeated boilerplate in a few other functions. However, many, many other instances in the code base continue to use the repeated boilerplate.

This is almost worse than just leaving the repeated boilerplate. There are now two idiomatic ways of doing the activity. Further, other developers who work on other parts of the code base probably won't find out about it, and might end up repeating your refactoring. Ugh. How is anyone new expected to get to grips with this code?

Similarly, if a class, function or concept is renamed, rename it everywhere, especially in the documentation.

It's difficult and often tedious, but it really is worth taking refactorings to completion. Apply them to the whole code base, or not at all.

Note that I'm referring to completeness, not perfection. If you block on perfection, you never get anything useful done. If you aim for frequent incremental improvements, you soar.

12. Read these books

I highly recommend "Refactoring" by Martin Fowler and "TDD by Example" by Kent Beck. I stole many of these ideas from them.



Over to you

This was very much just a dump of how I do refactoring when hacking on Launchpad. I'm always keen to learn more, and would love to hear about what works for you.

04 Nov 2011 6:29pm GMT

31 Oct 2011

feedPlanet Twisted

Jp Calderone: Don't Use Buildbot EC2 Features

I just noticed that Buildbot spun up one EC2 instance 31 days ago and another one 14 days ago and left them both running.

31 Oct 2011 5:15pm GMT

28 Oct 2011

feedPlanet Twisted

Stephen Thorne: Gmail Keyboard Shortcuts

F3e04bbd80a5c4d9_gmail-magnet

I use Gmail a lot now. I also have come to know and love a few keyboard shortcuts. I can't imagine dealing with a large volume of email without them.

You have to turn them on under the mail settings first.
u will refresh the current mail search or your inbox if you're looking at your inbox.
u will return you to your mail search, label or inbox if you're looking at an email.
j/k to navigate between emails in your inbox view.
x selects an email, and dumps you back at the message list if you were in an email.
s applies a star to selected emails, or stars the email you're reading.
+/- will flag/unflag an email as important (if you're using priority inbox)
e archives selected emails, or archive an email you're reading.
r replies to an email.
m will mute selected email threads so it won't reappear in your inbox when more posts are made.
gl lets you type the name of a label you want to view.
? brings up the help.
Most important button to me is u, as it refreshes my inbox when my android phone has chimed to let me know that I've got an important email, and the gmail UI hasn't shown it yet. The androids. They always know.
(Photo from Geek Sugar)

28 Oct 2011 12:26pm GMT

Stephen Thorne: Appengine Development

I've been quite enjoying writing some code for Google Appengine this week. Just a small project for querying a variety of book retailer websites for information.I'm really impressed with it so far. Having the ability to simply import and use things like memcache and use it is excellent. It works locally with some kind of cache, and then when I deploy my app it uses a memcache on the server side seamlessly.I'm also liking that the urlfetch infrastructure built into appengine supports async downloading of webpages, so i can fire off 5+ API queries simultaneously and then collect the results, instead of doing them in sequence.Here's a random bit of code I threw together that lets me cache the results of doing API queries in memcache so that a user doing reloads won't cause more API requests to occur than are strictly necessary:
from google.appengine.api import urlfetch
from google.appengine.api import memcacheCACHE_TIMEOUT = 3600 # 1 hour class APIError(Exception):
passdef cache_result(rpc, url):
result = rpc.get_result()
if result.status_code == 200:
data = result.content
memcache.add(url, data, CACHE_TIMEOUT)def apiquery(url):
data = memcache.get(url)
if data is not None:
decoded_data = json.loads(data)
return lambda:decoded_data logging.debug("Cache miss for %r", url) rpc = urlfetch.create_rpc()
rpc.callback = lambda: cache_result(rpc, url)
urlfetch.make_fetch_call(rpc, url, method=urlfetch.GET) def answer():
result = rpc.get_result()
if result.status_code == 200:
data = result.content
return json.loads(data)
raise APIError, "Broken!"
return answer
With the above code I can now do something like:
queries = [apiquery(url) for url in generate_api_queries()]
for query in queries:
result = query()
... # do stuff with result
I don't really like the variable names I've used, but it's an interesting enough bunch of code linking together a few bits and pieces so I thought I'd throw it on my blog.

28 Oct 2011 12:25pm GMT

25 Oct 2011

feedPlanet Twisted

Twisted Matrix Laboratories: Sponsored Development, October 2011

I'm happy to report that I've just completed another two weeks of sponsored Twisted development. As usual, the issue tracker directed most of my work. I spent time on new development aimed at resolving tickets and I reviewed changes proposed by other developers with that aim.

All told, 59 tickets got some attention that they would not have received without the sponsored development. The result was 36 closed tickets and 17 other tickets unblocked for other developers to resume work on. Over the coming days or weeks a few more tickets will probably be resolved as a result of this work, as developers (including myself) respond to review feedback.

This work is made possible by the sponsorship of individuals and organizations which have donated to the Twisted project, part of the Software Freedom Conservancy, a not-for-profit organization that helps promote, improve, and develop open source software. Thanks!

Apart from working to resolve tickets in the issue tracker, I also did some work on infrastructure. I:

For more details about a ticket, see <http://tm.tl/NNNN>.

The tickets I reviewed, with tickets now closed in bold:

#78 - abortConnection() method for transports
#2674 - t.i.defer.inlineCallbacks documentation issues
#2675 - Test timeout policy should be decided by runner
#3077 - twistd --help plugin list isn't alphabetized
#3078 - Dynamic ZSH tab completion for any commands using t.p.usage
#3350 - make each peer register as a producer in portforward
#3453 - twisted.web._auth.digest mostly ignores the value of the uri field
#3896 - Passing a unicode object to request.write corrupts the entire response
#3926 - twisted.positioning -- a better positioning framework
#4159 - Add pre-exec hook to spawnProcess
#4519 - Delay parsing of request body until Request.args is accessed
#4751 - t.c.telnet.ITelnetProtocol.unhandledSubnegotiation doesn't reflect reality
#4849 - Add stream compression to twisted.words.jabber
#5024 - MSI buildbot patches version to avoid breaking distutils MSI builder, but does it in the wrong place
#5040 - twisted.web.template is lacking patterns
#5044 - Client documentation should use explicit buildProtocol()s rather than magic "protocol = MyProtocol" on factories
#5085 - Accept IPv6 address literals (with embedded scope ids) in IReactorTCP.connectTCP
#5139 - Replace usage of os.path.walk in favor of twisted.python.filepath.FilePath.walk
#5156 - no ambiguous antecedents (plugins documentation edition)
#5192 - 100-continue support for twisted.web.client.Agent
#5194 - log.msg can fail if someone removes an observer at the wrong time
#5250 - finger15.tac patch
#5252 - Manhole should support CTRL-A and CTRL-E for home/end
#5253 - Add support for input history persistence in twisted.conch.manhole
#5263 - add multipart/form-data support to twisted.web.client.Agent for easy form posts
#5267 - allow twisted.internet.ssl.[Private]Certificate (and friends) to be backed with TLSLite rather than pyOpenSSL
#5271 - trial test method timeout support is fragile
#5275 - t.w.template does not always properly escape comments
#5279 - pass mode to FilePath.createDirectory()
#5282 - ILogObserver, log.msg, log.err, and logging.html are insufficiently stern about log observer thread safety
#5283 - opt_user unused in manhole twistd plugin
#5285 - win32eventreactor misses disconnect events that happen too soon
#5286 - twisted.web.template documentation doesn't cover very basic use-case of rendering more than one of something
#5288 - twisted.web.template.Tag.fillSlots documentation doesn't mention it returns self
#5301 - test_tcp_internals.PlatformAssumptionsTestCase fails when we leak fds
#5312 - Create some release automation for building Sphinx documentation

And tickets I developed:

#581 - CopiedFailure.getTraceback should include the result of CopiedFailure.getErrorMessage even if unsafeTracebacks is not set
#745 - writeSequence not well-tested
#1946 - ErrorHolder is basically untested
#2838 - _dumbwin32proc.Process does not implement IProcessTransport completely
#4603 - names should handle unknown records
#4666 - t.i.task.Cooperator scheduler objects must return an IDelayedCall
#4671 - sometimes I want to have some constant values that are part of a set
#5040 - twisted.web.template is lacking patterns
#5062 - ProtocolWrapper masks the name of the wrapped protocol in the logs
#5075 - IntNStringReceiver copies too much data in dataReceived, causing AMP to be slow
#5140 - Deprecate twisted.scripts.tkunzip
#5158 - Installing on Pypy fails because of CPython-specific extension modules
#5233 - win32eventreactor doesn't notice TCP connection is lost in certain (hopefully rare) cases
#5237 - Add SFTP example
#5273 - Improve the API documentation for MultiService.addService to clarify its use in comparison to Service.setServiceParent
#5278 - When connecting with endpoints, the application factory used is neither started nor stopped, and irrelevant start/stop messages are logged instead
#5285 - win32eventreactor misses disconnect events that happen too soon
#5291 - twisted.trial TestCase should support assertDictEqual from Python 2.7 unittest
#5292 - Listening ports used with wrapping factories could log better start messages
#5293 - Remove mktap and corresponding twistd options (mktap is broken in trunk)
#5299 - udp.Port schedules connectionLost call in the global reactor
#5303 - Default transport implementation doesn't write out bytes immediately
#5304 - Win32Reactor.spawnProcess is redundant
#5308 - Make UDP ports identify themselves when stopping
#5316 - Incorrect API link in twisted.web.proxy.Proxy docstring

25 Oct 2011 6:28pm GMT

24 Oct 2011

feedPlanet Twisted

Thomas Vander Stichele: GStreamer Conference number 2

I'm in Prague right now for the second GStreamer conference. Prague is as pretty as I remember it from eighteen years ago when I was still in high school and we had our yearly school trip.

It's great to see a mix of familiar and new faces again. 11 years ago GStreamer was made public, and I joined a year later around the 0.1.1 release if I recall. And now it's this huge living breathing thing.

Tomorrow I will be giving a talk about Flumotion here, at 12.00 in the main room. If you're interested in GStreamer beyond mere playback, this talk is for you. The only sad part is that my good friend Jan Schmidt will be talking about Bluray at the same time, but I'm relying on Ubicast to record it properly so I can see it later!

24 Oct 2011 1:28pm GMT

21 Oct 2011

feedPlanet Twisted

Twisted Matrix Laboratories: October Sprint Report

After a brief rest, the Boston Twisted crew is back on the sprinting treadmill. Dave Sturgis came through for us once again for the venue, working with Smarterer to let us use their office space and eat their snacks last Saturday afternoon. We also took advantage of the Python Software Foundation's sprint funding program to buy all the sprinters dinner.

Eight sprinters turned out to fix bugs, add features, and review changes.

Chris (radix) (triumphantly returned from the inhospitable southwest) worked on adding a history-tracking mode to Deferreds to aid in debugging. This feature will let a developer see everything that has happened to a Deferred - what callbacks it has run, what values have passed through it, and what other Deferreds it has been associated with.

Itamar picked up a ticket relating to HTTP client support for persistent connections. He also finished up the work on adding a new transport method, abortConnection, fixed a logging bug in the new TLS implementation, and did some ticket triage and a review.

Allister (amacleod) heroically tackled the problem of our insufficient Windows testing infrastructure, getting introduced to our build farm and setting up a new Windows 7 virtual machine on it.

Jessica (jesstess) did a ton of follow-up on old tickets (1247 2115 2447 2498 2507 2513 2861), trying to learn if they are still valid and elicit further information from the original reporters. She also managed to close one which was no longer valid (but she also filed a new one for a documentation bug, in case you were worried we might run out of tickets).

Glyph knuckled down on a review for our conversion to Sphinx. We're so close I can practically taste it.

Ying (cyli) took a look at improving the authentication options for some of Twisted's built-in servers. She started with the FTP server and made great headway.

As for myself, I reviewed that abortConnection feature as well as a ticket for HTML5 support in twisted.web.template and the TLS logging bug that Itamar fixed.

Thanks to all the sprinters and to Smarterer and the PSF for their support of the sprint! We're going to do another one of these real soon, watch out for an announcement!

21 Oct 2011 5:02pm GMT

16 Oct 2011

feedPlanet Twisted

Moshe Zadka: A Transport-agnostic Process-agnostic System in Python

Abstract

Different communication mechanisms have different trade-offs: XML-RPC is ubiqutous, JSON-RPC is almost as popular but more space/time efficient, AMQP allows efficient routing and so on. It is advantageous to write application code which is completely transport-agnostic. Deciding how to split components into processes is a decision more efficiently done at the tail-end of a project, when the knowledge of which processes are more reliable, or need to communicate more, is known. Described here is the system we use to allow application code to be unaware of those distinction, pushing those decisions into deployment or installation time.

Introduction

When starting to implement a complicated system in Python, there are many unknowns: amount of communications between processes, typical process bugs and deployment targets. It is highly useful to allow writing the business logic code in a way that is agnostic to the answers to those questions. This also allows the communication layer to be written as a quick prototype, and optimized later. Since infrastructure costs are frequently minimized during the demo phase, this kind of approach allows quick demos during development without any cross-team code rewrite. The trade-off in using this kind of approach is that the feature set assumed from the transport has to be a least-common-denominator subset of all "possible" feature sets. This, in turn, pushes some of the work that could be done by more sophisticated transport to the Python-level infrastructure and sometimes, in the worst cases, into application code. The most egregious example is that any system that supports HTTP-based RPC mechanism (XML-RPC, JSON-RPC or SOAP) must not assume any kind of asynchronous communication mechanism.

Interface Definition Language

Many systems do not have an IDL of their own - or it is so weak (as in the case of XML-RPC Introspection) so as to be worthless. Interface definition languages give us two benefits: it reduces the amount of trust between processes, since processes can only send valid data to each other, and it allows us to declare explicitly which methods are supported by a given component. Our Interface Definition Language takes advantage of Python meta-classes to look like::

 class ThingyData(Data):
     """
     Data which identifies a target
     """
     uuid=lambda:str
     metadata=lambdaict

 class ThingiesOfType(Data):
     """
     Collection of targets of the same type
     """
     type=lambda:str
     thingies=lambda:array(ThingyData)

 class SearchInterface(Interface):
     """
     Interface to search functionality.
     """
     name='com.vmware.example.version1.Search'
     @public
     def RunQuery(query=str, maxResults=int):
         """run a query

         @param query: a query to run

         @param maxResults: maximum number of results to return

         @param results: query results
         """
         return dict(results=array(ThingiesOfType))

Since we cannot depend on the underlying transport to contain objects or references, we assume all communications happen in terms of "simple" (nested) data types: ints, strings, floats, uniformly typed lists, string to string dictionaries and string to vartype dictionaries. In order to do so in a dependable way, without application code having to translate manually, we have the Implementation metaclass::

 class SearchImplementation(object):
     __metaclass__ = Implementation
     interface = SearchInterface
     def RunQuery(self, query):
         d = defer.Deferred()
         myThingies= Structure('ThingiesOfType')
         myThingies.type = 'Something'
         myThingies.thingies = []
         reactor.callLater(5, d.callback, [myThingies])
         return d

It is possible to return either a deferred or a value from a method in an implementation. Note that there is no need for type checking inside application code - bad types would trigger an exception early on in the calling process.

Clients

Communication needs two sides. So far we have described how to write a server. We want client code to be equally transport-agnostic. In order to do that, we have a dynamic client object.

 def _simplifyObject(obj):
     if isinstance(obj, (int, float, str, bool, dict)):
         return obj
     elif isinstance(obj, list):
         return [_simplifyObject(x) for x in obj]
     elif isinstance(obj, Structure):
         return dict([(key, _simplifyObject(value))
                        for key, value in obj])
      else:
         raise ValueError("cannot simplify", obj)

 class MultiClient(object):
    def __init__(self):
        self.ifaces = set()
        self.routing = {}
    def add(self, iface, provider):
        self.ifaces.add(iface)
        for method in iface.methods:
            self.routing[iface.name+"."+method] = provider
    def callRemote(self, name, *args, **kwargs):
        args = map(_simplifyObject, args)
        provider = self.routing[name]
        return provider.callRemote(name, *args)

This means we can drop in any objects that support callRemote, commit to which interfaces are available for them, and the client code does all multiplexing. From inside the application code, we need no idea about which provider gives which interface - all we need is a callRemote.

Transport and Processes

The server code is the non-transport-agnostic part. It takes the fully-qualified name of a function, runs that function and treats the results as a (possibly nested) lists of implementations that should run in the given process. For the XML-RPC transport, it also expects on the command-line to have a list of which ports are running which interfaces. It creats XML-RPC proxies for those ports, and drops them into the singleton client. Running all servers manually means a lot of manual stitching up and repetition. In order to save that repetition, we have a so-called multiserver, using the Twisted Process Monitoring module (which takes care of starting and restarting processes). The multiserver gets a list of processes, specified in a JSON file. It then calculates what should be its client list, and runs all processes with the same client list, using the same server. We plan to add support for JSON-RPC and HTTPS as command-line switches to the multiserver, which would then run all processes, and assumptions, appropriately. Note, however, that a smarter multiserver could decide that some processes should use JSON-RPC and some XML-RPC, and run them all appropriately. Also note that currently the multiserver can only support one component per process, even though its underlying server is capable of more. We also plan to have the multiserver set a special environment variable "SECRET_KEY". This key would be a randomized string - all clients would send it, and all servers would require it before they process input. In this way, a malicious process running with different privileges on the same machine will not be able to get into the system.

Unified Front

We want to have a single port on which all public parts of the interface are exposed. We can do this in a way that is transport agnostic, even if it is not agnostic to some of the implementation details of Interface or the singleton client.

 def makeForwardingFunction(name, outargs):
     def retFunc(self, *args):
         remoteReturn = singletonClient.callRemote(name, *args)
         def cb(remoteReturnValue):
             return dict([(aname, argumentParse(tp, remoteReturnValue[aname]))
                              for aname, tp in outargs.iteritems()])
         remoteReturn.addCallback(cb)
         return remoteReturn
     return retFunc

 def makeForwarder(iface):
     d = dict(interface=iface)
     for name, func in iface.methods.iteritems():
         if func.isPublic:
             d[name] = makeForwardingFunction(iface.name+'.'+name, func.outargs)
     return Implementation('Forwarder', (), d)()

 def unifiedFront():
     for iface in singletonClient.ifaces:
         yield makeForwarder(iface, client.client)

Note that we could have set our multiserver to serve the unified front using one protocol (e.g., XML-RPC over HTTP) while running all clients using another one (e.g., AMQP). While this requires some complicated code in the multiserver, it does not require *any* kind of application code support.

Conclusion

With the right kind of separation between application code and transport code, it is possible to support complicated interaction topologies while completely hiding them from the application code. Python metaclasses, duck-typing and high level of dynamism makes writing this kind of code fairly easy.


16 Oct 2011 1:53pm GMT

13 Oct 2011

feedPlanet Twisted

Jp Calderone: Garlic Followup

Previously I shared some pictures of a garden bed Jericho and I have been working on. Over the long weekend we had some more time to spend on it. While we were away, we got some help preparing the rest of the garden, which would have otherwise taken weeks or months to do by hand:

You can see the oats we put in at the beginning of September in the garlic ged on the left side there. They didn't grow as much as I had hoped:

Perhaps due to some nutrient or mineral deficiency. To rectify that (and based on a soil test), we spread a number of amendments, starting with greensand to provide potassium:

We also spread rock phosphate (for phosphorus) and pelletized lime for calcium and to adjust the pH to be less acidic. And, importantly, compost - about 1 cubic yard over the entire bed (with which task my dad helped us out):

As you can see, we just left the oats in place. They are not cold hardy and will die soon enough without any help. With the bed thusly prepped, we began breaking up our "seed" garlic:

Garlic is most often grown by sowing cloves in the autumn for harvest the following summer. The winter encourages the clove to split and grow into a new bulb. We planted four varieties of garlic, but mostly Inchelium, a softneck variety.

These seed bulbs each had around a dozen cloves in them.

We planted the largest undamaged cloves. We also planted three varieties of hardneck garlic. Compared to the inchelium, these all look pretty similar to each other. Here's some Siberian Red:

The hardneck varieties have bulbs with fewer, larger cloves. After we broke up the cloves, we planted them! While one of us dropped cloves in pre-marked locations, the other followed behind and planted them.

The cloves are planted right-side-up about one inch deep. Finally we mulched them with straw to even out temperature variations and retain moisture.

Now the garlic sits tight until next year.

13 Oct 2011 1:26am GMT

08 Oct 2011

feedPlanet Twisted

Christopher Armstrong: Kingdoms of Amalur: Reckoning interactive story

The community people at Big Huge Games and 38 Studios are doing a neat little project on Facebook - a kind of crowdsourced choose-your-own-adventure story.

Here's the blurb from the initial post:

We wanted to give you a quick heads-up about an initiative we're starting on our Facebook page today. It is an interactive story that you can help us write! We post a screenshot and a poll, you help us decide what happens next. Muse has posted more info on our forums, so come take a look: link


And here's the first paragraph of the first post:

You bolt awake in an unfamiliar place, sweating and out of breath from the dream. You were in the tower again. The decay. The filth. And the Tuatha…. Panic. Fire. The gnome giving his life to save yours. So many dead. As you wipe the sleep from your eyes, you swear that you can still smell the moldy decay of Allestar Tower and the smoke of the fires that destroyed it.


It reminds me of things like Action Castle, which is kind of an interactive fiction party game. Oh, here's a great video: I was actually in the audience for this!



(edit: I see that the embedded video didn't get displayed for Planet IF readers -- here's the link: The "Action Castle" Incident at PAX East 2010)

Anyway, this is like that, but with thousands of people on Facebook. Neat!

08 Oct 2011 2:24am GMT

06 Oct 2011

feedPlanet Twisted

Glyph Lefkowitz: ἁγιολογία for r0ml

I have the rare distinction of being a second-generation software developer. Most recently, I mentioned this in an interview when asked who my programming heroes are. It might sound kind of corny, but I'm serious when I say that my father is my programming hero.

Robert "r0ml" Lefkowitz

My dad had a cool hacker alias in the seventies. He's been known as "r0ml" around the web since before there was a web. If you are in a particularly typographically hip part of the internet, it might even be "RØML". How many of your parents have a nom de plume with a digit, or a non-ASCII character in it? Or, for that matter, any kind of hacker pseudonym?
I had the good fortune to work with one of r0ml's colleagues, Amir Bakhtiar. Amir paid me one of the highest compliments I've ever received: he said that the code for systems I've worked on is similar to r0ml's in its style and exposition. My dad taught me how to program in x86 assembler, and in that process, I learned a lot about the way he thought about solving problems and building systems. I regard thinking that well, or even comparably well, as a real achievement.
That's not to say that I would do everything exactly the way that he does. For example, he writes a lot of networking code in Java. He doesn't use Twisted, for the most part. If you know me and you know my dad, you know that we disagree on plenty of stuff.
Unlike the stereotypical, often-satirized filial argument, these discussions are something I look forward to. Disagreeing with my dad is still one of the most intellectually challenging activities I've ever engaged in. Whenever I have a conversation with him about a topic where he has a different view, I come away enlightened - if not necessarily convinced.
Conversations among my friends occasionally turn to the topic of our respective upbringings, as they do in any close group. One of the recurring themes of my childhood is that, while my siblings and I were sometimes told to be quiet, we were never told to be quiet because our opinions weren't valuable. Sometimes we were told in unequivocal terms that we were wrong, of course. However, my dad always encouraged us to present our thoughts. Then, he wouldn't pull any punches in relentlessly refuting our arguments, using a combination of facts, estimates, calculations, and rhetorical flourishes. I learned more about influencing people and thinking clearly around the dinner table than in my entire formal education.
r0ml always questions glib answers, challenges the official version of events, distrusts things that are "intuitively obvious" or "common sense". The skepticism I've developed as a result of his consistent example has rarely led me astray. Glib answers, official versions, and common sense are frequently, if not always, wrong. He taught me to search for the non-intuitive answer, the surprising inflection point in the data.
In a roundabout way, he also taught my siblings and I how to perform some delightful rhetorical flourishes of our own, but also not to trust them. Pretty phrases can be deployed equally effectively in the service of illustration or deception. Although I can appreciate that parents often come to a point where they've had enough and a little deception can be a useful thing.
One cannot be a practiced rhetorician without a heaping helping of eclectic life experience; r0ml has that too. He's a fencer. And a juggler. He still has the highest score on Space Harrier of anyone I've ever met. (I can remember a crowd gathering in an arcade to see him start level 18.) He's an avid scholar of medieval thought and custom. For that matter, he's an avid scholar of a couple dozen other things, but listing them all would take a whole day.
He has the common occupational affliction of being a science fiction fan. However, fandom was never an identity for him. Again, by consistent example, he taught me to focus on my own creativity, and do something cool, never to just passively consume others' ideas. He treats entertainment as an inspiration, rather than an escape. For instance, one of the earliest memories I have about my father talking about software is a reference to the movie "Terminator". (Please keep in mind that this memory is ~20 years old at this point, so it might not be terribly accurate.) I remember him saying something like "All software should be relentless. If you remove its legs, it should use its arms. Whatever errors it encounters, it should deal with them, and keep going if it can."
Nevertheless, seeing "Tron: Legacy" with my dad, the hacker, in IMAX 3D, 20 years after we saw the original together... I didn't need to take a life lesson from that to think it was pretty rad[1].
Unlike many quiet geniuses who labor in obscurity, dispensing wisdom only to a fortunate few, r0ml is a somewhat notorious public speaker. You can see him this year at OSCON. If you hunt around the web, you can find some video examples of his previous talks, like this great 30-second interview[2] about the nature of open source process, from a talk he gave in 2008 (audio of the full talk here).
([1]: Although, jeez, what was the point of that whole open-source subplot at the beginning? It seemed like a great idea, but then it went absolutely nowhere!)
([2]: Speaking of not doing things exactly the way he does - where he uses a metaphor to "single-threading" and "multi-threading", I would have said "blocking" and "event-driven" - but more on that in a future post.)
Happy Father's Day, r0ml.

06 Oct 2011 10:12am GMT

04 Oct 2011

feedPlanet Twisted

Duncan McGreggor: Two Months at BlueLibris

Back for a Quick Write

Well, I've been at BlueLibris for two months and a day, now... and each one of those days I've wanted to blog about something that's happened. On occasion, I've G+'ed about something, but in general I've just been too busy to take a breath, much less use that breath to write some informal prose.

I've taken a few photos during this time, and put them up on flickr in this set; smaller versions are peppered throughout this post :-)

The First Day

Half Moon Bay, about 30 minutes from work.

I was born in California, but left when I was 8 years old. I returned after high school and lived for 2 years (1990-1992) in Northern California to attend language school. As a consultant and later an employee of various companies, I made regular trips out to the Valley and Bay Area, so was no stranger when I arrived in August this year. However, it was quite a shock to be actually working and living here!

After I left Canonical, I headed out from Colorado and took a few weeks off to go on a meditation retreat up in the Siskiyou Mountains of Oregon. This area was deeply reminiscent of the Colorado mountains, of which I am so fond. Such environs are my natural habitat... so imagine my shock when I was hurled 50 years into the future, deposited in something like a night scene from Bladerunner. 'Cause that's exactly how it felt :-) Not bad... just very, very different. Even though I'd been to the Valley a bunch of times in the past 10 years or so, it never really hit home... until it was my home.

Note the executive dress: shorts and flip-flops.

In the morning, the bright sun washed away visions of flying cars and robots. I was greeted by palm trees and traffic. After an hour commute from the East Bay to Menlo Park, what I had only seen in Google Maps was now staring me in the face: my new office building :-)

I had arrived early at 9:00am, so the place was locked up tight. I took that opportunity to meet some of the neighbors, walk around a bit, and then read up on the interesting facilities we were renting for our offices.

The text describes the facility's power.

I'd seen the solar panels of the building in the Google Maps satellite view, and had just assumed that this was a common accoutrement of California architecture. Turns out, our entire complex is soloar powered -- 100%! Our laptops, servers, monitors, microwave, fridge, and air condition all run on the Sun :-) This info was covered in great detail by one of the boards posted out in front of our small office complex.

Eventually, Soroush showed up, I introduced myself, and we made our way inside. The interior was even cooler than the exterior :-) There's a large open space where most people sit, a dark cave for the software engineers, an unused loft upstairs that looks down over the open space, and a cute little kitchen area adjacent to (and part of) the loft.

The Intervening Weeks

Within a day or two, I'd purchased some bean bags, nerf guns, nerf balls, action figures, and die cast cars. It was starting to feel like a start-up :-) (And the nerf gun fights reamain a regular brain-break for many of us!). A few days after that, Tim Hyland bought some sling-shot monkeys... I think we use these even more than the nerf products!

Playtime aside, the work has been very grueling. We all work late hours (but have a good time hanging out while we do so), wear many hats, and have more responsibilities than we care to enumerate. Equally, however, we're all deeply invested in the company, believe in the services we're providing to the health and other industries, and most of all are committed to the success of the company.

On that note, most of us were involved in ventures during the dot boom/bomb, and accrued wisdom from that area on how to conduct ourselves responsibly, even in the midst of all the excitement our innovations are generating.

I look forward to the time (soon!) when I'll be able to post more about these :-) For now, though, I can be very general and give a brief overview of our tech.

Our Technology

Matt playing along with a thumbs-up :-)

I worked on several device projects for Canonical, but never were we actually building our own hardware like we do at BlueLibris. This has been a fascinating introduction to a part of the tech world with which I am starting to gain some familiarity: circuitry design, component integration, plastics work -- the list goes on and on. Soroush Salehian and Matt Maibach lead these efforts, with Ram Fish weighing in regularly with his extensive EE background. These are the guys iterating on the hardware for our real-time, 3G health monitoring device.

As events are generated on the device, they get pushed up to "the cloud" where additional processing and heuristics are performed. We're working with lots of maths, machine learning, custom algorithms, etc., all in a distributed computing environment. We're using nosql solutions for long-term storage, and a Python-based framework for web management tools. We're in the process of converting the original application into a JavaScript-driven suite of applications. We're also giving the Python an overhaul in anticipation of significantly increased load and general usage over the next year (we've got some exciting partnerships and customers -- both existing and forth-coming). Henry Messenger and myself are working on these bits right now. To get the device talking to the cloud and collecting all the right data, we have the talented Kent Ryhorchuk hacking like a madman and consuming the APIs we're providing him.

There are other software development efforts in the works that I can't really talk about... but given the space, I'm sure you can imagine the possibilities :-)

Future Posts

I've run out of time for now, but I do want to mention that with BlueLibris, there is a tantalizing connection to where I see technology going in the next 10-20 years (ultra large-scale systems), and I'd like to say more about that soon. Furthermore, we should have some nice news-worthy items that will get their own press releases, but I'll be making community-friendly updates here on that as well. I'm also taking steps to make sure that we contribute back to the open source community, and will have posts relating to those efforts as well.

So watch this space, and look for more developments soon... :-)


04 Oct 2011 7:58pm GMT

28 Sep 2011

feedPlanet Twisted

Stephen Thorne: Cat Pictures

Cfbnb

Now that's one statically charged cat. I wonder if someone was
taunting him with catnip or tuna at the top of the slide? At least
he's getting some exercise.

28 Sep 2011 10:20am GMT