fidzu : Python

Here's an awesome trick: you can cast arrays to objects in PHP then access them like objects. The brilliant part about it is that it appears to make them act like objects, i.e. $b = $a makes $a and $b point to the same object (like pretty much every other dynamic language) rather than copying $a.

$ php

$a = array(1, 2, 3);
$b = $a;
$a[] = 4;
$b[] = 5;

$c = (object)$a;
$d = $c;
$c->>foo = "bar";
$d->>baz = "boz";

echo '$a: '.var_export($a, TRUE)."\n";
echo '$b: '.var_export($b, TRUE)."\n";
echo '$c: '.var_export($c, TRUE)."\n";
echo '$d: '.var_export($d, TRUE)."\n";
?>>
$a: array (
0 =>> 1,
1 =>> 2,
2 =>> 3,
3 =>> 4,
)
$b: array (
0 =>> 1,
1 =>> 2,
2 =>> 3,
3 =>> 5,
)
$c: stdClass::__set_state(array(
'foo' =>> 'bar',
'baz' =>> 'boz',
))
$d: stdClass::__set_state(array(
'foo' =>> 'bar',
'baz' =>> 'boz',
))

Comment

23 Jul 2008 11:28pm GMT

This post from John Cook, summarizing this post from Thomas Guest, is a pretty good description of what our students have spent the summer finding out…

23 Jul 2008 10:08pm GMT

One of the major challenges we face with the design of our new linguistic command-line project is that of trust. As Zittrain mentions in The Future of the Internet, this is really the fundamental problem of generative systems, and also their most valuable asset: the ability for a user to run arbitrary code is simultaneously what gives the personal computer its revolutionary power, but it's also its greatest vulnerability.

At present, because our project is still in the prototyping stage, we're opting for freedom of expressiveness and experimentation over security. That means that all the various verbs we write, while written in JavaScript, are always executed with Chrome privieges, meaning that they're capable of doing whatever they want to the end-user's computer.

So the particular dilemma that needs to be solved here is: how can an end-user trust that a verb won't do anything harmful to their data or privacy-be it intentional or accidental-while still providing a low barrier of entry for aspiring authors to write and distribute their own verbs?

We've considered some technical options so far. One is the idea of "security manifests" that come with verbs, specifying what a verb is capable of doing. For example, the "email" verb mentioned in my last post could specify in a manifest that it needed access to the user's email service and their current selection. This information could then be presented to a user when they choose to install a verb. At an implementation level, the code could run in a specially-constructed sandbox to ensure that the verb code never steps outside the bounds prescribed by its manifest. Alternatively, or in addition to this, an object-capabilities subset of JavaScript like Caja can be used. Such mechanisms ensure that untrusted code can only go as far as the end-user lets it-which, unfortunately, also puts a burden on said end-user. While I don't personally mind having such a burden myself, I know I wouldn't want to put it on my friends and family.

Digital certificates are another component of potential solutions, but they too have their own problems. While they're easy for centralized corporations to deal with, they're problematic for more distributed operations, and the monetary cost involved in obtaining one significantly increases the barrier to entry for individual software authors. And even signed code doesn't prevent the more privacy-invasive-but not outright malicious-classes of software like spyware.

As I've indicated earlier, this general issue of trust isn't a new problem, or even just an important issue for an experimental Firefox addon. It's what Zittrain believes is at the core of the future of the internet and the PC, and the solutions we create-or don't create-will determine whether their future is one that is sterile or generative. Windows Vista, being the most frequently exploited operating system as a natural result of its widespread use, is the harbinger of a future that relies entirely on technology and corporate trust heirarchies without taking any kind of social mechanisms into account; the result is a notoriously hard-to-use user interface that places an enormous burden on the end-user to constantly make informed security decisions about their computing experience. I don't think that the answer is merely a "less extreme" version of Vista-which is the model that most other operating systems and extensible applications seem to be following-but rather that a more effective solution is primarily a social one that is supported by technological tools.

I have a particular solution in mind that I'll be writing about soon. That said, I'd still love to hear any thoughts that anyone has on this topic.

23 Jul 2008 8:53pm GMT

But the result was worthwhile: you can now view a super-tree showing the evolution of the dinosaurs, or read about its creation. I think it's beautiful:

23 Jul 2008 7:24pm GMT

I am rather busy at the moment pushing my new company forward. Yes, for those of you who don't know yet, I have just started up my own company. It has been very rewarding so far, even considering all the hassle of tasks such as fund-seeking, registering the company, creating a website, develop the products, and making sure that everything work. Wow!, it's been a great challenge. The focus of

23 Jul 2008 6:38pm GMT

I'm gave sqlalchemy-migrate a try today. I like it, and I'm going to keep using it. The one downside is that it's a bit hard to find "the least you need to know" in the documentation, especially if you lean old-school like me and prefer to write your upgrade scripts in raw sql. So here's my stab at it.

Create your manage script. If you have development/production dbs with different connection urls, create two scripts with the same repository but different urls:

migrate manage dbmanage.py --repository=path/to/upgradescripts --url=db-connection-url

For each database, create the Migrate metadata (a migrate_version table):

./dbmanage.py version_control

Create an upgrade script. This will create a script [next version number]-[database type]-upgrade.sql in the "versions" subdirectory of your "repository." That's all, so you could certainly do this by hand if you prefer, but letting the script do it is less error-prone:

./dbmanage.py script_sql sqlite

Edit the script.

For each database, apply the upgrade:

./dbmanage.py upgrade

Repeat the script/upgrade process as needed. That's it! Everything else is optional!

(What this gives you is a process where all your developers can have their own local database for development, and all they have to do is "svn up; ./dbmanage.py upgrade" without having to worry about which upgrade scripts have been applied or not.)

23 Jul 2008 4:16pm GMT

TurboGears has used a tree of controller objects to do URL dispatch since 1.0. Which is nice and easy to understand, and makes getting started very quick. But, it wasn't always apparent how you should use it to do RESTful dispatch, since the HTTP verbs all ended up going to the same controller method.

You could dispatch within that method, but that never felt totally clean to me. And I've been thinking and talking about better ways to do this. The good news is that Rick Copland just "made it happen" after a conversation in atlanta last week.

Using his trick you can write code like this:

class Root(controllers.RootController):

    class person(RestMethod):
        @expose('json')
        def get(self, **kw):
            #...do stuff...
            return dict(method='GET', args=kw)

        @expose('json')
        def post(self, **kw):
             #...do stuff...
            return dict(method='POST', args=kw)

        # NOT exposed
        def delete(self, **kw):
            return dict(method='DELETE', args=kw)

And then HTTP GET and POST verbs will be routed to their respective methods. This makes working with RESTFul api's easier. And on that front I'm very much looking forward to Dojo 1.2 which has all of kinds of restful json data store goodness.

Hopefully Rick's recipe can find it's way into the 1.x core code somewhere so that it's even easier to do. But for now a couple dozen lines of code gives you everything you need for RESTful goodness.

23 Jul 2008 4:00pm GMT

Looks like the TG 1.x team has been doing good things. CherryPy 3 is support is a very, very good to hear that it's been integrated into TG1, because this sets the stage for a much longer support cycle for TG1.

CherryPy 3 is a more solid base to work with, and it's easier to invision supporting it for several years than it was with CherryPy 2.x. And the test-refactoring work that has gone into making this happen is a huge benefit as well, because you will be able to write application level tests that work in tg 1.5 and will still work in 2.0, easing that migration path.

Florent Aide may be on vacation, but things are still moving forward very quickly on the 1.5 front. I'm looking forward to seeing a release with all this great stuff in it.

23 Jul 2008 3:44pm GMT

ophirk has published a great essay on getting Python / IronPython accepted as a development language within the company he works for:

• Python for Executives

IronPython has worked well for them because of easy integration with C# and the .NET infrastructure.

He lists the four main fears that their managers had to face in adopting Python:

• Recruiting - It turns out that smart and innovative people are attracted by Python .Innovative smart people can also learn Python very quickly.
• Risk - in one year we had only five cases where a Python specific bug occurred. We had many more ASP. NET, JavaScript, HTML and even Java specific problems.
• Maturity - not only is Python very solid, so is IronPython. The code is stable, the smart features work, the community is responsive and the libraries are rich. The only downside is the IDE and Visual studio integration, but this is progressing as well.
• Performance - Python flexibility allowed extremely fast optimizations at the system and algorithmic levels, without requiring premature optimization. The system and algorithm changes are the ones critical in a real world environment. Some issues are showing up, but it seems they can be solved with a reasonable cost.

23 Jul 2008 2:12pm GMT

Ben Hall, a .NET developer from the UK, has announced a new project to create an IDE specifically targeting IronPython and IronRuby. In the future the project will also be extended to cover all languages that use the Dynamic Language Runtime.

• Announcing IronEditor - An Editor for IronRuby, IronPython and other DLR languages
• IronEditor Homepage and Download on Codeplex

Ben's goal is to create an IDE for Dynamic Languages that is lighter-weight than Visual Studio.

Out of the box the project already supports IronPython and IronRuby, including the creation of new projects or source files and executing code. IronEditor is based on Windows Forms and has a syntax highlighting code editor.

Currently to build IronEditor you will need Visual Studio 2008 (Express is fine), but there is a binary available for download. The blog entry shows screenshots of IronEditor running on Mono. Mono compatibility is an important goal for IronEditor. Currently the only feature missing on Mono is syntax highlighting.

23 Jul 2008 1:57pm GMT

Yesterday at OSCON I had the pleasure of announcing the 2008 Frank Willison award. This was instituted by O'Reilly in memory of their editor in chief, whose "Frankly Speaking" column was a regular joy on the web: guaranteed to entertain and inform. Frank was a great supporter of Python, and a believer in the value of communities.

Martin von Löwis continues to be a tireless worker on behalf of the Python community. He has been a long-term contributor to the Python core, and regularly answers questions on both the python-dev list and the comp.lang.python newsgroup. A PSF director since 2002 he was also the prime mover in transitioning the Python development infrastructure from SourceForge, and has created several Roundup issue trackers for various areas. He chaired the PSF grants committee, which among other achievements kept Jython alive when its future looked uncertain. I could go on, but you get the idea: when something needs doing, he rarely hesitates to step up to the plate.

Martin has, until today, been one of our unsung heroes. I hope all readers will join me in congratulating him on this well-deserved award.

23 Jul 2008 1:46pm GMT

Yesterday Jim an I had a tutorial session at EuroPython about dynamic compilation in Python. We brought up the topics of import hooks (PEP 302) since we have successfully used them as an opportunity to create code dynamically. The code example we demonstrated for that was one of the actual import hooks that we had used in a Jython setup. Even if it was no more than 80 lines of code, it might not have been the most accessible example. A few people asked me afterwords if I had a more simple example, so by public request, here is a simple meta_path hook that prevents the user from importing some modules.

import sys



class Restriction(object):

    __forbidden = set()

    @classmethod

    def add(cls, module_name):

       cls.__forbidden.add(module_name)

    def find_module(self, module_name, package_path):

       if package_path: return

       if module_name in self.__forbidden:

       return self

    def load_module(self, module_name):

       raise ImportError("Restricted")



sys.meta_path.append(Restriction())

add = Restriction.add

del Restriction

If we walk through this class from the top we first of all have a set containing the names of the modules that we will not allow the user to import and a method for adding modules to that set.
The next method, find_module, is the method invoked by the import subsystem. It is responsible for locating the requested module and return an object capable of loading it. The arguments passed to find_module are the fully qualified module name and, if the module is a sub module of a package, the path where that package was found. If the module cannot be found one should either return None or raise an ImportError. This is a handy way to delegate to the next import hook or the default import behavior. If it returns a loader object on the other hand, no other import mechanism will be attempted, which of course is useful in this case. So in this implementation we return self if the module name is found to be the one of the modules that we want to prevent the user from importing.
The loader object should implement the method load_module, which takes only one argument, the fully qualified module name and returns the corresponding module or raises an ImportError if the import fails. In this case we know that load_module is only ever invoked if we want the import to fail, therefore we always raise an ImportError.
There really isn't much more to it. It should be noted however that this isn't a good way to implement security restrictions in Python, since it is possible for any user code to remove the import hook from sys.meta_path, but I still think it makes for a good introductory example to import hooks.

Happy hacking!

23 Jul 2008 1:21pm GMT

Suppose you have a room full of scientists-hundreds of 'em-and want to find out how they actually use computers in their work. There isn't time to interview them individually, or to record their desktops during a typical working week, so you've decided to ask them to self-asses their understanding of some key terms on a scale of:

1. No idea what it is.
2. Use it/have used it infrequently.
3. Use it regularly.
4. Couldn't get through the day without it.

My list is below; what have I forgotten, and (more importantly) how would you criticize this assessment method?

1. A command-line shell
2. Shell scripts
3. Version control system (e.g., CVS, Subversion)
4. Bug tracker
5. Build system (e.g., Make, Ant)
6. Debugger (e.g., GDB)
7. Integrated Development Environment (e.g., Eclipse, Visual Studio)
8. Numerical Computing Environment (e.g., MATLAB, Mathematica)
9. Inverse analyzer (e.g., Inane)
10. Spreadsheet (e.g., Excel)
11. Relational database (e.g., SQLite, MySQL, Oracle)
12. Layout-based document formatting (e.g., LaTeX, HTML)
13. WYSIWYG document formatting (e.g., Word, PowerPoint, OpenOffice)

Now, you have the same room full of scientists, and you want to find out how much they know about software development. There still isn't time to interview them or have them solve some programming problems, so again you're falling back on self-assessment. This time, the scale is:

1. No idea what it means.
2. Have heard the term but couldn't explain it.
3. Could explain it correctly to a junior colleague.
4. Expert-level understanding.

and the terms themselves are:

Once again, my questions are (a) what have I forgotten, and (b) how "fair" is this as an assessment method?

23 Jul 2008 12:41pm GMT

Suppose you have a room full of scientists-hundreds of 'em-and want to find out how they actually use computers in their work. There isn't time to interview them individually, or to record their desktops during a typical working week, so you've decided to ask them to self-asses their understanding of some key terms on a scale of:

1. No idea what it is.
2. Use it/have used it infrequently.
3. Use it regularly.
4. Couldn't get through the day without it.

My list is below; what have I forgotten, and (more importantly) how would you criticize this assessment method?

1. A command-line shell
2. Shell scripts
3. Version control system (e.g., CVS, Subversion)
4. Bug tracker
5. Build system (e.g., Make, Ant)
6. Debugger (e.g., GDB)
7. Integrated Development Environment (e.g., Eclipse, Visual Studio)
8. Numerical Computing Environment (e.g., MATLAB, Mathematica)
9. Inverse analyzer (e.g., Inane)
10. Spreadsheet (e.g., Excel)
11. Relational database (e.g., SQLite, MySQL, Oracle)
12. Layout-based document formatting (e.g., LaTeX, HTML)
13. WYSIWYG document formatting (e.g., Word, PowerPoint, OpenOffice)

Now, you have the same room full of scientists, and you want to find out how much they know about software development. There still isn't time to interview them or have them solve some programming problems, so again you're falling back on self-assessment. This time, the scale is:

1. No idea what it means.
2. Have heard the term but couldn't explain it.
3. Could explain it correctly to a junior colleague.
4. Expert-level understanding.

and the terms themselves are:

Once again, my questions are (a) what have I forgotten, and (b) how "fair" is this as an assessment method?

23 Jul 2008 12:41pm GMT

I'm announcing two short training classes in Python programming for chemical informatics. I will teach the first in Leipzig, Germany on 6-7 October. It will be hosted by the Python Academy. I'm still working on the details of the second. It will be in the San Francisco Bay Area in early December.

For details about the price, topics and other information, see http://dalkescientific.com/training/ or send me an email.

I work with and develop software tools for computational chemists, mostly in small-molecule chemistry. These are scientists, not programmers, but they use computers every day as part of their research. Most need to do some programming, pehaps to implement a new algorithm, or more likely to handle something that's too tedious or error prone to do manually, like automating analysis of 10,000 compounds.

Most chemists have some training in programming, usually a semester or two of introductory programming classes in college and a lot of training from the school of experience. The latter usually means informal training from labmates, who are also not programming experts.

I've seen many of my chemists friends work hard at getting software to do what they want it to do. Chemists, like most other people who had to spend years of mostly isolated work to get a PhD, know how to perservere. But they would rather do science, not spend time figuring out how to use software.

My training classes are meant for them. I'll cover different aspects of how to be better at Python programming using examples that are directly relevant to doing small molecule in silico research.

23 Jul 2008 12:00pm GMT

I'm announcing two short training classes in Python programming for chemical informatics. I will teach the first in Leipzig, Germany on 6-7 October. It will be hosted by the Python Academy. I'm still working on the details of the second. It will be in the San Francisco Bay Area in early December.

For details about the price, topics and other information, see http://dalkescientific.com/training/ or send me an email.

I work with and develop software tools for computational chemists, mostly in small-molecule chemistry. These are scientists, not programmers, but they use computers every day as part of their research. Most need to do some programming, pehaps to implement a new algorithm, or more likely to handle something that's too tedious or error prone to do manually, like automating analysis of 10,000 compounds.

Most chemists have some training in programming, usually a semester or two of introductory programming classes in college and a lot of training from the school of experience. The latter usually means informal training from labmates, who are also not programming experts.

I've seen many of my chemists friends work hard at getting software to do what they want it to do. Chemists, like most other people who had to spend years of mostly isolated work to get a PhD, know how to perservere. But they would rather do science, not spend time figuring out how to use software.

My training classes are meant for them. I'll cover different aspects of how to be better at Python programming using examples that are directly relevant to doing small molecule in silico research.

23 Jul 2008 12:00pm GMT

If you've been following what's going on in the JVM-languages community you have probably stumbled across John Roses blog. One of his entries was about interface injection. In short wordings interface injection is the ability to at runtime add an interface to a class that was not precompiled as implementing that interface.
Interfaces are injected at one of 3 situations:

• When a method of the interface is invoked on an object.
• When an instanceof check for the interface is performed on an object.
• When the interface is queried for via reflection (I can see this working with Class#isAssignableFrom, but I have my doubts when it comes to Class#getInterfaces, although I'm sure someone smart will be able to solve this without having to know about all injectable interfaces beforehand).

When any of these occur a class can either have the interface implemented already (in the regular way) or a special static injection method on the interface class is invoked. It is up to this injection method to determine if the given class can implement the interface or not. If it determines that the class can implement the interface any missing methods have to be supplied at that time. John suggests that these methods are to be supplied as method handles. Since method handles, according to the EDR of the InvokeDynamic proposal (JSR 292), can be curried this would make it possible to attach an extra state object to the returned method handle, or to return a different implementation of the interface depending on the class they are injected to. Once an injection method of a specific interfaced has been invoked for a specific class, the injection method will never be invoked for that class again, meaning that once a class has been found to not implement an interface, that information will be final, and once an implementation of an interface has been supplied, this implementation can never be changed. This is important since it will allow the VM to perform all optimizations, such as inlining, as before.

What can this be used for?
As a language implementer on the Java platform I think interface injection would be a blessing. In fact I think it is the one thing that would simplify the implementation of languages on the JVM the most. Any language probably has a base interface (as Java has java.lang.Object and Jython has org.python.core.PyObject), let's be unbiased and call it "MyLangObject" for the sake of the continued discussion. There are two things that make the Java platform great:

1. There are a lot of really good toolkits an libraries implemented for the Java platform.
2. There are a lot of great languages for the Java platform in which even more great libraries and toolkits will be developed.

Therefore, if you are implementing a language for the Java platform you would want to interact with all of these libraries and toolkits. Problem is that most of them haven't been designed with your language in mind, and they shouldn't be. If MyLangObject was an injectable interface all you would need to do to be able to integrate with any object from another language would be to just interact with it through the MyLangObject interface, and the injection mechanism would take care of the rest.
The injection mechanism could even be used with the classes within your language. Instead of having a base class supplying the default implementation of the methods of MyLangObject you could let the injection method return the default implementation for your methods.
Or why not use interface injection to support function invocation with different argument counts. Each function in your language would implement a set of call methods, one for each argument count it can be invoked with. Your language would then have a set of injectable Callable interfaces one for each argument count that any function in your language can be invoked with, each with only one call method, with the appropriate number of arguments. These interfaces could be generated at runtime if your language supports runtime code generation. The default implementation of the call method in each Callable interface would of course raise an exception, since the function obviously doesn't support that argument count if it doesn't implement the appropriate method.
Interface injection really does provide a huge set of opportunities.

How could interface injection implement a Meta Object Protocol?
There is a great project initialized by Attila Szegedi of the JVM-languages comminuty to create a common meta object protocol (MOP) for all languages on the JVM. With interface injection this would be a simple task.

1. Let all objects in your language (that supports the MOP) implement the (probably not injectable) interface java.dyn.SupportsMetaObjectProtocol. An interface with only one method:
   java.dyn.MetaObjectProtocol getMetaObjectProtocol();
   This would return the implementation of the java.dyn.MetaObjectProtocol interface for your particular language.
2. The java.dyn.MetaObjectProtocol contains methods for getting method handles for each dynamic language construct that the community have agreed to be a good common construct, such as getters and setters for subscript operations. These method handles would come from the actual implementation of them for your particular language, and would therefore benefit from every imaginable optimization you have cooked up for your language.
3. When the main interface of my language is being injected into a class from your language it finds that your class implements java.dyn.SupportsMetaObjectProtocol and uses that to get the method handles for all dynamic language constructs supported by my language, rebinding them them to the method names used in my language.

And as simple as that interface injection has been used to implement a common ground for all languages on the Java platform with absolutely no overhead. I'm not saying that this is the way to implement a meta object protocol for the Java platform, I am just suggesting one way to do it, someone a lot smarter than me might come up with a much better implementation.

To sum things up: I can't wait until the JVM supports interface injection.

Edit: this post has also been re-posted on Javalobby.

23 Jul 2008 10:53am GMT

• 78 RPM record MP3s
  Great collection of authentic old music
• Nine ways to obfuscate e-mail addresses compared
  Useful results of a study on address harvesting

23 Jul 2008 5:00am GMT

Trying to get the RPC-server set up to handle multiple clients. This will allow me to request return statuses from the RPC-server to return via the sugarbot-launcher... which in turn allows the whole sugarbot/buildbot thing to work.

23 Jul 2008 12:30am GMT

Mark, an accomplished speaker, started by confessing that he had no doubt that Linux would be the platform of the future, and suggesting that tools like Firefox and bazaar had claimed ownership of their application space by providing extensibility in a modular way: conform to the interfaces, and the application becomes a platform. All the same, he insisted there is also a need to work with Windows (which makes sense since there are so many Windows systems around).

One of the issues Mark discussed was how we can extend agile techniques in ways inspired by community-driven processes. Architecting your tool set and your enterprise for collaboration and communication is essential. Ubuntu's governance is entirely separated from Canonical's, and Mark suspects that might turn out to be a best practice.

Tools must be extensible, and whether they are open source or proprietary they must be usable as components in diverse systems. The ultimate goal is "permissionless" development - new people coming in wanting to do new things need to be able to take the tools and run with them to implement their own ideas. His ideal for Ubuntu is for any developer worldwide to branch the Ubuntu code and publish their version in ways that other people can in turn pick up and run with.

Another best practice Mark has identified is time-based releases, which lend a rhythm to the development process and allow developers to keep track of what's happening in the trunk, which should be releasable at any time. More and more projects are thinking about synchronizing their releases. Imagine if a multitude of Linux distributions could collaborate to release at the same time, providing a "development cadence", encouraging individual tool developers to finalize their own releases for incorporation into a multitude of distributions.

Mark sees todays open source development world as struggling to provide free software, but the questions is how will the economic models support this? Free software is easy to love: it costs nothing to acquire. But who will fund the development of the free software? This will require innovations in economics that will, with a historical perspective, possibly be one of the more significant changes in these turbulent times. Having three almost-monopolies each controlling their own distribution would be counter-productive, and advertising on the desktop is offensive.

At Canonical they are hiring people to work on the desktop to build tools that benefit the providers of services. This will fundamentally change the world.

The challenge of the next two years is to lift the Linux desktop to a level where it is effectively art! How do Linux distributions not emulate Apple, but blow right past them. Jamie Zawinski says we should be aiming to build software tools that help their users get laid! The only doubts Mark had about that as a goal were related to Jamie's stated goal of making software that his mother could use ...

The Web 2.0 explosion has taught us that to survive in the web world a software product needs to be instantly attractive, and deliver functionality immediately in a profoundly usable way. Canonical have been investing in the process and the technology, and they are now trying to
realize the dream that Mark so elegantly elucidated in his presentation.

23 Jul 2008 12:21am GMT

Luke Petrolekas and I are thiiiiis close to having the Software Carpentry notes converted to a wiki. Once they are, I'm going to be working with Tina Yee to update them, do the examples in MATLAB as well as Python, and fix some longstanding bugs. I'm also going to resurrect the project's two mailing lists (one for occasional announcements, the other for people interested in developing new material and/or teaching the course). If you'd like to be on either or both, please let me know.

22 Jul 2008 11:35pm GMT

Luke Petrolekas and I are thiiiiis close to having the Software Carpentry notes converted to a wiki. Once they are, I'm going to be working with Tina Yee to update them, do the examples in MATLAB as well as Python, and fix some longstanding bugs. I'm also going to resurrect the project's two mailing lists (one for occasional announcements, the other for people interested in developing new material and/or teaching the course). If you'd like to be on either or both, please let me know.

22 Jul 2008 11:35pm GMT

We had our mid-summer student barbecue a couple of Fridays ago, and a good time was had by all. The photos are courtesy of Qiyu Zhu:

In transit

This must be the place.

Whozat?

Or we could use recursion!

Mmm…

Dancing, dancing!

22 Jul 2008 11:32pm GMT

I'm gave sqlalchemy-migrate a try today. I like it, and I'm going to keep using it. The one downside is that it's a bit hard to find "the least you need to know" in the documentation, especially if you lean old-school like me and prefer to write your upgrade scripts in raw sql. So here's my stab at it.

Create your manage script. If you have development/production dbs with different connection urls, create two scripts with the same repository but different urls:

migrate manage dbmanage.py --repository=path/to/upgradescripts --url=db-connection-url

For each database, create the Migrate metadata (a migrate_version table):

./dbmanage.py version_control

Create an upgrade script. This will create a script --upgrade.sql in the "versions" subdirectory of your "repository." That's all, so you could certainly do this by hand if you prefer, but letting the script do it is less error-prone:

./dbmanage.py script_sql sqlite

Edit the script.

For each database, apply the upgrade:

./dbmanage.py upgrade

Repeat the script/upgrade process as needed. That's it! Everything else is optional!

(What this gives you is a process where all your developers can have their own local database for development, and all they have to do is "svn up; ./dbmanage.py upgrade" without having to worry about which upgrade scripts have been applied or not.)

22 Jul 2008 11:19pm GMT

Launching a Startup in 3 Hours was a great talk given by Andrew Hyde (of techstars.org) and Gavin Doughtie (of Google). Both of the speakers are heavily involved in the recent trend of doing "Startup Weekends", and techstars.org is an organization that hosts startup weekends all around the US (and I think internationally as well - Andrew mentioned one in Germany if I heard correctly).

The first half of the talk was about the general concept of a startup weekend, the problems it avoids ("we've been working for 9 months and haven't launched anything"), the problems it brings up ("If you're not using Java, you're an idiot, so count me out!!"), and lots of details about how to organize, how to assign roles, and some common tools they use (like Basecamp and whatever your IM of choice is). There was also talk of legal issues, how (basically) to think about forming the company with the people involved, and decisions that need to be made at a business level aside from just the coding.

The second half of the talk wasn't a talk at all. Instead, people who had ideas stood up, presented their idea in a couple of sentences, and once the ideas were out there, we were told to break into groups and get to work! So people would get up and move over to the person whose idea they liked, and they'd start brainstorming. I decided to head out after about 30 minutes of observing and talking with people about ideas, but when I left, there were probably 6-8 groups of people engrossed in conversations, and the energy level was very high. Overall, it was a really exciting experience!

22 Jul 2008 10:40pm GMT

Our summer students have chosen a logo for their t-shirts. Yes, that is a hippo wearing a toque (knit cap, if you're American), a scarf, and an inflatable swim toy. You wouldn't believe how much discussion it took to get this far…

22 Jul 2008 7:43pm GMT

22 Jul 2008 6:48pm GMT

If you like font-related humor, you will like this: Font Conference. I have only two complaints: Comic Sans shouldn't be the hero, and it should have been called Type Cast.

22 Jul 2008 6:28pm GMT

22 Jul 2008 6:28pm GMT

Oh dear - I have fallen behind again.

• MyTTC.ca is up (Jorge's comments are more coherent than mine). Should this be the starting point for student projects in my fall CSC301 Software Engineering course?
• The science maps at Eigenfactor are pretty cool - this post talks about what's going on behind the scenes.
• How much of your code is comments? Meaningful identifiers? Spacing? Better yet, which open source project are you most like?
• Jon Udell is watching you, Senator Sununu.
• And finally, who owns science?

22 Jul 2008 6:16pm GMT

22 Jul 2008 5:18pm GMT

Wow, my name is on the DjangoCon schedule on Sunday, 2:25 pm. Jim Baker told me about it just a few days ago. This may be a bit surprising, but looks like the whole DjangoCon is a bit surprising.

I can hardly believe that I'm going to be there, and I'm extremely happy to have this opportunity to meet the Django community and to show what we are doing on this GSoC-funded project of getting Django running on Jython and integrating it with some cool JVM stuff.

Well, I still have to do all the paper-work (I don't have a US visa yet, this is going to be my first visit to the country), and there is not too much time to do it. Not to mention that I've to practice my spoken English. But we humans are optimistic by nature, and I think that such optimism let us do most of the great things we do!

22 Jul 2008 12:45pm GMT

Put away your IDE, code using a pencil, and do a big design up front. Get it right the first time. It can be done. Naturally, I'm trolling a bit, but that link is a very short, very fun read.

It reminds me of an interview with Donald Knuth that I've been thinking about a lot lately. Knuth said:

With the caveat that there's no reason anybody should care about the opinions of a computer scientist/mathematician like me regarding software development, let me just say that almost everything I've ever heard associated with the term "extreme programming" sounds like exactly the wrong way to go...with one exception. The exception is the idea of working in teams and reading each other's code. That idea is crucial, and it might even mask out all the terrible aspects of extreme programming that alarm me.

Somehow André Bensoussan and Donald Knuth are both capable of designing everything up front, and just getting it right. Since I've always been an agile coder, I find such stories amazing. I wonder if Bensoussan would still work that way these days. The systems are larger, the libraries are less stable, the timelines are shorter, and there are more people you have to integrate with.

22 Jul 2008 10:22am GMT

It's been a while since I used Python for anything larger than scripts few tens lines long, so it feels really great to do it again. I did discover however that I became a bit rusty. Not so much in not being able to achieve what I want as not being sure that I do it in a sensible and pythonic way.

I've been working on a private project where I came to a following problem. API calls can trigger various responses, somewhat like HTTP, containing status codes together with a short description. Every faulty response should trigger its own exception, which led me to my first implementation:

class Unauthorized(Exception):
    status = 101
    value = "Unauthorized."

I didn't like it even though it looks and behaves like it should. What I wanted was a better overlook of possible responses in a way where I have to make any possible changes easily and only at one place.

My second attempt was auto-generating exception classes using type. Since class definition took only a line instead of three, it certainly achieved better transparency, but I still had to make changes at two places.

Final step was to auto-generate classes in a loop. To do this I attached them to module namespace using globals() dictionary. Actually I used __builtin__ one at first, but it obviously didn't work that great.

So this is what I have now. It works and achieves my goals. I only need to change dictionary to add a new response or change existing one and it could hardly be more readable.

But is it pythonic enough? If not, what would be, apart from traditional way described in first step?

22 Jul 2008 7:39am GMT

The evening plans didn't wait for talks to be done. The IRC channel (#oscon on irc.freenode.net) was alive with talk of prospects for dinner and drinks after the conference. I myself was torn between a group going out for Lebanese and another going to Henry's, but opted to go with my buddies from home to Henry's.

It was worth it. If you haven't been, Henry's Tavern boasts 100 beers and hard ciders on tap (oddly, the beer list is the only menu *not* online - guess it changes too frequently). There are a ton of local beers that you can't even get on the east coast just waiting for you to try, but there are also some rare treats, like the Belgian Lambic beers, which you don't often see on tap. The food is a little pricey, but is really good, and the staff is very friendly. A couple of us were in a rush to get back by 7 for the BoF sessions, and when we asked the waittress how easy it was to catch a cab, she immediately informed us that she would have the hostess call one for us. About 2 minutes later we were in a cab on our way back (we wouldn't have made it back in time if we had to walk back to catch the light rail).

I was not one of those rushing to a BoF, so I did a little poking around the area near the convention center. It was getting dark, and I didn't want to stray too far, but I did find a couple of points of interest. First, there's a bank right across the street from the convention center. I'd be willing to bet that the ATM there is less than the $3 the ATM inside the center charges.

Beyond that is a paintball place. It was closed by the time I found it, and I don't know if they run every day, or anything else, but interested parties might find it open during the lunch breaks or something if you wanted to check it out. The paintball place is located behind a building that is directly across the street from the conv. center. If you see the bank, it's on the other side of the side street the bank sits on.

Tonight appears to be low-key from what I can tell. There's currently no chatter on irc, the hotel bar had a few people chatting, and I might go down to catch the rush of people as they return from dinner and BoF sessions. Stay tuned tomorrow for more!

22 Jul 2008 3:23am GMT

In accordance with the Django 1.0 release roadmap, tonight we've released the first "alpha" testing version of Django 1.0. This release includes all of the major features due for inclusion in the final Django 1.0, though some lower-priority items are still scheduled to be included before the 1.0 feature freeze, which will occur with the first beta release next month.

To grab a copy of the 1.0 alpha, head over to the Django downloads page, and be sure to read the release notes. Please keep in mind, though, that this release is not meant for production use, and is intended primarily for developers who are interested in checking out the new features in 1.0 and helping to identify and resolve bugs prior to the final release. The 1.0 alpha will not receive long-term support and will not be updated with security fixes, since its main purpose is to serve as a stepping-stone on the path to the final Django 1.0 release.

The next step on that path will be the first Django 1.0 beta release, currently scheduled for August 5. If you'd like to help out, please review our documentation for contributors and feel free to join in one of the development sprints scheduled for the run up to 1.0; the full schedule is available in the Django 1.0 release roadmap.

22 Jul 2008 1:48am GMT

Twenty lines of Python, an Excel chart, and here we are:

22 Jul 2008 12:02am GMT

From the OSCON'08 schedule:

People for Geeks

People, whether other geeks or "normal" people, we're bad at them. They're such an important part of our lives and jobs and the point of all the technology we love so much. Misunderstandings between people can be the root cause of so many vast, deep, frustrating, and expensive mistakes. Yet we put so little effort into understanding them, or learning how to work with people.

You can learn how to deal with people, and you can even learn to enjoy it. This tutorial gathers together some of the best speakers on "people topics" and teaches you-the programmer, the sysadmin, the DBA, the geek-how better to deal with people.

…

People planning to attend this session also want to see:

• Programming Vim

21 Jul 2008 11:48pm GMT

First, I have arrival Haikou tonight, and then have eaten some delicious. :)

Secondly, the GSoC payment has sent to my store value card right now! Hopefully every GSoCers have got it. :)

21 Jul 2008 7:27pm GMT

It isn't even published yet, but there is already a review of IronPython in Action! Noel, who works in 'cheminformatics' and is an avid Python user, has been reading the Early Access version (currently the first eleven chapters) and has posted his thoughts:

• Review of IronPython in Action

The obligatory introduction to Python is quickly and efficiently dealt with before moving on to the main subject - accessing .NET classes and creating a GUI application. For those coming from C# to IronPython, there is a whole chapter on unit testing with Python, as well as another that covers everything from Python magic methods to metaprogramming.

In short, for any serious users of IronPython, this book is a must have. It may also convince those using C# to make the switch to a better and more productive life with Python..

21 Jul 2008 5:21pm GMT

This blog entry by Terence Chao is very short, and barring about two words all in Chinese. However it links to a PDF document (by the author of the blog) with examples of embedding IronPython 1 in C# and using IronPython to automate Excel. More Chinese text, but straighforward code examples:

• Embedded IronPython 1 - The Simple Way

His Excel automation example takes command line arguments to convert Excel files into CSV files.

If you are (or can read) Chinese, then you might also be interested in these entries:

• Dynamic Language Console: Create Python Host
  

A simple example of the IronPython 2 hosting API, executing code from a string. This entry is mostly code, with screenshots and diagram.

• IronPython Tutorial

This one is mainly text, but appears to be a pretty impressive IronPython tutorial. It looks like it is the start of a translation of the tutorial (by Microsoft) that comes with IronPython. You can find it on the web here.

• IronPython and Silverlight

Another blog entry with screenshots and code (XAML plus IronPython). This one on using IronPython inside Silverlight.

21 Jul 2008 1:28pm GMT

Mark Ramm and I were talking about how to differentiate GET, POST, PUT, and DELETE in TurboGears 2 and we came up with a syntax that's pretty cool. That's not the reason for this post, though. This morning, I noticed that our syntax is completely compatible with TurboGears 1.x -- so here's how you do it. What we wanted was to expose a RESTful method like this: class Root(

21 Jul 2008 1:18pm GMT

Well, finally I have came back to the earth!

After a super busy month, many things get done - the exams, TOEFL, GSoC midterm evaluation, rent room in Singapore, ... Another good news is my GSoC project still on track! Though, well, I have a little trouble of implementing the Python buffer support - I havn't a clear plan about what I am going to do...

Tonight I will go to Haikou, and then contiune my summer in this hot beatiful city. :)

21 Jul 2008 12:32pm GMT

I've just upgraded to WordPress 2.6. As I have claims to be a technologist I install and upgrade my blog software using Subversion. An upgrade simply requires issuing an 'svn switch' command and then running the WordPress database upgrade via the page that presents. Usually this works like a charm. Today I had issues.

I've logged a support ticket but thought I should mention it here in case anyone else sees the same issue.

When I tried to log in after the database upgrade the login page just kept re-directing back to itself. The main page of this blog was showing, but with an error message stating (in part) - "error in akismet.php on line 487″.

After a little checking around I discovered that Akismet (the WordPress spam fighting plugin) needed to be upgraded at the same time as the core WordPress code and hadn't. The old version (2.1.4) is not compatible with the latest WordPress release and needed to be upgraded (to 2.1.6). A simple workaround was to download and install the latest version of Akismet by hand so it wasn't a huge problem, but it would have been nice if the WordPress Subversion repository had been updated to reflect the change.

Update: OK, it's probably me. A fresh checkout of WordPress (release 2.6 or 2.5) comes complete with the correct version of the Akismet plug in. It must have a problem with my 'svn switch' that I didn't catch.

21 Jul 2008 11:56am GMT

Motivation

I think the coolest feature of Jython is the seamless integration with Java. Let say you have the following java class:

package com.leosoto;

public class HelloWorld {

    public void hello() {

        System.out.println("Hello World");

    }

    public void hello(String name) {

        System.out.printf("Hello %s!", name);

    }

}

If the class is on the classpath when you start Jython, using it from python code is straightforward:

>>> from com.leosoto import HelloWorld

>>> h = HelloWorld()

>>> h.hello()

Hello World

>>> h.hello("joe")

Hello joe!

Now, did you knew that if the class was not pointed by the classpath, we could also package it on a jar, and the following would also work:

>>> import sys 

>>> sys.path.append('/path/to/helloworld.jar')

>>> from com.leosoto import HelloWorld

Until yesterday, I didn't knew!

Since part of my GSoC project is to come with a way to package Django projects in a single distributable war file, I've spent a complete day reading and playing with the Jython import logic, and here is what I got.

Not much different than Python, right?

First of all, Jython is an implementation of the Python language. So the import mechanism follow strictly what is know as PEP 302: import hooks. I don't want to repeat what is documented there, but a quick explanation is in order:

• 
  
• First, try custom importers registered on sys.meta_path. If one of them is capable of importing the requested module, we are done.
• 
  
• For each entry of sys.path:
  • 
    
  • Find the first hook registered on sys.path_hookthat can handle the path entry (for example, zipimport is registered there and handle all "*.zip" paths)
  • 
    
  • If a importer hook was found, try it. If the importer loaded the module, we are done
  • 
    
  • If a importer hook was not found, use the builtin import logic (good old *.py files inside directories). If the module is successfully loaded, we are done.
    
  
  

How are java classes loaded then?

With a built-in import hook, naturally ;-)

If you start CPython and look at sys.path_hooks you get:

>>> import sys

>>> sys.path_hooks

[<type 'zipimport.zipimporter'>]

On Jython, the result is slightly different:

>>> import sys

>>> sys.path_hooks

[<type 'JavaImporter'>, <type 'zipimport.zipimporter'>]

The JavaImporter only recognizes the '__classpath__' entry on sys.path, so it is fired after looking at path components before '__classpath__'. This gives us some control over which namespaces will end up containing python modules and which will contain java packages/classes, if some conflict occurs (such as the very real issue of having the 'test' python module and the 'test' java package). Naturally, the '__classpath__' entry is added automagically to sys.path on Jython startup.

But...

>>> sys.path_hooks = []

>>> sys.path_importer_cache = {}

>>> del sys.modules['java']

>>> import java

>>> dir(java)

['__name__', 'applet', 'awt', 'beans' ... ]

This should have failed, after removing the JavaImporter hook and all the involved caches. Well, there is also some magic going on here...

Jython, JavaImporter and Java Packages

When an import is going to fail (that is, after searching on all sys.path entries and having no results), Jython tries to load a java package or java class. But wasn't that the task of the JavaImporter?

Well, sort of. Half of such job is the responsability of the JavaImporter. The other half is managed by the SysPackageManager, which keeps in memory a tree of discovered java packages.

When the Jython interpreter starts, the SysPackageManager looks for all jars and directores on the classpath and build the tree of java packages. You can also explicitely add a Java package into the PackageManager by calling sys.add_package("package.which.was.not.autodiscovered"). This is useful on environments where Jython is not allowed to look at the system classpath, or doesn't get the right information (as maybe the case when running inside a JavaEE container).

Back to JavaImporter, its job is to just look into the SysPackageManager's loaded packages and check if the requested name is present there.

And here is the magic

Another way to get packages loaded into SysPackageManager is to add a zip or jar to sys.path. The next time the import logic runs, it automatically add the contents of the new jar (or zip) to the tree of known java packages.

This is a little weird, because if you have the following in your sys.path:

['__classpath__', '/foo', 'foo.jar']

Then, if java packages on foo.jar conflicts with python modules from /foo then the java packages will prevail, because the '__classpath__' entry is before '/foo', and then the JavaImporter will do its magic.

And the other bit of magic is what we have already seen: Jython does a last attempt to load a Java package, or to be more precise, to add a package to the SysPackageManager if the imported name is know to the JVM as a class or package name. If this operation is successful, the module is directly imported by this Jython builtin import logic addition (no way to go back to the JavaImporter at this time).

Some observations

Here ends the objective part of this post. What follow now are my observations on the whole process:

• 
  
• I don't quite understand why Jython tries to load Java classes or packages at the end of the import logic after trying the standard procedure. Seems like such fallback would make the calls to sys.add_package unnecessary, but then, why does add_package exists? And, in any case, I think that JavaImporter should do this
• 
  
• The confusing situation of jar files (and java classes) in sys.path is well... confusing. The good news is that namespace conflicts aren't that common in practice, so just remembering that all java "modules" come from the magic '__classpath__' element is enough.
• 
  
• It would be nice if the Jython standard loader were installed on the

meta_path. Then, JavaImporter could be added there too, just after the default python code loader. This way, we would have a more clear precedence rule (Python modules first, Java packages/classes later), instead of the current "first python modules before __classpath__, followed by java "modules", followed by python modules after __classpath__, followed bt java "modules" wich weren't registered yet on the PackageManager).

That's all

OK, that was a long post. Now that I've dumped all that info here, I can go back to coding and try to make distributable WAR files for django projects, containing the complete Jython, modjy and Django runtime.

21 Jul 2008 11:40am GMT

Ludumdare 48H is coming - Aug 8 - Aug 10.

Ludum Dare(ld48) is a regular community driven game development competition. The goal is, given a theme and 48 hours, to develop a game from scratch. Ludum Dare aims to encourage game design experimentation, and provide a platform to develop and practice rapid game prototyping.

Here's the time-horizon:
- July 19 - July 25: You may submit themes to the wiki
- July 26 - Aug 2: Themes will be edited for awesomeness by the #ludumdare regulars
- Aug 3 - Aug 8: Various rounds of voting. Check back daily so you don't miss any!
- Aug 8 - Aug 10: Ludum Dare #12 !!! (Starting at 8pm PST)


www.ludumdare.com - the super awesome website, refresh hourly
irc://irc.afternet.org/ludumdare #ludumdare - the compo IRC channel, waste your time here
http://www.ludumdare.com/compo/ - the compo blog, be sure to sign up here
http://www.ludumdare.com/wiki/ - the compo wiki, submit themes & read rules here
http://www.ludumdare.com/planet/ - the compo planet, read about our exciting lives here

21 Jul 2008 9:48am GMT

Some extremely large changes today. The old system has been completely removed (all of the manual-parsing stuff) and replaced with the native Python stuff. I have most of the stuff for buildbot reporting in place, but have not finished it yet. I need to re-do a bunch of the nose-testing stuff tonight before I get too far along.

20 Jul 2008 9:26pm GMT

I won't as much as go to the extent of declaring "Batman 2 - The Dark Knight" as the greatest movie to ever have been filmed, but its right up there all right. After witnessing an awesome performance by the late Heath Ledger, which no words could possibly describe, the Batman movie has now outclassed every other movie of its genre by miles (at least).

The Joker has mesmerized me back to life, and back to blogging.( the fact that my exams have now come to an end must have helped just a wee bit).

20 Jul 2008 9:23pm GMT