fidzu : Python

The Apache Software Foundation has finally put Apache 1.3 out to pasture. This has been a long time coming and quite overdue in my mind. Even though Apache 1.3 is quite antiquated, mod_wsgi has continued supporting it at the same time as supporting newer versions of Apache. For the record, mod_python gave up supporting Apache 1.3 about six years ago, but then, mod_python hasn't seen a release in

11 Mar 2010 12:54pm GMT

For Max's birthday, we made a cake for when his friends were over on Friday, and one for his real birthday on Tuesday. Because we're all watching Lost, and Max is obsessed with it, both are from the show. One styled after the temple:

And the other based on the Dharma logo, with a marshmallow peep as the Swan:

11 Mar 2010 2:38am GMT

For Max's birthday, we made a cake for when his friends were over on Friday, and one for his real birthday on Tuesday. Because we're all watching Lost, and Max is obsessed with it, both are from the show. One styled after the temple:

And the other based on the Dharma logo, with a marshmallow peep as the Swan:

11 Mar 2010 2:38am GMT

I guess two posts on lifestreaming clients isn't enough?.

Yesterday MacHeist started offering pre public beta access to Tweetie 2 for Mac. That caught my eye because Syrinx, my primary Twitter client has been a little slow at keeping up with Twitter features. I didn't really want to get the MacHeist bundle (don't want to hassle with packages that I don't want) just to get the private beta, but I mentioned on Twitter that I was thinking about it. Several folks suggested that I try Echofon. I gave it a whirl, found some things that I like and other that I didn't. I started keeping notes about Syrinx vs Echofon, and now it's turned into a blog post.

My usage style / requirements

I follow a bunch of people, including many people who live in Europe who tweet while I am asleep. I need a client that can remember unread tweets from overnight. I've found very few clients that are able to do this. My reading style tends to be bursty as well, so I want the client to do a good job of keeping track of what I've read and what I have not. These two requirements are what has kept me on Syrinx - it can hold days worth of tweets without a problem. Syrinx's bookmark also gives me definite way of marking what has been read and what has not, and puts control of that mark directly in my hands.

The other major requirement is that I spend some time (probably too much) on airplanes, without net access. I want a client (mostly on my iPhone) that can go back in fill in the gaps left by being in the air. Tweetie 2 for the iPhone can do this, but the experience of switch back and forth between reading the stream on desktop Syrinx and iPhone Tweetie 2 is annoying.

A minor requirement is to be able to monitor a number of Twitter searches at once - that means opening a window for each search, something that Syrinx also does.

Now, let's have a look at how Syrinx and Echofon stack up for me.

Syrinx

The obvious things that I like about Syrinx are that it can hold as many tweets as I want, as well as the bookmark. I've also grown accustomed to the way that it displays time in absolute format, something which Tweetie 2 / iPhone also does. One other nicety in Syrinx is that it can display real names in addition to Twitter handles, because sometimes handles and people are hard to match up. When you have tons of tweets lying around in the? client, sometimes you want to go back to one, and Syrinx obliges with the ability to search all the tweets that it currently has in memory.

So what are the problems with Syrinx? It's been occasionally unstable, but not in a show stopping fashion. It doesn't have good support for lists, but I still haven't made much use of lists. Syrinx does great on opening windows for searches, but it doesn't remember what searches you have open, so you have to keep track of that yourself. Probably the biggest drawback of Syrinx is that its development is going slowly because its author has a day job.

Echofon

When I compare Echofon and Syrinx, I realize that a lot of the things that I prefer in Echofon are niceties. I like that it can open browser links in the background. I like the way that the drawer is used for dealing with Twitter users and profiles and for displaying conversations. I just wish it could display more than one conversation at once - but that's hard in the drawer model. The ability to colorize tweets matching keywords makes it easier to pick out tweets on high priority topics. As a photographer, I appreciate the ability to display pictures without going all the way to the browser. I do wish there was a way to get some kind of preview of those pictures right in the tweet stream. Echofon does this clever thing where it combines "rapid-fire" tweets from the same person. This seems to work really well, and the visual cue is definitely helpful.

Looking at the tweet authoring side, I love the "retweet with comment" option. One reason that I stopped commenting on retweets was that it was annoying to do it. No more. Echofon can tab complete Twitter id's when @replying or direct messaging. I still wish for a direct message "rolodex" - there are some people who have hard to remember Twitter id's. bit.ly is my preferred URL shortener because of the analytics, but you have to be logged in to bit.ly in order for that to work well. Fortunately Echofon is able to log into bit.ly accounts so that your analytics work.

In theory, I like the idea of an Echofon ecosystem that syncs the desktop and mobile clients. I haven't tried this yet because I have iPhone Twitter client fatigue, and because as much as I like Echofon, there are some issues that make it hard for me to switch over.

The first of these issues is that Echofon won't hold all of the tweets that happen overnight. It looks like Echofon will hold about 5 hours of tweets before it starts to drop them on the floor. There go some of those European tweets.

The next big issue is that marking read/unread doesn't work for me. If I am scrolling up through my home tweets and I hit the top, everything gets marked read. It's easy to do that by accident. Switching to the @, DM, or search tabs also marks my home tweets as all read, and that doesn't work for me at all.

Compared to those two issues, everything else is just nits, but here goes, just to be complete. Echofon doesn't display absolute time or real names. Also, Echofon doesn't let you search your home tweets.

Wild and crazy wishes

Certain URL shortening services (su.pr and ow.ly come to mind) wrap the page in a header bar, which is annoying. I'd love if my client would route through those services so that the URL that I got in the browser was the actual content.

Sometimes there are links that are retweeted a bunch. I would love it if a client could compress all those retweets into a single entry which showed how many / which people I follow retweeted a link, along with an indication of whether or not I had already "read" an earlier retweeter (which would mean I had already read the link).

I guess I'll have to do another version of this post when Tweetie 2 for Mac finally ships. Or maybe it's still early enough for some of these ideas to make the cut.

11 Mar 2010 1:08am GMT

I guess two posts on lifestreaming clients isn't enough?.

Yesterday MacHeist started offering pre public beta access to Tweetie 2 for Mac. That caught my eye because Syrinx, my primary Twitter client has been a little slow at keeping up with Twitter features. I didn't really want to get the MacHeist bundle (don't want to hassle with packages that I don't want) just to get the private beta, but I mentioned on Twitter that I was thinking about it. Several folks suggested that I try Echofon. I gave it a whirl, found some things that I like and other that I didn't. I started keeping notes about Syrinx vs Echofon, and now it's turned into a blog post.

My usage style / requirements

I follow a bunch of people, including many people who live in Europe who tweet while I am asleep. I need a client that can remember unread tweets from overnight. I've found very few clients that are able to do this. My reading style tends to be bursty as well, so I want the client to do a good job of keeping track of what I've read and what I have not. These two requirements are what has kept me on Syrinx - it can hold days worth of tweets without a problem. Syrinx's bookmark also gives me definite way of marking what has been read and what has not, and puts control of that mark directly in my hands.

The other major requirement is that I spend some time (probably too much) on airplanes, without net access. I want a client (mostly on my iPhone) that can go back in fill in the gaps left by being in the air. Tweetie 2 for the iPhone can do this, but the experience of switch back and forth between reading the stream on desktop Syrinx and iPhone Tweetie 2 is annoying.

A minor requirement is to be able to monitor a number of Twitter searches at once - that means opening a window for each search, something that Syrinx also does.

Now, let's have a look at how Syrinx and Echofon stack up for me.

Syrinx

The obvious things that I like about Syrinx are that it can hold as many tweets as I want, as well as the bookmark. I've also grown accustomed to the way that it displays time in absolute format, something which Tweetie 2 / iPhone also does. One other nicety in Syrinx is that it can display real names in addition to Twitter handles, because sometimes handles and people are hard to match up. When you have tons of tweets lying around in the? client, sometimes you want to go back to one, and Syrinx obliges with the ability to search all the tweets that it currently has in memory.

So what are the problems with Syrinx? It's been occasionally unstable, but not in a show stopping fashion. It doesn't have good support for lists, but I still haven't made much use of lists. Syrinx does great on opening windows for searches, but it doesn't remember what searches you have open, so you have to keep track of that yourself. Probably the biggest drawback of Syrinx is that its development is going slowly because its author has a day job.

Echofon

When I compare Echofon and Syrinx, I realize that a lot of the things that I prefer in Echofon are niceties. I like that it can open browser links in the background. I like the way that the drawer is used for dealing with Twitter users and profiles and for displaying conversations. I just wish it could display more than one conversation at once - but that's hard in the drawer model. The ability to colorize tweets matching keywords makes it easier to pick out tweets on high priority topics. As a photographer, I appreciate the ability to display pictures without going all the way to the browser. I do wish there was a way to get some kind of preview of those pictures right in the tweet stream. Echofon does this clever thing where it combines "rapid-fire" tweets from the same person. This seems to work really well, and the visual cue is definitely helpful.

Looking at the tweet authoring side, I love the "retweet with comment" option. One reason that I stopped commenting on retweets was that it was annoying to do it. No more. Echofon can tab complete Twitter id's when @replying or direct messaging. I still wish for a direct message "rolodex" - there are some people who have hard to remember Twitter id's. bit.ly is my preferred URL shortener because of the analytics, but you have to be logged in to bit.ly in order for that to work well. Fortunately Echofon is able to log into bit.ly accounts so that your analytics work.

In theory, I like the idea of an Echofon ecosystem that syncs the desktop and mobile clients. I haven't tried this yet because I have iPhone Twitter client fatigue, and because as much as I like Echofon, there are some issues that make it hard for me to switch over.

The first of these issues is that Echofon won't hold all of the tweets that happen overnight. It looks like Echofon will hold about 5 hours of tweets before it starts to drop them on the floor. There go some of those European tweets.

The next big issue is that marking read/unread doesn't work for me. If I am scrolling up through my home tweets and I hit the top, everything gets marked read. It's easy to do that by accident. Switching to the @, DM, or search tabs also marks my home tweets as all read, and that doesn't work for me at all.

Compared to those two issues, everything else is just nits, but here goes, just to be complete. Echofon doesn't display absolute time or real names. Also, Echofon doesn't let you search your home tweets.

Wild and crazy wishes

Certain URL shortening services (su.pr and ow.ly come to mind) wrap the page in a header bar, which is annoying. I'd love if my client would route through those services so that the URL that I got in the browser was the actual content.

Sometimes there are links that are retweeted a bunch. I would love it if a client could compress all those retweets into a single entry which showed how many / which people I follow retweeted a link, along with an indication of whether or not I had already "read" an earlier retweeter (which would mean I had already read the link).

I guess I'll have to do another version of this post when Tweetie 2 for Mac finally ships. Or maybe it's still early enough for some of these ideas to make the cut.

11 Mar 2010 1:08am GMT

10 Mar 2010 11:46pm GMT

10 Mar 2010 11:46pm GMT

I've succumbed and created a Twitter account for myself - not really out of a burning desire to start using it, but rather because last week I gave a lecture to my students on how Python could be used for Internet programming at the application level. I already had examples of how to access Flickr and Google services, but Twitter seemed like another good example to use because students are familiar with it as a 'Web 2.0' service and because there is a nice, easy-to-use library available for its API: python-twitter.

I showed the students a simple program to post a status update. I ran it using my newly-created Twitter account (python33r) and then showed them the tweet appearing on my Twitter page.

I didn't think I'd want to continue using the service, but I happened to start following a few people/organisations of interest to me and it has provided a couple of useful notifications already.

10 Mar 2010 10:39pm GMT

I've succumbed and created a Twitter account for myself - not really out of a burning desire to start using it, but rather because last week I gave a lecture to my students on how Python could be used for Internet programming at the application level. I already had examples of how to access Flickr and Google services, but Twitter seemed like another good example to use because students are familiar with it as a 'Web 2.0' service and because there is a nice, easy-to-use library available for its API: python-twitter.

I showed the students a simple program to post a status update. I ran it using my newly-created Twitter account (python33r) and then showed them the tweet appearing on my Twitter page.

I didn't think I'd want to continue using the service, but I happened to start following a few people/organisations of interest to me and it has provided a couple of useful notifications already.

10 Mar 2010 10:39pm GMT

March 9th (that's tomorrow at the time of typing) IronPython in Action is the Manning deal of the day. This is a one day offer with a special discount.

You can get the discount by buying IronPython in Action from the Manning website and using the discount code dotd0310tw.

It isn't only IronPython in Action that is on offer, you can also get Quick Python by Vern Ceder.

10 Mar 2010 5:17pm GMT

March 9th (that's tomorrow at the time of typing) IronPython in Action is the Manning deal of the day. This is a one day offer with a special discount.

You can get the discount by buying IronPython in Action from the Manning website and using the discount code dotd0310tw.

It isn't only IronPython in Action that is on offer, you can also get Quick Python by Vern Ceder.

10 Mar 2010 5:17pm GMT

The second release candidate for Python 2.6.5 is now available.

10 Mar 2010 3:56pm GMT

The second release candidate for Python 2.6.5 is now available.

10 Mar 2010 3:56pm GMT

[Warning! This is complete newbie information!]

One of the things I found difficult when first using Python on a Mac was simply getting my scripts to run as programmes, with a simple double-click. The nub of the problem is that one must run a script using the python interpreter, like so:
python script.py

This is easy enough to do in a command window, terminal window, at the shell prompt etc. (The terminology is different depending on the operating system, but the reality is the same.)

10 Mar 2010 3:05pm GMT

[Warning! This is complete newbie information!]

One of the things I found difficult when first using Python on a Mac was simply getting my scripts to run as programmes, with a simple double-click. The nub of the problem is that one must run a script using the python interpreter, like so:
python script.py

This is easy enough to do in a command window, terminal window, at the shell prompt etc. (The terminology is different depending on the operating system, but the reality is the same.)

10 Mar 2010 3:05pm GMT

There is one interesting problem to solve from the guys on Project Euler. I think that maybe there are some problems running a naive approach to the problem on the 2002, but right now, even a inefficient approach can give you results in not much time. I've made the following program in Python to measure the differences between [...]

10 Mar 2010 12:27pm GMT

There is one interesting problem to solve from the guys on Project Euler. I think that maybe there are some problems running a naive approach to the problem on the 2002, but right now, even a inefficient approach can give you results in not much time. I've made the following program in Python to measure the differences between [...]

10 Mar 2010 12:27pm GMT

I create and work with several Python tools that manipulate scenes in 3ds Max. These are usually floating dialogs linked to the main 3ds Max window that send MaxScript commands via COM. This works well until I need the tool's UI to update when something happens in 3ds Max. Like refresh an object list when the selection in Max changes.

You would think doing a COM connection the other way would work. However, since Python-registered COM servers run separately in their own instance of the interpreter, there's no native connection to the original tool.

Nathaniel Albright, a fellow TA at Volition, recently created a Python COM server that communicated to his Python tool via TCP/IP socket. So it went 3ds Max -> Python COM server -> TCP/IP -> Python tool. This works well, but I wondered if the DotNet facilities in MaxScript offered a direct way to use sockets.

I had yet to touch DotNet in MaxScript, so this seemed like a good opportunity to learn a few things. After a lot of searching online I only turned up a few scraps of info, no complete recipe. However, I did find enough to get MXS DotNet sockets working, and assemble a comprehensive example.

I created a little Python tool that displays the names of all selected objects in the 3ds Max scene. As the scene selection changes, the list of names automatically updates. I won't go over all the code in this post, but the full working example tool is included in the zipfile below.

There's three main points of interest in the example:

1. Using DotNet in MaxScript to communicate via TCP/IP socket
2. Listening on a socket in a background thread in Python
3. Creating and posting custom wxPython events

The MaxScript Client

I made a MaxScript struct called "mxs_socket". The code follows, and also included in the zipfile below.

struct mxs_socket (
   ip_address = "127.0.0.1", -- "localhost" also valid
   port       = 2323,        -- default port

   -- <dotnet>connect <string>ip_string <int>port_int
   --
   -- Description:
   -- Takes IP address, port and connects to socket listener at that
   -- address
   fn connect ip_string port_int = (
      socket = dotNetObject "System.Net.Sockets.Socket" ( dotnetclass "System.Net.Sockets.AddressFamily" ).InterNetwork ( dotnetclass "System.Net.Sockets.SocketType" ).Stream ( dotnetclass "System.Net.Sockets.ProtocolType" ).Tcp
      socket.Connect ip_string port_int

      socket   -- return
   ),

   -- <int>send <string>data
   --
   -- Description:
   -- Converts a string (or any object that can be converted
   -- to a string) to dotnet ASCII-encoded byte sequence and
   -- sends it via socket. Uses ip_address and port defined
   -- in struct above, or set by client.
   -- Returns integer of how many bytes were sent.
   fn send data = (
      -- Convert string to bytes
      ascii_encoder = dotNetObject "System.Text.ASCIIEncoding"
      bytes = ascii_encoder.GetBytes ( data as string )

      -- Connect, send bytes, then close
      socket = connect ip_address port
      -- result is # of bytes sent
      result = socket.Send bytes
      socket.Close()

      result  -- return # of bytes sent
   )
)

Using this, I can send bytes to any socket listener on port 5432 by doing the following:

socket = mxs_socket port:5432
socket.send "Hello, World!"

The connect method was pretty simple in the end. The only twist turned out to be converting the socket integer into a DotNet socket object.

The send method converts the string into an ASCII-encoded DotNet bytes object, connects to the listener, sends the bytes, then closes the connection. The value returned is the number of bytes sent.

The last lines of the above code sets up a MaxScript callback that fires when the object selection changes in the scene. That uses mxs_socket to send a string containing the names of all the selected objects to any tool that's listening on that port.

Now I just need to make my Python tool listen.

The Python Server

My Python server/listener (also in the zipfile below) is a typical wxPython frame, but with two added qualities... It uses a background thread to listen on a socket, and posts a custom wx.Event when data is received. I had never used either of these techniques before, but it was fun getting it working.

Since a typical wxPython app sits in its main loop waiting for user input, I created a Socket_Listen_Thread class, a subclass of threading.Thread. This does the listening in a background thread while the main UI thread waits on the user. The run method here does the real work:

def run( self ):
   self.running = True

   while ( self.running ):
      # Starting server...
      # Listen for connection.  We're in non-blocking mode so it can
      # check for the signal to shut down from the main thread.
      try:
         client_socket, clientaddr = self.socket.accept( )
         data_received = True
      except socket.error:
         data_received = False

      if ( data_received ):
         # Set new client socket to block.  Otherwise it will
         # inherit the non-blocking mode of the server socket.
         client_socket.setblocking( True )

         # Connection found, read its data then close
         data = client_socket.recv( self.buffer_size )
         client_socket.close( )

         # Create wx event and post it to our app window
         event = self.event_class( data = data )
         wx.PostEvent( self.window, event )

This listening thread runs quietly in the background until it receives data. At that point I need a way to break into the main UI thread. There's other ways to do this, but using a custom wx.Event seemed to be the best fit here.

First, when the wx.Frame is opened, I create a custom wxPython event.

(Max_Update_Event, EVT_3DS_MAX_UPDATE) = wx.lib.newevent.NewEvent()

Calling NewEvent returns both a new Event class and an object to bind the event handler to. I pass the event class to the listener thread, bind an event handler to it, and that's all.

When data comes in over that TCP/IP port from our MaxScript tool, the listening thread receives it and posts our custom event to the main wx.Frame. That in turn fires the event handler to update the UI.

My example MaxScript client and Python listener described above can be found in the following ZIP file. Drop me a line if you do something useful with them, I'd love to hear about it.

MaxScript_DotNet_Sockets_Python.zip (4 KB)

Thanks to Nate Albright and everyone contributing to the "dotNet + MXS" and "Python + MXS" threads on the CGSociety forums. Those long-running threads have been very inspiring, and contain several tips that were key in getting the MaxScript DotNet socket stuff hammered out.

10 Mar 2010 8:53am GMT

I create and work with several Python tools that manipulate scenes in 3ds Max. These are usually floating dialogs linked to the main 3ds Max window that send MaxScript commands via COM. This works well until I need the tool's UI to update when something happens in 3ds Max. Like refresh an object list when the selection in Max changes.

You would think doing a COM connection the other way would work. However, since Python-registered COM servers run separately in their own instance of the interpreter, there's no native connection to the original tool.

Nathaniel Albright, a fellow TA at Volition, recently created a Python COM server that communicated to his Python tool via TCP/IP socket. So it went 3ds Max -> Python COM server -> TCP/IP -> Python tool. This works well, but I wondered if the DotNet facilities in MaxScript offered a direct way to use sockets.

I had yet to touch DotNet in MaxScript, so this seemed like a good opportunity to learn a few things. After a lot of searching online I only turned up a few scraps of info, no complete recipe. However, I did find enough to get MXS DotNet sockets working, and assemble a comprehensive example.

I created a little Python tool that displays the names of all selected objects in the 3ds Max scene. As the scene selection changes, the list of names automatically updates. I won't go over all the code in this post, but the full working example tool is included in the zipfile below.

There's three main points of interest in the example:

1. Using DotNet in MaxScript to communicate via TCP/IP socket
2. Listening on a socket in a background thread in Python
3. Creating and posting custom wxPython events

The MaxScript Client

I made a MaxScript struct called "mxs_socket". The code follows, and also included in the zipfile below.

struct mxs_socket (
   ip_address = "127.0.0.1", -- "localhost" also valid
   port       = 2323,        -- default port

   -- <dotnet>connect <string>ip_string <int>port_int
   --
   -- Description:
   -- Takes IP address, port and connects to socket listener at that
   -- address
   fn connect ip_string port_int = (
      socket = dotNetObject "System.Net.Sockets.Socket" ( dotnetclass "System.Net.Sockets.AddressFamily" ).InterNetwork ( dotnetclass "System.Net.Sockets.SocketType" ).Stream ( dotnetclass "System.Net.Sockets.ProtocolType" ).Tcp
      socket.Connect ip_string port_int

      socket   -- return
   ),

   -- <int>send <string>data
   --
   -- Description:
   -- Converts a string (or any object that can be converted
   -- to a string) to dotnet ASCII-encoded byte sequence and
   -- sends it via socket. Uses ip_address and port defined
   -- in struct above, or set by client.
   -- Returns integer of how many bytes were sent.
   fn send data = (
      -- Convert string to bytes
      ascii_encoder = dotNetObject "System.Text.ASCIIEncoding"
      bytes = ascii_encoder.GetBytes ( data as string )

      -- Connect, send bytes, then close
      socket = connect ip_address port
      -- result is # of bytes sent
      result = socket.Send bytes
      socket.Close()

      result  -- return # of bytes sent
   )
)

Using this, I can send bytes to any socket listener on port 5432 by doing the following:

socket = mxs_socket port:5432
socket.send "Hello, World!"

The connect method was pretty simple in the end. The only twist turned out to be converting the socket integer into a DotNet socket object.

The send method converts the string into an ASCII-encoded DotNet bytes object, connects to the listener, sends the bytes, then closes the connection. The value returned is the number of bytes sent.

The last lines of the above code sets up a MaxScript callback that fires when the object selection changes in the scene. That uses mxs_socket to send a string containing the names of all the selected objects to any tool that's listening on that port.

Now I just need to make my Python tool listen.

The Python Server

My Python server/listener (also in the zipfile below) is a typical wxPython frame, but with two added qualities... It uses a background thread to listen on a socket, and posts a custom wx.Event when data is received. I had never used either of these techniques before, but it was fun getting it working.

Since a typical wxPython app sits in its main loop waiting for user input, I created a Socket_Listen_Thread class, a subclass of threading.Thread. This does the listening in a background thread while the main UI thread waits on the user. The run method here does the real work:

def run( self ):
   self.running = True

   while ( self.running ):
      # Starting server...
      # Listen for connection.  We're in non-blocking mode so it can
      # check for the signal to shut down from the main thread.
      try:
         client_socket, clientaddr = self.socket.accept( )
         data_received = True
      except socket.error:
         data_received = False

      if ( data_received ):
         # Set new client socket to block.  Otherwise it will
         # inherit the non-blocking mode of the server socket.
         client_socket.setblocking( True )

         # Connection found, read its data then close
         data = client_socket.recv( self.buffer_size )
         client_socket.close( )

         # Create wx event and post it to our app window
         event = self.event_class( data = data )
         wx.PostEvent( self.window, event )

This listening thread runs quietly in the background until it receives data. At that point I need a way to break into the main UI thread. There's other ways to do this, but using a custom wx.Event seemed to be the best fit here.

First, when the wx.Frame is opened, I create a custom wxPython event.

(Max_Update_Event, EVT_3DS_MAX_UPDATE) = wx.lib.newevent.NewEvent()

Calling NewEvent returns both a new Event class and an object to bind the event handler to. I pass the event class to the listener thread, bind an event handler to it, and that's all.

When data comes in over that TCP/IP port from our MaxScript tool, the listening thread receives it and posts our custom event to the main wx.Frame. That in turn fires the event handler to update the UI.

My example MaxScript client and Python listener described above can be found in the following ZIP file. Drop me a line if you do something useful with them, I'd love to hear about it.

MaxScript_DotNet_Sockets_Python.zip (4 KB)

Thanks to Nate Albright and everyone contributing to the "dotNet + MXS" and "Python + MXS" threads on the CGSociety forums. Those long-running threads have been very inspiring, and contain several tips that were key in getting the MaxScript DotNet socket stuff hammered out.

10 Mar 2010 8:53am GMT

Have spent rather a lot of time trying to wrestle the deployment docs into shape for the TurboGears 2.1 release. Even with all that work, only the "standard deployment pattern" is really in "good" shape. Alternative servers, alternative deployment styles, etceteras are all still needing lots of love. I am, however, I think, finished with that section for a while unless someone else chips in to help.

Also ran into one of those "argh" inducing issues this evening. A user was trying to follow through the "Using who.ini" discussion on his way to doing OpenID with TurboGears and ran into a weird, subtle bug. Turns out, when you disable TurboGears authentication repoze.what "fails open" as far as much TurboGears code is concerned. The predicates just all silently start evaluating to True. The @require() decorator thankfully does the right thing, but users who do:

tg.predicates.has_permission('manage')

Now always get a True value (the predicate object), so if you're using that to control display of information... oops. Not likely to have bitten too many users, but it just seems we should have something raising errors on _nonzero_() if the predicate's aren't "booleanized()".

In other news, given that someone else is trying to get OpenID working, I suppose I should look at the branch of repoze.openid. Upstream wants changes, but I haven't yet figured out what's supposed to happen wrt registration to make those changes work. Repoze just doesn't seem to have a registration story AFAICT.

10 Mar 2010 7:51am GMT

Have spent rather a lot of time trying to wrestle the deployment docs into shape for the TurboGears 2.1 release. Even with all that work, only the "standard deployment pattern" is really in "good" shape. Alternative servers, alternative deployment styles, etceteras are all still needing lots of love. I am, however, I think, finished with that section for a while unless someone else chips in to help.

Also ran into one of those "argh" inducing issues this evening. A user was trying to follow through the "Using who.ini" discussion on his way to doing OpenID with TurboGears and ran into a weird, subtle bug. Turns out, when you disable TurboGears authentication repoze.what "fails open" as far as much TurboGears code is concerned. The predicates just all silently start evaluating to True. The @require() decorator thankfully does the right thing, but users who do:

tg.predicates.has_permission('manage')

Now always get a True value (the predicate object), so if you're using that to control display of information... oops. Not likely to have bitten too many users, but it just seems we should have something raising errors on _nonzero_() if the predicate's aren't "booleanized()".

In other news, given that someone else is trying to get OpenID working, I suppose I should look at the branch of repoze.openid. Upstream wants changes, but I haven't yet figured out what's supposed to happen wrt registration to make those changes work. Repoze just doesn't seem to have a registration story AFAICT.

10 Mar 2010 7:51am GMT

In previous posts, I've written about the possibilities offered by Python 3.1's Unicode identifier capability, as well as the new challenges posed when one tries to display them on screen.

As the final project for my Java course, I set about trying to create an editor that would allow the user to enter text right to left, but save it as valid interpretable (is that a word?) Python 3.x code:

There's still a good deal of work to be done on this, but this is a start. Hopefully I'll have something more robust next time.

10 Mar 2010 3:55am GMT

In previous posts, I've written about the possibilities offered by Python 3.1's Unicode identifier capability, as well as the new challenges posed when one tries to display them on screen.

As the final project for my Java course, I set about trying to create an editor that would allow the user to enter text right to left, but save it as valid interpretable (is that a word?) Python 3.x code:

There's still a good deal of work to be done on this, but this is a start. Hopefully I'll have something more robust next time.

10 Mar 2010 3:55am GMT

Those of you that have been paying attention to the Django release roadmap will have noticed that the original estimated release date for Django 1.2 final has passed, but we haven't actually made a final release.

Although Django's release cycle is generally date-based, we also try to keep our release dates flexible to account for bugfixing time. At the beginning of the development sprints at PyCon a few weeks ago, over 300 tickets were still open on the Django 1.2 milestone. Now it's down to 120 (we've been clearing out, on average, about ten tickets a day), but that's still a lot more than we're comfortable shipping; as a result, we're pushing back the final 1.2 release a bit.

Some of the tickets still open for 1.2 are documentation or translation updates; these will be dealt with before the final 1.2 release. Others are minor bugs or edge cases which are difficult to trigger or unlikely to cause serious problems in actual deployment; these tickets will likely be bumped to a pure-bugfix release in the 1.2 series, or to 1.3 as warranted.

Over the next couple of days, the Django core team will be reviewing all of the currently-open tickets, and identifying those which:

• reveal a significant flaw in a feature added during the 1.2 cycle, or
• have the potential to cause data loss or other serious consequences in actual deployment.

Tickets which don't meet these criteria may be removed from the 1.2 milestone, or may simply be left out of the final release. We won't forget about these issues -- they'll still be in Trac, and they will be addressed -- but bugfix work prior to the 1.2 release will focus in major issues fitting the criteria above.

We're sensitive to the fact that during the Django 1.2 release cycle, we haven't paid as much attention to bugs and smaller features as we have done during previous releases. To address this, we're considering making Django 1.3 a "feature light" release -- that is, we will spend more time focussing on little features and long standing bugs, rather than adding lots of big features like we have done with Django 1.2. Once 1.2 lands, we'll have some more details about our exact plans for the 1.3 cycle.

Until then, we'll be posting here every few days to give you a status update, letting you know how many tickets remain, any problems we foresee, and to provide an updated estimate of the 1.2 final delivery date.

So: there are 120 tickets remaining, but quite a few of these of these will be bumped from the final release. It's difficult to know exactly how much work is left before we do the final ticket cull, but our first-cut revised estimate is for an RC1 release around March 22, with a final release around March 29. This is, for those of you who were following along during the early parts of the 1.2 cycle, roughly consistent with extra time added to the release schedule for the 1.2 alpha and beta milestones.

As always, any assistance preparing, reviewing or testing patches is most welcome; the more help we get, the sooner we can release. If you want to help out, check out the 1.2 todo list, find something that sounds interesting and dig in!

10 Mar 2010 12:36am GMT

Those of you that have been paying attention to the Django release roadmap will have noticed that the original estimated release date for Django 1.2 final has passed, but we haven't actually made a final release.

Although Django's release cycle is generally date-based, we also try to keep our release dates flexible to account for bugfixing time. At the beginning of the development sprints at PyCon a few weeks ago, over 300 tickets were still open on the Django 1.2 milestone. Now it's down to 120 (we've been clearing out, on average, about ten tickets a day), but that's still a lot more than we're comfortable shipping; as a result, we're pushing back the final 1.2 release a bit.

Some of the tickets still open for 1.2 are documentation or translation updates; these will be dealt with before the final 1.2 release. Others are minor bugs or edge cases which are difficult to trigger or unlikely to cause serious problems in actual deployment; these tickets will likely be bumped to a pure-bugfix release in the 1.2 series, or to 1.3 as warranted.

Over the next couple of days, the Django core team will be reviewing all of the currently-open tickets, and identifying those which:

• reveal a significant flaw in a feature added during the 1.2 cycle, or
• have the potential to cause data loss or other serious consequences in actual deployment.

Tickets which don't meet these criteria may be removed from the 1.2 milestone, or may simply be left out of the final release. We won't forget about these issues -- they'll still be in Trac, and they will be addressed -- but bugfix work prior to the 1.2 release will focus in major issues fitting the criteria above.

We're sensitive to the fact that during the Django 1.2 release cycle, we haven't paid as much attention to bugs and smaller features as we have done during previous releases. To address this, we're considering making Django 1.3 a "feature light" release -- that is, we will spend more time focussing on little features and long standing bugs, rather than adding lots of big features like we have done with Django 1.2. Once 1.2 lands, we'll have some more details about our exact plans for the 1.3 cycle.

Until then, we'll be posting here every few days to give you a status update, letting you know how many tickets remain, any problems we foresee, and to provide an updated estimate of the 1.2 final delivery date.

So: there are 120 tickets remaining, but quite a few of these of these will be bumped from the final release. It's difficult to know exactly how much work is left before we do the final ticket cull, but our first-cut revised estimate is for an RC1 release around March 22, with a final release around March 29. This is, for those of you who were following along during the early parts of the 1.2 cycle, roughly consistent with extra time added to the release schedule for the 1.2 alpha and beta milestones.

As always, any assistance preparing, reviewing or testing patches is most welcome; the more help we get, the sooner we can release. If you want to help out, check out the 1.2 todo list, find something that sounds interesting and dig in!

10 Mar 2010 12:36am GMT

At the backend of last I year I wrote a blog entry on decoding json on Silverlight. Well, the time has finally come and we're now encoding json to post back to our Django application. ... [292 words]

10 Mar 2010 12:01am GMT

At the backend of last I year I wrote a blog entry on decoding json on Silverlight. Well, the time has finally come and we're now encoding json to post back to our Django application. ... [292 words]

10 Mar 2010 12:01am GMT

I just got a call from an insistent headhunter, so to get him off the phone, I asked him to send me an email with the details of his fabulous opportunity. He said, "OK, sure! Your email address is ned @ Red Hat chelder.com?"

I guess I could create my own Linux distro now...

09 Mar 2010 7:51pm GMT

I just got a call from an insistent headhunter, so to get him off the phone, I asked him to send me an email with the details of his fabulous opportunity. He said, "OK, sure! Your email address is ned @ Red Hat chelder.com?"

I guess I could create my own Linux distro now...

09 Mar 2010 7:51pm GMT

Okay so I've been busy lately, with classes, the newspaper, and managing hordes of jabbering monkeys. I've been recently promoted to an op on the #sto and #stoqa irc channels(hence the jabbering monkeys).

I've observed something interesting though. Cryptic has been really stepping up their communication, thanks to people like Rehpic, Falkoren, and Jaguars. This has served to really benefit the game without much of a cost to the company.

Here's a hypothetical. Say someone asks some questions about personal shields. Falkoren then asks the powers designers how they work, and he tells us in #stoqa. This gives us the knowledge to properly test them, while some of us add this info to places like sto-intel.org

This then serves to enhance the game. Players have more knowledge, players can properly test complex things to ensure they're working right. For a minimal cost, of a few minutes of discussion between employees and customers, you have tremendously added to the perceived value of the game.

Encouraging communities are an excellent investment for game companies. Supporting them with actual employee interaction and communication is just pure gold.

However, Cryptic isn't perfect in their communication. For example, the other moderators have a set of new rules they want to add and enforce for the IRC channels, however, its been difficult reaching Rekhan or Phoxe to get approval. In addition, they don't rarely come in to the IRC channels for us to communicate community issues there are. I am thankful for Wishstone, she has been a good communication pipeline to Cryptic, but she's the OCR for the German community, not the english community! This needs to change.

09 Mar 2010 7:29pm GMT

ConFoo starts tomorrow but that it's not too late to register. Don't miss that unique opportunity to meet famous Pythonistas such as Mark Pilgrim (Dive into Python), Raymond Hettinger (core Python), Tarek Ziadé (Distribute), and Antoine Pitrou (new GIL).

09 Mar 2010 4:30pm GMT

ConFoo starts tomorrow but that it's not too late to register. Don't miss that unique opportunity to meet famous Pythonistas such as Mark Pilgrim (Dive into Python), Raymond Hettinger (core Python), Tarek Ziadé (Distribute), and Antoine Pitrou (new GIL).

09 Mar 2010 4:30pm GMT

Montréal-Python 12 will take place at UQAM, on Monday 2010-03-22 in the Sherbrooke building. The exact room number will be announced shortly. The SH building is located at 200 Sherbrooke west, Place-des-Arts metro station.

Here is our schedule for the evening:

• 18h00: Opening
• 18h20: Announcements
• 18h30: Flash presentations
• 19h00: Break
• 19h20: Main presentation
• 20h30: Discussions and refreshments at Benelux

Our main presenter is going to be Marcin Swiatek and he's going to talk about Generating control images for microscopy software with Python and Numpy.

Numpy is the mainstay of all things numerical in Python. I will use one of my past projects - a pipeline for testing image analysis software - to introduce Numpy. The presentation will focus on basic array manipulation, random number generators, and elementwise operations. Real math will be given silent treatment. PIL, or Python Imaging Library will appear in a supporting role.

Marcin has been working on development of life sciences software for the past 12 years. He has been a Python user since the version 1.5.2.

We still have a few spots for flash presentations so don't hesitate to contact us if you have something that you would like to present.

We want to thank our sponsors for making Montréal-Python 12 possible:

• UQAM
• Benelux
• O'Reilly
• AESS

09 Mar 2010 4:00pm GMT

LibSVM is a C++ library that implements several Support Vector Machine algorithms that are commonly used in machine learning. It is a fast library that has no dependencies and most machine learning frameworks bind it in some way or another. LibSVM comes with a Python interface written in swig, but this interface is inherently slow as it does not take into account numpy's array structure. Also, it does not wrap all the library's functionality. Some projects bind it using this bindings and other (such as PyMVPA) make its own wrap, binding some methods directly to numpy's array structure.

My approach was to code all algorithms that convert libsvm's data structures (sparse) to numpy arrays (dense) in pure C and wrap them in a very thin Cython layer. Special attention was given to minimize the overhead of converting between libsvm data structures and numpy arrays, as in my opinion this was the main source of bad performance in existing python bindings.

Benchmarks

As a first benchmark, I supposed a situation in which the dimension of the subspace is small and there are lots of points to classify. This is typically the case when your data is points in plane or in space and you want to draw the decision function by classifying every point in the grid. In this case, the bottleneck is not the classification algorithm, but the conversion of data from a dense representation used by python and numpy and a sparse representation used by libsvm. Not surprisingly, we get huge performance gains if we speed up the conversion dense/sparse.

Course of dimensionality

In the case of a huge number of dimensions, the speedup is not so spectacular, but we also get a performance boost by making training somewhat faster.

Bidirectional mapping

A feature that was needed and that I haven't found on other implementations is that you can tweak parameters in the SVM class and the classifier will reflect those changes (i.e. parameters are actually copied back and forth, not just passed as an opaque pointer).

Suppose you train an instance of the classifier and are interested in the coefficients that multiply the support vectors in the decision function. In scikits.learn, you can access this array under field .coef_:

Now, changing the value of these coefficients effectively changes the decision function:

Code

All code can be found in the scikit (you'll have to get the svn version), in file scikits/learn/svm.py and scikits/learn/src/. All plots are generated from this script

Notes

In the benchmarks, a Linear Kernel was used, as it is the most common. Other more computationally intensive kernels would probably narrow the difference.

Bugs

This code should be treated as alpha quality and has not being extensively tested. Please report any bugs that you encounter to the tracker

09 Mar 2010 1:49pm GMT

I have a homework doing some monte carlo experiments of an autoregressive process of order 1, and I thought I would use it as an example to demonstrate the sparse least squares solver that Stefan committed to scipy revision 6251.

All mistakes are mine...

Given an AR(1) process




We can estimate the following autoregressive coefficients.

I'm curious if there's any downside to using sparse least squares whenever the RHS of a least squares can be written in block diagonal form.

08 Mar 2010 10:06pm GMT

Yesterday we had an extremely productive coding sprint for the scikits.learn. The idea was to put people with common interests in a room and make them work in a single codebase.

Alexandre Gramfort and Olivier Grisel worked on GLMNet, Bertrand Thirion and Gaël Varoquaux worked on univariate feature selection and Vincent worked on Bayesian Regression.

I was supposed to work with Vincent, but as soon as Bertrand spot some bugs in my libsvm bindings, I could not think of anything except that, and eventually the day finished just as I fixed the bug …

You can find some cool examples of the things we did in directory examples:

04 Mar 2010 9:25am GMT

I've been here at the Delhi College of Engineering for approximately the last 4 years now. I've never really paid attention to what goes on in here. I entered this place with minimal hopes from myself and from this place, not exactly hoping that I'd find a space station parked in here. All the fuss about maintaining the attendance, submitting the assignments, socializing with people (I'm a little introvert, you see) was a little hard for me to keep up to. And I've had my moments of frustration. But then somewhere I knew I'd miss this place when I'll (possibly) graduate.

I had no idea I'd miss this place even before graduating.

The place is fun. Its really a very laid back place to be. The generally peaceful atmosphere in the academic blocks, the chirpy mech canteen, apparently also shared by the civil block, the quintessentially DCE elec canteen, the warm winter sun at the OAT, the wind-point, the T5 canteen, almost everything. I haven't set foot on each and every inch of the campus, but I've spent enough time in here to actually miss it after leaving.

I miss the times when we had seniors above us. We're the senior most students now, and watching our juniors acting like dudes playing guitars and chicks going all crazy about the newly opened (overpriced) nescafe outlet is just weird (in a bad way). Its after spending 3 years at this place that I finally made friends, this tiny little group of ours. Its unarguably the best thing that has happened to me in college, given my first class social ineptness. We've killed so much time at the mech canteen, eating the same stuff over and over again and still finding it good, wasting time at the elec canteen talking about completely random things, while two of the more intellectual people in the group talk about something more sensible. It has all been so good. Picture perfect. Now, the nescafe seems to have killed the elec canteen, just because most of the idiot juniors prefer to sit over there and have overpriced grilled sandwiches instead of the evergreen maggi at the elec. The mech canteen seems to have so many unfamiliar faces, that its just weird to go in there. The teachers are particular about everything (ranging from attendence to assignments to who's talking in class to almost everything) except for, well, leave it.

I cant really figure out what am I more upset about. Is it the fact that its my last semester at college and after it everyone goes their own way never meeting anyone else at least for the next 5 years I suppose? Or is it the fact that the college I'm leaving is not the same as what it was when I entered it? I really don't know.

27 Feb 2010 6:58pm GMT

This post is SUPPOSED to NOT make any sense. Its just me talking a lot of garbage.
Sometimes I wish I had gone for studying something other than computer science, or maybe something in addition to it. Not that studying computer science is less fancy in any way; learning what the crazy researchers in CS are upto always gives me goosebumps. But then, I have my reasons when I say that.

Here in my country, the system goes something like this - there are a set of entrance examinations that some (mostly top) engineering colleges conduct, and whoever sails through those exams, gets a seat. Pretty straightforward. The thing I love about this system is - it (IMHO) encourages students to study Physics, Chemistry, and Maths, as deeply as possible. I love Physics. I love Maths. And since I spent a year at home studying for those exams, I got an opportunity to study elementary Physics/Maths all by myself, for one full year, without any disturbances of any sort. Now I'm no genius, especially at those things, but I think I have a fairly good understanding of those subjects. For one full year, I had the chance to study what I love, without any constraints. That one year, even though some people say that I wasted, I say that it was the best year of my life. I studied interesting textbooks, solved crazy problems, brain stormed (at times even for days) on particular problems, and talked to people who shared my interests. Probably thats the reason why visiting the webpage of the physics department of some university gives me a strange kick.

What else gives me a strange kick? Computer Science.

22 Feb 2010 1:32pm GMT

Development has been progressing well on fwrap, although the first release is still a month in the future, at the very least.

Since last summer, it became clear that the core of fwrap - the code that generates the fortran wrappers, the C and Cython headers and the cython wrappers - was inextricably tied into the syntax tree generated by fparser, in a bad way. Fwrap had to muck around too much in the guts of fparser's classes to get stuff done. By the end of GSoC, fwrap was in a workable state (still not at first release, though), but it was very brittle and increasingly difficult to add functionality incrementally.

My efforts of late have been to extract fwrap from fparser, and fwrap's core is in a better state. I'm finding it easier to add in the necessary functionality, and the overall architecture is easier to grasp and adapt. I want to make it tolerably easy for others to understand and contribute to the code-base, and keeping fwrap's core separate from fparser shrinks the mental load considerably.

These are the main areas of functionality that I'm planning on addressing for the first release:

1. All intrinsic datatypes (integer, real, double precision, complex, logical and character) wrapped end-to-end.
2. Top-level functions & subroutines (those not inside a module) wrapped, end-to-end.
3. Assumed shape arrays (declared with 'integer, dimension(:,:) :: int_array') wrapped, end-to-end.
4. Explicit shape arrays with trivial expression extents ('integer, dimension(d1, d2, d3) :: int_array') wrapped, end-to-end. (Assumed-size arrays ('dimension(d1, d2, *)') are included here.)
5. Automatic discovery and compilation of kind-type-parameter values. For instance, if a module has a parameter 'SP' defined and a subroutine uses that module's parameter as a kind-type-parameter, fwrap will handle things automatically for you. This is mostly a parsing issue and one where fparser needs work.
6. Older constructs, like common blocks & implicit typing, shouldn't be too bad, either. Since common blocks map to C structs, these might wait until we get to derived-type support. We'll see.

With the decoupling between fwrap and fparser, it will be possible to write by hand a 'pyf' interface file that gives fwrap all the necessary bits to make the right wrappers. This makes testing easier and further helps to decouple fwrap from fparser.

There are many more fortran features that fwrap will support, if I can get the time or attract other contributors. High on the list are function/subroutine arguments (done well), Python callbacks, and full module wrapping. Module wrapping has the possibility to be rather interesting, since it is possible to have a nice correspondance between Fortran 9x modules and Python extension types, so long as a few guidelines are adhered to on the Fortran side. The C binding features supported in all major fortran compilers makes this Fortran module -> Python extension type mapping possible. But that is far in the future. I should mention that Fortran derived types will be supported soon after the first release, as well.

So there's an update for you - long on promises & ideas, but things are coming together nicely. Once the first release is out I'll put more effort into publicizing things and facilitating contributions, including a more detailed roadmap.

21 Feb 2010 9:15pm GMT

As from now, this blog will be available on abeocracy.be

17 Feb 2010 4:37pm GMT

While I find no time to blog, I thought I'd post our newest release announcement here.

We are happy to announce the 0.2.0 (beta) release of scikits.statsmodels. This is both a bug-fix and new feature release.

Download
------------

You can easy_install (or PyPI URL:
http://pypi.python.org/pypi/scikits.statsmodels/)

Source downloads: http://sourceforge.net/projects/statsmodels/

Development branches: http://code.launchpad.net/statsmodels

Note that the trunk branch on launchpad is almost always stable and has the most up to date changes since our releases are so few and far between.

Documentation
------------------

http://statsmodels.sourceforge.net/

We invite you to install, kick the tires, and make bug reports and feature requests.

Feedback can either be on scipy-user or the mailing list at
http://groups.google.com/group/pystatsmodels?hl=en
Bug tracker: https://bugs.launchpad.net/statsmodels

Main Changes in 0.2.0
---------------------------------

* Improved documentation and expanded and more examples
* Added four discrete choice models: Poisson, Probit, Logit, and Multinomial Logit.
* Added PyDTA. Tools for reading Stata binary datasets (*.dta) and putting them into numpy arrays.
* Added four new datasets for examples and tests.
* Results classes have been refactored to use lazy evaluation.
* Improved support for maximum likelihood estimation.
* bugfixes
* renames for more consistency
RLM.fitted_values -> RLM.fittedvalues
GLMResults.resid_dev -> GLMResults.resid_deviance

Sandbox
-------------

We are continuing to work on support for systems of equations models, panel data models, time series analysis, and information and entropy econometrics in the sandbox. This code is often merged into trunk as it becomes more robust.

16 Feb 2010 2:24pm GMT

Earning a PhD at a great university has been a dream I've been cherishing for the last 4 years. And for all I know, the motive for that has been nothing more than "doing good work in a field I love". Now that the 4 years of my undergraduate education are coming close to an end, I've started to feel like graduate school admissions are nothing more than good, rather *great* academic performance. Sure it might matter if the work experience on your application is *exceptionally research oriented*, but the fact that a moderate profile gets rejected just because of moderate grades makes me a little sad at times. For one simple reason - I've never wanted something so badly for myself.

08 Feb 2010 5:22pm GMT

Today I released the first public version of Scikit-Learn (release notes).

It's a python module implementing some machine learning algorithms, and it's shaping quite good. For this release I did not want to do any incompatible changes, so most of them are just bug fixes and updates.

For the next release, however, some more radical changes are planned, and definitely something should be done about the (incredibly long) namespace, having to tape

each time you want to perform a nearest-neighbor algorithms is just not practical!

Here is a nice screenshot,

01 Feb 2010 1:32pm GMT

It's been a while since an update.

Events out of my control have consumed my discretionary time, events that have conspired to delay fwrap's delivery. The usual story. I lean on your patience, and I'm glad to know there is still interest out there.

A critical component of fwrap is its compilation infrastructure. I've been thinking about this a lot lately; any input you care to share with me is very welcome. At first I was using a simple Makefile for the automated integration tests on my box here. That's fine for testing, but won't cut it for all the different systems out there. I've since moved to a monkeypatched Cython-numpy distutils script that gets the job done, but isn't the shiniest bit of code I've written.

The compilation system has to do a number of disparate things:

1. Somehow generate the ISO C BINDING mappings from Fortran datatype to C datatype. For some setups, this may require compiling and running a Fortran source file. I'm informed by some (David Cournapeau, who has some expertise in this area) that doing this on some platforms is less than ideal. There will be other setups, though. More on this in a future post.
2. Cythonize the Cython source files.
3. Compile the Fortran source with the appropriate flags.
4. Compile the C source.
5. Create the extension module.

So the compilation script will, in an ideal world, be able to compile and run a Fortran source file, cythonize Cython code, compile C & Fortran code, and compile an extension module. All while being intelligently adaptable to different command-line flags to customize different Fortran compilers & compilation flags, etc. A tall order.

Here are the candidates, as I see it:

1. Cython-numpy distutils - get this in shape and make it robust. This encapsulates lots of work done in numpy.distutils.fcompiler to automate all the different fortran compilers out there on all different platforms. I'd be a fool to not make use of it. On the other hand, distutils stuff can be ugly and error-prone, in my limited experience. A benefit is that it would require just Cython and Numpy; Cython will be a dependency of fwrap, and Numpy very well might be. So no extra dependencies, which is great. All in all, though, I'd like something more robust out of the box, something like…
2. Scons - I've been favorably disposed to what I've seen of scons, and I'd love to use it. I've not experimented with its fortran support yet, although browsing its source shows some promise. I see a number of different fortran compilers there that are supported, although not all (pathscale, xlf, others…) It would be an extra dependency - is it worth it? And how hard would it be to support other fortran compilers? For that I'm looking into…
3. Numscons - (http://projects.scipy.org/numpy/wiki/NumScons) This likely is the bees' knees of what I'm looking for, a combination of scons with numerical compilation. We know it works for numpy so there's one success already. Is it mature? Will it be supported in the future?
4. Makefiles (on systems that would support it) - I'm Makefile literate, but no guru. And it may be a minefield with all the different systems out there, i.e. it might require a guru.

If anyone has the competence to comment on the above I'm all ears.

15 Jan 2010 12:12am GMT

This week we created a sourceforge project to host our development of scikit-learn. Although the project already had a directory in scipy's repo, we needed more flexibility in the user management and in the mailing list creation, so we opted for SourceForge.

To be honest, after using git and Google Code for bug tracking, I was not very excited about using subversion/sourceforge again. On the other hand, we needed some sort of compromise that would allow a very heterogeneous range of developers to work together, and after some (surprisingly civilized) emails and some chatting with Gael, we agreed that SourceForge was indeed the best choice.

In case you are interested, there's a (preliminary) web page with more info. You might also want to have a look at the previous project's web page.

07 Jan 2010 1:17pm GMT

I like XCode, and I use it to edit all of my source for SymPy. But, like many editors, it likes to auto-indent new lines to the level of indentation of the previous line. This is a useful feature, but it makes for training whitespace out the wazoo, since blank lines will be indented in. I am constantly finding myself using SymPy's strip_whitespace script to clean up my files.

This bugged me enough that I Googled a solution, and found this. It is a simple XCode plugin that, among other things, adds an option to strip trailing whitespace on save. Just install in the PlugIns folder in the XCode package and enable the option in the new Google pane of the XCode preferences.

29 Dec 2009 11:56pm GMT

This week I arrived to the place where I will be working the following two years: Neurospin.

It's a research center located 20 km from Paris, and so far things are going smoothly: the place is beautiful, work is great and food is excellent. Well OK, I do miss some things from Spain and weather is horrible, but from now on it can only get better, I suppose.

22 Dec 2009 5:36pm GMT

School this semester was hard. I have a hard time with exam environments. Even my programming course took a lot of my time due to our (pretty cool) service learning project.

We made a J2ME app for a local community centre in our school which involved allowing a user to see his or her timetable and set reminders to the calendar. J2ME was a pain to work with, so some of the group members and I vowed (sort of) to complete it. We may end up making it blackberry specific. Code signing for J2ME is pretty expensive and we aren't getting paid.

So lately I had some free time in between exams and played around with the new widget api for the blackberry. It was fun and easy, but the resulting app, maybe due to me never having used javascript before, was painfully slow. I used jquery and learned quite a lot about javascript dev, and managed to improve it quite a bit, but its clear a native app would be faster and not that much harder to make. I conclude that if you're fine with Java, there is no harm in sticking with it. it was a fun experience, though, and I'm glad I gave it a try.

20 Dec 2009 5:33pm GMT