19 Apr 2010
Planet Turbogears
Plumbling Life's Depths: Duh! But upstart on 1.5 year old OS not so great...
19 Apr 2010 12:10pm GMT
Plumbling Life's Depths: Setting up a DHCP test environment in QEMU
My current project must run against a DHCP server which has to have a fairly exotic configuration (not really one I'd want running on my home network), and I want to be able to blow away and re-create the client constantly as I test installation procedures and the like. So, since we're on Fedora 10 (server), I've set up a little QEMU environment like this:
- base read-only Fedora 10 minimal install with users and basic config (qcow2)
- overlay one, the DHCP server, with one "user stack" nic and one vlan nic
- overlay two, the client server, same network config
but after getting all that set up, I can't get the silly vlan nic to come up on the DHCP server
. The idea is that I'll write scripts to blow away and recreate the second overlay then install the software to test into it... but this silly bug has just blocked the whole process.
The ifup scripts are complaining that another machine already has the 10.220.* ip addresses I'm configuring (unlikely, given that there's just to the two machines on the vlan). I expect it is some silly Fedora, SELinux, or Qemu thing, or maybe a conflict between those... now I just have to narrow that down a little
.
[Update] Fixed. Don't get "cute" with your MAC addresses in QEMU. I'd assigned the MAC 01:02:03:04:05:06 to the DHCP server's back-end nic, letting QEMU auto-assign the MAC instead got it working.
19 Apr 2010 12:04pm GMT
13 Apr 2010
Planet Turbogears
Plumbling Life's Depths: Fedora 10 without the Cruft
13 Apr 2010 6:43pm GMT
Plumbling Life's Depths: Upstart seems kinda nice
13 Apr 2010 7:35am GMT
10 Apr 2010
Planet Turbogears
Plumbling Life's Depths: Extract all fields of a DER-encoded PKCS7 file?
Say you have a PKCS#7 file with an embedded code-image as a field, how would you go about extracting that field's contents? pyOpenSSL doesn't seem to have mechanisms for extracting/viewing arbitrary fields from PKCS7. pyasn1 can load the structures, but of course it doesn't have access to the meaning of the fields, and it doesn't seem to see the final blob field as an ASN1 field at all... I can just stop parsing at that point and consider the result "the file", but that's... ahem... a little sub-optimal.
The real question being: what do you use when they want to work with the arbitrary fields of a PKCS7 file? I'll eventually need to generate such files (with lots of specified fields) as well as "consume" them, so I'm looking more for robust real-world solutions than hacks.
10 Apr 2010 4:15am GMT
07 Apr 2010
Planet Turbogears
Michael Pedersen: What To Do When It's Time To Find A New Maintainer
This post is going to ramble a bit, and I apologize. Almost two years ago, I got an Asus EEE 901 netbook. This is a great little device, letting me have a highly portable laptop style computer. I then got the idea to use it for GeoCaching, and wrote Cache 901. The problem I face now is that it's time for me to hand Cache 901 over to somebody else.
When I wrote that program, GeoCaching was something that would be done every couple of months. We'd go out, find a couple caches, have some fun while doing so, record them, and get back on with our lives. Since then, something's changed, and we haven't done it for over a year by now at all. Since we're not GeoCaching, and it doesn't look like it will happen again anytime soon, I've stopped working on Cache 901.
Now, Cache 901 is a good program, and fairly popular (as of now, it's been downloaded nearly 900 times so far this year). I don't want the program to die. I also don't want to be responsible for taking it to the next level. I want to hand it off to somebody else. I want it to become active again. The worst part is that I don't know how to find someone to take it over.
If you have any suggestions, please let me know.
07 Apr 2010 10:36pm GMT
05 Apr 2010
Planet Turbogears
Plumbling Life's Depths: Generic Binary-Data Pack/Unpack Module?
Having just written yet another module to pack/unpack data into a particular binary format, I'm wondering if someone has a more effective method they'd like to share? I'm thinking some module that can handle RLE/TLV type stuff, complex "encodings" such as utf-8 continued charcters, struct-like pack/unpack for static elements, efficient loading of arrays, and bit-field pack/unpack. Some way to customize handling of a given element within the format (so you could write constraints or what have you), some way to provide defaults, both static and calculated, some way to specify derivative fields, checksums and the like, some way to...
Whenever I look at generalizing this pattern, it always seems to turn into "you should have the equivalent of a compiler/parser-generator" produce this code, but if someone has a good, generic solution, I'd love to hear about it.
05 Apr 2010 5:26am GMT
29 Mar 2010
Planet Turbogears
Plumbling Life's Depths: tgext.command 0.2 is out...
Adds support for instantiating Beaker's Cache from the config file (i.e. you can do cache-specific operations such as edge-triggered-cache-clearing in your utility scripts). That's mostly useful if you have content-modifying scripts and want to clear/populate the cache when the script runs. Available in PyPI.
29 Mar 2010 9:48pm GMT
Plumbling Life's Depths: Don't make me guess, Gentle Spec Writer
Say you were writing a major standard, something implemented by thousands of companies. Say, one which specified, among other things, a format for public key files. Might it be appropriate to say "use a BER-encoded x509" certificate (with some (standard) x509 extensions)? Or would you:
- declare that you require a "special" binary format for your x509 certificates
- provide a large table breaking out an octet-by-octet dump (of an example BER certificate) as an example of how one would go about writing a certificate
- omit any explanation of what the various "key" or "header" values are in said binary dump (or how you calculate them from the x509 cert)
- leave it to the user to recognize that this "special" format must just be regular-old BER by the absurdly under-specified nature of the encoding
What implementer is going to write their own BER encoder/decoder? Why would you need to know the particular byte sequences in an example certificate? It would be a lot more useful to most people to include a PEM-format file and instructions on how to encode/decode it via openssh.
29 Mar 2010 3:05am GMT
26 Mar 2010
Planet Turbogears
Michael Pedersen: Emacs As 3Way Diff Tool, Use With Mercurial
So, I've switched recently to EMacs. Mostly doing okay with the switch, but today I wanted to do something I'd not seen mentioned elsewhere: I wanted to use it as a 3 way diff tool, and integrate this with Mercurial. Turns out it's not too hard to do at all.
First, you'll need to make the following into a script. I named it "ediff3":
#!/bin/bash
if [ $# -ne 3 ]; then
echo Usage: $0 local base other
exit 1
fi
emacs --eval '(ediff-merge-with-ancestor "'$1'" "'$2'" "'$3'")'
26 Mar 2010 9:16pm GMT
20 Mar 2010
Planet Turbogears
Plumbling Life's Depths: Remote networking is a PITA
20 Mar 2010 4:31pm GMT
17 Mar 2010
Planet Turbogears
Plumbling Life's Depths: RunSnakeRun 2.0.0b6 (and SquareMap 1.0.0b25) released
17 Mar 2010 12:32am GMT
16 Mar 2010
Planet Turbogears
Michael Pedersen: wxPython in a virtualenv
virtualenv is an absolutely wonderful tool. I use it every day in so many ways, and enjoy the freedom it gives to experiment with anything. Found a new tool, and want to try it out, but don't want to screw up your system's Python libraries? Try it out in a virtualenv!
Unfortunately, getting wxPython in a virtualenv is a painful process, and definitely takes some work. Here's the steps to do so:
- Download this patch, and save it as $HOME/wxpatch.txt.Without this patch, wxPython2.8.10.1 will fail to compile properly. Hopefully, this will be fixed in the next release.
- Make your virtualenv. If you're like me, you're going to be using virtualenvwrapper, so you'll run this command:
mkvirtualenv --no-site-packages wxpython
- Activate your virtualenv. If you are using the virtualenv wrapper I linked above, you would do "workon wxpython". Otherwise, it will be "source $PATH_TO_VENV/bin/activate".
- Download the wxPython source and save it as $HOME/wxPython-src-2.8.10.1.tar.bz2
- Now comes the messy command. Yes, all of these are genuinely necessary in order to make this work properly. Make sure to substitue "$PATH_TO_VENV" with the appropriate real path to the root of the virtual environment you made in step 2. Warning: The next steps will take a long time.
cd $HOME
tar -xjf wxPython-src-2.8.10.1.tar.bz2
cd wxPython-src-2.8.10.1
patch -p0 < $HOME/wxpatch.txt
./configure --prefix=$PATH_TO_VENV --with-gtk=2 --enable-unicode --with-opengl
make install
cd contrib/src/stc
make install
cd ../gizmos
make install
cd ../../../wxPython
python setup.py install
- Finally, you will need to ensure that your LD_LIBRARY_PATH variable is configured. If you are using the virtualenv wrappers mentioned above, create a script named $PATH_TO_VENV/bin/postactivate that has the following contents (no substitutions of any sort here at all!):
export LD_LIBRARY_PATH=$VIRTUAL_ENV/lib:$LD_LIBRARY_PATH
Now, set the script to be executable.
- If you're not using the virtualenv wrappers mentioned above, edit $PATH_TO_VENV/bin/activate and add that same line at the end of the file.
With all of the above done, you only have to activate the virtualenv, and you can then import and use wx inside of it without problem.
16 Mar 2010 7:14pm GMT
Plumbling Life's Depths: Say it with me: Errors should never pass...
Say you were writing a program that takes a "config-file" parameter and you were (explicitly) passed a filename that, for whatever reason, you couldn't load. Is the proper operation to exit and report an error, log a warning and continue with defaults, or load a default filename silently?
In this particular case, app-armour was configured to prevent dhclient from accessing most of the filesystem. It (app-armour) was dutifully logging errors in /var/log/messages, but dhclient was just silently defaulting back to the very-subtly different default config... queue lots of very subtle debugging effort. Maybe there's some good reason for this behaviour, but at the moment I can't see it.
import this
16 Mar 2010 5:56pm GMT
15 Mar 2010
Planet Turbogears
Plumbling Life's Depths: Infinite Deployment Docs and OpenID Issues
Have spent rather a lot of time trying to wrestle the deployment docs into shape for the TurboGears 2.1 release. Even with all that work, only the "standard deployment pattern" is really in "good" shape. Alternative servers, alternative deployment styles, etceteras are all still needing lots of love. I am, however, I think, finished with that section for a while unless someone else chips in to help.
Also ran into one of those "argh" inducing issues this evening. A user was trying to follow through the "Using who.ini" discussion on his way to doing OpenID with TurboGears and ran into a weird, subtle bug. Turns out, when you disable TurboGears authentication repoze.what "fails open" as far as much TurboGears code is concerned. The predicates just all silently start evaluating to True. The @require() decorator thankfully does the right thing, but users who do:
tg.predicates.has_permission('manage')
Now always get a True value (the predicate object), so if you're using that to control display of information... oops. Not likely to have bitten too many users, but it just seems we should have something raising errors on _nonzero_() if the predicate's aren't "booleanized()".
In other news, given that someone else is trying to get OpenID working, I suppose I should look at the branch of repoze.openid. Upstream wants changes, but I haven't yet figured out what's supposed to happen wrt registration to make those changes work. Repoze just doesn't seem to have a registration story AFAICT.
15 Mar 2010 9:33pm GMT
12 Mar 2010
Planet Turbogears
Michael Pedersen: JSOn vs XML vs YAML, and Python Parsing
EDIT: I added YAML testing data to the results. I'm not posting my original files, as I don't have a license to be converting this game, and don't want to trip any legal issues. Suffice to say that the names in the file give it away.
EDIT: I added ElementTree testing data to the results.
I've begun working on translating a board game into a computer game. In order to do so, I need to be able to represent the state of the game and its many (many) tokens. In addition, I need to be able to store and load that state quickly. Finally, I need that state to be something that I can represent with a single string.
Storing the state presents a simple hierarchical structure: Multiple decks of cards are shuffled, and the shuffled state must be saved. This allows for a web front end to the game. The cards are just one aspect: I have to store information about what cards the player has, what state the cards are in, where the player's token is, and what feels like about 500 pieces of information (no joke: I scanned 494 images for this game).
So, how do I represent this data? How do I make it so that a web server can load and save the state quickly? XML is good for hierarchical data, as are JSON and YAML. But which one is right this time?
I needed to test it out. So, I wrote three equivalent files, one in XML, one in JSON, one in YAML. I then ran the timeit module against those files, using the following commands:
- python -m timeit -n 100 -s 'from xml.dom.minidom import parse' 'd=parse("allboard.xml")'
- python -m timeit -n 100 -s 'from xml.etree.ElementTree import parse' 'd=parse("allboard.xml")'
- python -m timeit -n 100 -s 'import simplejson' 'd=simplejson.load(open("allboard.json"))'
- python -m timeit -n 100 -s 'import yaml; from yaml import CLoader as Loader' ' d=yaml.load(open("allboard.yaml"), Loader=Loader)'
- python -m timeit -n 1000 -s 'from lxml import etree' 'd=etree.parse(open("allboard.xml"))'
- python -m timeit -n 100 -s 'from xml.etree.cElementTree import parse' 'd=parse("allboard.xml")'
The results were simply staggeringly different.
XML (minidom) took 15.3ms per run. XML (ElementTree) took 19.6ms per run. XML(lxml) took 925usec per run. XML (cElementTree) took 1.35ms per fun.JSON took 250usec per run. YAML took 10.1ms per run when using the CLoader. Without the CLoader, it took 115ms per run.
JSON wins this one hands down.
Is it possible I chose poorly? Of course. There are many XML parsers for Python, just as there are many JSON parsers. I wanted pure Python here, so as to maximize portability. I chose these modules since they should be completely standard. If I'm wrong, tell me, and I'll revisit this.
12 Mar 2010 8:54pm GMT