07 Nov 2011

feedBSD Planet * BSD People

The Julipedia (Blog): Kyua: Weekly status report

Many things have happened this week, but they can all be summarized in one single sentence: &lt;tt&gt;&lt;b&gt;kyua test&lt;/b&gt;&lt;/tt&gt;&lt;b&gt; now records the results of the execution of a test suite into the SQLite database&lt;/b&gt;.&lt;br /&gt;&lt;br /&gt;"Why is this important?", you ask. Well, that's a good question. Recording test results opens the gate to many long-awaited features that should be coming soon, such as the ability to inspect the history of a particular test, to query all available data of a test result and/or to generate a dashboard of test results. It's interesting to realize that &lt;i&gt;most of these features are just one SQL query away&lt;/i&gt;. If you install Kyua, you can already run a few tests and then use &lt;tt&gt;kyua db-exec&lt;/tt&gt; to issue arbitrary SQL queries against the database; the schema (see &lt;tt&gt;store/schema.sql&lt;/tt&gt;) might look a bit convoluted, but a bunch of &lt;tt&gt;NATURAL JOIN&lt;/tt&gt;s will yield the desired output.&lt;br /&gt;&lt;br /&gt;The feature requests that have the highest priority at this point are the ability to generate a report of the last tests run both as a text file and as an HTML dashboard, because having these features means we can finally kill the &lt;tt&gt;atf-run&lt;/tt&gt; and &lt;tt&gt;atf-report&lt;/tt&gt; pair. At this point I'm, once again, "stuck" while figuring out how to best organize the code to make all these things possible while still keeping a nice separation across the existing layers (&lt;tt&gt;cli&lt;/tt&gt;, &lt;tt&gt;engine&lt;/tt&gt; and &lt;tt&gt;store&lt;/tt&gt;)... all without introducing much unnecessary complexity. But exciting times lie ahead!&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-5051367421950896129?l=blog.julipedia.org' alt='' /&gt;&lt;/div&gt;</content>

07 Nov 2011 3:48pm GMT

31 Oct 2011

feedBSD Planet * BSD People

The Julipedia (Blog): Kyua: Weekly status report

&lt;br /&gt;&lt;ul&gt;&lt;li&gt;Submitted a placeholder implementation of the new persistence layer (&lt;tt&gt;store&lt;/tt&gt; top-level directory). This only supports opening a database and ensuring its metadata is valid. &lt;a href="http://code.google.com/p/kyua/source/detail?r=253"&gt;See r253&lt;/a&gt;.&lt;/li&gt;&lt;li&gt;Added a &lt;tt&gt;db-exec&lt;/tt&gt; CLI command to execute arbitrary SQL commands onto the database. This is handy during development and testing, but may also help users to extract information out of the database in those cases where the CLI does not cover their needs just yet. &lt;a href="http://code.google.com/p/kyua/source/detail?r=255"&gt;See r255&lt;/a&gt;.&lt;/li&gt;&lt;li&gt;Miscellaneous fixes and improvements to &lt;tt&gt;utils::env&lt;/tt&gt; and &lt;tt&gt;utils::sqlite&lt;/tt&gt;.&lt;/li&gt;&lt;li&gt;Preliminary code to support putting objects (like actions and contexts) into the database. I've been thinking about this for a while and finally came up with a design that completely decouples the persistence needs from the higher-level classes in the &lt;tt&gt;engine&lt;/tt&gt; layer. I haven't submitted the code yet though, as it lacks tests. (Still thinking how to write the loading of objects though.)&lt;/li&gt;&lt;/ul&gt;&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-5304169474623962697?l=blog.julipedia.org' alt='' /&gt;&lt;/div&gt;</content>

31 Oct 2011 2:48pm GMT

24 Oct 2011

feedBSD Planet * BSD People

The Julipedia (Blog): Kyua: Weekly status report

&lt;br /&gt;&lt;ul&gt;&lt;li&gt;Moved all the logic code of the "debug", "list" and "test" commands from the CLI layer to the engine layer.&lt;/li&gt;&lt;ul&gt;&lt;li&gt;Up until now, the CLI modules implementing these commands contained all the logic to load Kyuafiles and iterating over them to find matching tests and applying the desired operation to them. This kind of code belongs in "driver" modules (aka controllers) of the engine layer, because there is nothing UI-related in them.&lt;/li&gt;&lt;li&gt;After this refactoring, the code left in the CLI modules is purely presentation-related, and the code in the engine implements all the logic.&lt;/li&gt;&lt;li&gt;The goal of these changes is to be able to hide the interactions with the database in these controllers. The CLI layer has no business in dealing with the database connection (other than allowing the user to specify which database to talk to, of course).&lt;/li&gt;&lt;/ul&gt;&lt;li&gt;Implemented a very simple RAII model for SQLite transactions.&lt;/li&gt;&lt;li&gt;Some additions to the utils::sqlite bindings to simplify some common calling patterns (e.g. binding statement parameters by name).&lt;/li&gt;&lt;li&gt;Preliminary prototypes at database initialization. This involves creating new databases and populating them with the initial schema, plus dealing with database metadata to, e.g. detect if we are dealing with the correct schema version.&lt;/li&gt;&lt;ul&gt;&lt;li&gt;The code for this is still too crappy to be submitted, so don't look for it in the repository just yet!&lt;/li&gt;&lt;li&gt;The design document details many things that should be part of the schema (e.g. "sessions"0, but I've decided that I'll start easy with a simplified schema and later build on top of it. Otherwise there will be too many clunky moving parts to deal with while the fundamental ideas are not yet completely clear.&lt;/li&gt;&lt;/ul&gt;&lt;li&gt;Fixes to let the code build and run again in NetBSD (macppc at least).&lt;/li&gt;&lt;/ul&gt;&lt;div&gt;I've now been stuck for a few days trying to figure out what the best way to implement the conversion of (new) in-memory objects to database objects is, and how to later recover these objects. E.g. what the correct abstractions are to take test case results and put them in the database, and how to retrieve these results to generate reports later on. I now start to have a clear mental picture on how this should look like, but I have yet to see how it will scale.&lt;/div&gt;&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-402139290656767976?l=blog.julipedia.org' alt='' /&gt;&lt;/div&gt;</content>

24 Oct 2011 2:48am GMT

22 Oct 2011

feedBSD Planet * BSD People

Hubertf's NetBSD Blog: Enlarging a (virtual) disk

I've tried to build NetBSD-current at various points in the past few months, and always hit one of two bugs: -current blows up with a gcc Internal Compiler Error when crossbuilding on Mac OS X, and kernel panics with native NetBSD builds with sources on NFS. This stinks, and I've successfully managed to do a successful -current build with sources on (local) disk. With NetBSD running within VMware Fusion on Mac OS X.

To go on from there, I found that my NetBSD VM's only disk was too small to do anything useful. Options for enlarging that came to mind:

  1. NFS - see 'panic' above, no go.
  2. Adding another (virtual) disk - easily doable, but I felt like not adding one
  3. Extending the existing disk - adventure time!

Option #3 was it, and after removing all VMware snapshots, enlarging the disk was easy with VMware Fusion, going from 10GB to 20GB. After growing the disk itself, the next question was how to use the newly gained disk. Of course some file system needs to use it, and in theory there are the following options:

  1. Enlarge the last file system on disk
  2. Fix the partition table to add another partition for the new space

The disk was resized from 10GB to 20GB. The partition table (disklabel) was created by a standard NetBSD install, and first had the root file system, followed by the swap partition. From that, adding 10GB more swap was not useful, so I've decided to change the disklabel to add the new disk space as a new partition behind the existing partitions. This is also an excuse to not frob with growfs and resize_ffs. (And of course I'm ignoring the option of backing up the full file system, doing a full rebuild of the filesystem and then doing a restore :-)

For those in a similar situation, here are the steps to use the newly gained space on an enlarged (virtual) disk:

  1. Prepare: save the old output of "dmesg" (/var/run/dmesg.boot is OK)
  2. Enlarge - VMware Fusion wants a shutdown for that, you cannot suspend the machine
  3. After booting, run a diff on the saved "dmesg" output, to learn what the old and new size of the disk is, in sectors. My diff looks like this, note the size change in sectors:
    -wd0: 10240 MB, 22192 cyl, 15 head, 63 sec, 512 bytes/sect x 20971520 sectors
    +wd0: 20480 MB, 44384 cyl, 15 head, 63 sec, 512 bytes/sect x 41943040 sectors 
  4. Backup the existing/old disklabel, just in case: disklabel wd0 >disklabel.BAK
  5. Edit the disklabel: disklabel -e wd0
  6. In the editor, adjust the disk size in sectors from 20971520 to 41943040:
    total sectors: 41943040 
  7. Partition 'd' is the full disk on i386/amd64, it starts at sector 0 and is 41943040 sectors big
    #        size    offset     fstype [fsize bsize cpg/sgs]
     d:  41943040         0     unused      0     0        # (Cyl.      0 -  44384*)
  8. Partition 'c' is the NetBSD part of the disk. As this VM only has NetBSD, all the usable space is used. Note that "usable" space excludes the first 63 sectors of the disk (mbr etc.), i.e. it is 41943040 - 63 = 41942977 sectors:
    #        size    offset     fstype [fsize bsize cpg/sgs]
     c:  41942977        63     unused      0     0        # (Cyl.      0*-  44384*)
  9. After this everything is in sync with the new disk again, and the remaining/new space can be used for new partition 'e'. As the new space starts where the disk used to end, its offset is the old size, 20971520 sectors.

    The size of the new partition expands from the offset sector 20971520 to the end of the disk at sector 41943040, i.e. the partition size is:

    % expr 41943040 - 20971520
    In total, this gives for the new partition:
    #        size    offset     fstype [fsize bsize cpg/sgs]
     e:  20971520  20971520     4.2BSD   2048 16384     0  # (Cyl.  22192*-  44384*)
  10. Last, create file system, mount and populate:
    # newfs /dev/rwd0e
    # mkdir /disk2
    # echo '/dev/wd0e /disk2 ffs rw,log 2 2' >>/etc/fstab
    # mount /disk2
    # cd /usr ; pax -rw -pe -v stuff /disk2
    # rm -fr stuff ; ln -s /disk2/stuff .

Now let's see if I get things far enough to get a build of g4u going... wish me luck!

P.S.: I'm offering choccolate to anyone fixing crossbuilding of NetBSD-current from Mac OS X. Any takers?

22 Oct 2011 9:04pm GMT

17 Oct 2011

feedBSD Planet * BSD People

The Julipedia (Blog): Kyua: Weekly status report

&lt;br /&gt;&lt;ul&gt;&lt;li&gt;Added support to prepare and execute statements to &lt;tt&gt;utils::sqlite&lt;/tt&gt;.&lt;/li&gt;&lt;li&gt;Added the &lt;tt&gt;UTILS_PURE&lt;/tt&gt; definition to tag functions as pure. I should now sweep through old code to apply the attribute where possible.&lt;/li&gt;&lt;li&gt;Created a pkgsrc package for Vera++. Investigating if I can use this tool for coding style validation, as the current code of Kyua is a bit messy in this regard.&lt;/li&gt;&lt;li&gt;Made a quick attempt at getting &lt;tt&gt;kyua test&lt;/tt&gt; record test results in a simple database; success! This was just a proof of concept, so it is not submitted yet.&lt;/li&gt;&lt;li&gt;Started to refactor the code to move many logic constructions from the &lt;tt&gt;cli&lt;/tt&gt; to the &lt;tt&gt;engine&lt;/tt&gt;. With the need to store test results in the database, it's clear that these semantics do not belong in the CLI, but the current code structure do not easily allow this. Need to add some "controller" classes in the &lt;tt&gt;engine&lt;/tt&gt; to hide all the interaction among internal components.&lt;/li&gt;&lt;/ul&gt;&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-4275543087243170349?l=blog.julipedia.org' alt='' /&gt;&lt;/div&gt;</content>

17 Oct 2011 3:48am GMT

09 Oct 2011

feedBSD Planet * BSD People

The Julipedia (Blog): Kyua: Weekly status report

&lt;br /&gt;&lt;ul&gt;&lt;li&gt;Some more reading on SQLite. The &lt;a href="http://www.amazon.com/Using-SQLite-Jay-Kreibich/dp/0596521189"&gt;Using SQLite&lt;/a&gt; book is close to awesome; very easy to read and full of insightful tips.&lt;/li&gt;&lt;li&gt;Cleaned up my preliminary SQLite wrapper code. Still very incomplete, but I've bitten the bullet and decided to commit the new library as is; otherwise I'll just keep&amp;nbsp;procrastinating. So, ladies and gentlemen, welcome &lt;a href="http://code.google.com/p/kyua/source/detail?r=228"&gt;the new utils::sqlite module in Kyua&lt;/a&gt;. At the moment, this just provides the barebones to open and close a database. I need to start playing around with preparing statements before I can model these in my wrapper classes. (Yes, the API structure is extremely similar to that in &lt;a href="http://code.google.com/p/lutok/"&gt;Lutok&lt;/a&gt;.)&lt;/li&gt;&lt;/ul&gt;&lt;div&gt;&lt;b&gt;Update (23:40)&lt;/b&gt;: I sent this too early, thinking I would not have anything nice to report this week either. But as it turns out, I finally had some spare time this late evening and got a chance to submit the extremely-incomplete SQLite C++ bindings I've been toying around with. See the update above! :-)&lt;/div&gt;&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-5370918960000243408?l=blog.julipedia.org' alt='' /&gt;&lt;/div&gt;</content>

09 Oct 2011 11:48pm GMT

03 Oct 2011

feedBSD Planet * BSD People

The Julipedia (Blog): Kyua: Weekly status report

&lt;ul&gt;&lt;li&gt;Unfortunately, not much time this week either :-(&lt;/li&gt;&lt;li&gt;I am currently working on some adjustments to the design document of the database to describe new ideas; the previous design was incomplete in some areas and/or not accurate enough to support the described use cases.&amp;nbsp; However, I've not had the to time to finish these edits and publish them.&lt;/li&gt;&lt;/ul&gt;&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-2063946456167536880?l=blog.julipedia.org' alt='' /&gt;&lt;/div&gt;</content>

03 Oct 2011 7:48pm GMT

26 Sep 2011

feedBSD Planet * BSD People

The Julipedia (Blog): Kyua: Weekly status report

Unfortunately, no activity this week other than some brainstorming on the database design.&lt;br /&gt;&lt;br /&gt;Why? My shipment container from Dublin arrived and I spent most of the weekend organizing stuff, setting up my little NetBSD box and attending a friend's wedding!&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-1017676619421586594?l=blog.julipedia.org' alt='' /&gt;&lt;/div&gt;</content>

26 Sep 2011 10:48pm GMT

19 Sep 2011

feedBSD Planet * BSD People

The Julipedia (Blog): VirtualBox and port forwarding (and Fusion)

I have been a &lt;a href="http://www.vmware.com/products/fusion/overview.html"&gt;VMware Fusion&lt;/a&gt; user for a long time and I think I am about to be a &lt;a href="http://www.virtualbox.org/"&gt;VirtualBox&lt;/a&gt; convert. Let me explain you my story.&lt;br /&gt;&lt;br /&gt;With the release of the &lt;a href="http://msdn.microsoft.com/en-us/windows/home/br229518"&gt;Windows 8 Developer Preview&lt;/a&gt;, I felt like giving the system a try. Not because I particularly like or care about Windows, but just because I am genuinely curious about operating systems and wanted to check out their new Metro interface. So I went ahead, downloaded the ISO image, created a new virtual machine under Fusion 3, started the virtual machine, and saw Fusion crash while the Windows installer was booting.&lt;br /&gt;&lt;br /&gt;Oh well, an emulation problem. I tried to fiddle with the virtual machine settings a bit, selecting different CPU virtualization settings and different hardware devices. No luck. After that, I resorted to Google to look for a solution to the problem and found that other people were experiencing the same issue. But, at the same time, I also found that VMware had just released Fusion 4 and that this new release works with Windows 8 just fine. Ah, cool! Or not... $50 for the upgrade. For my use cases (which amount to booting a few BSD and Linux virtual machines in console mode), it is hard to justify the upgrade - specially when it would just be to fulfill my curiosity and then delete the virtual machine 10 minutes later.&lt;br /&gt;&lt;br /&gt;But wait! There is that thing called VirtualBox. I had used this product a while ago and did not enjoy doing so because its interface was utterly (really) confusing. It was nowhere near the polish of Fusion or Parallels. However, to my surprise, most of my major itches have been fixed and the new interface of VirtualBox is bearable. And it boots the Windows 8 preview just fine!&lt;br /&gt;&lt;br /&gt;Right. So after installing Windows 8, playing with it for 10 minutes, and later deleting it, I decided to look around and see what else VirtualBox has to offer. It has &lt;i&gt;many&lt;/i&gt; settings. Yes, I know, Fusion has many settings too, but most are hidden from the UI and can only be accessed by modifying the &lt;tt&gt;vmx&lt;/tt&gt; text file. In the case of VirtualBox they are easily accessible.&lt;br /&gt;&lt;br /&gt;I kept looking around... and there it was, the "&lt;b&gt;Port Forwarding&lt;/b&gt;" option staring at me under the Network settings.&lt;br /&gt;&lt;br /&gt;One of the features I've been wishing for a long, long, &lt;i&gt;long&lt;/i&gt; time is the ability to easily configure a NATed (not bridged) virtual machine in my laptop, assign a host name to it from the graphical interface and then be able to SSH into the virtual machine from the laptop. As it turns out, this is doable in Fusion but it requires some nasty tinkering: one has to edit a &lt;i&gt;dhcpd&lt;/i&gt; configuration file by hand to add a new stanza for the virtual machine (&lt;tt&gt;/Library/Application Support/VMware Fusion/vmnet8/dhcpd.conf&lt;/tt&gt;). This involves manually copy/pasting the MAC address of the virtual machine into that file (which you must first extract from the &lt;tt&gt;vmx&lt;/tt&gt; file), assigning it an IP in the corresponding subnet, and sticking the IP plus host name in the &lt;tt&gt;/etc/hosts&lt;/tt&gt; file. This gets boring pretty quickly.&lt;br /&gt;&lt;br /&gt;VirtualBox may or may not support this kind of setting, but it has the aforementioned "Port Forwarding" option which is interesting on its own. As you can imagine, this sets up forwarding of a port from the host system to a port of the virtual machine. Creating an entry for SSH is as easy as clicking a couple of buttons, choosing a local port on the host system (e.g. 2022) and forwarding it to port 22 of the guest system. Voila. You can now access your virtual machine locally &lt;i&gt;and remotely&lt;/i&gt; without having to go through any of the host name nor DHCP madness I experienced before:&lt;br /&gt;&lt;br /&gt;&lt;tt&gt;$ ssh -p 2022 localhost&lt;/tt&gt;&lt;br /&gt;&lt;br /&gt;You can later create an alias in your &lt;tt&gt;~/.ssh/config&lt;/tt&gt; file to make accessing the virtual machine a breeze:&lt;br /&gt;&lt;br /&gt;&lt;tt&gt;Host myvm&lt;/tt&gt;&lt;br /&gt;&lt;tt&gt;Hostname localhost&lt;/tt&gt;&lt;br /&gt;&lt;tt&gt;Port 2022&lt;/tt&gt;&lt;br /&gt;&lt;br /&gt;And later:&lt;br /&gt;&lt;br /&gt;&lt;tt&gt;$ ssh myvm&lt;/tt&gt;&lt;br /&gt;&lt;br /&gt;It couldn't be easier. This trivial feature is a killer for the kind of environment I am interested in. And I also know that VirtualBox provides some other interesting features, like headless mode, that are not (easily) available with Fusion.&lt;br /&gt;&lt;br /&gt;So yes, I think I am a convert.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-6942172427809398844?l=blog.julipedia.org' alt='' /&gt;&lt;/div&gt;</content>

19 Sep 2011 10:48pm GMT

18 Sep 2011

feedBSD Planet * BSD People

The Julipedia (Blog): Kyua: Weekly status report (db designdoc edition!)

&lt;ul&gt;&lt;li&gt;Moved the code of &lt;tt&gt;utils::lua&lt;/tt&gt; to a new project, &lt;a href="http://code.google.com/p/lutok/"&gt;Lutok&lt;/a&gt;.&lt;/li&gt;&lt;li&gt;Attempted to integrate a copy of Lutok into the Kyua source code to simplify installing Kyua. I have been playing with Subversion externals and Autoconf/Automake hacks to make this happen, but unfortunately haven't got a pleasant solution yet.&lt;/li&gt;&lt;li&gt;Modified Lutok to not expose the Lua C API at all from header files and cleaned up the Kyua code to cope with the changes.&lt;/li&gt;&lt;li&gt;Been chewing through the first chapters of the "&lt;a href="http://www.amazon.com/Using-SQLite-Jay-Kreibich/dp/0596521189"&gt;Using SQLite&lt;/a&gt;" book to refresh my SQL skills. &lt;/li&gt;&lt;li&gt;And, most importantly, wrote a &lt;a href="http://code.google.com/p/kyua/wiki/DatabaseDesign"&gt;preliminary design document for the database store&lt;/a&gt; of Kyua and the reporting features. Comments certainly welcome! Be aware that this is how &lt;tt&gt;atf-report&lt;/tt&gt; will be replaced, so once this is done we should be able to finally kill &lt;tt&gt;atf-run&lt;/tt&gt; and &lt;tt&gt;atf-report&lt;/tt&gt; altogether :-)&lt;/li&gt;&lt;/ul&gt;&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-7580227422862832024?l=blog.julipedia.org' alt='' /&gt;&lt;/div&gt;</content>

18 Sep 2011 11:48pm GMT

The Julipedia (Blog): Name your C++ auto-cleaners

As you may already know, RAII is a very powerful and popular pattern in the C++ language. With RAII, you can wrap non-stack-managed resources into a stack-managed object such that, when the stack-managed object goes out of scope, it releases the corresponding non-stack-managed object. &lt;a href="http://onlamp.com/pub/a/onlamp/2006/05/04/smart-pointers.html?page=1"&gt;Smart pointers&lt;/a&gt; are just one example of this technique, but so are IO streams too.&lt;br /&gt;&lt;br /&gt;Before getting into the point of the article, bear with me for a second while I explain what the&amp;nbsp; &lt;a href="http://code.google.com/p/lutok/source/browse/trunk/stack_cleaner.hpp"&gt;&lt;tt&gt;stack_cleaner&lt;/tt&gt;&lt;/a&gt; object of &lt;a href="http://code.google.com/p/lutok/"&gt;Lutok&lt;/a&gt; is. The "stack cleaner" takes a reference to a Lua state and records the height of the Lua stack on creation. When the object is destroyed (which happens when the declaring function exits), the stack is returned to its previous height thus ensuring it is clean. It is always a good idea for a function to prevent side-effects by leaving its outside world as it was - and, like it or not, the Lua state is part of the outside world because it is an input/output parameter to many functions.&lt;br /&gt;&lt;br /&gt;Let's consider a piece of code &lt;i&gt;without&lt;/i&gt; using the stack cleaner:&lt;br /&gt;&lt;br /&gt;&lt;pre&gt;void&lt;br /&gt;my_function(lutok::state&amp;amp; state, const int foo)&lt;br /&gt;{&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; state.push_integer(foo);&lt;/pre&gt;&lt;pre&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; ... do something else in the state ...&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; const int bar = state.to_integer();&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; if (bar != 3) {&lt;br /&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; state.pop(1); &lt;br /&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; throw std::runtime_error("Invalid data!");&lt;br /&gt;&amp;nbsp; &amp;nbsp; }&lt;br /&gt;&amp;nbsp; &amp;nbsp; state.pop(1);&lt;br /&gt;&lt;br /&gt;}&lt;/pre&gt;&lt;br /&gt;Note that we have had to call &lt;tt&gt;state.pop(1)&lt;/tt&gt; from "all" exit points of the function to ensure that the stack is left unmodified upon return of &lt;tt&gt;my_function&lt;/tt&gt;. Also note that "all exit points" may not be accurate: in a language that supports exceptions, any statement may potentially raise an exception so to be really safe we should do:&lt;br /&gt;&lt;br /&gt;&lt;pre&gt;void&lt;br /&gt;my_function(lutok::state&amp;amp; state, const int foo)&lt;br /&gt;{&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; state.push_integer(foo);&lt;br /&gt;&amp;nbsp; &amp;nbsp; try { &lt;br /&gt;&amp;nbsp;&amp;nbsp; &amp;nbsp; &amp;nbsp;&amp;nbsp; ... do something else in the state ...&lt;br /&gt;&amp;nbsp;&amp;nbsp; &amp;nbsp; &amp;nbsp;&amp;nbsp; const int bar = state.to_integer();&lt;br /&gt;&amp;nbsp;&amp;nbsp; &amp;nbsp; &amp;nbsp;&amp;nbsp; if (bar != 3 &lt;br /&gt;&amp;nbsp; &amp;nbsp;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; throw std::runtime_error("Invalid data!");&lt;br /&gt;&amp;nbsp; &amp;nbsp; } catch (...) {&lt;br /&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; state.pop(1);&lt;br /&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; throw;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; }&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; state.pop(1);&lt;br /&gt;}&lt;/pre&gt;&lt;br /&gt;... which gets old very quickly. Writing this kind of code is error-prone and boring.&lt;br /&gt;&lt;br /&gt;With an "auto-cleaner" object such as the &lt;tt&gt;stack_cleaner&lt;/tt&gt;, we can simplify our code like this:&lt;br /&gt;&lt;br /&gt;&lt;pre&gt;void&lt;br /&gt;my_function(lutok::state&amp;amp; state, const int foo)&lt;br /&gt;{&lt;br /&gt;&amp;nbsp; &amp;nbsp; lutok::stack_cleaner cleaner(state);&lt;br /&gt;&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; state.push_integer(foo);&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; ... do something else in the state ...&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; const int bar = state.to_integer();&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; if (bar != 3) &lt;br /&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; throw std::runtime_error("Invalid data!"); &lt;br /&gt;}&lt;/pre&gt;&lt;br /&gt;And we leave the boring task of determining when to actually call &lt;tt&gt;state.pop(1)&lt;/tt&gt; to the compiler and the runtime environment. In this particular case, no matter how the &lt;tt&gt;my_function&lt;/tt&gt; terminates, we ensure that the Lua stack will be left as the same size as it was before.&lt;br /&gt;&lt;br /&gt;But, as I said earlier, all this was just an introduction to the idea that made me write this post.&lt;br /&gt;&lt;br /&gt;When you declare an auto-cleaner object of any kind, &lt;i&gt;be sure to give it a name&lt;/i&gt;. It has happened to me a few times already that I have written the following construct:&lt;br /&gt;&lt;br /&gt;&lt;pre&gt;lutok::stack_cleaner(state);&lt;/pre&gt;&lt;br /&gt;... which is syntactically correct, harmless and "looks good" if you don't look closely. The compiler will chew along just fine because, even though we are declaring an anonymous object, its constructor and destructor may be doing who-knows-what, so their code must be called and thus the "unused variable" warning cannot really be raised.&lt;br /&gt;&lt;br /&gt;&lt;b&gt;However&lt;/b&gt; this does not give us the desired behavior. The cleaner object will be constructed and destructed in the same statement without having a chance to wrap any of the following code, because its scope is just the statement in which it was defined. In other words, the cleaner will have absolutely no effect on the rest of the function and thus will be useless.&lt;br /&gt;&lt;br /&gt;So, moral of the story: always give a name to your auto-cleaner objects so that their scope is correctly defined and their destructor is run when you actually expect:&lt;br /&gt;&lt;br /&gt;&lt;pre&gt;lutok::stack_cleaner ANY_NAME_HERE(state);&lt;/pre&gt;&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-7544613249446930628?l=blog.julipedia.org' alt='' /&gt;&lt;/div&gt;</content>

18 Sep 2011 3:48am GMT

15 Sep 2011

feedBSD Planet * BSD People

The Julipedia (Blog): Introducing Lutok: A lightweight C++ API for Lua

It has finally happened. &lt;a href="http://code.google.com/p/lutok/"&gt;Lutok&lt;/a&gt; is the result of what was promised in the "&lt;a href="http://blog.julipedia.org/2011/09/splitting-utilslua-from-kyua.html"&gt;Splitting utils::lua from Kyua&lt;/a&gt;" web post.&lt;br /&gt;&lt;br /&gt;Quoting the project web page:&lt;br /&gt;&lt;blockquote&gt;Lutok provides thin C++ wrappers around the Lua C API to ease the interaction between C++ and Lua. These wrappers make intensive use of RAII to prevent resource leakage, expose C++-friendly data types, report errors by means of exceptions and ensure that the Lua stack is always left untouched in the face of errors. The library also provides a small subset of miscellaneous utility functions built on top of the wrappers.&lt;br /&gt;&lt;br /&gt;Lutok focuses on providing a clean and safe C++ interface; the drawback is that it is not suitable for performance-critical environments. In order to implement error-safe C++ wrappers on top of a Lua C binary library, Lutok adds several layers or abstraction and error checking that go against the original spirit of the Lua C API and thus degrade performance.&lt;br /&gt;&lt;br /&gt;Lutok was originally developed within Kyua but was later split into its own project to make it available to general developers.&lt;/blockquote&gt;Coming up with a name for this project was quite an odyssey, and is what has delayed is release more than I wanted. My original candidate was "luawrap" which, although not very original, was to-the-point and easy to understand. Unfortunately, that name did not clear with the legal department and I had to propose several other names, some of which were not acceptable either. Eventually, I settled with "Lutok", which comes from "LUa TOolKit".&lt;br /&gt;&lt;br /&gt;At this point, the source tree of Lutok provides pretty much the same code as the &lt;tt&gt;utils::lua&lt;/tt&gt; module of Kyua. While it may be enough to get you started, I'm pretty sure you will lack some functions in the &lt;tt&gt;state&lt;/tt&gt; class. If that is the case, don't hesitate to file a bug report to let me know what is missing.&lt;br /&gt;&lt;br /&gt;In case you missed the link above, the project page is here: &lt;a href="http://code.google.com/p/lutok/"&gt;Lutok in Google Code&lt;/a&gt;. &lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-7627076009086318140?l=blog.julipedia.org' alt='' /&gt;&lt;/div&gt;</content>

15 Sep 2011 7:48pm GMT

13 Sep 2011

feedBSD Planet * BSD People

The Julipedia (Blog): Using va_copy to safely pass ap arguments around

A long time ago, while I was preparing an ATF release, I faced many failing tests and crashes in one of the platforms under test. My memory told me this was a problem in OpenSolaris, but the &lt;a href="http://www.blogger.com/mtn-host.prjek.net/viewmtn/atf/revision/info/655c6c51f2155076f482a038042c67fd25adc934"&gt;repository logs&lt;/a&gt; say that the problem really happened in Fedora 8 x86_64.&lt;br /&gt;The problem manifested itself as segmentation faults pretty much everywhere, and I could trace such crashes down to pieces of code like the following, of which the C code of ATF is full of:&lt;br /&gt;&lt;pre&gt;void&lt;br /&gt;foo_fmt(const char *fmt, ...)&lt;br /&gt;{&lt;br /&gt; va_list ap;&lt;br /&gt;&lt;br /&gt; va_start(ap, fmt);&lt;br /&gt; foo_ap(fmt, ap);&lt;br /&gt; va_end(ap);&lt;br /&gt;}&lt;br /&gt;&lt;br /&gt;void&lt;br /&gt;foo_ap(const char *fmt, va_list ap)&lt;br /&gt;{&lt;br /&gt; char buf[128];&lt;br /&gt;&lt;br /&gt; vsnprintf(buf, sizeof(buf), fmt, ap);&lt;br /&gt;&lt;br /&gt; ... now, do something with buf ...&lt;br /&gt;}&lt;/pre&gt;The codebase of ATF provides &lt;tt&gt;_fmt&lt;/tt&gt; and &lt;tt&gt;_ap&lt;/tt&gt; variants for many functions to give more flexibility to the caller and, as shown above, the &lt;tt&gt;_fmt&lt;/tt&gt; variant just relies on the &lt;tt&gt;_ap&lt;/tt&gt; variant to do the real work.&lt;br /&gt;Now, the crashes that appeared from the code above seemed to come from the call that consumes the &lt;tt&gt;ap&lt;/tt&gt; argument, which in this case is &lt;tt&gt;vsnprintf&lt;/tt&gt;. Interestingly, though, all the tests in other platforms but Linux x86_64 worked just fine, and this included OpenSolaris, other Linux distributions, some BSDs and even different hardware platforms.&lt;br /&gt;As it turned out, you &lt;b&gt;cannot blindly pass &lt;tt&gt;ap&lt;/tt&gt; arguments around&lt;/b&gt; because they are not "normal" parameters (even though, unfortunately, they look like so!). In most platforms, the &lt;tt&gt;ap&lt;/tt&gt; element will be just an "absolute" pointer to the stack, so passing the variable to an inner function calls is fine because the caller's stack has not been destroyed yet and, therefore, the pointer is still valid. But... the &lt;tt&gt;ap&lt;/tt&gt; argument can have other representations. It'd be an offset to the stack instead of a pointer, or it'd be a data structure that holds all the variable parameters. If, for example, the &lt;tt&gt;ap&lt;/tt&gt; argument held an offset, passing it to an inner function call would make such offset point to "garbage" because the stack would have been grown due to the new call frame. (I haven't investigated what specific representation is x86_64 using.)&lt;br /&gt;The solution is to use the &lt;tt&gt;va_copy&lt;/tt&gt; function to generate a new &lt;tt&gt;ap&lt;/tt&gt; object that is valid for the current stack frame. This is easy, so as an example, we have to rewrite the &lt;tt&gt;foo_ap&lt;/tt&gt; function above as follows:&lt;br /&gt;&lt;pre&gt;void&lt;br /&gt;foo_ap(const char *fmt, va_list ap)&lt;br /&gt;{&lt;br /&gt; char buf[128];&lt;br /&gt; va_list ap2;&lt;br /&gt;&lt;br /&gt; va_copy(ap2, ap);&lt;br /&gt; vsnprintf(buf, sizeof(buf), fmt, ap2);&lt;br /&gt; va_end(ap2);&lt;br /&gt;&lt;br /&gt; ... now, do something with buf ...&lt;br /&gt;}&lt;/pre&gt;This duplication of the &lt;tt&gt;ap&lt;/tt&gt; argument pointing to the variable list of arguments ensures that &lt;tt&gt;ap2&lt;/tt&gt; can be safely used from the new stack frame.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-2547470671733170941?l=blog.julipedia.org' alt='' /&gt;&lt;/div&gt;</content>

13 Sep 2011 2:48am GMT

12 Sep 2011

feedBSD Planet * BSD People

The Julipedia (Blog): Kyua: Weekly status report

&lt;ul&gt;&lt;li&gt;My plan for this week was to release utils::lua as a separate module. Unfortunately, this has not been possible because I've been fighting with legal to clear the name for the project. I don't have an approved name yet, so this will have to wait a bit more :-(&lt;/li&gt;&lt;li&gt;On another order of things, I have started writing a design document for the database that will collect test case results and other information. Will share it as soon as it is readable and more or less complete.&lt;/li&gt;&lt;/ul&gt;&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-2227882804331509588?l=blog.julipedia.org' alt='' /&gt;&lt;/div&gt;</content>

12 Sep 2011 2:48pm GMT

06 Sep 2011

feedBSD Planet * BSD People

The Julipedia (Blog): Diversions in Autoconf (actually, in M4sugar)

Have you ever wondered how Autoconf reorganizes certain parts of your script regardless of the order in which you invoke the macros in your &lt;tt&gt;configure.ac&lt;/tt&gt; script? For example, how come you can define &lt;tt&gt;--with-*&lt;/tt&gt; and &lt;tt&gt;--enable-*&lt;/tt&gt; flags anywhere in your script and these are all magically moved to the option-processing section of the final shell script? After all, Autoconf is just a collection of M4 macros, and a macro preprocessor's only work is to expand macros in the input with predefined output texts. Isn't it?&lt;br /&gt;&lt;br /&gt;Enter &lt;a href="http://www.gnu.org/s/hello/manual/autoconf/Diversion-support.html#Diversion-support"&gt;M4sugar's diversions&lt;/a&gt;. Diversions are a mechanism that allows M4 macros to output code to different text blocks, which are later concatenated in a specific order to form the final script.&lt;br /&gt;&lt;br /&gt;Let's consider an example (based on a few M4 macros to &lt;a href="http://mtn-host.prjek.net/viewmtn/atf/revision/file/7ae5fa10200ec66c26f08c27e4aeaf3facf3f031/atf-c/atf-common.m4"&gt;detect the ATF bindings&lt;/a&gt; from your own configure scripts). Suppose you want to define a macro &lt;tt&gt;FROB_ARG&lt;/tt&gt; to provide a &lt;tt&gt;--with-frob&lt;/tt&gt; argument whose argument must be either "yes" or "no". Also suppose you want to have another macro &lt;tt&gt;FROB_CHECK&lt;/tt&gt; to detect whether &lt;tt&gt;libfrob&lt;/tt&gt; exists. Lastly, you want the user to be able to use these two independently: when &lt;tt&gt;FROB_CHECK&lt;/tt&gt; is used &lt;i&gt;without&lt;/i&gt; invoking &lt;tt&gt;FROB_ARG&lt;/tt&gt; first, you want it to unconditionally look for the library; otherwise, if &lt;tt&gt;FROB_ARG&lt;/tt&gt; has been used, you want to honor its value.&lt;br /&gt;&lt;br /&gt;We could define these macros as follows:&lt;br /&gt;&lt;br /&gt;&lt;pre&gt;AC_DEFUN([FROB_ARG], [&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; AC_ARG_WITH([frob],&lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; &amp;nbsp; &amp;nbsp;&amp;nbsp; &amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; [AS_HELP_STRING([--with-frob=yes|no], [enable frob])],&lt;br /&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; [with_frob=${withval}, [with_frob=yes]) &lt;br /&gt;])&lt;br /&gt;&lt;br /&gt;AC_DEFUN([FROB_CHECK], [&lt;br /&gt;&amp;nbsp; &amp;nbsp; m4_divert_text([DEFAULTS], [with_frob=yes])&lt;br /&gt;&lt;br /&gt;&amp;nbsp; &amp;nbsp; if test "${with_frob}" = yes; then&lt;br /&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; ... code to search for libfrob ... &lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; elif test "${with_frob}" = no; then&lt;br /&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; :&amp;nbsp; # Nothing to do. &lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; else&lt;br /&gt;&amp;nbsp; &amp;nbsp; &amp;nbsp; &amp;nbsp; AC_MSG_ERROR([--with-frob must be yes or not]) &lt;br /&gt;&amp;nbsp;&amp;nbsp;&amp;nbsp; fi&lt;br /&gt;])&lt;/pre&gt;&lt;br /&gt;Note the &lt;tt&gt;m4_divert_text&lt;/tt&gt; call above: this macro invocation tells M4sugar to store the given text (&lt;tt&gt;with_frob=yes&lt;/tt&gt;) in the &lt;tt&gt;DEFAULTS&lt;/tt&gt; diversion. When the script is later generated, this text will appear at the beginning of the script before the command-line options are processed, completely separated from the shell logic that consumes this value later on.&lt;br /&gt;&lt;br /&gt;With this we ensure that the &lt;tt&gt;with_frob&lt;/tt&gt; shell variable is always defined regardless of the call to the &lt;tt&gt;FROB_ARG&lt;/tt&gt; macro. If this macro is called, &lt;tt&gt;with_frob&lt;/tt&gt; will be defined during the processing of the options and will override the value of the variable defined in the &lt;tt&gt;DEFAULTS&lt;/tt&gt; section. However, if the macro has not been called, the variable will keep its default value for the duration of the script.&lt;br /&gt;&lt;br /&gt;Of course, this example is fictitious and could be simplified in other ways. But, as you can see in the &lt;a href="http://mtn-host.prjek.net/viewmtn/atf/revision/file/7ae5fa10200ec66c26f08c27e4aeaf3facf3f031/atf-c/atf-common.m4"&gt;referred change&lt;/a&gt; and in the Autoconf code itself, diversions are extensively used for trickier purposes. In fact, Autoconf uses diversions to topologically sort macro dependencies in your script and output them in a specific order to satisfy cross-dependencies.&lt;br /&gt;&lt;br /&gt;Isn't that cool?&amp;nbsp; I can't cease to be amazed, but I also don't dare to look at how this works internally for my own sanity...&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-8210083915264683632?l=blog.julipedia.org' alt='' /&gt;&lt;/div&gt;</content>

06 Sep 2011 10:48pm GMT

05 Sep 2011

feedBSD Planet * BSD People

The Julipedia (Blog): Introducing "Dirt and the City"

A friend of mine recently started a blog called &lt;a href="http://dirtandthecity.blogspot.com/"&gt;Dirt and the City&lt;/a&gt; and, because he has invited me to contribute to it, I feel like posting a reference here ;-)&lt;br /&gt;&lt;br /&gt;Dirt and the City is a blog that intends to show you the "ugly" (literally) side of NYC. The blog is named after a famous TV show filmed in this same city; I haven't watched the show myself, but I bet that there is no dirt to be seen in it. If you are wondering what the city really looks like, this blog is your place.&lt;br /&gt;&lt;br /&gt;But, before visiting the blog, be sure to take it as it is meant to be: just a fun collection of photos! If you ever want to visit NYC, don't let the contents of the blog change your plans. You will definitely enjoy your visit regardless.&lt;br /&gt;&lt;br /&gt;Without further ado, and just in case you missed the link above, &lt;a href="http://dirtandthecity.blogspot.com/"&gt;click here to visit Dirt and the City&lt;/a&gt;.&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-1340557741979574423?l=blog.julipedia.org' alt='' /&gt;&lt;/div&gt;</content>

05 Sep 2011 12:48am GMT