fidzu : BSD

Wow, it has been a long time... 5 years ago, I created the &lt;a href="http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/devel/monotone-server/?only_with_tag=MAIN"&gt;monotone-server&lt;/a&gt; package in pkgsrc, a package that provided an interactive script to set up a &lt;a href="http://monotone.ca/"&gt;monotone&lt;/a&gt; server from scratch with, what I though, minimal hassle.&lt;br /&gt;&lt;br /&gt;My package did the job just fine, but past year I was blown away by the simplicity of the same package in &lt;a href="http://www.fedoraproject.org/"&gt;Fedora&lt;/a&gt;: their &lt;tt&gt;init.d&lt;/tt&gt; script provides a set of extra commands to initialize the server before starting it up, and that is it. No need to mess with a separate interactive script; no need to create and memorize passphrases that you will never use; and, what's more, all integrated in the only single place that makes sense: in the init.d "service management" script.&lt;br /&gt;&lt;br /&gt;It has been a while since I became jealous of their approach, but I've finally got to it: I've spent the last few days rewriting the monotone-server package in pkgsrc and came up with a similar scheme. And this &lt;a href="http://mail-index.netbsd.org/pkgsrc-changes/2010/03/12/msg038844.html"&gt;new package&lt;/a&gt; just made its way &lt;a href="http://mail-index.netbsd.org/pkgsrc-changes/2010/03/12/msg038844.html"&gt;&lt;/a&gt;to pkgsrc-HEAD! The new package comes with what I think is a detailed manual page that explains how to configure the server from scratch. Take a look and, if you find any mistakes, inconsistencies or improvements to be done, let me know!&lt;br /&gt;&lt;br /&gt;In the meantime, I will log into my home server, rebuild the updated package and put it in production :-)&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-1599070089952057305?l=blog.julipedia.org' alt='' /&gt;&lt;/div&gt;</content>

12 Mar 2010 9:48am GMT

Despite my time for free software being virtually zero these days, I have managed to implement a prototype of what ATF would look like if it didn't implement forking and isolation in test programs. This feature has been often requested by users to simplify their life when debugging test cases.&lt;div&gt;&lt;div&gt;&lt;br /&gt;&lt;/div&gt;&lt;div&gt;I shouldn't repeat everything I posted on the atf-devel mailing list regarding this announcement, so please &lt;a href="http://mail-index.netbsd.org/atf-devel/2010/03/05/msg000070.html"&gt;refer to that email for details&lt;/a&gt;. But I must say that the results look promising: the overall code of ATF is much simpler and also faster. (An execution I just tried cuts the run time of the ATF test suite from 1m 41s to 1m 16s.) Expect more simplifications and speed-ups!&lt;/div&gt;&lt;/div&gt;&lt;div class="blogger-post-footer"&gt;&lt;img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-1501363386005733392?l=blog.julipedia.org' alt='' /&gt;&lt;/div&gt;</content>

06 Mar 2010 8:48pm GMT

I've come across a couple sudo tricks during the past few days and thought I would pass them along. The first thing you need to know before using sudo is that you should use the visudo command to edit the sudo config file. This script does a number of useful...

28 Feb 2010 6:30pm GMT

I've been thinking of a neat-o project for this year's Google Summer of Code:

Overview of operation

The opencrypto(9) framework exists to coordinate hardware acceleration in NetBSD. Applications of the framework can be inside the kernel like the FAST_IPSEC IPsec implementation, or in userland like OpenSSL with the "cryptodev" engine. Crypto drivers can be realized in software or in hardware. Hardware drivers can be used to instruct e.g. the AMD Geode LX's AES block or a HIFN chip to perform cryptographic operations. Upon system startup, the crypto drivers at the opencrypto(9) framework, telling what operations they can perform. When an operation is required later, the framework will look which crypto device is currently not busy, and offload the operation to that device. Upon completion, the result is fed back to the application.

The following image illustrates the components and their interaction.

Limitations

Offloading the cryptographic requests involves some overhead. Data needs to be transferred to the hardware and back. On systems with a slow CPU, this overhead is relatively small compared to the operation speed of the CPU. On faster CPUs, the overhead becomes more of a burden, making the benefit of the crypto hardware negligible.

As examples, while a hifn(4) chip can provide worthwhile speedups on 500MHz and 1GHz CPUs, no performance win is experienced on a 2.4GHz CPU.

Proposal

The communication overhead involves data transfers over a PCI bus, which is of relatively low speed compared to today's modern CPUs. Preventing the data transfer is a worthwhile goal. In coordination with today's modern multi-core CPUs, using one or more CPUs solely for the purpose of crypto acceleration, a measurable improvement of crypto performance is expected. At the same time, no special hardware requirements beyond the CPU exist. This allows turning standard contemporary systems into fast crypto systems easily.

The following image illustrates the idea of interoperation between a CPU core that runs the kernel and application codes and three cores that are dedicated to crypto code.

Implementation Roadmap

This is where it gets fishy. ;) The existing opencrypto(4) framework probably needs to be make MP-aware at the same time, employing proper use of NetBSD's locking framework. (Already done) The existing swcrypto(4) needs to be adjusted for operation on multiple CPUs at the same time. A way to decide how many CPUs are dedicated to run swcrypto(4) instances. CPUs that run swcrypto(4) need to be taken out from the usual NetBSD CPU scheduling so that they are available exclusively for crypto.

Requirements

In no particular order:

• Know how to build and install a kernel
• Understanding of fine grained SMP and locking
• How to use NetBSD's kernel threads, code-wise
• How to interact with NetBSD's scheduler, code-wise
• Tell the scheduler to pin a specific kernel thread to a specific CPU
• Interaction between applications (IPsec, OpenSSL) with opencrypto(9), code-wise
• Interaction of crypto providers with opencrypto(9), code-wise
• Hardware! You won't be able to do this without at least two CPU cores in your machine. The more the better.
• Benchmarking & a test setup for it

Project Applications

Please follow the NetBSD Project Application/Proposal HowTo if you're serious to work on this project.

If you have any questions let me know, public discussion should be led on the tech-crypto@ list.

22 Feb 2010 10:57pm GMT

My inbox is still overflowing with NetBSD related news, so here is the next chunk for you:

• People owning a Marvell PCMCIA WiFi card can how have a look at the malo(4) driver. If you use this on a gumstix ARM board, have a look as some fiddling with drvctl(8) is needed to get the card recognized properly.
• Speaking about Gumstix, KIYOHARA Takashi has declared the porting effort as finished. Support for the baex, connex, verdex and verdex-pro modules is available, as is support for the support modules.
• Staying in the "embedded" corner, KIYOHARA Takashi has announced that Plathome's OpenBlockS600 (AMCC 405EX) can now boot NetBSD via NFS. See the posting for dmesg output. The OpenBlockS600 comes with a AMCC 405EX PowerPC CPU, two GigE ethernet ports and a bunch of other goodies for a price of about $600US.
• Coming from hardware to software, pkgsrc-2009Q4 was released some time ago, and of course binary packages are available for a number of platforms: 5.0/macppc, 4.0.1/sparc and 5.0.1/sparc, 4.0.1/i386, 5.0.1/i386 and the same for 4.0.1/amd64 5.0.1/amd64. Also, binaries of pkgsrc-2009Q3 are available for 5.0/shark.
• Google's Summer of Code was a big success for NetBSD and all of the Open Source community, and it seems there will be one again this summer.

  Preparations are in an early stage, but there are already a FAQ and a timeline as well as the Program Terms of Service. From the NetBSD side, we're always happy for project suggestions (please use our mailing lists for discussions), and in NetBSD, we are currently working on out projects page. If you plan to submit a proposal for a project with NetBSD, please see our project application/proposal form If you plan to submit a proposal for a project with NetBSD, please see our Project Application/Proposal HowTo.

• Getting back to the NetBSD code, a number of interesting changes were made in the previous weeks. The first to mention is that David Young has continued is work on the new shutdown order for device drivers: ``cgd, dk, dm, md, raid, and vnd gracefully detach from the device tree during shutdown. I believe that ccd is the only virtual disk that does not detach.'' This allows having arbitrary stack of file systems, and still have them unconfigured properly in the right order on system shutdown.
• Another major change that went into NetBSD recently is that terminfo was imported into NetBSD-current. Terminfo replaces termcap, but provides a backward compatible termcap interface. This move follows discussion from last summer, and docs by The Open Group (the people who make things like the POSIX standard and the Single Unix Specification), which indicates that the termcap specification will be withdrawn in the future.
• Moving from userland inside the kernel, David Holland has proceeded with work to unhook LFS from UFS. Historically, the Log structured File System was written after the Berkeley Fast File System. With the idea of sharing the core "Unix File System" code for both file systems, this resulted in a strong relationship between LFS and FFS, which was/is not always the best for the advantage and stability of either one: ``sharing ufs between both ffs and lfs has made all three entities (but particularly lfs) gross. ffs and lfs are not similar enough structurally for this sharing to really be a good design.''
• Another major addition to the NetBSD kernel was made recently by Darren Hunt: ``Courtesy of CoyotePoint Systems, I've been working on a port of DTrace [...] to NetBSD for i386.'' Citing Wikipedia, ``DTrace is a comprehensive dynamic tracing framework created by Sun Microsystems for troubleshooting kernel and application problems on production systems in real time. Originally developed for Solaris, it has since been released under the free Common Development and Distribution License (CDDL) and has been ported to several other Unix-like systems.

  DTrace can be used to get a global overview of a running system, such as the amount of memory, CPU time, filesystem and network resources used by the active processes. It can also provide much more fine-grained information, such as a log of the arguments with which a specific function is being called, or a list of the processes accessing a specific file. ''

  The code is available in NetBSD-current. I haven't looked into this yet, but I'm looking forward of reports and blog postings if the wikipedia command line examples work.

  (I think like with ZFS, Dtrace could use a hand with documenting the NetBSD side of things. Any takers?)

• The last kernel change to mention is related to security: mapping the address 0 from userland was disabled. This issue went through the press late last year, and it this is now addressed in NetBSD, too. Those that still beed to map address 0 can do so via the USER_VA0_DISABLED_DEFAULT kernel option or the vm.user_va0_disable sysctl.
• So much about about the NetBSD code for now. Of course having all those fine features added screams for an immediate (*cough*) release, which brings me to the fact that NetBSD 5.0.2 has been released: `` NetBSD 5.0.2 is the second critical/security update of the NetBSD 5.0 release branch. It represents a selected subset of fixes deemed critical for security or stability reasons.

  Please note that all fixes in critical/security updates (i.e., NetBSD 5.0.1, 5.0.2, etc.) are cumulative, so the latest update contains all such fixes since the corresponding minor release. These fixes will also appear in future minor releases (i.e., NetBSD 5.1, 5.2, etc.), together with other less-critical fixes and feature enhancements. ''

I'm closing for today by pointing to three NetBSD-related events:

• There's a hackathon going this weekend (Feb 20/21 2010)
• Volunteers are wanted to setup & man a NetBSD booth at FrOSCamp 2010 Zurich, Switzerland, on Sep 17/18 2010
• pkgsrcCon 2010 will be held in Basel, Switzerland, from May 28ths to 30ths 2010.

Have fun meeting the gang!

21 Feb 2010 2:20pm GMT

I had an urge to configure PPPoE today, but didn't have a DSL line handy to test it. So, with some help from martin@, I got to prod NetBSD into becoming a PPPoE server.

I've put my notes here.

Open issues:

• Whenever I terminate a client session ("ifconfig pppoe0 down"), the server's pppoe0 interface goes down as well. Before new connections are served, I have to do a manual "ifconfig pppoe0 up".
• When I add "query-dns=3" to the client side, the connection is terminated immediately after it's established. I don't know how to offer DNS servers via PPPoE/LCP, but at least just leaving the connection open instead of terminating it would be nice.

I understand this method is only for one/few PPPoE connections. If more sessions are needed, this may not be the best option. It still served my purpose well. YMMV.

15 Feb 2010 9:56pm GMT

I have been using emacs since the beginning of computer time, way back in the mid-80s. My .emacs file has been carried around for almost as long. Each major rev causes me headaches, as I never know what archaic options are going to break things. And yet, I'm still learning...

15 Feb 2010 3:54pm GMT

I'm currently looking at git to see what it can and cannot do, and one thing I've looked today is how effective the backing store mechanism is. To recall: CVS stores a list of patches between versions in a single file, and git stores each new revision in full in a separate file in the so-called object store. Is that an issue for NetBSD? Let's see;

One of the more frequently updated files is the i386 port's GENERIC kernel config file, which is at revision 1.963 right now. This means that since it's import into CVS, 963 different revisions have been made. In CVS, all those files are kept in a single GENERIC,v file. In git, this puts 963 files on the file system. A bit of a difference.

Looking at the space requirements for storing the repository data itself, the GENERIC,v file is 883,233 bytes[1]. Extracting all 963 versions from revision 1.1 to revision 963 results in disk space usage of 32,805,828 bytes[2,3]. And that's not counting the overhead of 962 inodes and the related directory bookkeeping.

In other words, the git model requires about 37 times the space that CVS does.

Sure the example file is not exactly one with an average number of revisions, and I know that git offers some more efficient storage methods via "pack" files, but investigating those is left as an exercise to the reader. :-)

[1] Obtained via rsync from cvs.netbsd.org:

% ls -la GENERIC,v 
-r--r--r--  1 feyrer  wheel  883233 Feb 12 16:57 GENERIC,v 

[2]

% mkdir extracted
% chdir extracted
% sh -c 'for i in `jot 964`; do echo $i ; co -p -r1.$i ../GENERIC >GENERIC-`printf %04d $i` ; done'

[3]

% cat extracted/* | wc -c
 32805828 

12 Feb 2010 5:26pm GMT

A couple of links posted to my Linux Tumblr last week: Wicked Cheap Hosting - some great web hosting deals, courtesy of All About Linux TermBuilder - a simple web app to build a commandline command...

08 Feb 2010 6:07pm GMT

So early in December, I went on a quest for a new Linux distro. It wasn't so much that I was unhappy with my openSUSE 11.0 installation, but I knew I was probably going to install the new 11.2 version and so I figured I would cast about to see...

07 Feb 2010 3:37pm GMT

Aleksej Saushev writes on a bunch of lists:

We're running 13th Hackathon February 19-22 2010,
come and join us on IRC channel #netbsd-code at FreeNode (irc.freenode.net).
You may choose other ways to participate, if you find it more convenient.
See Hackathon page for updates: http://wiki.netbsd.se/Hackathon13


Goal

Fix as many bugs as possible, close as many PRs as possible.


Details

In previous years we have accumulated a lot of problem reports.
Some of them relate to no longer supported branches (e.g. 2.0)
and were occasionally fixed during regular code work.
Some of them relate to hardware developers don't have access to.
Some of them may be too easy to fix, but noone looks at that part
of code (e.g. documentation).
Some of them relate to packages already removed or heavily reworked.
Some of them relate to packages in exotic environment (Solaris, Interix)
and developers cannot test if the bug is present or not.

You can find more in PR database at http://netbsd.org/support/query-pr.html


More specifically, David Holland maintains categorized lists of PRs at

  http://www.netbsd.org/~dholland/buglists/

of which "Wanted for 5.1" 

(http://www.netbsd.org/~dholland/buglists/51-WANTED.html)
and "Stuck" (http://www.netbsd.org/~dholland/buglists/STUCK.html)
are of particular interest.

Another hot area is pkgsrc 
(http://www.netbsd.org/~dholland/buglists/pkgsrc.html).


IMPORTANT NOTE: You don't have to be programmer to be able to help us,
you can help us with generic testing support.  

Come, bring your favourite booze, and join in!

05 Feb 2010 11:00pm GMT

A few things I have recently thrown down on my Linux Love Tumblr blog, which I use for quick little Linux links and notes: 7 Best Linux Apps for Ripping CDs and DVDs | Maximum PC 50+ Ultimate Collections of Planet Wallpapers TermBuilder: a graphical Linux command line generator...

02 Feb 2010 8:46pm GMT

Drop the following script into /etc/rc.d/alixkitt, put "alixkitt=yes" into /etc/rc.conf and enjoy:

#!/bin/sh
# Turn ALIX2c front LEDs into running LED
# http://www.gifninja.com/Workspace/59f3f22e-5f40-4937-936c-1dc0d6fbe690/output.gif
#
# Copyright (c) 2010 Hubert Feyrer <hubertf@NetBSD.org>

# PROVIDE: alixkitt

$_rc_subr_loaded . /etc/rc.subr

name="alixkitt"
start_cmd="alixkitt_start"
stop_cmd="alixkitt_stop"
pidfile="/var/run/${name}.pid"

alixkitt_sleeptime=1.0          # 0.5

alixkitt_setled()
{
        gpioctl -q -d /dev/gpio 6 $1
        gpioctl -q -d /dev/gpio 25 $2
        gpioctl -q -d /dev/gpio 27 $3

        sleep $alixkitt_sleeptime
}

alixkitt_start()
{
        touch $pidfile
        (
                alixkitt_setled 0 0 0

                while [ -f $pidfile ] ; do
                        alixkitt_setled 0 1 1
                        alixkitt_setled 1 0 1
                        alixkitt_setled 1 1 0
                        alixkitt_setled 1 0 1
                done
        ) &;
        echo $! >$pidfile
        echo Started pid $!
}

alixkitt_stop()
{
        if [ -f /var/run/alixkitt.pid ]; then
                read pid <$pidfile
                echo Stopped pid $pid
                kill $pid
                rm $pidfile
                alixkitt_setled 1 1 1
        fi
        exit 0
}

load_rc_config $name
run_rc_command "$1"

P.S.: There's still a bug left in that stopping immediately re-starts the process. Got a patch? Update: Axel Scheepers suggested doing the while-loop only as long as the PID-file exists. Changed above.

02 Feb 2010 10:18am GMT

Those of us who have multiple hard drives in our computers will inevitably boot up one morning to find the naming scheme for these drives has changed. What was once /dev/sda is now /dev/sdb and vice versa. Your computer won't boot and fsck complains about an uknown or mismatched filesystem...

01 Feb 2010 2:28pm GMT

I've found that the three LEDs at the front of a PC Engines Alix2C board can be toggled on and off via their attachment to the GPIO bus. Here are the commands for this:

31 Jan 2010 11:50pm GMT

Herre are some more things that I've caught in my inbox for too long, and I'm finally finding some time to sum them up here:

• NetBSD's "let's move kernel parts to the userland" RUMP project is still under heavy development, and in order to make testing of compatibility after kernel changes easier, a new command "rumptest" was added to build.sh: ``Basically you say:

              ./build.sh ${yourargs} tools ; ./build.sh ${yourargs} rumptest
          
  

  Where yourargs are what have you, e.g. '-U -u -o -O /objs'.

  The latter builds only the rump kernel libs and uses some ld+awk magic to figure out if things go right or not. This is to avoid having to install headers and build libs (which is too slow since a full build is too slow). The magic is not a substitute for a full build, but it is n+1 times faster and works probably 99.9% of the time.

  The scheme uses a number of predefined component sets (e.g. tmpfs+vfs+rumpkern) to test linkage. They are currently listed in build.sh. This area probably needs some work in the future. It would be nice to autogenerate the combinations somehow.

  If things go well, you get something like this:

          ===> Rump build&link tests successful
          ===> build.sh ended:   Wed Nov 18 20:10:59 EET 2009 
  

  '' See Antti's Antti's mail to tech-kern: on how to tell if things didn't go so well, and what to do in that case.
• According to Wikipedia, ``Non-Uniform Memory Access or Non-Uniform Memory Architecture (NUMA) is a computer memory design used in multiprocessors, where the memory access time depends on the memory location relative to a processor. Under NUMA, a processor can access its own local memory faster than non-local memory, that is, memory local to another processor or memory shared between processors.''

  Supporting NUMA in a contemporary (i.e.: Intel centric) SMP-enabled operating system requires following a bunch of standards, two of which are parsing of two tables, the System Resource Affinity Table (SRAT) and the System Locality Information Table (SLIT). Both tables are accessible via the Advanced Configuration and Power Interface (ACPI), and according to the German-language Wikipedia, the SRAT is used to assign local memory to local threads to boost their performance, and the SLIT defines the "distance" of the nodes among themselves, which is used to determine the "nearest" memory if local memory is not enough.

  Now, Christop Egger has posted patches to add an ACPI SLIT parser and an ACPI SRAT parser. See the two postings for dmesg pr0n from his tests on an 8-node system.

• Staying with ACPI and Christoph Egger, he found that even though the ACPI spec defines an ACPI device for fans, BIOS vendors and OEMs do their own thing. To accommodate things like the fan sensor found in the ACPI Thermal Zone in his HP Pavillion DV9700 laptop he has proposed a driver to extend the acpitz(4) driver with fan information. That way, envstat(8) can be used to display the ran's RPMs:

  [acpitz0]
    Processor Thermal Zone:     56.000   95.000                       degC
                       fan:       2840                                 RPM 
  

• Staying with driver games, iMil writes me that there's documentation on getting DRI, AIGLX, Composite and Compiz going with NetBSD 5.0 available in the O(ther)NetBSD Wiki now.

  The documentation covers how to enable the Direct Rendering Manager (DRI), setting up and configuring Modular X.org, assuring that everything's in place, and how to get Compitz going. Mmm, wobbly windows at last! :-)

• While we're talking funky desktop stuff: Marc Balmer has submitted a patch to get touchpanel support for ums(4). ums(4) is for USB mice, and in contrast to mice, touch panels need to deal with absolute numbers, not relative numbers.
• Back to the guts of the kernel, another patch suggested by Christop Egger was for adding x2apic. What is x2apic? X2APIC is ``an Intel-only feature but can also be found in virtual environments with support for CPU apic id's > 0xff.

  I.e. Xen 4.0 (not yet released) supports 128 CPUs in HVM guests with the CPUs enumerated with even apic id's. That means you need x2apic for the 128th CPU :) ''

• While speaking of Xen: Xen 4.0 is coming soon, and there's a call to help testing it on NetBSD!

  Install Mercurial, check out latest Xen sources, apply a bunch of patches, build and install. Examples of commands are given, in addition to changes required for /boot.cfg etc.

  Report your findings to port-xen!

• Last one for today: Michal Gladecki, Editor-in-Chief of BSD Magazine writes: ``We are happy to announce that BSD Magazine is transforming into a free monthly online publication. The online version of BSD Magazine will stay in the same quality and form. It will look like the BSD magazine one is familiar and comfortable with. Please sign up to our newsletter at www.bsdmag.org and get every issue straight to your inbox. Also, you can now download any of the previous issues from our website. The first online issue -- 2/2010 -- is coming out in February. Please spread the word about BSD Magazine. '' Click!

So much for today. I still have a bunch of news items in my inbox for next time, but let's call it good for today.

Unrelated, I've been playing with git a bit over the past few days, and wile I have a number of questions building up (which will be subject to tech-repository or so), what I can say today is that the speed of "git pull" with NetBSD's git repository and my 1MBit DSL line reminds me a lot of the times when I used SUP with my 56k modem - it took forever, too. :-(

31 Jan 2010 4:54pm GMT