fidzu : Solaris

I have been really slacking off with my blogging and really need to get back into it. What better way than with something amusing that happened today to a colleague.

He picked up an task today where the customer had the following issue and question. Unfortunately I don't recall the exact issue and patch number but they are perepheral to the humour.

We've noticed that this problem occurs on a system with patch XXXXXX-02, but not on those with XXXXXX-04. Can you tell us if there is a patch or workaround to the problem?

How do you answer a question like that? An overseas colleague came up with the suggestion of "Sir, you really need a holiday".

My suggestion was more prosaic, that he simply look in the patch README for the bugs that were fixed between the -02 and -04 revision and reply "Yes, that was bug YYYYYYY which was fixed in XXXXXX-04", and try to keep a straight face.

Of course the really worrying thing about this whole incident, is that the current revision of the patch in question was -57!

23 Jul 2008 10:42am GMT

In my previous post about the IP Provider I got the following comment: "There is nothing unpleasant about the wonderfulness that is tcpdump! You'll need to put a lot of work in to match tcpdump's usefulness with Dtrace…"

That just sounds like a challenge. Bring it on! Can snoop or tcpdump do this?

root@ultra ~$ ./ip_whosent.d 
Packet sent to 192.168.100.4: 88 byte packet on behalf of ssh (PID: 1075)
Packet sent to 192.168.100.4: 88 byte packet on behalf of ssh (PID: 1075)
Packet sent to 208.67.222.222: 56 byte packet on behalf of nscd (PID: 152)
Packet sent to 208.67.222.222: 71 byte packet on behalf of nscd (PID: 152)
Packet sent to 208.67.222.222: 56 byte packet on behalf of nscd (PID: 152)
Packet sent to 72.14.207.99: 52 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 8.12.32.9: 52 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 54 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 87 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 58 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 64 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 8.12.32.9: 65 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 208.67.219.230: 644 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 208.67.219.230: 637 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 72.14.207.99: 660 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 208.67.219.230: 52 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 208.67.219.230: 664 byte packet on behalf of firefox-bin (PID: 1944)
Packet sent to 8.12.32.9: 48 byte packet on behalf of thunderbird-bin (PID: 1133)
Packet sent to 72.14.207.99: 40 byte packet on behalf of firefox-bin (PID: 1944)
^C

Here is the script:

#!/usr/sbin/dtrace -qs 



ip:ip:*:send
/execname != "sched"/
{ 
        printf("Packet sent to %s: %d byte packet on behalf of %s (PID: %d)n", 
                        args[2]->ip_daddr, args[4]->ipv4_length, execname, pid ); 
}

Oh but wait....... how about a full call stack on each sent packet? Just add a new line to the above script: stack();

root@ultra ~$ ./ip_sentstack.d 
Packet sent to 72.14.207.99: 84 byte packet on behalf of ping (PID: 2020)

              ip`ip_wput_ire+0x21f5
              ip`ire_send+0x1c9
              ip`ire_add_then_send+0x2b9
              ip`ip_newroute+0xa0a
              ip`ip_output_options+0x18c7
              ip`icmp_wput+0x44a
              unix`putnext+0x22b
              genunix`strput+0x1ad
              genunix`kstrputmsg+0x261
              sockfs`sosend_dgram+0x26e
              sockfs`sotpi_sendmsg+0x4a8
              sockfs`sendit+0x160
              sockfs`sendto+0x8e
              sockfs`sendto32+0x2d
              unix`sys_syscall32+0x101

Or check out one of the examples on the IP Provider wiki page (this is almost certainly by Brendan Gregg):

# ./ipio.d
 CPU  DELTA(us)          SOURCE               DEST      INT  BYTES
   1     598913    10.1.100.123 ->   192.168.10.75  ip.tun0     68
   1         73   192.168.1.108 ->     192.168.5.1     nge0    140
   1      18325   192.168.1.108 <-     192.168.5.1     nge0    140
   1         69    10.1.100.123 <-   192.168.10.75  ip.tun0     68
   0     102921    10.1.100.123 ->   192.168.10.75  ip.tun0     20
   0         79   192.168.1.108 ->     192.168.5.1     nge0     92

Here is the script:

#!/usr/sbin/dtrace -s

#pragma D option quiet
#pragma D option switchrate=10hz

dtrace:::BEGIN
{
        printf(" %3s %10s %15s    %15s %8s %6sn", "CPU", "DELTA(us)",
            "SOURCE", "DEST", "INT", "BYTES");
        last = timestamp;
}

ip:::send
{
        this->elapsed = (timestamp - last) / 1000;
        printf(" %3d %10d %15s -> %15s %8s %6dn", cpu, this->elapsed,
            args[2]->ip_saddr, args[2]->ip_daddr, args[3]->ill_name,
            args[2]->ip_plength);
        last = timestamp;
}

ip:::receive
{
        this->elapsed = (timestamp - last) / 1000;
        printf(" %3d %10d %15s <- %15s %8s %6dn", cpu, this->elapsed,
            args[2]->ip_daddr, args[2]->ip_saddr, args[3]->ill_name,
            args[2]->ip_plength);
        last = timestamp;
}

Can DTrace decrypt IPsec ESP payloads? No. Ok, so tcpdump isn't dead yet, but the capabilities offered by DTrace are far deeper. I've got a ton of ideas more that I could put here, but don't have time atm. DTrace for the win!

23 Jul 2008 9:01am GMT

One of the perennial problems of sponsoring an open source conference is that the organisers always seem to want the sponsorship to pay for an exhibition booth. Exhibition booths need furnishing and decorating. They need things to exhibit. They need staffing. Most of this would be fine at a traditional exhibition, but at an open source conference there aren't many people attending to choose things to buy and thus the sales staff aren't keen to do all the above.

So what should we do with that booth? An approach we first tried at FISL a few years ago was to stop treating it as a selling space and start treating it as a social space. This year at OSCON in Portland we've decided to open up and dedicate our booth to hosting a micro-unconference. We've set it up with whiteboards, tables, electrical outlets and fresh coffee. And if having a place to veg isn't enough, we've invited all comers to deliver lightning talks throughout the two days. There are still a few slots on the agenda if you want to deliver a talk, but the quality of the speakers already listed is high (check out Monty's talk on Maria for example).

By the way, the legendary (or is that "mythical") Sun FOSS Party is back again this year, 8pm in the parking garage at the Doubletree hotel on Wednesday (July 23). Loads of cool diversions and I gather there is plenty more to drink this year than last. All welcome.

23 Jul 2008 5:54am GMT

Solaris Application Programming author Darryl Grove will be in Japan on Friday to present at the Solaris Night Seminar in Tokyo. Hisayoshi Kato will also present. Key topic is DTrace.

23 Jul 2008 12:02am GMT

In an effort to gain visibility in a world of information overload, the media invented the "ALERT". However, there seems to be no consensus as to what an "ALERT" means. The result is "The Boy Who Cried Wolf" syndrome, where an ALERT more often than not adds to the noise.

I made the mistake of subscribing to ZDNet mailing Open Source ALERT mailing list. Note to ZDNet, a post by Dana Blankenhorn is not worthy of an alert. That is not to discredit Dana whatsoever. I subscribe to Dana's blog. However, a daily blog post is not ALERT-worthy. I unsubscribed. ZDNet lost a communication channel to me, their customer.

Note that cable news is not immune, where they spend endless hours of analyzing a single topic. An ALERT is intended to notify the viewer of something eventful, although it rarely does so.

My thoughts on good vs bad alerts.

Good alerts - including mandatory Yahoo, Microsoft, Google, Apple references:

• Yahoo buys Microsoft
  Microsoft buying Yahoo lost its alert-ability months ago
  
• Google Trends shows googling Google on the decline
  Not that it will ever happen
  
• Microsoft places [anything significant] under an open source license
• Apple opens up iPhone to developers, and AT&T Unlimited data plan is really unlimited, enabling SlingBox owners to Sling.
  GlassFish & MySQL Unlimited really is Unlimited.
• Tsunami. Earthquake.

Bad alerts:

• Microsoft buys Yahoo
  See "endless hours of analyzing a single topic" above
• new iPhone model released
  "... endless hours ..." yada, yada, yada
• Anything about Britany Spears
  Alerts should be newsworthy
  
• Alien Invasion
  They're already here
  

22 Jul 2008 11:19pm GMT

No, it's not my phrase, but i think it resumes the sysadmin's work… Actually that phrase is a citation from Seu Jorge, talking about a theater school where the motto was the title of this post. Seu Jorge is a singer, song writer, actor, and soundtrack composer. You can see him in "City of God" (wonderful movie), and in many Jazz festivals around the world.
peace.

22 Jul 2008 8:16pm GMT

Inside T. Boone Pickens' Brain: "I'd rather surround myself with sharp young minds than play golf and gin rummy all day." -- T. Boone Pickins. That attitude isn't just talk from a cocky oil billionaire. It just may be a critical component to staying young as you grow old. Very interesting article on brain research, and specifically about how this guy thinks.

22 Jul 2008 3:49pm GMT

NOTE: Phillip "Flip" Russell is no longer working at Sun, and has helped
for the past few years with SVOSUG. I would like to thank Phillip for
helping out and hope that you'll keep in touch with us.

I had been planning to host the meeting last week when Lori Alt was
visiting, but Lori was busy on Thurs., but we did tape a video for ZFS
Boot that can be seen on John Weeks' ustream account:

(first 10 minutes have some white noise in the audio)

http://www.ustream.tv/recorded/563014

I would also like the let folks know about some dates coming up.

The August meeting will be moved up to PenLUG in Redwood City, please join
along in meeing with PenLUG. This will be on the same night SVOSUG
normally meets on, 4th Thurs., or August 28th.

http://www.penlug.org/twiki/bin/view/Home/MeetingAgenda20080828

Sept. 3rd, James Gosling is speaking at the Silicon Valley Linux User's
Group. Here's a good chance to hear James speak at a small local venue.

http://www.svlug.org/meetings.php

Sept. SVOSUG meeting will be back at the Mansion, and I have a tentative
speaker but don't have it confirmed yet. Stay tuned for this announcement
soon.

Oct. SVOSUG meeting is being planned as a Arduino build-a-thon. The parts
to build an Arduino are only about $10-$15. We need to have a cross
compiler for OpenSolaris (gcc-avr) to support the Arduino, which is a full
open hardware and software platform to develop on. John Plocher recently
wrote a couple snazzy programs on the Arduino, one uses 2 servos with a
web cam to allow panning of the webcam with 2 potentiometers. This was
done with about 25 lines of c code, compiled and uploaded to the Arduino.
He also wrote a musical type program. Yes, John has discovered the
Arudino...and so can you...stay tuned for more info, we'll help you build
one for youself out of parts. This will be based on the Freeduino, bare
bones board. We will be inviting the local LUGs and BayLISA to join us.

http://www.moderndevice.com/

Have a nice summer break, I'm hoping for one myself.

22 Jul 2008 12:40pm GMT

Michelle posted the new OpenSolaris Trademark Policy. Nice to see this document out there. The FAQ has been updated too. Trademark discussions take place on trademark-policy-dev.

22 Jul 2008 12:30pm GMT

Arrgh, so far behind it seems. Anyway, the latest Firefox builds can be found at:

What's new:

http://www.mozilla.com/en-US/firefox/3.0.1/releasenotes/#whatsnew

Download

http://www.mozilla.com/en-US/firefox/3.0.1/releasenotes/#contributedbuilds

http://releases.mozilla.org/pub/mozilla.org/firefox/releases/3.0.1/contrib/

22 Jul 2008 6:31am GMT

Recently introduced (snv_92) is the first piece of the DTrace Network Providers, the DTrace IP Provider. Here is a taste:

root@ultra include$ dtrace -qn 'ip:ip:*:receive{ printf("Packet recieved from %s: %d byte packetn", args[2]->ip_saddr, args[4]->ipv4_length ); }'
Packet recieved from 74.125.15.85: 40 byte packet
Packet recieved from 74.125.15.85: 40 byte packet
Packet recieved from 8.11.47.20: 88 byte packet
Packet recieved from 8.11.47.20: 216 byte packet
Packet recieved from 8.11.47.20: 200 byte packet
Packet recieved from 8.11.47.20: 136 byte packet
Packet recieved from 8.11.47.20: 104 byte packet
^C

Pretty soon snoop and tcpdump will be nothing more than unpleasant memories. :)

A big thank you to the DTrace Team!!!

22 Jul 2008 1:43am GMT

In a previous posting I introduced the use of direct bindings within the OSNet consolidation. A comment to this posting questioned the difference between the two options -z direct and -B direct, and pointed out that runtime errors can occur during process execution if a lazy dependency (typically enabled with -B direct) can not be found. In this entry, I'll discuss the difference between the -z direct and -B direct options, and offer a useful technique for handling the case where the lazy dependency is not present at runtime.

First, the difference between -z direct and -B direct. A full discussion of these options can be found in the Direct Binding Appendix of the Linker and Libraries Guide. Aside from lazy loading being enabled by -B direct, the essential difference between these options is a trade off between ease of use, and of control. -B direct can be specified anywhere on the command-line, and results in any external and internal symbol bindings being established as direct. This means that if libX.so defines xy() and references xy(), then a direct binding will be established within the same object.

    % cc -G -o libxy.so xy.c -Bdirect -Kpic
    % elfdump -y libxy.so | fgrep xy
          [7]  DB          <self>             xy

Hence, -B direct is a blunt club that hits everything. In contrast, -z direct is sensitive to its position in the command line, and can therefore be used in a more precise manner. Only external references that are resolved to dependencies that follow -z direct are established as direct. In the following example, only the references to libX.so and libY.so will have direct bindings established.

    % cc -o libxy.so xy.c -lA -lB -z direct -lX -lY

But the real question is why would you use one option over the other? -B direct is recommended where possible, due to its simplicity and ease of use. However, there are cases where finer grained control is needed, and -z direct is more appropriate. One example is libproc. This library contains many routines that users (typically debugging tools) wish to interpose upon. We want libproc to have direct bindings to any of the dependencies it requires (libc, libelf, etc.), but we do not wish libproc to directly bind to itself. Therefore, but using -z direct we can build libproc to bind directly to its own dependencies while freely binding to any interposers, for any of the interfaces libproc defines. This interposition is provided regardless of the interposers being explicitly defined (a requirement as we do not have control over all the consumers of libproc). Note, we even went a little bit further, and defined all the libproc interfaces as NODIRECT, which prevents any direct binding to libproc. This was to prevent any dependencies binding to libproc instead of to an interposer.

The comment to my previous blog entry also raised the issue of how lazy loading can be compromised if a lazy dependency can not be found. Typically, lazy loading is used to locate dependencies that are expected to exist. Historically, interfaces like dlopen(3c) have been used to test for the occurrence of dependencies that might not exist. However, a useful technique is to use lazy loading and test for the existence of a dependency with dlsym(3c). By testing for the existence of a known interface with a lazy dependency you can verify the dependency exists and then feel free to call any other interface within that dependency.

When a dependency is bound to, the SONAME of that dependency is recorded in the caller.

    % cc -G -o libxy.so -hlibxy.so xy.c -Kpic
    % elfdump -d libxy.so | fgrep SONAME
         [2]  SONAME            0x1                 libxy.so

    % cc -o main main.c -z lazyload -L. -lxy
    % elfdump -d main | egrep "NEEDED|POSFLAG"
         [0]  POSFLAG_1         0x1                 [ LAZY ]
         [1]  NEEDED            0x163               libxy.so

With this dependency established, you can protect yourself from calling the interfaces within the dependency unless the interface family you are interested in are known to exist.

    if (dlsym(RTLD_PROBE, "symbol-in-libxy-1") {
        /*
         * feel free to call any-and-all interfaces in libxy
         */
        symbol-in-libxy-1();
        symbol-in-libxy-2();
        ....

With this model you don't need to know the name of the object that provides the interfaces, as the name was recorded at link-time. And, the dlsym() will trigger an attempt to load the dependency associated with the symbol. All other references can be made directly through function calls rather than through dlsym(). This allows the compiler, or verification tools like lint, to ensure that you are calling the function with the proper argument and return types, and will therefore lead to safer and more robust code.

The use of dlopen() is still appropriate for selecting between differing objects, or when the caller is not knowledgeable of the dependency, such as the case with plugins. In other cases, the use of lazy loading together with dlsym(), as outlined above, is recommended, as the implementation is usually easier to write, debug and deploy.

---

Technorati Tag: OpenSolaris
Technorati Tag: Solaris

21 Jul 2008 11:04pm GMT

Hacking around with luupgrade on a Nevada SXCE (OpenSolaris) server when /opt is a ZFS filesystem but the root file system is UFS. In this example the file system (mounted at /opt) is called optpool:

YMMV...

comments, corrections, improvements welcome...

---

Disable services that use /opt and any others that make sense.

Backup /opt (e.g. with rsync)

Unmount (zfs unmount) the optpool filesystem

Run luupgrade

Change the mountpoint property (zfs set) of optpool to a temporary mountpoint. And make sure it mounted there, or zfs mount it manually.

Go to that mountpoint and rename directories that need to be moved out of the way, e.g. perhaps SUNWmlib.

Mount the new BE (lumount)

Note that luupgrade, being a bit confused, probably created a new /opt and installed some stuff there. Copy that stuff over to where you temporarily mounted optpool. E.g:

cd /.alt.new_be/opt && /bin/tar -cpf - SUNWmlib | (cd /tmpmountpoint && /bin/tar -xpvf -) # YMMV

Repeat as necessary, some possibilities (i.e. these showed up in /opt in a b64a->b88 luupgrade for example): SUNWtvnc SUNWvgl SUNWjavadb TurboVNC VirtualGL

Now do luactivate and reboot (init 6)

Move (mv) the bogus /opt directory out of the way.

Change the mountpoint property (zfs set) of optpool back to /opt, and make sure it mounted.

init 6 again

Review the state of SMF services, svcs -x, etc. of course.

21 Jul 2008 6:43pm GMT

We were trying to limit the number of regular users who could use xterm or cmdtool for security reasons. A user-defined group was created and admins/non-regulars were assigned to it. Unfortunately, the admins/non-regulars were mysteriously denied xterm/cmdtool execution, which definitely was not the desired effect. Permission denied. That feedback was a bit perplexing because the admins/non-regulars were supposeably assigned to the newly created group per niscat. After a little troubleshooting, it was discovered the newly created group ID matched an existing group's GID in a different name service database. The GID issue was corrected and xterm worked like a champ.

21 Jul 2008 1:08pm GMT

Everyone probably knows this (I'm late in processing my emails) but the Firefox builds for 2.0.0.16 are at

http://www.mozilla.com/en-US/firefox/2.0.0.16/releasenotes/#contributedbuilds

http://releases.mozilla.org/pub/mozilla.org/firefox/releases/2.0.0.16/contrib/

21 Jul 2008 2:52am GMT

I'm in Portland, Oregon this weekend for an interesting meeting, but I'll not be able to stay on for OSCON this year because a change of my role at Sun necessitates attending a meeting in California that's a direct conflict with OSCON. It's not an especially closely-kept secret but I've now moved from Sun's software group and taken the Chief Open Source Officer role over to a newly-formed team reporting more directly to the CEO and working on Sun's relationships with communities globally.

The new team comprises some of Sun's best experts in open standards, open IPR and open source. It's called the Sun Open Technologies Practice, and in particular manages the Sun standards and open source websites. It allows us to take a more holistic approach to Sun's engagement in open standards and open source, especially in the area of influencing open standards bodies to have IPR policies that allow - or even encourage - open source implementation.

I'll let the new members of my team use their own channels to say they have joined, but suffice to say I'm excited by the challenging new opportunities this presents around the world.

20 Jul 2008 5:38pm GMT