fidzu : Linux

As we come to the end of Akkana Peck's excellent series on mastering GRUB2, it's not clear what advantages it has over legacy GRUB, or even good old LILO. It seems it's gone backwards. In today's installment we learn how to translate some common and mysterious error messages, and how to manage a multi-boot system with GRUB2.

01 Jan 1970 12:00am GMT

Back in 2007 we reported on Valve looking for a senior software engineer to port their Windows-base games to Linux, then in 2008 we said the Source Engine would be coming to Linux based upon our sources (something that we still believe in), later that year we also found a few Linux libraries with the Left 4 Dead game.

01 Jan 1970 12:00am GMT

CloudLinux Inc., a software company dedicated to serving the needs of hosting service providers, has announced that VPS.NET, a leader in the cloud hosting arena, will begin to offer CloudLinux as one of its standard Linux Operating Systems (OS) on all of its VPS cloud offerings. VPS.NET is an elastic cloud hosting provider offering instant scalable Virtual Private Server (VPS) solutions. By leveraging its Lightweight Virtual Environment (LVE) technology, the CloudLinux OS allows VPS.NET to deliver increased performance and flexibility to its suite of secure cloud service offerings.

01 Jan 1970 12:00am GMT

Few people understand Microsoft better than Tandy Trower, who worked at the company from 1981-2009. Trower was the product manager who ultimately shipped Windows 1.0, an endeavour that some advised him was a path toward a ruined career. Four product managers had already tried and failed to ship Windows before him, and he initially thought that he was being assigned an impossible task. In this follow-up to yesterday's story on the future of Windows, Trower recounts the inside story of his experience in transforming Windows from vaporware into a product that has left an unmistakable imprint on the world, 25 years after it was first released.

[Not Linux or FOSS related, but interesting history I think - Tracyanne]

01 Jan 1970 12:00am GMT

For a few years I have been trying to create a "distributed cluster storage system" (see below) on standard Linux hardware. I have been unsuccessful. I have looked into buying one and they do exist, but are so expensive I can't afford one. They also are designed for much larger enterprises and have tons of features I don't want or need. I am hoping the Linux community can help me create this low cost "distributed cluster storage system" which I think other small businesses could use. Please help me solve this so we can publish the solution to the open source community.

01 Jan 1970 12:00am GMT

The major goal of the regional server is to distribute the results of all service checks to the central server. The regional Nagios machines must use OCSP and OCHP in order to send the proper updates to the central server. This tutorial shows you how to set one up.

01 Jan 1970 12:00am GMT

Mini is designed to work on less-capable phones with smaller screens, slower network connections, and less-sophisticated user interface abilities. Version 5 offers support for features including Speed Dial, which is a grid of favorite Web pages, compression through Opera's servers to speed download of Web pages that aren't tailored for mobile phones, and tabbed browsing.

01 Jan 1970 12:00am GMT

Here's an interesting story: Microsoft deciding to release some code as open source for all the right reasons (although there's still a fly in the ointment...).

01 Jan 1970 12:00am GMT

The Linux Professional Institute (LPI), the world's premier Linux certification organization, announced that its affiliate organization LPI-Spain ) had partnered with Proyecto Universidad Empresa (PUE) to promote LPI certification and training within Spain's public schools. PUE is Spain's leading agency in the development of IT training and certification and provides academic programs for such major IT organizations as Microsoft, Cisco and Sun.

01 Jan 1970 12:00am GMT

Late last year we reviewed the Jetway NC92 Atom IPC motherboard that was a nice Mini ITX board with an Intel Atom N230 processor. A few weeks after that, at the Consumer Electronics Show in Las Vegas, the new Pineview processors were shown off. The Atom N400 and D510 Pineview CPUs are only a moderate upgrade from the very common Diamondville Atoms, but the newer Atoms are beginning to work their way into more nettops and netbooks. Jetway is one of the vendors that was quick to design a new IPC motherboard that bears the Intel Atom D510 dual-core processor with Intel GMA 3150 graphics and the NM10 Express Chipset.

01 Jan 1970 12:00am GMT

Mandriva began life in July 1998 as Linux Mandrake in France in Gael Duval's bedroom after he ported a KDE 1.0 desktop onto Red Hat Linux 5.1, uploaded the result onto two FTP servers, went away on holiday, and came back to find that he had a popular and successful Linux distribution on his hands...

01 Jan 1970 12:00am GMT

Ex-Sun open source veteran Simon Phipps has been elected to the board of directors at the Open Source Initiative (OSI) group. Phipps confirmed in a blog post today that he would start his new role on 1 April. On Monday he quit Oracle-owned Sun, where - prior to Larry Ellison's acquisition of the database firm - he had worked for nearly a decade. His resignation marked the latest in a long line of big guns at Sun who have left the firm following Oracle's $7bn takeover.

01 Jan 1970 12:00am GMT

The inability to play the latest off the shelf commercial games has been a thorn in the side of Linux for a long time. With companies such as Valve starting to embrace other platforms, will that be the catalyst Linux needs to become a first class citizen?

01 Jan 1970 12:00am GMT

Part of the repositioning of the KDE brands was choosing an appropriate 'KDE Software Label' for developers working on applications outside the main KDE Software Compilation. Technology developed by KDE is used far and wide, as can be witnessed on kde-apps.org and other sites. Some of these applications are developed by people closely entrenched in the KDE community, others by developers who just happened to like KDE technology and don't feel part of the KDE community in any big way. To allow authors to express their connection to the KDE community, the KDE community has chosen three appropriate labels. We are also presenting the first draft of a guide for distributors of KDE software on how to integrate their communications with our brands.

01 Jan 1970 12:00am GMT

Like we told you a few days ago, Banshee 1.5.5 (aka 1.6 RC1) got a very nice new YouTube extension but the new version also brings some other new features like Grid View and Gapeless Playback:

01 Jan 1970 12:00am GMT

Here is an account of my experience receiving training from Verizon on my Droid phone. Generally speaking, the training session went very well, but scheduling the training was a real annoyance.

01 Jan 1970 12:00am GMT

Hardware Central: "Advanced Micro Devices plans to release a processor in its "Fusion" line that will be positioned for the netbook market, putting it in competition with the Intel Atom, and, to a lesser degree, the ARM processor."

12 Mar 2010 9:02am GMT

Softpedia: "Canonical, the company behind the very popular Ubuntu Linux distribution, recognizes the potential of the emerging market and has partnered with Intel and Eucalyptus to promote its Ubuntu Enterprise Cloud (UEC) offering as part of Intel's Cloud Builder program."

12 Mar 2010 7:02am GMT

THIS IS A PROOF OF CONCEPT - its not going to be upstream unless someone else dedicates their life to it, (btw anyone know anyone in ASUS?)

So NVIDIA unveiled their optimus GPU selection solution for Windows 7, so I decided to see what it would take to implement something similar under DRI. I've named it PRIME for obvious reasons.

Goals:
1. Allow a second GPU to render 3D apps onto the screen of the first, pickable from the client side.
2. Just target the rendering side, I'm assuming the GPU power up/down is similiar to what was done for the older switching method.

Restrictions + limitations:
1. Must have compositing manager running
2. Must have second screen configured for slave card (doesn't need to be used)

Test system:
Intel 945 IGP + radeon r200 PCI card - yes this won't be a speed demon.

Terms:
Master: the IGP displaying the output - intel
Slave: the GPU rendering the app - radeon r200 in this case.

Step 1: kernel support

http://git.kernel.org/?p=linux/kernel/git/airlied/drm-testing.git;a=shortlog;h=refs/heads/drm-prime-test
http://cgit.freedesktop.org/~airlied/drm/log/?h=prime-test

The kernel requirements were simple, we needed a way to share a memory managed object between two kernel device drivers.
The kernel has a GEM namespace per device, however this isn't good enough to share with other devices, so I introduced a new PRIME namespace with two ioctls. One ioctl allows the master device to associate a device buffer handle with a name in the prime namespace, and the other allows the slave device to associate a prime namespace handle with a buffer. When the master creates a prime buffer the kernel associates the list of pages with the handle, and when the slave looks up the same handle it retrieves the list of pages and fakes up a TTM buffer populated with those pages as backing store. I've added the concept of slave object to TTM to allow for this.

The drm repo contains the API wrappers + intel + radeon pieces to call the association functions for buffer objects.

Step two: DRI2 Protocol
http://people.freedesktop.org/~airlied/prime/0001-dri2proto-add-prime-token.patch
http://people.freedesktop.org/~airlied/prime/0001-prime-support-for-mesa.patch

From the X server point of view a recent change to the DRI2 layer allowed for multiple device driver names to be associated with a DRI2 end point. The client can request either a DRI or VDPAU device name currently. I firstly extended the DRI2 protocol, to add a new buffer type, called PRIME, and added a hack to mesa's glx loader to request the prime driver if an environment variable was specified.

Step 3: X server DRI2 module + drivers

http://people.freedesktop.org/~airlied/prime/0001-intel-add-prime-master-support.patch
http://cgit.freedesktop.org/~airlied/xf86-video-ati/log/?h=prime-test
http://people.freedesktop.org/~airlied/prime/0001-dri2-prime-hackfest.patch

This was the messiest bit and still requires a lot of change. First up I added an interface for the drivers to register as PRIME master and slaves. Intel driver registers as master, radeon as slave for my demo. We store these in an array. When a client connects and requests prime driver, we mark the drawable and redirect the dri2 buffer creation requests to the slave screen driver. Also the drm authentication is sent to both kernel drms. It then hooks the swapbuffers command where it does a region copy, and redirects this to the slave driver, and damages the pixmap in the master driver. Now the "interesting" part, my original implementation simply grabbed the window pixmap at the dri2 create buffers time, however there is an ordering issue with compositing, this pixmap is pre-composite redirection so isn't actually the pixmap you want to tell the kernel to bind to both gpus. This turned out to function badly, I could see gears all stretched over the front buffer.

So a quick coke + chocolate break later, I had enough sugar to bash out the hack that now exists. DRI2 calls the slave driver copy region callback, which checks if the drawable pixmap is on the same screen, if its not, it checks if we've marked the pixmap as a prime pixmap (i.e. one that belongs to the master). It is, it swaps in the slaves copy, otherwise it callsback into DRI2. This callback calls the Intel driver to make the buffer object backing the pixmap, shareable, and returns the handle,then calls into radeon with the handle to create a new pixmap pointing at the shared buffer object. Once all that is done, radeon copies the back buffer to the shared front pixmap, we return and damage is posted and the compositor grabs the window pixmap and displays it.

So does it work?
On my blistering fast test system with X + xcompmgr running glxgears was going at 150fps from the r200 PCI card. Hopefully I can get some time on a faster system or one of the dual laptops.

Caveats:
- When a window manager is running the gears get all corrupted, this looks like the clipping and/or stride matching between
the drivers isn't correct. I suspect something with reparenting and decorations, I'm not enough of an X guru to understand this yet, hopefully one of the other hackers can fill me in. Also before it gets reparented and redirected a frame can land on the real front buffer, again clipping should take care of this, but isn't working yet. I need to workout how clipping and that stuff works in X/DRI2. - talk to ppl about clipping then JDI.
- Once a client has connected as a prime, we don't tear it down properly, so later clients can end marked as prime. - work out some sort of resources to turn stuff off
- Reference counting on the pages in the kernel is iffy, currently i915 ups the page list refcount but never drops it. solution JDI
- hardcoded /dev/dri paths in dri2 for slave device - solution JDI
- radeon driver could in theory be a prime master - solution JDI
- nouveau could support prime master/slave also. - solution nouveau guys JDI
- requires an ugly second screen in xorg.conf to load the slave driver. Can we have a 0 sized screen or maybe a rootless second screen. - solution : rearchitect X server to allow drivers without screens (6m-1yr work)
- pageflipping needs to be hacked off in intel driver. - work out and then JDI

Where is the video?
Once I get it working with a window manager on a useful machine I might do a video of two gears going.

Where now?
Well this is a purely academic exercise so far, after a week of kernel fighting I decided to do something new and cool. To make this as good as Windows we need to seriously re-architect the X server + drivers. At the moment you can't load an X driver without having a screen to attach it to, I don't really want a screen for the slave driver, however I still have to have one all setup and doing nothing and hopefully not getting in the way. We'd need to separate screen + drivers a lot better. Having some sort of dynamic screens would probably fall out of this work if someone decides to actually do it.

The kernel bits aren't as ugly as I thought but I'm not sure if upstreaming them is a good idea without the others bits. The refcounting definitely needs work also the cleanup when clients exit.

DRI2 needs some more changes, I might try and flesh it out a bit more and then talk to krh about a sane interface.

I'm probably going to get forced task switch quite soon, so I might just get to having this running on a W500 or T500, before dropping it for 6 months, so if anyone wants a neat project to play with and has the hw feel free to try and take this on.


ASUS feel free to send me one of the real optimus laptops and I'll get nouveau guys hooked up and try and RE the nvidia DMA engine.

12 Mar 2010 6:16am GMT

10 Things: "Legislation that affects the use of Internet-connected computers is springing up everywhere at the local, state and federal levels. You might be violating one of them without even knowing."

12 Mar 2010 5:02am GMT

The H Open: "Mandriva began life in July 1998 as Linux Mandrake in France in Gael Duval's bedroom after he ported a KDE 1.0 desktop onto Red Hat Linux 5.1, uploaded the result onto two FTP servers, went away on holiday, and came back to find that he had a popular and successful Linux distribution on his hands."

12 Mar 2010 3:02am GMT

The Bitsource: "In this article, I will describe how to bring up Debian Linux on an Embedded System. 
The hardware chosen for the embedded system is a PowerPC 440EPxX. "

12 Mar 2010 1:32am GMT

Netstat -vat: "The first alpha milestone of the Fedora 13 Linux distribution is now available and it's loaded with a number of innovative features."

12 Mar 2010 12:02am GMT

MakeTechEasier: "Creating a cool space scene can be done in a few minutes once you've got the basic method down, and Gimp provides several ways to go about adding random elements to keep things interesting."

11 Mar 2010 11:33pm GMT

Database Journal: "MySQL has an interesting architecture that sets it apart from some other enterprise database systems. It allows you to plug in different modules to handle storage. What that means to end users is that it is quite flexible, offering an interesting array of different storage engines with different features, strengths, and tradeoffs."

11 Mar 2010 11:03pm GMT

LWN.net: "Color management is sometimes unfairly characterized as a topic of interest only to print shops and video editors, but as Cruz explained at the top of his talk, anyone who shares digital content wants it to look correct, and everyone who uses more than one device knows how tricky that can be."

11 Mar 2010 10:33pm GMT

ars Technica: "As it turns out, there is no way to fully guarantee that identity theft won't happen, no matter what someone puts on the side of a truck."

11 Mar 2010 10:03pm GMT

Datamation: "The definition of virtual Linux is as fluid as the Linux platform itself. For the desktop user, virtual Linux translates into being able to use Linux without changing their existing operating system. For those working with servers however, virtual Linux can mean something very different altogether."

11 Mar 2010 9:33pm GMT

ars Technica: "Did you know that blocking ads truly hurts the websites you visit? We recently learned that many of our readers did not know this, so I'm going to explain why."

11 Mar 2010 9:03pm GMT

WMD Zone: "Novell have previously tried to leverage that market but did it all wrong. They didn't understand that there is just one killer feature ... that needs to be in there which is Exchange support."

11 Mar 2010 8:33pm GMT

Good Gear Guide: "Motorola will start loading Microsoft's search and map services onto its Android smartphones in China, bringing more non-Google services to the phones amid a row between Google and China"

11 Mar 2010 8:03pm GMT

Network World: "There's a large selection of free and open source (FOSS) operating systems available these days, and choosing the right one for any given circumstance can be quite a challenge. This article outlines the key factors you need to consider in order to pick the best operating system for your needs and experience level."

11 Mar 2010 7:33pm GMT

...sponsored by Microsoft, tommorow at 18h. I decided to take a look, so there will be some fun ;-). [And I guess I can always run away when it gets too bad.]

08 Mar 2010 9:09pm GMT

I've been going through the glibc sparc optimized assembler routines to see if anything can be improved. And I took a stab at seeing if strlen() could be made faster. Find first zero byte in string, pretty simple right?

The first thing we have to discuss is the infamous trick coined by Alan Mycroft, way back in 1987. It allows to check for the presence of a zero byte in a word in 3 instructions. There are 2 magic constants:

#define MAGIC1          0x80808080
#define MAGIC2          0x01010101

If you're checking 64-bits at a time simply expand the above magic values to 64-bits on 64-bit systems.

Then, given a word the check becomes:

        if ((val - MAGIC2) & ~val & MAGIC1)
                goto found_zero_byte_in_word;

Essentially we're subtracting MAGIC2 to induce underflow in each byte that has the value zero in it. Such underflows cause bit 8 to get set in that byte. Then we want to see if bit 8 is set after subtraction in any byte where bit 8 wasn't set before the subtraction.

To get the most parallelization on multi-issue cpus, we want to compute this using something like:

        tmp1 = val - MAGIC2;
        tmp2 = ~val & MAGIC1;
        if (tmp1 & tmp2)
                goto found_zero_byte_in_word;

to reduce the number of dependencies such that the computation of tmp1 and tmp2 can occur in the same cpu cycle.

Then there is all the trouble of getting the source buffer aligned so we can do the fast loop comparing a word at a time. The most direct implement is to read a byte at a time, checking for zero, until the buffer address is properly aligned. This is also the slowest implementation.

The powerpc code in glibc has a better idea. If dereferencing a non-word-aligned byte at address 'x' is valid, so is reading the word at 'x & ~3' (or 'x & ~7' on 64-bit). This is because page protection occurs on page boundaries, and x and 'x & ~3' are on the same page.

The only thing left to attend to is to make sure we don't match the alignment pad bytes with zero. This is solved by computing a mask of 1's and writing those 1's into the word we read before we do the Mycroft computation above. In C it looks something like:

        orig_ptr = ptr;
        align = (unsigned long) ptr & 3;
        mask = -1 >> (align * 8);
        ptr = (void *) ((unsigned long) ptr & ~3UL);
        val = *ptr;
        val |= ~mask;
        if ((val - MAGIC2) & ~val & MAGIC1)
                goto found_zero_byte_in_word;

At which point we can fall into the main loop.

Once we find the word containing a zero byte, we have to iteratively look for where it is in order to compute the return value. How to schedule this is not trivial, and it's especially cumbersome on 64-bit (where we have to potentially check 8 bytes as opposed to 4).

Anyways, let's analyze the 64-bit Sparc implementation I'm hacking on at the moment. I'm targetting UltraSPARC-III and Niagara2 for performance analysis. Simply speaking UltraSPARC-III can dual-issue integer operations, and Niagara2 is single issue and predicts all branches not taken (basically this means: minimize use of branches).

davem_strlen:
        mov     %o0, %o1
        andn    %o0, 0x7, %o0

        ldx     [%o0], %o5
        and     %o1, 0x7, %g1
        mov     -1, %g5

Save away the original string pointer in %o1. At the end we'll compute the return value as "%o1 - %o0". Align the buffer pointer and load a word as quickly as possible. We load the first word early so that we can hide the memory latency into all of the constant and mask formation we need to do before we can make the Mycroft test.

%g5 holds the initial part of the mask computation (-1, which gets expanded fully to 64-bits by this move instruction) and %g1 will have the shift factor.

        sethi   %hi(0x01010101), %o2
        sll     %g1, 3, %g1

        or      %o2, %lo(0x01010101), %o2
        srlx    %g5, %g1, %o3

        sllx    %o2, 32, %g1
        sethi   %hi(0x00ff0000), %g5

%o2 is going to hold the "0x01" expanded to 64-bits subtraction magic value. %o3 wil first hold the initial word mask, and then it will holds the "0x80" magic constant. We can compute the two 64-bit magic constants into registers in 5 instructions.

Pick either of the two constants, we choose the "0x01" here because we'll need it first. This is loaded first using "sethi", "or". This gives us the lower 32-bits of the constant, then we shift up a copy by 32-bits, then or that into the lower 32-bit copy to compute the final value. "0x80" is "0x01" shifted left by 7 bits so a simple shift is all we need to load the other 64-bit constant.

The "0x00ff0000" constant will be used while searching for the zero byte in the final word.

Next, we mask the initial word and fall through into the main loop.

        orn     %o5, %o3, %o5
        or      %o2, %g1, %o2

        sllx    %o2, 7, %o3

Mask in the pad bits using mask compute in %o3. Finish computation of 64-bit MAGIC1 into %o2, and finally put MAGIC2 into %o3. We're ready for the main loop:

10:     add     %o0, 8, %o0

        andn    %o3, %o5, %g1
        sub     %o5, %o2, %g2

        andcc   %g1, %g2, %g0
        be,a,pt %xcc, 10b
         ldx    [%o0], %o5

This is a real pain to schedule because there are many dependencies. But the "andn", "sub", "andcc" sequence is the Mycroft test, and those first two instructions can execute in one clock cycle on UltraSPARC-III. The ",a" annul bit on the branch means that we only execute the load in the branch delay slot if the branch is taken.

Now we have the code that searches for where exactly the zero byte is in the final word.

        srlx    %o5, 32, %g1
        sub     %o0, 8, %o0

We over advanced the buffer pointer in the main loop, so correct that by subtracting 8. Prepare a copy of the upper 32-bits of the word into %g1.

        andn    %o3, %g1, %o4
        sub     %g1, %o2, %g2

        add     %o0, 4, %g3
        andcc   %o4, %g2, %g0

        movne   %icc, %g1, %o5
        move    %icc, %g3, %o0

This is divide and conquer. Instead of doing 8 byte compares, we first see if the upper 32-bits have the zero byte. We essentially redo the Mycroft test on the upper 32-bits of the word.

If the upper 32-bits have the zero byte, we use %g1 for the comparisons. Otherwise we retain %o5 for the subsequent comparisons and advance the buffer pointer by 4 bytes. This is what the final two conditional move instructions are doing. Note that these conditional moves use '%icc', the 32-bit condition codes.

The astute reader may wonder why we just can't use the upper 32-bits of the Mycroft computation we made in the main loop? This doesn't work because the underflows can carry and cause false positives in upper bytes of the word. For example, consider a value where bits 35 down to 24 have hex value "0x0100". The subtraction of MAGIC2 will result in "0x8080". The real zero byte is the lower one, not the upper one. So we can't merely use the upper 32-bits of the already computed 64-bit Mycroft mask, we have to recompute it over 32-bits by hand.

Now we're left with 32-bits to check for a zero byte, we make extensive use of conditional moves to avoid branches:

        mov     3, %g2
        srlx    %o5, 8, %g1

        andcc   %g1, 0xff, %g0
        move    %icc, 2, %g2

        andcc   %o5, %g5, %g0
        srlx    %o5, 24, %o5
        move    %icc, 1, %g2

        andcc   %o5, 0xff, %g0
        move    %icc, 0, %g2

        add     %o0, %g2, %o0

We check starting at the low byte up to the highest byte. Because the highest byte, if zero, takes priority. We add the offset of the zero byte to the buffer pointer.

Finally:

        retl
         sub    %o0, %o1, %o0

We compute the length and return from the routine.

Many many moons ago, in 1998, Jakub Jelinek and his friend Jan Vondrak wrote the routines we use now on sparc. And frankly it's very hard to beat that code especially on multi-issue processors.

The powerpc trick to align the initial word helps us beat the existing code for all the unaligned cases. But for the aligned case the existing code holds a slight edge.

So now I've been trimming cycles as much as possible in the new code trying to reach the state where the aligned case executes at least as fast as the existing code. I'll check this work into glibc once I accomplish that.

The Mycroft trick extends to other libc string routines. For example for 'memchr' you replicate the search character into all bytes of a word, let's call it 'xor_mask' and in the inner loop you adjust each word by using:

        val ^= xor_mask;

Then use the Mycroft test as in strlen(). Another complication with memchr, however, is the need to check the given length bounds.

This can be done in one instruction by putting the far bounds into your base pointer register (called '%top_of_buffer' below), then using offsets starting at "0 - total_len" (referred to as '%negative_len' below).

Then your inner loop can do something like:

        ldx     [%top_of_buffer + %negative_len], %o5
        addcc   %negative_len, 8, %negative_len
        bcs     %xcc, len_exceeded
         ...

We exit the loop when adding 8 bytes to the negative len causes an overflow.

If you're interested in this kind of topic, bit twiddling tricks and whatnot, you absolutely have to own a copy of "Hacker's Delight" by Henry S. Warren, Jr.

08 Mar 2010 5:09pm GMT

Yep, that's me; and yes, I know what the cue to slow down the horse is -- lean back and use both reins. And yes, you can stop the horse by doing "slow down" three times...

But that's not a way to stop the horse. If you are going full gallop and need to stop, you want full stop now cue, not three slow down cues.

Now, I knew some horses that were actually very good at stopping, and yes, there's huge difference between stop now and slow down to full stop. Cue those horses were trained to was "whoa"...

So I tried teaching that cue to young stallion here, and it does not really work. Or rather... it works a bit too well.

I know many horses where "whoa" means slow down so I sometimes utter it when I want to just slow down... and then the horse comes abrubtly to full stop. What is worse, many other words trigger same response -- I guess they are too similar for stallion's ears.

There must be some reasonable cue, that is impossible to mistake for the horse, and unlikely to be given accidentally by the rider... unintended full stop is almost "and now climb back to the horse" event... but what is it? For now I know "whoa" is neither :-(.

(And for the record, I probably could teach horse to do full stop on something completely crazy -- like hand touching his tail -- he's learning almost too quick.)

07 Mar 2010 7:50pm GMT

I haven't had much time for blogging recently, too much exciting work going on at OsmocomBB:

• we now have simplistic support for Uplink (transmit) on SDCCH/4
• we have a minimal Layer2 (LAPDm) implementation
• we can send LOCATION UPDATING REQUEST to the network, and receive the respective response
• there's wireshark integration, i.e. all packets on the L1-L2 interface can be sent into wireshark for protocol analysis

There are still many limitations, but this is a major milestone in the project: We have working bi-directional communication from the phone to the network!

The limitations include:

• The cell has to use a combined CCCH (SDCCH/4 on timeslot 0)
• The cell has to use no encryption/authentication
• The layer2 is not finished, especially re-transmissions will not work yet
• There's no power control loop yet
• There's no timing advance correction

However, most of those are more or less simple we know what needs to be done, its just a matter of getting it done kind of tasks. There are no big unknowns involved, and particularly no further reverse-engineering of the hardware is required.

Also, the existence of a stable bi-directional communications channel between the network and the phone means that anyone interested in working on the higher layers can now actually do so. Completing and testing layer2 as well as RR/MM/CC on layer3 is a major task in itself, and it definitely requires the lower layers to be there.

The other good part is that development of layer2 and layer3 can happen entirely on the host PC, where debugging is much easier and there's no need for cross-compilation and we can use all the usual debugging options (gdb, valgrind, ...)

I'm now almost heading off for holidays (starting March 10), so don't expect any major progress from me anytime soon. I hope other interested developers will be able to take it from here and fill in some missing gaps until I'll get back.

05 Mar 2010 1:00am GMT

Folks,

Sorry for the delay. I should have updates out before the end of the week. Thanks. Remember, this is a spare time project and takes a lot of effort to do properly.

Jon.

04 Mar 2010 8:50am GMT

They are quite dramatical, but are very small yet - I committed search protocol changes. Now node stores transactions with IDs greater or equal than node's ID (it stored smaller or equal IDs previously), which is incompatible with current node searching, but allows to maintain human readable and logical (for humans) ID generation.

So, when node has ID, say, 0100..., it will host data transactions, which start from 01 (its the highest byte). It is much more convenient to configure nodes with this in mind, than to calculate what is less than 01, namely FF... IDs.

I also committed initial metadata support, but neither low level IO backend supports that yet, and I will leave only Tokyo Cabinet DB and file backends, BerkeleyDB support will be dropped, because of its slowliness. It is still in a development stage, since there is no clear vision on where this functionality should live - client or server.
I.e. it is possible that client will tell that it wants to insert metadata X into given object, and server will read/modify/write metadata blob itself, or it is possible that client will download whole metadata blob, update it locally and then write it back to server, which will replace old one with the new data. Likely I will use the former case, since it simplified client development, which should be a higher priority than server simplification.

We also found an interesting bug or feature of the storage - in some cases it is not possible to remove object, it will be recovered from the dead. Let's say we have two object copies and one node was turned off. Automatic recovery (not present yet though) will create another copy from the first one on alive nodes. Subsequent object removal will kill both copies on running nodes. When turned off node goes online again, autoamtic recovery tool will resurrect removed object from the copy presented on this node.

To date it is all a pure theory, since there is no separate metadata in the storage, thus no automatic recovery (admin should run special tool with properly crafted log file currently) and it does not remove objects from the storage. But still, described problem will hit us badly when we will actively use it.
And while there is no merge implemented either (it is kind of being materialized in my mind while we talk), solution will involve new history entry creation instead of actual data removal. Thus transaction log will contain a note that given object was removed. In case of network split and parallel object removal and update in different parts (which can not contact each other during this event) of the storage, this will also allow to implement correct and complete transaction history log by synchronization daemon.

Thus object will never be deleted from the storage, and instead its history will be updated to store a note about its status. File system checker will be extended to support a mode, when it will actually remove objects from the storage after they were marked (and resolved during merge with other logs if needed) as deleted after some timeout, which should be big enough to eliminate such ghost nodes appearence.

And the last but not least discussed issue concerns storage size and related limitations. Let's say that we reached our current storage capacity and want to add several another machines, which will add 50% of the current volume. We want to spread data equally between all nodes, thus we will need to update every node's ID to shift it a little, so that new nodes entered addressing ring and formed a fair ID distribution. Amount of transaction copies in this case is quite large - more than a half of all data will have to be transferred over the network, which will take a while.
Also, when we add new empty node into the storage, it will kind of hide data it is supposed to host (according to ID distribution) until it is copied to the new node from the neighbour. Thus there should be a poilicy, which will forbid simultaneous update of all servers, since there is a possibility that suddenly all added nodes hide all copies of some objects. It will be recovered of course, but it will take some time, which in some cases is not appropriate.

One of the solutions for the described storage size issue is different storage policy. We can implement multiple virtual datacenters, where each new virtual datacenter corresponds to newly added set of machines. In this case we will extend write application so that it could 'touch' old hash functions (and thus old virtual datacenters) first to determine whether it can store data there and move to the new machines if there is no space in the old ones. Reading can issue a parallel lookup to all virtual datacenters asking for given object ID.

This scheme has latency limitations as well as network traffic growing with new virtual datacenters involved, but it can be a good decision for smaller setups though.

Virtual datacenters (or configurable hash/transformation functions used to generate transaction ID) becomes one of the most flexible 'tools' to implement different storage setups.

Stay tuned, there will be more news soon!

04 Mar 2010 12:08am GMT

Although IT professionals should take care to avoid engineering envy, it is often useful to learn from the experiences of other engineering disciplines. In this posting, I will compare and contrast construction of a building to implementation of a large software project.

Leaving aside financial engineering, building construction starts with an architect, who lays out the general shape and look of the building. A structural engineer creates a detailed design, with an eye towards ensuring that the building will remain standing despite the best efforts of wind, gravity, and plate tectonics. A construction engineer works out the details of the construction process - for example, it is good if the building can support itself while being built as opposed to doing so only when completed. Other engineering specialties may be required as well, for example, HVAC (heating, ventilating, and air conditioning).

Once the building is built, different skills are needed, including operating engineers, maintenance personnel, and janitors.

A very similar sequence of events can play out for a large software application. Software architects (for better or worse) lay out the general shape of the project, developers design and code it, and others ensure that it is built, tested, and safely ensconced in some source-code management system.

However, once the application is completed, it is likely that its care and feeding will be taken over by application, database, and system administrators. The architects and developers will switch to other projects (possibly version N+1 of this same application), and perhaps even retire or otherwise move on. Of course, if the application runs at multiple sites, there might well be a separate set of administrators for each site. But for simplicity, let's assume that this application runs at only one site.

Now suppose that it is necessary to parallelize this application.

This is tantamount to major structural change to the building, such as adding several new floors. A structural change of this nature is clearly not a job that you would normally entrust to operating engineers, maintenance personnel, or janitors.

But what else can you do if the original architects and developers are gone?

01 Mar 2010 1:43am GMT

In the Motorola/Compal C155 phone supported by OsmocomBB, we have found a ringtone melody chip called SPMA100B from sunplus.

As strange as it might seem, this is the only part used in the phone for which we have not been able to find any kind of programming information. So if you know anything about how to program this part from software (register map, programming manual, ...) please let me know!

And no, we don't need electrical/mechanical data sheets, thanks :)

01 Mar 2010 1:00am GMT

Visited an old mine today... Actually for an orienteering run. And seen some pretty impressive tech...

Mine is actually from 1890 or so, and it was running up to 1997 or so. They had some wonderful hacks -- like steam engine, still powering the elevator up to 1997, but running on compressed gas.

And because they did not use the computers to control the elevator, they had to use two-operators, and blackbox type device recording elevator speeds and communication over single rope. Speedometer used mercury. Impressive.

But... on the other hand they kept things simple. Steel pipe was used for communications 500 meters underground. Single part. In 1980, they'd probably use two analog phones and a battery, about 10 parts total. Today, we'd probably use two computers, running VOIP over ethernet, for about 1000 milion parts total. Is not progress wonderful?

28 Feb 2010 9:30pm GMT

I used to hate skiing - I wasted 3 years in running ski section in univercity, while I could play football or, let's say, chess. Well, there was no chess section, but whatever else it could be more interesting than ski.

And this year I opened myself alpine ski. I did it about 15-20 years ago previously when was in school, and it was simple small plastic skis. Technology made a significan progress since then and I got ability to test real skis.

That's what I did this and previous weekends - two days in Stepanovo ski resort. It was essentially the first time I tried big slope (not that big compared to real resorts in Europe of course, just about a kilometer or less and 100 meters drop) and real snow. And it was fucking incredible - it is fast, it is long enough to feel the speed and ground, it is quite different - there are multiple traces and a lot of small roads from main trace, where one can ride over hummocks and small ski jumps.

I bought myself all equipment except skis itself - want to touch different things first, but I believe I will get my own next time. With the proper equipment it is not cold, warm or wet, it is just ubercool. Getting that I basically have no technique, I open lots of cases for myself all the time. And I believe that I have some progress, maybe not that good, but very pleasant for myself.

I tried long blue trace previously, but today I started a red one. And it was fucking beautiful - so fast and so strong. No boring places and long waits, just pure pleasure of speed and control. On this trace I found myself moving noticebly more technically than on a simpler trace.

I started to sit lower, put legs closer and change ski edges using mass center and not ass or legs, pipe changing arcs became shorter and with longer radius, which increased speed compared to plain skiing.

Of course it was not always perfect, and frankly I believe it looked like crap and was a real crap from good technique point of view, but it was very pleasant for me, and that's what matters. I want to get another hour or so with good teacher, who will tell me where main problems are, since I can not see how I made a slope. Sometimes I flew over the trace couple of meters and than landed in 'different positions' usually already without skis moving on my body another dozen of meters. But I like it too - it shows complex cases and sharps instincts.

Currently I believe there are no somewhat big parts of my body, which do not try to scream and ache. Especially shine bones (hard to move or stay long enough) and various leg muscles, but it is not a problem - I will be fresh again in a day, and hundred or so of "The Glenrothes" and couple of hours playing piano and trumpet will quickly help me. So plan is to make another turn next weekend or preferably move to ski resort couple times.

Fucking incredible. Just love it!

28 Feb 2010 6:43pm GMT

Due to the big storm Thursday night, we spent two days without power. After freezing ourselves on Friday, we decided to spend Saturday at a friend's place (thank you Aris, Chris and Sarah). While checking on our house, there were always crews at work trying to clear up the fallen trees, reopen roads and reconnect power and communications lines. A big thank you goes out to the power and telco crews who are working around the clock to clear up the mess and reconnect New England.

28 Feb 2010 5:24pm GMT

I've uploaded man-pages-3.24 into the release directory (or view the online pages). The most notable changes in man-pages-3.24 are the following:

• The addition of three pages by David Howells describing the kernel key management facility: add_key(2), request_key(2), and keyctl(2). (These pages were formerly part of the keyutils package.)
• The fcntl(2) manual pages adds documention of F_SETOWN_EX and F_GETOWN_EX, which are new in Linux 2.6.32.
• Minor changes to many other pages.

27 Feb 2010 5:00pm GMT

Barnes and Noble released the nook source code last week. This includes the code to busybox, uboot and their kernel. Unfortunately, the uboot and kernel code both appear to be missing swathes of code found statically linked in the binaries that they're distributing. License compliance is hard, let's flail wildly.

26 Feb 2010 6:31pm GMT

Okay I've been busy elsewhere but dragged myself back to try and finish this for upstream

v10 of the patch is up
http://people.freedesktop.org/~airlied/vgaswitcheroo/0001-vga_switcheroo-initial-implementation-v10.patch

changes are mainly that mjg59 was right about keeping ugly things in the drivers.

adding ATRM support to get the ROMs on ATI hybrid for the discrete card was actually a pain with the previous code design,
so I moved lots of it around again, and now the discrete ROM can be retrieved via the ATRM method.

I've tested it on the W500 and it works as well as before, which means still the 3rd or 4th switch fails and locks the machine up,
I need to debug this further.

The refactored code should hopefully make it easier to fill in the nvidia/nvidia and intel/nvidia blanks for mjg59.

Update 1: v11 is now up
http://people.freedesktop.org/~airlied/vgaswitcheroo/0001-vga_switcheroo-initial-implementation-v11.patch
It should fix the failure to switch to IGD the 2nd time hopefully.

Update 2: v13 is now up, it blindly implements nvidia DSM changing, but I've no idea if it works. Hopefully someone can test it and give me some feedback. Its nearly all guesswork from work mjg59 did.

26 Feb 2010 5:04am GMT

Chris asks where OpenSolaris is headed. My reaction: nobody cares anymore. FreeBSD established itself as the alternative to Linux, and that leaves Solaris with no niche. So, whatever. It is much more important what is going to happen to OpenOffice and MySQL. Also, Sun carried a pretty large assortment of lesser projects, such as Lustre.

25 Feb 2010 11:13pm GMT