31 Jul 2014

feedPlanet Debian

Russell Coker: Links July 2014

Dave Johnson wrote an interesting article for Salon about companies ripping off the tax system by claiming that all their income is produced in low tax countries [1].

Seb Lee-Delisle wrote an insightful article about how to ask to get paid to speak [2]. I should do that.

Daniel Pocock wrote an informative article about the reConServer simple SIP conferencing server [3]. I should try it out, currently most people I want to conference with are using Google Hangouts, but getting away from Google is a good thing.

François Marier wrote an informative post about hardening ssh servers [4].

S. E. Smith wrote an interesting article "I Am Tired of Hearing Programmers Defend Gender Essentialism [5].

Bert Archer wrote an insightful article about lazy tourism [6]. His initial example of "love locks" breaking bridges was a bit silly (it's not difficult to cut locks off a bridge) but his general point about lazy/stupid tourism is good.

Daniel Pocock wrote an insightful post about new developments in taxis, the London Taxi protest against Uber, and related changes [7]. His post convinced me that Uber is a good thing and should be supported. I checked the prices and unfortunately Uber is more expensive than normal taxis for my most common journey.

Cory Doctorow wrote an insightful article for The Guardian about the moral issues related to government spying [8].

The Verge has an interesting review of the latest Lytro Lightbox camera [9]. Not nearly ready for me to use, but interesting technology.

Prospect has an informative article by Kathryn Joyce about the Protestant child sex abuse scandal in the US [10]. Billy Graham's grandson is leading the work to reform churches so that they protect children instead of pedophiles. Prospect also has an article by Kathryn Joyce about Christians home-schooling kids to try and program them to be zealots and how that hurts kids [11].

The Daily Beast has an interesting article about the way that the extreme right wing in the US are trying to kill people, it's the right wing death panel [12].

Jay Michaelson wrote an informative article for The Daily Beast about right-wing hate groups in the US who promote the extreme homophobic legislation in Russia and other countries [13]. It also connects to the Koch brothers who seem to be associated with most evil. Elias Isquith wrote an insightful article for Salon about the current right-wing obsession with making homophobic discrimination an issue of "religious liberty" will hurt religious people [14]. He also describes how stupid the right-wing extremists are in relation to other issues too.

EconomixComix.com has a really great comic explaning the economics of Social Security in the US [15]. They also have a comic explaining the TPP which is really good [16]. They sell a comic book about economics which I'm sure is worth buying. We need to have comics explaining all technical topics, it's a good way of conveying concepts. When I was in primary school my parents gave me comic books covering nuclear physics and other science topics which were really good.

Mia McKenzie wrote an insightful article for BlackGirlDangerous.com about dealing with racist white teachers [17]. I think that it would be ideal to have a school dedicated to each minority group with teachers from that group.

Related posts:

  1. Links July 2013 Wayne Mcgregor gave an interesting TED talk about the creative...
  2. Links May 2014 Charmian Gooch gave an interesting TED talk about her efforts...
  3. Links June 2014 Russ Albery wrote an insightful blog post about trust, computer...

31 Jul 2014 1:38pm GMT

Craig Small: Linux Capabilities

I was recently updating some code that uses fping. Initially it used exec() that was redirected to a temporary file but I changed it to use popen. While it had been a while since I've done this sort of thing, I do recall there was an issue with running popen on setuid binary. A later found it is mainly around setuid scripts which are very problematic and there are good reasons why you don't do this.

Anyhow, the program worked fine which surprised me. Was fping setuid root to get the raw socket?

$ ls -l /usr/bin/fping
-rwxr-xr-x 1 root root 31464 May  6 21:42 /usr/bin/fping

It wasn't which at first all I thought "ok, so that's why popen is happy". The way that fping and other programs work is they bind to a raw socket. This socket sits below the normal type sockets such as the ones used for TCP and UDP and normal users cannot use them by default. So how did fping work it's magic and get access to this socket? It used Capabilities.

Previously getting privileged features had a big problem; it was an all or nothing thing. You want access to a raw socket? Sure, be setuid but that means you also could, for example, read any file on the system or set passwords. Capabilites provide a way of giving programs some better level of access, but not a blank cheque.

The tool getcap is the way of determining what capabilities are found on a file. These capabilities are attributes on the file which, when the file is run, turn into capabilities or extra permissions. fping has the capability cap_net_raw+ep applied to it. This gives access to the RAW and PACKET sockets which is what fping needs. The +ep after the capability name means it is an Effective and Permitted capability, which describes what happens with child processes and dropping privileges.

I hadn't seen these Capabilities before. They are a nice way to give your programs the access they need, but limiting the risk of something going wrong and having a rouge program running as root.

Related articles

31 Jul 2014 12:58pm GMT

Steve Kemp: luonnos viesti - 31 heinäkuu 2014

Yesterday I spent a while looking at the Debian code search site, an enormously useful service allowing you to search the code contained in the Debian archives.

The end result was three trivial bug reports:

#756565 - lives

Insecure usage of temporary files.

A CVE-identifier should be requested.

#756566 - libxml-dt-perl

Insecure usage of temporary files.

A CVE-identifier has been requested by Salvatore Bonaccorso, and will be added to my security log once allocated.

756600 - xcfa

Insecure usage of temporary files.

A CVE-identifier should be requested.

Finding these bugs was a simple matter of using the code-search to look for patterns like "system.*>.*%2Ftmp".

Perhaps tomorrow somebody else would like to have a go at looking for backtick-related operations ("`"), or the usage of popen.

Tomorrow I will personally be swimming in a loch, which is more fun than wading in code..

31 Jul 2014 12:54pm GMT

Russell Coker: BTRFS Status July 2014

My last BTRFS status report was in April [1], it wasn't the most positive report with data corruption and system hangs. Hacker News has a brief discussion of BTRFS which includes the statement "Russell Coker's reports of his experiences with BTRFS give me the screaming heebie-jeebies, no matter how up-beat and positive he stays about it" [2] (that's one of my favorite comments about my blog).

Since April things have worked better. Linux kernel 3.14 solves the worst problems I had with 3.13 and it's generally doing everything I want it to do. I now have cron jobs making snapshots as often as I wish (as frequently as every 15 minutes on some systems), automatically removing snapshots (removing 500+ snapshots at once doesn't hang the system), balancing, and scrubbing. The fact that I can now expect that a filesystem balance (which is a type of defragment operation for BTRFS that frees some "chunks") from a cron job and expect the system not to hang means that I haven't run out of metadata chunk space. I expect that running out of metadata space can still cause filesystem deadlocks given a lack of reports on the BTRFS mailing list of fixes in that regard, but as long as balance works well we can work around that.

My main workstation now has 35 days of uptime and my home server has 90 days of uptime. Also the server that stores my email now has 93 days uptime even though it's running Linux kernel 3.13.10. I am rather nervous about the server running 3.13.10 because in my experience every kernel before 3.14.1 had BTRFS problems that would cause system hangs. I don't want a server that's an hour's drive away to hang…

The server that runs my email is using kernel 3.13.10 because when I briefly tried a 3.14 kernel it didn't work reliably with the Xen kernel 4.1 from Debian/Wheezy and I had a choice of using the Xen kernel 4.3 from Debian/Unstable to match the Linux kernel or use an earlier Linux kernel. I have a couple of Xen servers running Debian/Unstable for test purposes which are working well so I may upgrade my mail server to the latest Xen and Linux kernels from Unstable in the near future. But for the moment I'm just not doing many snapshots and never running a filesystem scrub on that server.

Scrubbing

In kernel 3.14 scrub is working reliably for me and I have cron jobs to scrub filesystems on every system running that kernel. So far I've never seen it report an error on a system that matters to me but I expect that it will happen eventually.

The paper "An Analysis of Data Corruption in the Storage Stack" from the University of Wisconsin (based on NetApp data) [3] shows that "nearline" disks (IE any disks I can afford) have an incidence of checksum errors (occasions when the disk returns bad data but claims it to be good) of about 0.42%. There are 18 disks running in systems I personally care about (as opposed to systems where I am paid to care) so with a 0.42% probability of a disk experiencing data corruption per year that would give a 7.3% probability of having such corruption on one disk in any year and a greater than 50% chance that it's already happened over the last 10 years. Of the 18 disks in question 15 are currently running BTRFS. Of the 15 running BTRFS 10 are scrubbed regularly (the other 5 are systems that don't run 24*7 and the system running kernel 3.13.10).

Newer Kernels

The discussion on the BTRFS mailing list about kernel 3.15 is mostly about hangs. This is correlated with some changes to improve performance so I presume that it has exposed race conditions. Based on those discussions I haven't felt inclined to run a 3.15 kernel. As the developers already have some good bug reports I don't think that I could provide any benefit by doing more testing at this time. I think that there would be no benefit to me personally or the Linux community in testing 3.15.

I don't have a personal interest in RAID-5 or RAID-6. The only systems I run that have more data than will fit on a RAID-1 array of cheap SATA disks are ones that I am paid to run - and they are running ZFS. So the ongoing development of RAID-5 and RAID-6 code isn't an incentive for me to run newer kernels. Eventually I'll test out RAID-6 code, but at the moment I don't think they need more bug reports in this area.

I don't have a great personal interest in filesystem performance at this time. There are some serious BTRFS performance issues. One problem is that a filesystem balance and subtree removal seem to take excessive amounts of CPU time. Another is that there isn't much support for balancing IO to multiple devices (in RAID-1 every process has all it's read requests sent to one device). For large-scale use of a filesystem these are significant problems. But when you have basic requirements (such as a mail server for dozens of users or a personal workstation with a quad-core CPU and fast SSD storage) it doesn't make much difference. Currently all of my systems which use BTRFS have storage hardware that exceeds the system performance requirements by such a large margin that nothing other than installing Debian packages can slow the system down. So while there are performance improvements in newer versions of the BTRFS kernel code that isn't an incentive for me to upgrade.

It's just been announced that Debian/Jessie will use Linux 3.16, so I guess I'll have to test that a bit for the benefit of Debian users. I am concerned that 3.16 won't be stable enough for typical users at the time that Jessie is released.

Related posts:

  1. BTRFS Status March 2014 I'm currently using BTRFS on most systems that I can...
  2. BTRFS Status April 2014 Since my blog post about BTRFS in March [1] not...
  3. Starting with BTRFS Based on my investigation of RAID reliability [1] I have...

31 Jul 2014 10:45am GMT

Wouter Verhelst: Using your Belgian eID with SSH mark two

Several years ago, I blogged about how to use a Belgian electronic ID card with SSH. I never really used it myself, but was interested in figuring out if it would still work.

The good news is that since then, you don't need to recompile OpenSSH anymore to get PKCS#11 support; this is now compiled in by default.

The slightly bad news is that there will be some more typework. Rather than entering ssh-add -D 0 (to access the PKCS#11 certificate in slot 0), you should now enter something along the lines of ssh-add -s /usr/lib/libbeidpkcs11.so.0. This will ask for your passphrase, but it isn't necessary to enter the correct pin code at this point in time. The first time you try to log on, you'll get a standard beid dialog box where you should enter your pin code; this will then work. The next time, you'll be logged on and you can access servers without having to enter a pin code.

The worse news is that there seems to be a bug in ssh-agent, making it impossible to unload a PKCS#11 library. Doing ssh-add -D will remove your keys from the agent; the next time you try to add them again, however, ssh-agent will simply report SSH_AGENT_FAILURE. I suspect the dlopen()ed modules aren't being unloaded when the keys are removed.

Unfortunately, the same (or at least, a similar) bug appears to occur when one removes the card from the cardreader.

As such, I don't currently recommend trying to use this.

Update: fix command-line options to ssh-add invocation above.

31 Jul 2014 10:28am GMT

Petter Reinholdtsen: Debian Edu interview: Bernd Zeitzen

The complete and free "out of the box" software solution for schools, Debian Edu / Skolelinux, is used quite a lot in Germany, and one of the people involved is Bernd Zeitzen, who show up on the project mailing lists from time to time with interesting questions and tips on how to adjust the setup. I managed to interview him this summer.

Who are you, and how do you spend your days?

My name is Bernd Zeitzen and I'm married with Hedda, a self employed physiotherapist. My former profession is tool maker, but I haven't worked for 30 years in this job. 30 years ago I started to support my wife and become her officeworker and a few years later the administrator for a small computer network, today based on Ubuntu Server (Samba, OpenVPN). For her daily work she has to use Windows Desktops because the software she needs to organize her business only works with Windows . :-(

In 1988 we started with one PC and DOS, then I learned to use Windows 98, 2000, XP, …, 8, Ubuntu, MacOSX. Today we are running a Linux server with 6 Windows clients and 10 persons (teacher of children with special needs, speech therapist, occupational therapist, psychologist and officeworkers) using our Samba shares via OpenVPN to work with the documentations of our patients.

How did you get in contact with the Skolelinux / Debian Edu project?

Two years ago a friend of mine asked me, if I want to get a job in his school (Gymnasium Harsewinkel). They started with Skolelinux / Debian Edu and they were looking for people to give support to the teachers using the software and the network and teaching the pupils increasing their computer skills in optional lessons. I'm spending 4-6 hours a week with this job.

What do you see as the advantages of Skolelinux / Debian Edu?

The independence.

First: Every person is allowed to use, share and develop the software. Even if you are poor, you are allowed to use the software included in Skolelinux/Debian Edu and all the other Free Software.

Second: The software runs on old machines and this gives us the possibility to recycle computers, weeded out from offices. The servers and desktops are running for more than two years and they are working reliable.

We have two servers (one tjener and one terminal server), 45 workstations in three classrooms and seven laptops as a mobile solution for all classrooms. These machines are all booting from the terminal server. In the moment we are installing 30 laptops as mobile workstations. Then the pupils have the possibility to work with these machines in their classrooms. Internet access is realized by a WLAN router, connected to the schools network. This is all done without a dedicated system administrator or a computer science teacher.

What do you see as the disadvantages of Skolelinux / Debian Edu?

Teachers and pupils are Windows users. <Irony on> And Linux isn't cool. It's software for freaks using the command line. <Irony off> They don't realize the stability of the system.

Which free software do you use daily?

Firefox, Thunderbird, LibreOffice, Ubuntu Server 12.04 (Samba, Apache, MySQL, Joomla!, … and Skolelinux / Debian Edu)

Which strategy do you believe is the right one to use to get schools to use free software?

In Germany we have the situation: every school is free to decide which software they want to use. This decision is influenced by teachers who learned to use Windows and MS Office. They buy a PC with Windows preinstalled and an additional testing version of MS Office. They don't know about the possibility to use Free Software instead. Another problem are the publisher of school books. They develop their software, added to the school books, for Windows.

31 Jul 2014 6:30am GMT

30 Jul 2014

feedPlanet Debian

Bits from Debian: Jessie will ship Linux 3.16

The Debian Linux kernel team has discussed and chosen the kernel version to use as a basis for Debian 8 'jessie'.

This will be Linux 3.16, due to be released in early August. Release candidates for Linux 3.16 are already packaged and available in the experimental suite.

If you maintain a package that is closely bound to the kernel version - a kernel module or a userland application that depends on an unstable API - please ensure that it is compatible with Linux 3.16 prior to the freeze date (5th November, 2014). Incompatible packages are very likely to be removed from testing and not included in 'jessie'.

  1. My kernel module package doesn't build on 3.16 and upstream is not interested in supporting this version. What can I do?
    The kernel team might be able to help you with forward-porting, but also try Linux Kernel Newbies or the mailing list(s) for the relevant kernel subsystem(s).

  2. There's an important new kernel feature that ought to go into jessie, but it won't be in 3.16. Can you still add it?
    Maybe - sometimes this is easy and sometimes it's too disruptive to the rest of the kernel. Please contact the team on the debian-kernel mailing list or by opening a wishlist bug.

  3. Will Linux 3.16 get long term support from upstream?
    The Linux 3.16-stable branch will not be maintained as a longterm branch at kernel.org. However, the Ubuntu kernel team will continue to maintain that branch, following the same rules for acceptance and review, until around April 2016. Ben Hutchings is planning to continue maintenance from then until the end of regular support for 'jessie'.

30 Jul 2014 9:10pm GMT

Konstantinos Margaritis: SIMD book, update 3! Addition/Subtraction mostly finished

Finally. Apologies for the delay, but it's been a busy month. This time I will hold true to my word and upload a PDF for people to see (attached to this page).

So, what's new? Here is a list of things done:

* Finished ALL addition/subtraction related instructions for all engines and major derivatives (SSE*/AVX, VMX/VSX, NEON/armv8 NEON). With diagrams (these were the reasons it has taken so long).
* Reorganized the structure (split the book into Parts I/II, the instruction index will be in Part II, Part I will carry the design analysis of each SIMD engine.
* Added an TOC/index.
* So far, with just Addition/Subtraction Chapter and the rest empty sections, it has reached 175 pages (B5, again I'm not fixed on the size, it might actually change)! My estimate is that the whole book will surpass 800 pages with everything included.

TODO:

30 Jul 2014 7:08pm GMT

Steinar H. Gunderson: Blehnovo, follow-up

A few weeks back, I posted about my adventures in getting Lenovo Switzerland/Germany to repair my laptop after I dropped it onto the floor and cracked the screen. (I'm sure they would be thrilled to hear that it sits next to my Wi-Fi on Linux rant, which got 39000 readers after reaching the front page Hacker News and then /r/linux.) Now that I'm leaving Norway tomorrow (via Assembly 2014 and then on to Zürich), I thought it would be a good time to wrap up the story.

In case you don't remember, we last left the situation July 7th, when I had called Lenovo Norway, who accepted my laptop for repair pretty much immediately after Lenovo Germany had staunchly refused for over a month. What happened next:

July 8th: DHL comes and picks up my laptop.

July 10th: I get an email saying that my laptop is repaired, and will be returned to me the next day.

July 11th: DHL calls me and asks if I'm at home. I say no, I'm at Solskogen, can they deliver it there instead? They redirect the package, no problems. Just as they call the second time, I'm away to shower after a run, but I ask them to just deliver it to someone there. A friend signs for it.

I start up the laptop, and woo, the screen works again. But the EFI menu has been tampered with, probably for diagnostic purposes, so it no longer boots Linux (but rather has a Windows entry that doesn't work). I spend half an hour fiddling with d-i to get it back again. During this process, I notice something strange -- my beautiful 1920x1080 screen has been replaced with a much less beautiful 1366x768 one! Also, if I lift the laptop in the wrong spot (at first I think it's related to the monitor bezel, but later understand that it's not), it promptly dies and spews garbage all over the screen. In any case, it's good enough to fix my backup job (which hadn't been properly running for a while--boo) and thus get the source code I've been working on out.

I call Lenovo Norway. They are confused. They try to figure out what happened, but are having problems finding the right internal part number for the 1920x1080 screen. They say they'll call me back.

July 16th: I call Lenovo Norway, wondering why they haven't called me back. I also send them a video showing the issue. The guy says he'll need to check with his colleague and will contact me back, although he can't guarantee anything will happen the same day. They don't call me back.

July 17th: A guy from InfoCare calls and wonders if I'm available. Turns out he's sent out to fix my laptop. (He doesn't speak Norwegian, but he speaks English, so no problem.) He arrives in my house, and replaces the 1366x768 panel with a 1920x1080 one. I can't reproduce the crash issue offhand, but shows him the video. He tries to figure out what the crashes are due to, but can't immediately see anything being loose.

I call Lenovo Norway again, and say it's great they've fixed my panel, but that the issue still persists. The guy doesn't own my case but has seen the video. After some discussion and thinking, they can't remotely diagnose it, and we agree that I'll send it in again, because they can seemingly not send a full array of parts with InfoCare (so they'd potentially go five trips, which I understand it not so cool for them).

July 18th: DHL picks up the laptop again. The guy looks a bit bewildered.

July 23rd: The laptop arrives at Lenovo. I get an email saying it's fixed.

July 24th: DHL arrives with the laptop. The repair sheet says they have replaced the "primary memory" (which I suppose means RAM). I try to trigger the issue. It's gone!

So, that's the status. I have a perfectly working laptop, with a gorgeous 1920x1080 display. I'm basically in the state I thought I would be over the weekend when I dropped it May 30th, it just took almost two months instead of one working day.

So, is my warranty correctly registered in the system now? I have no idea, because while Lenovo Norway was busy repairing my laptop three times, Lenovo Germany appears to have done nothing at all (not even closed my case, because that should have triggered an email). So as far as I know, there's still an open case on me somewhere, where they try to figure out why system A shows I have accident coverage and system B doesn't.

30 Jul 2014 3:00pm GMT

29 Jul 2014

feedPlanet Debian

Ian Campbell: Debian Installer ARM64 Dailies

It's taken a while but all of the pieces are finally in place to run successfully through Debian Installer on ARM64 using the Debian ARM64 port.

So I'm now running nightly builds locally and uploading them to http://www.hellion.org.uk/debian/didaily/arm64/.

If you have CACert in your CA roots then you might prefer the slightly more secure version.

Hopefully before too long I can arrange to have them building on one of the project machines and uploaded to somewhere a little more formal like people.d.o or even the regular Debian Installer dailies site. This will have to do for now though.

Warning

The arm64 port is currently hosted on Debian Ports which only supports the unstable "sid" distribution. This means that installation can be a bit of a moving target and sometimes fails to download various installer components or installation packages. Mostly it's just a case of waiting for the buildd and/or archive to catch up. You have been warned!

Installing in a Xen guest

If you are lucky enough to have access to some 64-bit ARM hardware (such as the APM X-Gene, see wiki.xen.org for setup instructions) then installing Debian as a guest is pretty straightforward.

I suppose if you had lots of time (and I do mean lots) you could also install under Xen running on the Foundation or Fast Model. I wouldn't recommend it though.

First download the installer kernel and ramdisk onto your dom0 filesystem (e.g. to /root/didaily/arm64).

Second create a suitable guest config file such as:

name = "debian-installer"
disk = ["phy:/dev/LVM/debian,xvda,rw"]
vif = [ '' ] 
memory = 512
kernel = "/root/didaily/arm64/vmlinuz"
ramdisk= "/root/didaily/arm64/initrd.gz"
extra = "console=hvc0 -- "

In this example I'm installing to a raw logical volume /dev/LVM/debian. You might also want to use randmac to generate a permanent MAC address for the Ethernet device (specified as vif = ['mac=xx:xx:xx:xx:xx:xx']).

Once that is done you can start the guest with:

xl create -c cfg

From here you'll be in the installer and things carry on as usual. You'll need to manually point it to ftp.debian-ports.org as the mirror, or you can preseed by appending to the extra line in the cfg like so:

mirror/country=manual mirror/http/hostname=ftp.debian-ports.org mirror/http/directory=/debian

Apart from that there will be a warning about not knowing how to setup the bootloader but that is normal for now.

Installing in Qemu

To do this you will need a version of http://www.qemu.org which supports qemu-system-aarch64. The latest release doesn't yet so I've been using v2.1.0-rc3 (it seems upstream are now up to -rc5). Once qemu is built and installed and the installer kernel and ramdisk have been downloaded to $DI you can start with:

qemu-system-aarch64 -M virt -cpu cortex-a57 \
    -kernel $DI/vmlinuz -initrd $DI/initrd.gz \
    -append "console=ttyAMA0 -- " \
    -serial stdio -nographic --monitor none \
    -drive file=rootfs.qcow2,if=none,id=blk,format=qcow2 -device virtio-blk-device,drive=blk \
    -net user,vlan=0 -device virtio-net-device,vlan=0

That's using a qcow2 image for the rootfs, I think I created it with something like:

qemu-img create -f qcow2 rootfs.qcow2 4G

Once started installation proceeds much like normal. As with Xen you will need to either point it at the debian-ports archive by hand or preseed by adding to the -append line and the warning about no bootloader configuration is expected.

Installing on real hardware

Someone should probably try this ;-).

29 Jul 2014 7:36pm GMT

Daniel Pocock: Pruning Syslog entries from MongoDB

I previously announced the availability of rsyslog+MongoDB+LogAnalyzer in Debian wheezy-backports. This latest rsyslog with MongoDB storage support is also available for Ubuntu and Fedora users in one way or another.

Just one thing was missing: a flexible way to prune the database. LogAnalyzer provides a very basic pruning script that simply purges all records over a certain age. The script hasn't been adapted to work within the package layout. It is written in PHP, which may not be ideal for people who don't actually want LogAnalyzer on their Syslog/MongoDB host.

Now there is a convenient solution: I've just contributed a very trivial Python script for selectively pruning the records.

Thanks to Python syntax and the PyMongo client, it is extremely concise: in fact, here is the full script:

#!/usr/bin/python

import syslog
import datetime
from pymongo import Connection

# It assumes we use the default database name 'logs' and collection 'syslog'
# in the rsyslog configuration.

with Connection() as client:
    db = client.logs
    table = db.syslog
    #print "Initial count: %d" % table.count()
    today = datetime.datetime.today()

    # remove ANY record older than 5 weeks except mail.info
    t = today - datetime.timedelta(weeks=5)
    table.remove({"time":{ "$lt": t }, "syslog_fac": { "$ne" : syslog.LOG_MAIL }})

    # remove any debug record older than 7 days
    t = today - datetime.timedelta(days=7)
    table.remove({"time":{ "$lt": t }, "syslog_sever": syslog.LOG_DEBUG})

    #print "Final count: %d" % table.count()

Just put it in /usr/local/bin and run it daily from cron.

Customization

Just adapt the table.remove statements as required. See the PyMongo tutorial for a very basic introduction to the query syntax and full details in the MongoDB query operator reference for creating more elaborate pruning rules.

Potential improvements

  • Indexing the columns used in the queries
  • Logging progress and stats to Syslog


LogAnalyzer using a database backend such as MongoDB is very easy to set up and much faster than working with text-based log files

29 Jul 2014 6:27pm GMT

Gunnar Wolf: Editorial process starting in 3... 2... 1...

Yay!

Today I finally submitted our book, Fundamentos de Sistemas Operativos, for the Editorial Department of our institute. Of course, I'm not naïve enough to assume there won't be a heavy editorial phase, but I'm more than eager to dive into it... And have the book printed in maybe two months time!

Of course, this book is to be published under a free license (CC-BY-SA). And I'm talking with the coauthors, we are about to push the Git repository to a public location, as we believe the source for the text and figures can also be of interest to others.

The book itself (as I've already boasted about here :-} ) is available (somewhat as a preprint) for download.

[update] Talked it over with the coauthors, and we finally have a public repository! Clone it from:

https://github.com/gwolf/sistop.git

Or just browse it from Github's web interface.

29 Jul 2014 6:09pm GMT

Christian Perrier: Developers per country (July 2014)

This is time again for my annual report about the number of developers per country.

This is now the sixth edition of this report. Former editions:

So, here we are with the July 2014 version, sorted by the ratio of *active* developers per million population for each country.

Act: number of active developers
Dev: total number of developers
A/M: number of active devels per million pop.
D/M: number of devels per million pop.
2009: rank in 2009
2010: rank in 2010
2011: rank in 2011 (June)
2012: rank in 2012 (June)
2013: rank in 2012 (July)
2014: rank now
Code Name Population Act Dev Dev Act/Million Dev/Million 2009 2010 June 2011 June 2012 July 2013 July 2014
fi Finland 5259250 19 31 3,61 5,89 1 1 1 1 1 1
ie Ireland 4670976 13 17 2,78 3,64 13 9 6 2 2 2
nz New Zealand 4331600 11 15 2,54 3,46 4 3 5 7 7 3 *
mq Martinique 396404 1 1 2,52 2,52

3 4 4 4
se Sweden 9088728 22 37 2,42 4,07 3 6 7 5 5 5
ch Switzerland 7870134 19 29 2,41 3,68 2 2 2 3 3 6 *
no Norway 4973029 11 14 2,21 2,82 5 4 4 6 6 7 *
at Austria 8217280 18 29 2,19 3,53 6 8 10 10 10 8 *
de Germany 81471834 164 235 2,01 2,88 7 7 9 9 8 9 *
lu Luxemburg 503302 1 1 1,99 1,99 8 5 8 8 9 10 *
fr France 65350000 101 131 1,55 2 12 12 11 11 11 11
au Australia 22607571 32 60 1,42 2,65 9 10 12 12 12 12
be Belgium 11071483 14 17 1,26 1,54 10 11 13 13 13 13
uk United-Kingdom 62698362 77 118 1,23 1,88 14 14 14 14 14 14
nl Netherlands 16728091 18 40 1,08 2,39 11 13 15 15 15 15
ca Canada 33476688 34 63 1,02 1,88 15 15 17 16 16 16
dk Denmark 5529888 5 10 0,9 1,81 17 17 16 17 17 17
es Spain 46754784 34 56 0,73 1,2 16 16 19 18 18 18
it Italy 59464644 36 52 0,61 0,87 23 22 22 19 19 19
hu Hungary 10076062 6 12 0,6 1,19 18 25 26 20 24 20 *
cz Czech Rep 10190213 6 6 0,59 0,59 21 20 21 21 20 21 *
us USA 313232044 175 382 0,56 1,22 19 21 25 24 22 22
il Israel 7740900 4 6 0,52 0,78 24 24 24 25 23 23
hr Croatia 4290612 2 2 0,47 0,47 20 18 18 26 25 24 *
lv Latvia 2204708 1 1 0,45 0,45 26 26 27 27 26 25 *
bg Bulgaria 7364570 3 3 0,41 0,41 25 23 23 23 27 26 *
sg Singapore 5183700 2 2 0,39 0,39


33 33 27 *
uy Uruguay 3477778 1 2 0,29 0,58 22 27 28 28 28 28
pl Poland 38441588 11 15 0,29 0,39 29 29 30 30 30 29 *
jp Japan 127078679 36 52 0,28 0,41 30 28 29 29 29 30 *
lt Lithuania 3535547 1 1 0,28 0,28 28 19 20 22 21 31 *
gr Greece 10787690 3 4 0,28 0,37 33 38 34 35 35 32 *
cr Costa Rica 4301712 1 1 0,23 0,23 31 30 31 31 31 33 *
by Belarus 9577552 2 2 0,21 0,21 35 36 39 39 32 34 *
ar Argentina 40677348 8 10 0,2 0,25 34 33 35 32 37 35 *
pt Portugal 10561614 2 4 0,19 0,38 27 32 32 34 34 36 *
sk Slovakia 5477038 1 1 0,18 0,18 32 31 33 36 36 37 *
rs Serbia 7186862 1 1 0,14 0,14



38 38
tw Taiwan 23040040 3 3 0,13 0,13 37 34 37 37 39 39
br Brazil 192376496 18 21 0,09 0,11 36 35 38 38 40 40
cu Cuba 11241161 1 1 0,09 0,09
38 41 41 41 41
co Colombia 45566856 4 5 0,09 0,11 41 44 46 47 46 42 *
kr South Korea 48754657 4 6 0,08 0,12 39 39 42 42 42 43 *
gt Guatemala 13824463 1 1 0,07 0,07



43 44 *
ec Ecuador 15007343 1 1 0,07 0,07
40 43 43 45 45
cl Chile 16746491 1 2 0,06 0,12 42 41 44 44 47 46 *
za South Africa 50590000 3 10 0,06 0,2 38 48 48 48 48 47 *
ru Russia 143030106 8 9 0,06 0,06 43 42 47 45 49 48 *
mg Madagascar 21281844 1 1 0,05 0,05 44 37 40 40 50 49 *
ro Romania 21904551 1 2 0,05 0,09 45 43 45 46 51 50 *
ve Venezuela 28047938 1 1 0,04 0,04 40 45 50 49 44 51 *
my Malaysia 28250000 1 1 0,04 0,04

49 50 52 52
pe Peru 29907003 1 1 0,03 0,03 46 46 51 51 53 53
tr Turkey 74724269 2 2 0,03 0,03 47 47 52 52 54 54
ua Ukraine 45134707 1 1 0,02 0,02 48 53 58 59 55 55
th Thailand 66720153 1 2 0,01 0,03 50 50 54 54 56 56
eg Egypt 80081093 1 3 0,01 0,04 51 51 55 55 57 57
mx Mexico 112336538 1 1 0,01 0,01 49 49 53 53 58 58
cn China 1344413526 10 14 0,01 0,01 53 53 57 56 59 59
in India 1210193422 8 9 0,01 0,01 52 52 56 57 60 60
sv El Salvador 7066403 0 1 0 0,14

36 58 61 61































969 1561 62,08%







A few interesting facts:

29 Jul 2014 4:34pm GMT

Russell Coker: Android Screen Saving

Just over a year ago I bought a Samsung Galaxy Note 2 [1]. About 3 months ago I noticed that some of the Ingress menus had burned in to the screen. Back in ancient computer times there were "screen saver" programs that blanked the screen to avoid this, then the "screen saver" programs transitioned to displaying a variety of fancy graphics which didn't really fulfill the purpose of saving the screen. With LCD screens I have the impression that screen burn wasn't an issue, but now with modern phones we have LED displays which have the problem again.

Unfortunately there doesn't seem to be a free screen-saver program for Android in the Google Play store. While I can turn the screen off entirely there are some apps such as Ingress that I'd like to keep running while the screen is off or greatly dimmed. Now I sometimes pull the notification menu down when I'm going to leave Ingress idle for a while, this doesn't stop the screen burning but it does cause different parts to burn which alleviates the problem.

It would be nice if apps were designed to alleviate this. A long running app should have an option to change the color of it's menus, it would be ideal to randomly change the color on startup. If the common menus such as the "COMM" menu would appear in either red, green, or blue (the 3 primary colors of light) in a ratio according to the tendency to burn (blue burns fastest so should display least) then it probably wouldn't cause noticable screen burn after 9 months. The next thing that they could do is to slightly vary the position of the menus, instead of having a thin line that's strongly burned into the screen there would be a fat line lightly burned in which should be easier to ignore.

It's good when apps have an option of a "dark" theme, that involves less light coming from the screen that should reduce battery use and screen burn. A dark theme should be at least default and probably mandatory for long running apps, a dark theme is fortunately the only option for Ingress.

I am a little disappointed with my phone. I'm not the most intensive Ingress player so I think that the screen should have lasted for more than 9 months before being obviously burned.

Related posts:

  1. Maintaining Screen Output In my post about getting started with KVM I noted...
  2. Android Device Service Life In recent years Android devices have been the most expensive...
  3. Cheap Android Tablet from Aldi I've just bought a 7″ Onix tablet from Aldi....

29 Jul 2014 12:28pm GMT

Russell Coker: Happiness and Lecture Questions

I just attended a lecture about happiness comparing Australia and India at the Australia India Institute [1]. The lecture was interesting but the "questions" were so bad that it makes a good case for entirely banning questions from public lectures. Based on this and other lectures I've attended I've written a document about how to recognise worthless questions and cut them off early [2].

As you might expect from a lecture on happiness there were plenty of stupid comments from the audience about depression, as if happiness is merely the absence of depression.

Then they got onto stupidity about suicide. One "question" claimed that Australia has a high suicide rate, Wikipedia however places Australia 49th out of 110 countries, that means Australia is slightly above the median for suicide rates per country. Given some of the dubious statistics in the list (for example the countries claiming to have no suicides and the low numbers reported by some countries with extreme religious policies) I don't think we can be sure that Australia would be above the median if we had better statistics. Another "question" claimed that Sweden had the highest suicide rate in Europe, while Greenland, Belgium, Finland, Austria, France, Norway, Denmark, Iceland, and most of Eastern Europe are higher on the list.

But the bigger problem in regard to discussing suicide is that the suicide rate isn't about happiness. When someone kills themself because they have a terminal illness that doesn't mean that they were unhappy for the majority of their life and doesn't mean that they were any unhappier than the terminally ill people who don't do that. Some countries have a culture that is more positive towards suicide which would increase the incidence, Japan for example. While people who kill themselves in Japan are probably quite unhappy at the time I don't think that there is any reason to believe that they are more unhappy than people in other countries who only keep living because suicide is considered to be wrong.

It seems to me that the best strategy when giving or MCing a lecture about a potentially contentious topic is to plan ahead for what not to discuss. For a lecture about happiness it would make sense to rule out all discussion of suicide, anti-depressants, and related issues as they aren't relevant to the discussion and can't be handled in an appropriate manner in question time.

Related posts:

  1. Length of Conference Questions After LCA last year I wrote about "speaking stacks" and...
  2. Questions During Lectures An issue that causes some discussion and debate is the...
  3. Ziggy's Lecture about Nuclear Power The Event I just attended a lecture by Dr Ziggy...

29 Jul 2014 10:57am GMT

Johannes Schauer: bootstrap.debian.net temporarily not updated

I'll be moving places twice within the next month and as I'm hosting the machine that generates the data, I'll temporarily suspend the bootstrap.debian.net service until maybe around September. Until then, bootstrap.debian.net will not be updated and retain the status as of 2014-07-28. Sorry if that causes any inconvenience. You can write to me if you need help with manually generating the data bootstrap.debian.net provided.

29 Jul 2014 8:46am GMT