12 Nov 2018

feedLXer Linux News

Avoiding senior moments with command-line functions

The trick is to make the documentation available on the CLI. Also, how to get a "yes" or "no" answer from grep.

12 Nov 2018 8:00pm GMT


Installing Vundle: The Plugin Manager For Vim

This guide will present a step-by-step guide of how to install and configure Vundle (Vim Bundle) from GitHub, and what issues you may face when installing Vundle.

12 Nov 2018 7:00pm GMT

feedLXer Linux News

Debian GNU/Linux 9.6 "Stretch" Released with Hundreds of Updates, Download Now

The Debian Project announced the general availability of the sixth point release to the latest stable Debian GNU/Linux 9 "Stretch" operating system series.

12 Nov 2018 6:46pm GMT


How to Mount S3/Wasabi/Digital Ocean Storage Bucket on CentOS and Ubuntu using S3FS

Learn how to use S3FS to mount cloud storage services locally on a CentOS or Ubuntu box.

12 Nov 2018 6:00pm GMT

feedLXer Linux News

Model the brain with the NEST simulator on Fedora

The latest version of the NEST simulator is now available in Fedora as part of the NeuroFedora initiative. NEST is a standard tool used by computational neuroscientists to make large scale computer models of the brain that are needed to investigate among other things, how the brain processes information. The NEST Eco-system NEST offers a […]

12 Nov 2018 5:32pm GMT


Stealth and hacking game 'OFF GRID' fully funded and heading to Linux

GamingOnLinux: OFF GRID, a stealth and hacking game about privacy and mass surveillance managed to get funded on Kickstarter and so it's coming to Linux.

12 Nov 2018 5:00pm GMT

feedLXer Linux News

System76 Thelio, Fedora 29, Solus Update, Samsung DeX, i3wm, ProtonDB | This Week in Linux 43

on TWinL43: Lutris, KDE Connect, openSUSE's Legal Review System, Sailfish 3.0, Cinnamon 4.0, GIMP, System76 Thelio, Fedora 29, Solus Update, Samsung DeX, i3wm, ProtonDB & much more!

12 Nov 2018 4:17pm GMT


VMware Buys Kubernetes-based Heptio to Boost Its Multi-Cloud Strategy

itprotoday: Another important open source company has been picked up by a big proprietary player.

12 Nov 2018 4:00pm GMT

feedLXer Linux News

A Trash-Bin for the Ubuntu Command Line

In this article, we will explain a few ways to safely remove files through the Ubuntu command line and also install a Trash CLI so that files can be recovered if we have deleted them by mistake.

12 Nov 2018 3:03pm GMT


How to Install Open Source Social Network (OSSN) on Ubuntu 18.04 LTS

HowToForge: Open Source Social Network (OSSN) is a free and open source social networking software written in PHP.

12 Nov 2018 3:00pm GMT

Import your files from closed or obsolete applications

An interview with Italo Vignoli of the Document Liberation Project.

12 Nov 2018 1:00pm GMT

feedLXer Linux News

Behind the scenes with Linux containers

Can you have Linux containers without Docker? Without OpenShift? Without Kubernetes?read more

12 Nov 2018 12:49pm GMT


Developers are the new kingdom builders

opensource.com: Today's developers aren't just kingmakers; thanks to blockchain, they're building their own kingdoms.

12 Nov 2018 12:00pm GMT

feedLXer Linux News

AsciiDoc – text document format for writing

AsciiDoc is a lightweight markup language for writing notes, documentation, articles, books, ebooks, slideshows, web pages, man pages and blogs. It's a plain text human readable/writable document format that dates back to 2002.

12 Nov 2018 11:28am GMT

7 Best free & Open source Linux Mint & Ubuntu music player

we have created this list of top Linux music player those work on both Ubuntu and Linux Mint... So, without further delay let's see the Top & Best free plus open-source Linux Mint and Ubuntu Music player.

12 Nov 2018 6:00am GMT

How To Install and Configure GitLab on CentOS 7

GitLab is a web-based open source Git repository manager written in Ruby including wiki, issue management, code review, monitoring and continuous integration and deployment. It enables developers to build,deploy and run their applications. This tutorial covers the steps necessary for installing and configuring GitLab (CE) on a CentOS 7 system using the Omnibus packages.

12 Nov 2018 2:11am GMT


7 reasons I love open source

Being a part of the open source community is a huge win for many reasons.

12 Nov 2018 2:00am GMT

11 Nov 2018


Install and Deploy Kubernetes on Ubuntu 18.04 LTS

Kubernetes is an open-source container management system for Docker containers.

11 Nov 2018 10:00pm GMT

feedLXer Linux News

Installing Vundle: The Plugin Manager For Vim

Today, this guide will present a step-by-step guide of how to install and configure Vundle (Vim Bundle) from GitHub, and what issues you may face when installing Vundle. As many of you may know, vim is a console-based text editor that has numerous advanced features. One such feature is that its functionality can be extended and customized using plugins written by other people. Managing these plugins, however, can be rather tedious. Vundle attempts to assist users in managing these plugins for you by providing an interface.

11 Nov 2018 8:22pm GMT

How to Install MariaDB on Ubuntu 16.04

MariaDB is a community-developed fork of MySQL. It is a free and open source, very fast, stable and scalable database server, which makes it one of the most used database servers in the world. It is an improved, drop-in replacement for popular MySQL Database Server. In this tutorial, we will show you how to install MariaDB on Ubuntu 16.04. Installing MariaDB on Ubuntu 16.04 is an easy task, and if you carefully follow the steps of this tutorial, you should have it installed on your system in less than 10 minutes. Let's get started.

11 Nov 2018 1:56pm GMT

How to install Apache Maven on CentOS 7

Apache Maven is a free and open source project management and comprehension tool used primarily for Java projects. Maven uses a Project Object Model (POM) which is essentially a XML file containing information about the project, configuration details, the project's dependencies, and so on. In this tutorial we will show you two different ways to install Apache Maven on CentOS 7.

11 Nov 2018 11:04am GMT

Snapdragon 2100 dev kit arrives as Fossil debuts smartwatch for new Snapdragon 3100

Intrinsyc has launched an Android-based Open-Q 2500 SOM module and development kit for smartwatches based on the Snapdragon 2500. Meanwhile, Fossil unveiled the first watch to run Wear OS on the next-gen Snapdragon 3100. Intrinsyc has followed up on last year's smartwatch oriented Open?Q 2100 SOM, featuring Qualcomm's Snapdragon Wear 2100 SoC, with a new […]

11 Nov 2018 8:13am GMT

Snapshot Games have cancelled the Linux version of Phoenix Point

Some news that I'm not particularly happy about. Snapshot Games, which includes X-COM creator Julian Gollop, have announced they've cancelled the Linux version of Phoenix Point.

11 Nov 2018 5:21am GMT

Linux-driven 96Boards SBC features AI and RISC-V companion chips

Bitmain announced a "Sophon BM1880 EDB" 96Boards CE SBC featuring its new Sophon BM1880 AI chip plus dual Cortex-A53 cores that run Linux. There's also a RISC-V chip and optional Raspberry Pi and Arduino modules. Beijing-based Bitmain, which is known primarily as a leading vendor of bitcoin mining chips and computers, also has a "Sophon" […]

11 Nov 2018 2:30am GMT

Paging Linux Users: What Made You Give Up on Windows?

Once described by Microsoft as "a cancer," the Linux world has become a key focus for Microsoft these days, especially as part of the company's transformation started shortly after Satya Nadella took over the CEO role from Steve Ballmer.

11 Nov 2018 12:35am GMT

07 Nov 2018

feedKernel Planet

Linux Plumbers Conference: Sold out LPC 2018 starts in a week — info for attendees

In just one week, the 2018 Linux Plumbers Conference will begin on November 13 with microconferences, a refereed track, Networking Summit track, Kernel Summit track, BoFs, and more. The conference is completely sold out at this point, sadly we cannot accommodate those on the waiting list. Below is some information for conference attendees.

We look forward to seeing all of the attendees in Vancouver next week …

Pick Up Your Badge Early:

Registration is located on the Junior Ballroom Foyer (North Tower Third Floor) of the Sheraton Wall Centre. Pre-registration will open Monday from 3:00pm to 5:00pm. General Registration will be open from 8:00am to 5:00pm Tuesday to Thursday.

View the Schedule:

Overview schedule
Detailed schedule

Please email contact@linuxplumbersconf.org if you'd like to request any changes.

Opening Reception:

Date: Tuesday, 13 November
Time: 6:30 - 9:30 pm
Location: Craft Beer Market, 85 W 1st Ave
Buses will depart from outside the North Tower from 6:30pm

Note: Craft Beer Market is at least 30 minutes walk from the hotel but is possible for the intrepid (and waterproof).

Closing Party at Blue Water Café:

Date: Thursday, 15 November
Time: 6:30 - 10:00 pm
Location: Blue Water Café: 1095 Hamilton St
Buses will depart from Outside the North Tower from 6:15pm

Note: The venue is 8 minutes walk from the Hotel for those who wish to brave the November weather

Lunch Details:

Lunch is on your own from 12:30 to 2:00 each day (though some microconferences may alter slightly). This year, we opted not to do lunch cards because of difficulties finding a Canadian card vendor and problems with currency conversion. The hotel does have an on-site restaurant in the North Tower, but it's too small for all our attendees. A map with nearby restaurants is available (and hard copies can be had at the registration desk). For the adventurous, Davie Street, as the centre of the Gay Pride neighbourhood, has a lot of interesting restaurants and bars (Google is pretty up to date) but beware, they're all rather small and some are cash only.

Venue Details:

Event Venue

1088 Burrard Street
British Columbia
V6Z 2R9

Hotel, Parking & Transportation

Please refer to the Hotel Web page.

07 Nov 2018 5:20pm GMT

04 Nov 2018

feedKernel Planet

Paul E. Mc Kenney: Book review: "Skin in the Game: Hidden Asymmetries in Daily Life"

"Antifragile" was the last volume in Nassim Taleb's Incerto series, but it has lost that distinction with the publication of "Skin in the Game: Hidden Asymmetries in Daily Life". This book covers a great many topics, but I will focus on only a few that relate most closely to my area of expertise.

Chapter 2 is titled "The Most Intolerant Wins: The Dominance of a Stubborn Minority". Examples include kosher and halal food, the English language (I plead guilty!!!), and many others besides. In all cases, if the majority is not overly inconvenienced by the strongly expressed needs or desires of the minority, the minority's preferences will prevail. On the one hand, I have no problem eating either kosher or halal food, so would be part of the compliant majority in that case. On the other hand, although I know bits and pieces of several languages, the only one I am fluent in is English, and I have attended gatherings where the language was English solely for my benefit. But there are limits. For example, if I were to attend a gathering in certain parts of (say) rural India or China, English might not be within the realm of possibility.

But what does this have to do with parallel programming???

This same stubborn-minority dominance appears in software, including RCU. Very few machines have more than a few tens of CPUs, but RCU is designed to accommodate thousands. Very few systems run workloads featuring aggressive real-time requirements, but RCU is designed to support low latencies (and even more so the variant of RCU present in the -rt patchset). Very few systems allow physical removal of CPUs while the systems is running, but RCU is designed to support that as well. Of course, as with human stubborn minorities, there are limits. RCU handles systems with a few thousand CPUs, but probably would not do all that well on a system with a few million CPUs. RCU supports deep sub-millisecond real-time latencies, but not sub-microsecond latencies. RCU supports controlled removal and insertion of CPUs, but not surprise removal or insertion.

Chapter 6 is titled Intellectual Yet Idiot (with the entertaining subtext "Teach a professor how to deadlift"), and, as might be expected from the title, takes a fair number of respected intellectual to task, for but two examples, Cass Sunstein and Richard Thaler. I did find the style of this chapter a bit off-putting, but I happened to read Michael Lewis's "The Undoing Project" at about the same time. This informative and entertaining book covers the work of Daniel Kahneman and Amos Tversky (whose work helped to inform that of Sunstein and Thaler), but I found the loss-aversion experiments to be unsettling. After all, what does losing (say) $100 really mean? That I will be sad for a bit? That I won't be able to buy that new book I was looking forward to reading? That I don't get to eat dinner tonight? That I go hungry for a week? That I starve to death? I just might give a very different answer in these different scenarios, mightn't I?

This topic is also covered by Jared Diamond in his most excellent book entitled "The World Until Yesterday". In the "Scatter your land" section, Diamond discusses how traditional farmers plant multiple small and widely separated plots of land. This practice puzzled anthropologists for some time, as it does the opposite of optimize yields and minimize effort. Someone eventually figured out that because these traditional farmers had no way to preserve food and limited opportunities to trade it, there was no value in producing more food than they could consume. But there was value in avoiding a year in which there was no food, and farming different crops in widely separated locations greatly decreased the odds that all their crops in all their plots would fail, thus in turn minimizing the probability of starvation. In short, these farmers were not optimizing for maximum average production, but rather for maximum probability of survival.

And this tradeoff is central to most of Taleb's work to date, including "Skin in the Game".

But what does this have to do with parallel programming???

Quite a bit, as it turns out. In theory, RCU should just run its state machine and be happy. In practice, there are all kinds of things that can stall its state machine, ranging from indefinitely preempted readers to long-running kernel threads refusing to give up the CPU to who knows what all else. RCU therefore contains numerous forward-progress checks that reduce performance slightly but which also allow RCU to continue working when the going gets rough. This sort of thing is baked even more deeply into the physical engineering disciplines in the form of the fabled engineering factor of safety. For example, a bridge might be designed to handle three times the heaviest conceivable load, thus perhaps surviving a black-swan event such as a larger-than-expected earthquake or tidal wave.

Returning to Skin in the Game, Taleb makes much of the increased quality of decisions when the decider is directly affected by them, and rightly so. However, I became uneasy about cases where the decision and effect are widely separated in time. Taleb does touch obliquely on this topic in a section entitled "How to Put Skin in the Game of Suicide Bombers", but does not address this topic in more prosaic settings. One could take a survival-based approach, arguing that tomorrow matters not unless you survive today, but in the absence of a very big black swan, a large fraction of the people alive today will still be alive ten years from now.

But what does this have to do with parallel programming???

There is a rather interesting connection, especially when you consider that Linux-kernel RCU's useful lifespan probably exceeds my own. This is not a new thought, and is in fact why I have put so much energy into speaking and writing about RCU. I also try my best to make RCU able to stand up to whatever comes its way, with varying degrees of success over the years.

However, beyond a certain point, this practice is labeled "overengineering", which is looked down upon within the Linux kernel community. And with good reason: Many of the troubles one might foresee will never happen, and so the extra complexity added to deal with those troubles will provide nothing but headaches for no benefit. In short, my best strategy is to help make sure that there are bright, capable, and motivated people to look after RCU after I am gone. I therefore intend to continue writing and speaking about RCU. :-)

04 Nov 2018 3:54am GMT

30 Oct 2018

feedKernel Planet

Pete Zaitcev: Where is Amazon?

Imagine, purely hypothetically, that you were a kernel hacker working for Red Hat and for whatever reason you wanted to find a new challenge at a company with a strong committment to open source. What are the possibilities?

To begin with, as the statistics from the Linux Foundation's 2016 report demonstrate, you have to be stark raving mad to leave Red Hat. If you do, Intel and AMD look interesting (hello, Alan Cox). IBM is not bad, although since yesterday, you don't need to quit Red Hat to work for IBM anymore. Even Google, famous for being a black hole that swallows good hackers who are never heard from again, manages to put up a decent showing, Fuchsia or no. Facebook looks unimpressive (no disrespect to DaveJ intended).

Now, the no-shows. Both of them hail from Seattle, WA: Microsoft and Amazon. Microsoft made an interesting effort to adopt Linux into its public cloud, but their strategy was to make Red Hat do all the work. Well, as expected. Amazon, though, is a problem. I managed to get into an argument with David "dwmw2" Woodhouse on Facebook about it, where I brought up a somewhat dated article at The Register. The central claim is, the lack of Amazon's contribution is the result of the policy rolled all the way from the top.

(...) as far as El Reg can tell, the internet titan has submitted patches and other improvements to very few projects. When it does contribute, it does so typically via a third party, usually an employee's personal account that is not explicitly linked to Amazon.

I don't know if this culture can be changed quickly, even if Bezos suddenly changes his mind.

30 Oct 2018 3:26am GMT

25 Oct 2018

feedKernel Planet

Davidlohr Bueso: Linux v4.19: Performance Goodies

This post marks one year since I began doing these kernel performance goodies write ups, starting from v4.14. And this week Greg released Linux v4.19, so here are some of the changes related to software optimizations, performance and scalability topics across various subsystems.

epoll: loosen irq safety when possible

The epoll code uses an irq-safe spinlock to protect concurrent operations to the ready-event linked list. However, with the exception of the callback done from the wakequeues, the calls to the spinlock are never done in irq context, and therefore there is really no need to save and restore interrupts each time the lock is acquired and released. For example, on x86, a POPF (irqrestore) instruction can be quite expensive as it changes all the flags and therefore potentially heavy on dependencies. These changes yield some measurable results on a range of epoll_wait(2) microbenchmarks, around 7-20% in raw throughput. This is unsurprising as PUSHF + POPF is more expensive than STI + CLI.
[Commit 002b343669c4, 304b18b8d6af, 92e641784055, 679abf381a18]

sched/numa: migrate pages to local nodes quicker early in the lifetime of a task

Automatic NUMA Balancing uses a multi-stage pass to decide whether a page should migrate to a local node. This filter avoids excessive ping-ponging if a page is shared or used by threads that migrate cross-node frequently. Threads inherit both page tables and the preferred node ID from the parent. This means that threads can trigger hinting faults earlier than a new task which delays scanning for a number of seconds. As it can be load balanced very early in its lifetime there can be an unnecessary delay before it starts migrating thread-local data. This patch migrates private pages faster early in the lifetime of a thread using the sequence counter as an identifier of new tasks.
[Commit 37355bdc5a12]

rcu: check if GP already requested

This commit makes rcu_nocb_wait_gp() check to see if the current CPU already knows about the needed grace period having already been requested. If so, it avoids acquiring the corresponding leaf rcu_node structure's lock, thus decreasing contention. This optimization is intended for cases where either multiple leader rcu kthreads are running on the same CPU or these kthreads are running on a non-offloaded (e.g., housekeeping) CPU.
[Commit ab5e869c1f7a]

cpufreq/schedutil: take into account time spent in irq

Time being spent in interrupt handlers was not being accounted for in the CPU utilization when selecting an operating performance point. This can be a significant amount of time which is reported in the normal context time window. The new CPU utilization is yields a 10% performance boost on iperf workloads.
[Commit 9033ea11889f]

mm/page_alloc: enlarge zone's batch size

The page allocator will first try to use a percpu set of pages, then if all used up, ask the Buddy for a batch of pages. The size of this batch can have a number of consequences, including performance. The last time this magic number was increased was 13 years ago, and there have been numerous hardware improvements since then. As such a recent study with allocator intensive benchmarks, shows that doubling the size of the batch can yield improvements on larger/modern machines.
[Commit d8a759b57035]

mm: skip invalid pages block at a time in zero_resv_unresv()

The role of zero_resv_unavail() is to make sure that every struct page that is allocated but is not backed by memory that is accessible by kernel is zeroed and not in some uninitialized state. Since struct pages are allocated in blocks we can skip pageblock_nr_pages at a time, when the first one is found to be invalid. This optimization may help since now on x86 every hole in e820 maps is marked as reserved in memblock, and thus will go through this function.
[Commit 720e14ebec64]

kvm, x86: implement paravirt "send IPI" hypercall

Replace sending IPIs one by one for xAPIC physical mode by a single hypercall (vmexit). This patchset lets a guest send multicast IPIs, with at most 128 destinations per hypercall in 64-bit mode and 64 vCPUs per hypercall in 32-bit mode. An IPI microbenchmark shows non-trivial performance improvements for broadcast IPIs (send IPI to all online CPUs and force them to take/drop a spinlock).
[Commit 4180bf1b655a]

arm64: use queued spinlocks

Similar to x86, replace the old ticket spinlocks with fair qspinlocks and make use of MCS features as well as better performance under virtualization. This is particularly suitable for larger multicore machines.
[Commit c11090474d70]

25 Oct 2018 6:19pm GMT

22 Oct 2018

feedKernel Planet

Kees Cook: security things in Linux v4.19

Previously: v4.18.

Linux kernel v4.19 was released today. Here are some security-related things I found interesting:

L1 Terminal Fault (L1TF)

While it seems like ages ago, the fixes for L1TF actually landed at the start of the v4.19 merge window. As with the other speculation flaw fixes, lots of people were involved, and the scope was pretty wide: bare metal machines, virtualized machines, etc. LWN has a great write-up on the L1TF flaw and the kernel's documentation on L1TF defenses is equally detailed. I like how clean the solution is for bare-metal machines: when a page table entry should be marked invalid, instead of only changing the "Present" flag, it also inverts the address portion so even a speculative lookup ignoring the "Present" flag will land in an unmapped area.

protected regular and fifo files

Salvatore Mesoraca implemented an O_CREAT restriction in /tmp directories for FIFOs and regular files. This is similar to the existing symlink restrictions, which take effect in sticky world-writable directories (e.g. /tmp) when the opening user does not match the owner of the existing file (or directory). When a program opens a FIFO or regular file with O_CREAT and this kind of user mismatch, it is treated like it was also opened with O_EXCL: it gets rejected because there is already a file there, and the kernel wants to protect the program from writing possibly sensitive contents to a file owned by a different user. This has become a more common attack vector now that symlink and hardlink races have been eliminated.

syscall register clearing, arm64

One of the ways attackers can influence potential speculative execution flaws in the kernel is to leak information into the kernel via "unused" register contents. Most syscalls take only a few arguments, so all the other calling-convention-defined registers can be cleared instead of just left with whatever contents they had in userspace. As it turns out, clearing registers is very fast. Similar to what was done on x86, Mark Rutland implemented a full register-clearing syscall wrapper on arm64.

Variable Length Array removals, part 3

As mentioned in part 1 and part 2, VLAs continue to be removed from the kernel. While CONFIG_THREAD_INFO_IN_TASK and CONFIG_VMAP_STACK cover most issues with stack exhaustion attacks, not all architectures have those features, so getting rid of VLAs makes sure we keep a few classes of flaws out of all kernel architectures and configurations. It's been a long road, and it's shaping up to be a 4-part saga with the remaining VLA removals landing in the next kernel. For v4.19, several folks continued to help grind away at the problem: Arnd Bergmann, Kyle Spiers, Laura Abbott, Martin Schwidefsky, Salvatore Mesoraca, and myself.

shift overflow helper
Jason Gunthorpe noticed that while the kernel recently gained add/sub/mul/div helpers to check for arithmetic overflow, we didn't have anything for shift-left. He added check_shl_overflow() to round out the toolbox and Leon Romanovsky immediately put it to use to solve an overflow in RDMA.

Edit: I forgot to mention this next feature when I first posted:

trusted architecture-supported RNG initialization

The Random Number Generator in the kernel seeds its pools from many entropy sources, including any architecture-specific sources (e.g. x86's RDRAND). Due to many people not wanting to trust the architecture-specific source due to the inability to audit its operation, entropy from those sources was not credited to RNG initialization, which wants to gather "enough" entropy before claiming to be initialized. However, because some systems don't generate enough entropy at boot time, it was taking a while to gather enough system entropy (e.g. from interrupts) before the RNG became usable, which might block userspace from starting (e.g. systemd wants to get early entropy). To help these cases, Ted T'so introduced a toggle to trust the architecture-specific entropy completely (i.e. RNG is considered fully initialized as soon as it gets the architecture-specific entropy). To use this, the kernel can be built with CONFIG_RANDOM_TRUST_CPU=y (or booted with "random.trust_cpu=on").

That's it for now; thanks for reading. The merge window is open for v4.20! Wish us luck. :)

© 2018, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

22 Oct 2018 11:17pm GMT

16 Oct 2018

feedKernel Planet

Matthew Garrett: Initial thoughts on MongoDB's new Server Side Public License

MongoDB just announced that they were relicensing under their new Server Side Public License. This is basically the Affero GPL except with section 13 largely replaced with new text, as follows:

If you make the functionality of the Program or a modified version available to third parties as a service, you must make the Service Source Code available via network download to everyone at no charge, under the terms of this License. Making the functionality of the Program or modified version available to third parties as a service includes, without limitation, enabling third parties to interact with the functionality of the Program or modified version remotely through a computer network, offering a service the value of which entirely or primarily derives from the value of the Program or modified version, or offering a service that accomplishes for users the primary purpose of the Software or modified version.

"Service Source Code" means the Corresponding Source for the Program or the modified version, and the Corresponding Source for all programs that you use to make the Program or modified version available as a service, including, without limitation, management software, user interfaces, application program interfaces, automation software, monitoring software, backup software, storage software and hosting software, all such that a user could run an instance of the service using the Service Source Code you make available.

MongoDB admit that this license is not currently open source in the sense of being approved by the Open Source Initiative, but say:We believe that the SSPL meets the standards for an open source license and are working to have it approved by the OSI.

At the broadest level, AGPL requires you to distribute the source code to the AGPLed work[1] while the SSPL requires you to distribute the source code to everything involved in providing the service. Having a license place requirements around things that aren't derived works of the covered code is unusual but not entirely unheard of - the GPL requires you to provide build scripts even if they're not strictly derived works, and you could probably make an argument that the anti-Tivoisation provisions of GPL3 fall into this category.

A stranger point is that you're required to provide all of this under the terms of the SSPL. If you have any code in your stack that can't be released under those terms then it's literally impossible for you to comply with this license. I'm not a lawyer, so I'll leave it up to them to figure out whether this means you're now only allowed to deploy MongoDB on BSD because the license would require you to relicense Linux away from the GPL. This feels sloppy rather than deliberate, but if it is deliberate then it's a massively greater reach than any existing copyleft license.

You can definitely make arguments that this is just a maximalist copyleft license, the AGPL taken to extreme, and therefore it fits the open source criteria. But there's a point where something is so far from the previously accepted scenarios that it's actually something different, and should be examined as a new category rather than already approved categories. I suspect that this license has been written to conform to a strict reading of the Open Source Definition, and that any attempt by OSI to declare it as not being open source will receive pushback. But definitions don't exist to be weaponised against the communities that they seek to protect, and a license that has overly onerous terms should be rejected even if that means changing the definition.

In general I am strongly in favour of licenses ensuring that users have the freedom to take advantage of modifications that people have made to free software, and I'm a fan of the AGPL. But my initial feeling is that this license is a deliberate attempt to make it practically impossible to take advantage of the freedoms that the license nominally grants, and this impression is strengthened by it being something that's been announced with immediate effect rather than something that's been developed with community input. I think there's a bunch of worthwhile discussion to have about whether the AGPL is strong and clear enough to achieve its goals, but I don't think that this SSPL is the answer to that - and I lean towards thinking that it's not a good faith attempt to produce a usable open source license.

(It should go without saying that this is my personal opinion as a member of the free software community, and not that of my employer)

[1] There's some complexities around GPL3 code that's incorporated into the AGPLed work, but if it's not part of the AGPLed work then it's not covered

comment count unavailable comments

16 Oct 2018 10:44pm GMT

15 Oct 2018

feedKernel Planet

Davidlohr Bueso: Linux v4.18: Performance Goodies

Linux v4.18 has been out a two months now; making this post a bit late, but still in time before the next release. Also so much drama in the CoC to care about performance topics :P As always comes with a series of performance enhancements and optimizations across subsystems.

locking: avoid pointless TEST instructions

A number of places within locking primitives have been optimized to avoid superfluous test instructions for the CAS return by relying on try_cmpxchg, generating slightly better code for x86-64 (for arm64 there is really no difference). Such have been the cases for mutex fastpath (uncontended case) and queued spinlocks.
[Commit c427f69564e2, ae75d9089ff7]

locking/mcs: optimize cpu spinning

Some architectures, such as arm64, can enter low-power standby state (spin-waiting) instead of purely spinning on a condition. This is applied to the MCS spin loop, which in turn directly helps queued spinlocks. On x86, this can also be cheaper than spinning on smp_load_acquire().
[Commit 7f56b58a92aa]

mm/mremap: reduce amount of TLB shootdowns

It was discovered that on a heavily dominated mremap workload, the amount of TLB flushes was excessive causing overall performance issues. By removing the LATENCY_LIMIT magic number to handle TLB flushes on a PMD boundary instead of every 64 pages, the amount of shootdowns can be redced by a factor of 8 in the ideal case. The LATENCY_LIMIT was almost certainly used originally to limit the PTL hold times but the latency savings are likely shadowed by the cost of IPIs in many cases.
[Commit 37a4094e828f]

mm: replace mmap_sem to protect cmdline and environ procfs files

Reducing (ab)users of the mmap_sem is always good for general address space performance. Introduce a new mm->arg_lock to protect against races when handling /proc/$PID/{cmdline,environ} files, this removes (mostly) the semaphore's requirements.
[Commit 88aa7cc688d4]

mm/hugetlb: make better use of page clearing optimization

Pass the fault address (address of the sub-page to access) to the nopage fault handler to better use the general huge page clearing optimization. This allows the sub-page to access to be cleared last to avoid the cache lines of to access sub-page to be evicted when clearing other sub-pages. Performance improvements were reported for vm-scalability.anon-w-seq workload under hugetlbfs, reducing ~30% throughput.
[Commit 285b8dcaacfc]

sched: don't schedule threads on pre-empted vCPUs

It can be determined whether a vCPU is running to prioritize CPUs when scheduling threads. If a vCPU has been pre-empted, it will incur the extra cost of VMENTER and the time it actually spends to be running on the host CPU. If we had other vCPUs which were actually running on the host CPU and idle we should schedule threads there.
[Commit 247f2f6f3c70, 943d355d7fee]

sched/numa: Stagger NUMA balancing scan periods for new threads

It is redundant and counter productive for threads sharing an address space to change the protections to trap NUMA faults. Potentially only one thread is required but that thread may be idle or it may not have any locality concerns and pick an unsuitable scan rate. This patch uses independent scan period but they are staggered based on the number of address space users when the thread is created.

The intent is that threads will avoid scanning at the same time and have a chance to adapt their scan rate later if necessary. This reduces the total scan activity early in the lifetime of the threads. The different in headline performance across a range of machines and workloads is marginal but the system CPU usage is reduced as well as overall scan activity.
[Commit 137844759843]

block/bfq: postpone rq preparation to insert or merge

A lock contention point is removed (see patch for details and justification) by postponing request preparation to insertion or merging, as lock needs to be grabbed any longer in the prepare_request hook.
[Commit 18e5a57d7987]

btrfs: improve rmdir performance for large directories

When checking if a directory can be deleted, instead of ensuring all its children have been processed, this optimization keeps track of the directory index offset of the child last checked in the last call to can_rmdir(), and then use it as the starting point for future calls. The changes were shown to yield massive performance benefits; for test directory with two million files being deleted the runtime is reduced from half an hour to less than two seconds.
[Commit 0f96f517dcaa]

KVM: VMX: Optimize tscdeadline timer latency

Add the advance tscdeadline expiration support to which the tscdeadline timer is emulated by VMX preemption timer to reduce the hypervisor lantency (handle_preemption_timer -> vmentry). The guest can also set an expiration that is very small in that case we set delta_tsc to 0, leading to an immediately vmexit when delta_tsc is not bigger than advance ns. This patch can reduce ~63% latency for kvm-unit-tests/tscdeadline_latency when testing busy waits.
[Commit c5ce8235cffa]

net/sched: NOLOCK qdisc performance enhancements and fixes

There have been various performance related core changes to the NOLOCK qdisc code. The first begins with reducing the atomic operations of __QDISC_STATE_RUNNING. The bit is flipped twice per packet in the uncontended scenario with packet rate below the line rate: on packed dequeue and on the next, failing dequeue attempt. The changes simplify the qdisc. The changes moves the bit manipulation into the qdisc_run_{begin,end} helpers, so that the bit is now flipped only once per packet, with measurable performance improvement in the uncontended scenario.

Later, the above is actually replaced by using a sequence spinlock instead of the atomic approach address pfifo_fast performance regressions. There is also a reduction in the Qdisc struct memory footprint (spanning a cacheline less).
[Commit 96009c7d500e, 021a17ed796b, e9be0e993d95]

lib/idr: improve scalability by reducing IDA lock granularity

Improve the scalability of the IDA by using the per-IDA xa_lock rather than the global simple_ida_lock. IDAs are not typically used in performance-sensitive locations, but since we have this lock anyway, we can use it.
[Commit b94078e69533]

x86-64: micro-optimize __clear_put()

Use immediate constants and saves two registers.
[Commit 1153933703d9]


It is probably safe to assume that all Armv8-A implementations have a multiplier whose efficiency is comparable or better than a sequence of three or so register-dependent arithmetic instructions. Select ARCH_HAS_FAST_MULTIPLIER to get ever-so-slightly nicer codegen in the few dusty old corners which care.
[Commit e75bef2a4fe2]

15 Oct 2018 8:19pm GMT

11 Oct 2018

feedKernel Planet

Pete Zaitcev: I'd like to interject for a moment

In a comment on the death of G+, elisteran brought up something that long annoyed me out of all proportion with its actual significance. What do you call a collection of servers communicating through NNTP? You don't call them "INN", you call them "Usenet". The system of hosts communicating through SMTP is not called "Exim", it is called "e-mail". But when someone wants to escape G+, they often consider "Mastodon". Isn't it odd?

Mastodon is merely an implementation of Fediverse. As it happens, only one of my Fediverse channels runs on Mastodon (the Japanese language one at Pawoo). Main one still uses Gnusocial, the anime one was on Gnusocial and migrated to Pleroma a few months ago. All of them are communicating using the OStatus protocol, although a movement is afoot to switch to ActivityPub. Hopefully it's more successful than the migration from RSS to Atom was.

Yet, I noticed that a lot of people fall to the idea that Mastodon is an exclusive brand. Rarely one has to know or care what MTA someone else uses. Microsoft was somewhat successful in establishing Outlook as such a powerful brand to the exclusion of the compatible e-mail software. The maintainer of Mastodon is doing his hardest to present it as a similar brand, and regrettably, he's very successful at that.

I guess what really drives me mad about this is how Eugen uses his mindshare advanage to drive protocol extensions. All of Fediverse implementations generaly communicate freely with one another, but as Pleroma and Mastodon develop, they gradually leave Gnusocial behind in features. In particular, Eugen found a loophole in the protocol, which allows to attach pictures without using up the space in the message for the URL. When Gnusocial displays a message with attachment, it only displays the text, not the picture. This acutally used to be a server setting, in case you want to safe your instance from NSFW imagery and save a little bandwidth. But these days pictures are so prevalent, that it's pretty much impossible to live without receiving them. In this, Eugen has completed the "extend" phase and is moving onto "extinguish".

I'm not sure if this a lost cause by now. At least I hope that members of my social circle migrate to Fediverse in general, and not to Mastodon from the outset. Of course, the implementation does matter when they make choices. As I mentioned, for anything but Linux discussions, pictures are essential, so one cannot reasonably use a Gnusocial instance for anime, for example. And, I can see some users liking Mastodon's UI. And, Mastodon's native app support is better (or not). So yes, by all means, if you want to install Mastodon, or join an instance that's running Mastodon, be my guest. Just realize that Mastodon is an implementation of Fediverse and not the Fediverse itself.

11 Oct 2018 1:16pm GMT

09 Oct 2018

feedKernel Planet

Pete Zaitcev: Ding-dong, the witch is dead

Reactions by G+ inhabitants were better than expected at times. Here's Jon Masters:

For the three people who care about G+: it's closing down. This is actually a good thing. If you work in kernel or other nerdy computery circles and this is your social media platform, I have news for you...there's a world outside where actual other people exist. Try it. You can then follow me on Twitter at @jonmasters when you get bored.

Rock on. Although LJ was designed as a shitty silo, it wasn't powerful enough to make itself useless. For example, outgoing links aren't limited. That said, LJ isn't bulletproof: the management is pushing the "new" editor that does not allow HTML. The point is though, there's a real world out there.

And, some people are afraid of it, and aren't ashamed to admit it. Here's Steven Rostedt in Jon's comments:

In other words, we are very aware of the world outside of here. This is where we avoided that world ;-)

So weak. Jon is titan among his entourage.

Kir enumerated escape plans thus (in my translation):

Where to run, unclear. Not wanting to Facebook, Telegram is kinda a marginal platform (although Google+ marginal too), too lazy to stand up a standalone. Nothing but LJ comes to mind.

One thing that comes across very strongly is how reluctant people are to run their own infrastructure. For one thing, the danger of a devastating DDoS is absolutely real. And then you have to deal with spam. Those who do not have the experience also tend to over-estimate the amount of effort you have to put into running "dnf update" once in a while.

Personally, I think that although of course it's annoying, the time wasted on the infra is not that great, or at least it wasn't for me. The spam can be kept under control with a minimal effort. Or, could be addressed in drastic ways. For example, my anime blog simply does not have comments at all. As far as DoS goes, yes, it's a lottery. But then the silo platform can easily die (like G+), or ban you. This actually happens a lot more than those hiding their heads in the sand like to admit. And you don't need to go as far as to admit to your support of President Trump in order to get banned. Anything can trigger it, and the same crazies that DoS you will also try to deplatform you.

One other idea I was very successful with, and many people have trouble accepting, is having several channels for social posting (obviously CKS was ahead of the time with separating pro and hobby). Lots and lots of G+ posters insist on dumping all the garbage into one bin, instead of separating the output. Perhaps now they'll find a client or device that allows them switch accounts easily.

09 Oct 2018 2:46pm GMT

07 Oct 2018

feedKernel Planet

Pete Zaitcev: Python and journalism

Back in July, Economist wanted to judge a popularity of programming languages and used ... Google Trends. Python is rocketing up, BTW. Go is not even mentioned.

07 Oct 2018 8:04pm GMT

04 Oct 2018

feedKernel Planet

Andy Grover: Stratis 1.0 released!

We just tagged Stratis 1.0.

I can't believe I haven't blogged about Stratis before, although I've written in other places about it. We've been working on it for two years.

Basically, it's a fancy manager of device-mapper and XFS configuration, to provide a similar experience as ZFS and Btrfs, but completely different under the hood.

Four things that took the most development time (so far)
  1. Writing the design doc. Early on, much of the work was convincing people the approach we wanted was a good one. We spent a lot of time discussing details among ourselves and winning over internal stakeholders (or not), but most of all, showing that we had given serious thought to various alternatives, and had spent some time to comprehend the consequences of initial design choices. Having the design doc made these discussions easier, and solicited feedback that resulted in a much better design than what we started with.
  2. Implementing on-disk metadata formats and algorithms to protect maximally against corruption and over-write. People said it would take more time than we thought and they…weren't wrong! I still think implementing this was the right call, however.
  3. The hordes of range lists Stratis manages internally. It was probably inevitable that using multiple device-mapper layers involves a lot of range mapping. Stratis does a lot of it now, and it will be doing way more in the future, once we start using DM devices like integrity, raid, and compression. Rust really came through for us here I think. Rust's functional aspects work very well for things like mapping and allocating.
  4. The D-Bus interface was a big effort in the pre-0.5 timeframe, but now that it is up and running it's easy to maintain and update. We owe much of this to the quality of the dbus-rs library, and the receptivity of its author, diwic, to help us understand how to use it, and also helping to add small bits that aided our usage of D-Bus.
People to thank

Thanks to Igor Gnatenko and Josh Stone, two people who played a large part in making Rust on Fedora a reality. When I started writing the prototype for Stratis, this was a big question mark! I just hoped that the value of Rust would ensure that sooner or later Rust would be supported on Fedora and RHEL, and thanks to these two (and others, and, oh, you know, Firefox needing it…) it worked out.

I'd also like to thank the Rust community, for making such a compelling, productive systems language through friendliness and respect, sweating the details, and sharing! Like I alluded to before, Rust's functional style was a good match for our problem space, and Rust's intense focus on error handling also was perfect for a critical piece of software like stratisd, where what to do about errors is the most important part of what it does.

Finally, I'd like to thank the other members of the Stratis core team: Todd, Mulhern, and Tony. Stratis 1.0 is immeasurably better because of the different backgrounds and strengths we each brought to bear on developing this new piece of software. Thanks, everybody. You made 1.0 happen.

The Future

The 1.0 release marks the end of the beginning, so to speak. We just left the Shire, Frodo! Stratis is a viable product, but there's so much more to do. Integrating more high-value device-mapper layers, more integration with other storage APIs (both "above" and "below"), more flexibility around adding and removing storage devices, while keeping the UI clean and the admin work low, is the challenge.

Stratis is going to need some major help to get there. For people interested in doing development, testing, packaging, or using Stratis, I invite you to visit our website and GitHub, or just keep tabs by following the project on Google Plus or Twitter.

04 Oct 2018 10:44pm GMT

02 Oct 2018

feedKernel Planet

Linux Plumbers Conference: 2018 Linux Plumbers Conference is almost completely full

Due to overwhelming demand for tickets to the Linux Plumbers Conference, there are no additional registrations available at this time.

As we finalize the makeup of microconferences, refereed talks, and so on, there will be some spots available. We will be making them available to those who have expressed interest as fairly as we can and as soon as we can. We plan to contact the recipients of the first batch of released slots by October 8. There may be another, likely smaller, batch notified thereafter.

Those interested in attending the conference, should send a request to contact@linuxplumbersconf.org to get on the waiting list. In the unlikely event that the waiting list has been exhausted, we will release any remaining registrations on a first-come-first-served basis by mid-late October.

LPC [1] will be held in Vancouver, British Columbia, Canada from Tuesday, November 13 through Thursday, November 15.

[1] https://linuxplumbersconf.org/

02 Oct 2018 9:34pm GMT

Linux Plumbers Conference: CLANG/GCC/GLIBC Toolchain Microconference Accepted into 2018 Linux Plumbers Conference

The interaction between toolschain components such as GCC, GLIBC, and CLANG/LLVM with the Linux kernel and with underlying hardware has evolved rapidly. The corresponding communities continue to push on the limits of what is possible, due to new silicon as well as the performance and security changes of the past year.

Specific topics include support for control-flow enforcement technologies (CET), loop-nest optimization flag changes, optimized x86_64 math functions, unified API for new ports, emulation fallback for system calls, handling deprecated kernel support (such as PowerPC HTM support), building the Linux kernel with CLANG, and ARMv8.5 features.

If you would like to contribute to this discussion, please feel free to contact Victor Rodriguez (vm.rod25atgmail.com), H.J. Lu (hjl.toolsatgmail.com), Adhemerval Zanella (adhemerval.zanellaatlinaro.org), David Edelsohn (dje.gccatgmail.com), or Siddhesh Poyarekar (siddheshatgotplt.org).

We hope to see you there!

02 Oct 2018 3:46am GMT

27 Sep 2018

feedKernel Planet

James Morris: 2018 Linux Security Summit North America: Wrapup

The 2018 Linux Security Summit North America (LSS-NA) was held last month in Vancouver, BC.

Attendance continued to grow this year, with a record of 220+ attendees. Our room was upgraded as a result, with spectacular views.

LSS-NA 2018 Vancouver BC

Linux Security Summit NA 2018, Vancouver,BC

We also had many great proposals and the schedule ended up being a very tight fit. We've asked for an extra day for LSS-NA next year - here's hoping.

Slides of all presentations are available here: https://events.linuxfoundation.org/events/linux-security-summit-north-america-2018/program/slides/

Videos may be found in this youtube playlist.

Once again, as is typical, the conference was focused around development, somewhat uniquely in the world of security conferences. It's interesting to see more attention seemingly being paid to the lower parts of the stack: secure booting, firmware, and hardware roots of trust, as well as the continued efforts in hardening the kernel.

LWN provided some excellent coverage of LSS-NA:

Paul Moore has a brief writeup here.

Thanks to everyone involved in the event for 2018: the speakers, attendees, the program committee, the sponsors, and the organizing team at the Linux Foundation. LSS-NA would not be possible without all of you!

27 Sep 2018 8:06pm GMT

26 Sep 2018

feedKernel Planet

Pete Zaitcev: Postres vs MySQL

Unexpectedly in the fediverse:

[...] in my experience, postgres crashes less, and the results are less devastating if it does crash. I've had a mysql crash make the data unrecoverable. On the other hand I have a production postgres 8.1 installation (don't ask) that has been running without problems for over 10 years.

There is more community information and more third-party tools that require mysql, it has that advantage. the client tools for mysql are easier to use because the commands are in plain english ("show tables") unlike postgres that are commands like "\dt+". but if I'm doing my own thing though, I use postgres.

Reiser, move over. There's a new game in town.

26 Sep 2018 1:15am GMT

25 Sep 2018

feedKernel Planet

Linux Plumbers Conference: Regular Registration Quota Reached

Thank you all for the extremely strong interest in participation to the 2018 Linux Plumbers Conference this year.

At this point, all of the regular registration slots for LPC 2018 have sold out.

There will be a very limited number of registrations available on a first come first serve basis going forward.

Those interested in attending the conference, should send a request to contact@linuxplumbersconf.org to get on the waiting list.

We will process people as quickly as possible as slots initially allocated to sponsors, microconferences and speakers get released.

25 Sep 2018 8:52pm GMT