03 Jan 2020

feedPlanet Gentoo

Gentoo News: FOSDEM 2020

FOSDEM logo

It's FOSDEM time again! Join us at Université libre de Bruxelles, Campus du Solbosch, in Brussels, Belgium. This year's FOSDEM 2020 will be held on February 1st and 2nd.

Our developers will be happy to greet all open source enthusiasts at our Gentoo stand in building K where we will also celebrate 20 years compiling! Visit this year's wiki page to see who's coming.

03 Jan 2020 12:00am GMT

28 Dec 2019

feedPlanet Gentoo

Alexys Jacob: Scylla Summit 2019

I've had the pleasure to attend again and present at the Scylla Summit in San Francisco and the honor to be awarded the Most innovative use case of Scylla.

It was a great event, full of friendly people and passionate conversations. Peter did a great full write-up of it already so I wanted to share some of my notes instead…

This a curated set of topics that I happened to question or discuss in depth so this post is not meant to be taken as a full coverage of the conference.

Scylla Manager version 2

The upcoming version of scylla-manager is dropping its dependency on SSH setup which will be replaced by an agent, most likely shipped as a separate package.

On the features side, I was a bit puzzled by the fact that ScyllaDB is advertising that its manager will provide a repair scheduling window so that you can control when it's running or not.

Why did it struck me you ask?

Because MongoDB does the same thing within its balancer process and I always thought of this as a patch to a feature that the database should be able to cope with by itself.

And that database-do-it-better-than-you motto is exactly one of the promises of Scylla, the boring database, so smart at handling workload impacts on performance that you shouldn't have to start playing tricks to mitigate them… I don't want this time window feature on scylla-manager to be a trojan horse on the demise of that promise!

Kubernetes

They almost got late on this but are working hard to play well with the new toy of every tech around the world. Helm charts are also being worked on!

The community developed scylla operator by Yannis is now being worked on and backed by ScyllaDB. It can deploy, scale up and down a cluster.

Few things to note:

Change Data Capture

Oh boy this one was awaited… but it's now coming soon!

I inquired about it's performance impact since every operation will be written to a table. Clearly my questioning was a bit alpha since CDC is still being worked on.

I had the chance to discuss ideas with Kamil, Tzach and Dor: one of the thing that one of my colleague Julien asked for was the ability for the CDC to generate an event when a tombstone is written so we could actually know when a specific data expired!

I want to stress a few other things too:

LightWeight Transactions

Another so long awaited feature is also coming from the amazing work and knowledge of Konstantin. We had a great conversation about the differences between the currently worked on Paxos based LWT implementation and the maybe later Raft one.

So yes, the first LWT implementation will be using Paxos as a consensus algorithm. This will make the LWT feature very consistent while having it slower that what could be achieved using Raft. That's why ScyllaDB have plans on another implementation that could be faster with less data consistency guarantees.

User Defined Functions / Aggregations

This one is bringing the Lua language inside Scylla!

To be precise, it will be a Lua JIT as its footprint is low and Lua can be cooperative enough but the ScyllaDB people made sure to monitor its violations (when it should yield but does not) and act strongly upon them.

I got into implementation details with Avi, this is what I noted:

Text search

Dejan is the text search guy at ScyllaDB and the one who kindly implemented the LIKE feature we asked for and that will be released in the upcoming 3.2 version.

We discussed ideas and projected use cases to make sure that what's going to be worked on will be used!

Redis API

I've always been frustrated about Redis because while I love the technology I never trusted its clustering and scaling capabilities.

What if you could scale your Redis like Scylla without giving up on performance? That's what the implementation of the Redis API backed by Scylla will get us!

I'm desperately looking forward to see this happen!

28 Dec 2019 7:04pm GMT

24 Dec 2019

feedPlanet Gentoo

Michał Górny: Handling PEP 517 (pyproject.toml) packages in Gentoo

So far, the majority of Python packages have either used distutils, or a build system built upon it. Most frequently, this was setuptools. All those solutions provided a setup.py script with a semi-standard interface, and we were able to handle them reliably within distutils-r1.eclass. PEP 517 changed that.

Instead of a setup script, packages now only need to supply a declarative project information in pyproject.toml file (fun fact: TOML parser is not even part of Python stdlib yet). The build system used is specified as a combination of a package requirement and a backend object to use. The backends are expected to provide a very narrow API: it's limited to building wheel packages and source distribution tarballs.

The new build systems built around this concept are troublesome to Gentoo. They are more focused on being standalone package managers than build systems. They lack the APIs matching our needs. They have large dependency trees, including circular dependencies. Hence, we've decided to try an alternate route.

Instead of trying to tame the new build systems, or work around their deficiencies (i.e. by making them build wheel packages, then unpacking and repackaging them), we've explored the possibility of converting the pyproject.toml files into setup.py scripts. Since the new formats are declarative, this should not be that hard.

We've found poetry-setup project which seemed to have a similar goal. However, it was already discontinued at the time in favor of dephell. The latter project looked pretty powerful but the name was pretty ominous. We did not need most of the functions, and it was hell to package.

Finally, I've managed to dedicate some time into building an in-house solution instead. pyproject2setuppy is a small-ish (<100 SLOC) pyproject.toml-to-setuptools adapter which allows us to run flit- or poetry-based projects as if they used regular distutils. While it's quite limited, it's good enough to build and install the packages that we needed to deal with so far.

The design is quite simple - it reads pyproject.toml and calls setuptools' setup() function with the metadata read. As such, the package can even be used to provide a backwards-compatible setup.py script in other packages. In fact, this is how its own setup.py works - it carries flit-compatible pyproject.toml and uses itself to install itself via setuptools.

dev-python/pyproject2setuppy is already packaged in Gentoo. I've sent eclass patches to easily integrate it into distutils-r1. Once they are merged, installing pyproject.toml packages should be as simple as adding the following declaration into ebuilds:

DISTUTILS_USE_SETUPTOOLS=pyproject.toml

This should make things easier both for us (as it saves us from having to hurriedly add new build systems and their NIH dependencies) and for users who will not have to suffer from more circular dependencies in the Python world. It may also help some upstream projects to maintain backwards compatibility while migrating to new build systems.

24 Dec 2019 10:59pm GMT