04 Sep 2015

feedPlanet Grep

Dries Buytaert: Digital Distributors: The Supermarkets of the Web

It's hard to ignore the strong force of digital distributors on the open web. In a previous post, I focused on three different scenarios for the future of the open web. In two of the three scenarios, the digital distributors are the primary way for people to discover news, giving them an extraordinary amount of control.

When a small number of intermediary distributors get between producers and consumers, history shows us we have to worry -- and brace ourselves for big industry-wide changes. A similar supply chain disruption has happened in the food industry, where distributors (supermarkets) changed the balance of power and control for producers (farmers). There are a few key lessons we can take from the food industry's history and apply them to the world of the web.

Historical lessons from the food industry

When food producers and farmers learned that selling directly to consumers had limited reach, trading posts emerged and began selling farmers' products more than their own. As early as the 14th century, these trading posts turned into general stores, and in the 18th century, supermarkets and large-scale grocery stores continued that trend in retailing. The adoption of supermarkets was customer-driven; customers benefit from being able to buy all their food and household products in a single store. Today, it is certainly hard to imagine going to a dozen different speciality stores to buy everything for your day-to-day needs.

But as a result, very few farmers sell straight to consumers, and a relatively small amount of supermarkets stand between thousands of suppliers and millions of consumers. The supermarkets have most of the power; they control how products are displayed, and which ones gain prominent placement on shelves. They have been accused of squeezing prices to farmers, forcing many out of business. Supermarkets can also sell products at lower prices than traditional corner shops, leading to smaller grocery stores closing.

In the web's case, digital distributors are the grocery stores and farmers are content producers. Just like supermarket consumers, web users are flocking to the convenience and relevance of digital distributors.

Farmers supermarkets

This graph illustrates the market power of supermarkets / digital distributors. A handful of supermarkets / digital distributors stand between thousands of farmers / publishers and millions of consumers.

Control of experience

Web developers and designers devote a tremendous amount of time and attention to creating beautiful experiences on their websites. One of my favorites was the Sochi Olympics interactive stories that The New York Times created.

That type of experience is currently lost on digital distributors where everything looks the same. Much like a food distributor's products are branded on the shelves of a supermarket to stand out, publishers need clear ways to distinguish their brands on the "shelves" of various digital distribution platforms.

While standards like RSS and Atom have been extended to include more functionality (e.g. Flipboard feeds), there is still a long way to go to support rich, interactive experiences within digital distributors. As an industry, we have to develop and deploy richer standards to "transport" our brand identity and user experience from the producer to digital distributor. I suspect that when Apple unveils its Apple News Format, it will come with more advanced features, forcing others like Flipboard to follow suit.

Control of access / competition

In China, WeChat is a digital distributor of different services with more than 0.5 billion active users. Recently, WeChat blocked the use of Uber. The blocking comes shortly after Tencent, WeChat's parent company, invested in rival domestic car services provider Didi Kuaidi. The shutdown of Uber is an example of how digital distributors can use their market power to favor their own businesses and undermine competitors. In the food industry, supermarkets have realized that it is more profitable to exclude independent brands in favor of launching their own launching grocery brands.

Control of curation

Digital distributors have an enormous amount of power through their ability to manipulate curation algorithms. This type of control is not only bad for the content producers, but could be bad for society as a whole. A recent study found that the effects of search engine manipulation could pose a threat to democracy. In fact, Google rankings may have been a deciding factor in the 2014 elections in India, one of the largest elections in the world. To quote the study's author: "search rankings could boost the proportion of people favoring any candidate by more than 20 percent -- more than 60 percent in some demographic groups". By manipulating its search results, Google could decide the U.S. presidential election. Given that Google keeps its curation algorithms secret, we don't know if we are being manipulated or not.

Control of monetization

Finally, a last major fear is control of monetization. Like supermarkets, digital distributors have a lot of control over pricing. Individual web publishers must maintain their high quality standards to keep consumers demanding their work. Then, there is some degree of leverage to work into the business model negotiation with digital distributors. For example, perhaps The New York Times could offer a limited-run exclusive within a distribution platform like Facebook's Instant Articles or Flipboard.

I'm also interested to see what shapes up with Apple's content blocking in iOS9. I believe that as an unexpected consequence, content blocking will place even more power and control over monetization into the hands of digital distributors, as publishers become less capable of monetizing their own sites.

04 Sep 2015 1:30pm GMT

Xavier Mertens: How to Kick-Out the Bad Guy?

[The post How to Kick-Out the Bad Guy? has been first published on /dev/random]

Kicked OutA quick blog post about an issue I faced this morning. While drinking my morning coffee and reviewing what happened during the last night in my logs, I detected that one of my website (leakedin.com) was entirely mirrored by a guy from Brazil. I'm not against sharing information but in this case, it was consuming bandwidth and server resources for nothing… I was time to kick him off!

The web crawling was smoothly performed and he did not trigger any counter-measures in place like, by example, to automatically blacklist his IP address for a few minutes. This makes the script kiddies' task more difficult. In this case, I just blacklisted his IP definitively via a specific firewall rule. Case closed! Hélas, no… The guy was clever enough to keep the traffic within a set of established TCP/IP sessions. What does it mean? By default, a firewall won't block them due to their "ESTABLISHED" state. The reason is the following rule present in almost all iptables rulebase:

-A INPUT -m tcp -p tcp -m state --state RELATED,ESTABLISHED -j ACCEPT

The quick way could be to restart the web server (or the firewall) but with a side impact for all the users… I remembered that the dsniff package has an interesting tool for this situation. dsniff is a nice toolbox for pentesters and should not be installed on a production server… definitively! But one of the tool is called tcpkill. It allows you to kill specific established TCP sessions based on an "expression" compatible with the tcpdump/libpcap format. Some examples:

# tcpkill -i eth0 port 80
# tcpkill -i eth0 host
# tcpkill -i eth0 port 22 and not host

The ESTABLISHED sessions were killed and the firewall is now doing its job: to block new TCP sessions from this IP. Very efficient!

[The post How to Kick-Out the Bad Guy? has been first published on /dev/random]

04 Sep 2015 7:25am GMT

03 Sep 2015

feedPlanet Grep

Frank Goossens: Music from Our Tube; Carole King feeling the Earth Move

Oh Carole, no wonder Neil was head over heels in love with you …

YouTube Video
Watch this video on YouTube or on Easy Youtube.

Possibly related twitterless twaddle:

03 Sep 2015 3:57am GMT

02 Sep 2015

feedPlanet Grep

Dieter Plaetinck: Transplanting Go packages for fun and profit

crazy Gopher scientist

A while back I read coders at work, which is a book of interviews with some great computer scientists who earned their stripes, the questions just as thoughtful as the answers. For one thing, it re-ignited my interest in functional programming, for another I got interested in literate programming but most of all, it struck me how common of a recommendation it was to read other people's code as a means to become a better programmer. (It also has a good section of Brad Fitzpatrick describing his dislike for programming languages, and dreaming about his ideal language. This must have been shortly before Go came about and he became a maintainer.)

I hadn't been doing a good job reading/studying other code out of fear that inferior patterns/style would rub off on me. But I soon realized that was an irrational, perhaps slightly absurd excuse. So I made the decision to change. Contrary to my presumption I found that by reading code that looks bad you can challenge and re-evaluate your mindset and get out with a more nuanced understanding and awareness of the pros and cons of various approaches.

I also realized if code is proving too hard to get into or is of too low quality, you can switch to another code base with negligible effort and end up spending almost all of your time reading code that is worthwhile and has plenty of learnings to offer. There is a lot of high quality Go code, easy to find through sites like Github or Golang weekly, just follow your interests and pick a project to start reading.

It gets really interesting though once you find bodies of code that are not only a nice learning resource, but can be transplanted into your code with minimal work to solve a problem you're having, but in a different context then the author of the code originally designed it for. Components often grow and mature in the context of an application without being promoted as reusable libraries, but you can often use them as if they were. I would like to share 2 such success cases below.

Nsq's diskqueue code

I've always had an interest in code that manages the same binary data both in memory and on a block device. Think filesystems, databases, etc. There's some interesting concerns like robustness in light of failures combined with optimizing for performance (infrequent syncs to disk, maintaining the hot subset of data in memory, etc), combined with optimizing for various access patterns, this can be a daunting topic to get into.

Luckily there's a use case that I see all the time in my domain (telemetry systems) and that covers just enough of the problems to be interesting and fun, but not enough to be overwhelming. And that is: for each step in a monitoring data pipeline, you want to be able to buffer data if the endpoint goes down, in memory and to disk if the amount of data gets too much. Especially to disk if you're also concerned with your software crashing or the machine power cycling.

This is such a common problem that applies to all metrics agents, relays, etc that I was longing for a library that just takes care of spooling data to disk for you without really affecting much of the rest of your software. All it needs to do is sequentially write pieces of data to disk and have a sequential reader catching up and read newer data as it finishes processing the older.

NSQ is a messaging platform from bitly, and it has diskqueue code that does exactly that. And it does so oh so elegantly. I had previously found a beautiful pattern in bitly's go code that I blogged about and again I found a nice and elegant design that builds further on this pattern, with concurrent access to data protected via a single instance of a for loop running a select block which assures only one piece of code can make changes to data at the same time (see bottom of the file), not unlike ioloops in other languages. And method calls such as Put() provide a clean external interface, though their implementation simply hooks into the internal select loop that runs the code that does the bulk of the work. Genius.

func (d *diskQueue) Put(data []byte) error {
  // some details
  d.writeChan <- data
  return <-d.writeResponseChan

In addition the package came with extensive tests and benchmarks out of the box.

After finding and familiarizing myself with this diskqueue code about a year ago I had an easy time introducing disk spooling to Carbon-relay-ng, by transplanting the code into it. The only change I had to make was capitalizing the Diskqueue type to export it outside of the package. It has proven a great fit, enabling a critical feature through little work of transplanting mature, battle-tested code into a context that original authors probably never thought of.

Note also how the data unit here is the []byte, the queue does not deal with the higher level nsq.Message (!). The authors had the foresight of keeping this generic, enabling code reuse and rightfully shot down a PR of mine that had a side effect of making the queue aware of the Message type. In NSQ you'll find thoughtful and deliberate api design and pretty sound code all around. Also, they went pretty far in detailing some lessons learned and providing concrete advice, a very interesting read, especially around managing goroutines & synchronizing their exits, and performance optimizations. At Raintank, we had a need for a messaging solution for metrics so we will so be rolling out NSQ as part of the raintank stack. This is an interesting case where my past experience with the NSQ code and ideas helped to adopt the full solution.

Bosun expression package

I'm a fan of the bosun alerting system which came out of Stack Exchange. It's a full-featured alerting system that solves a few problems like no other tool I've seen does (see my linked post), and timeseries data storage aside, comes with basically everything built in to the one program. I've used it with success. However, for litmus I needed an alerting handler that integrated well into the Grafana backend. I needed the ability to do arbitrarily complex computations. Graphite's api only takes you so far. We also needed (desired) reduction functions, boolean logic, etc. This is where bosun's expression language is really strong. I found the expression package quite interesting, they basically built their own DSL for metrics processing. so it deals with expression parsing, constructing AST's, executing them, dealing with types (potentially mixed types in the same expression), etc.

But bosun also has incident management, contacts, escalations, etc. Stuff that we either already had in place, or didn't want to worry about just yet. So we could run bosun standalone and talk to it as a service via its API which I found too loosely coupled and risky, hook all its code into our binary at once - which seemed overkill - or the strategy I chose: gradually familiarize ourself and adopt pieces of Bosun on a case by case basis, making sure there's a tight fit and without ever building up so much technical debt that it would become a pain to move away from the transplanted code if it becomes clear it's not/no longer well suited. For the foreseeable future we only need one piece, the expression package. Potentially ultimately we'll adopt the entire thing, but without the upfront commitment and investment.

So practically, our code now simply has one line where we create a bosun expression object from a string, and another where we ask bosun to execute the expression for us, which takes care of parsing the expression, querying for the data, evaluating and processing the results and distilling everything down into a final result. We get all the language features (reduction functions, boolean logic, nested expressions, …) for free.

This transplantation was again probably not something the bosun authors expected, but for us it was tremendously liberating. We got a lot of power for free. The only thing I had to do was spend some time reading code, and learning in the process. And I knew the code was well tested so we had zero issues using it.

Much akin to the NSQ example above, there was another reason the transplantation went so smoothly: the expression package is not tangled into other stuff. It just needs a string expression and a graphite instance. To be precise, any struct instance that satisfies the graphiteContext interface that is handily defined in the bosun code. While the bosun design aims to make its various clients (graphite, opentsdb, …) applicable for other projects, it also happens to let us do opposite: reuse some of its core code - the expression package - and pass in a custom graphite Context, such as our implementation which has extensive instrumentation. This lets us use the bosun expression package as a "black box" and still inject our own custom logic into the part that queries data from graphite. Of course, once we want to change the logic of anything else in the black box, we will need come up with something else, perhaps fork the package, but it doesn't seem like we'll need that any time soon.


If you want to become a better programmer I highly recommend you go read some code. There's plenty of good code out there. Pick something that deals with a topic that is of interest to you and looks mature. You typically won't know if code is good before you start reading but you'll find out really fast, and you might be pleasantly surprised, as was I, several times. You will learn a bunch, possibly pretty fast. However, don't go for the most advanced, complex code straight away. Pick projects and topics that are out of your comfort zone and do things that are new to you, but nothing too crazy. Once you truly grok those, proceed to other, possibly more advanced stuff.

Often you'll read reusable libraries that are built to be reused, or you might find ways to transplant smaller portions of code into your own projects. Either way is a great way to tinker and learn, and solve real problems. Just make sure the code actually fits in so you don't end up with the software version of Frankenstein's monster. It is also helpful to have the authors available to chat if you need help or have issues understanding something, though they might be surprised if you're using their code in a way they didn't envision and might not be very inclined to provide support to what they consider internal implementation details. So that could be a hit or miss. Luckily the people behind both nsq and bosun were supportive of my endeavors but I also made sure to try to figure out things by myself before bothering them. Another reason why it's good to pick mature, documented projects.

Gopher frankenstein

Part of the original meaning of hacking, extended into open source, is a mindset and practice of seeing how others solve a problem, discussion and building on top of it. We've gotten used to - and fairly good at - doing this on a project and library level but forgot about it on the level of code, code patterns and ideas. I want to see these practices come back to life.

We also apply this at Raintank: not only are we trying to build the best open source monitoring platform by reusing (and often contributing to) existing open source tools and working with different communities, we realize it's vital to work on a more granular level, get to know the people and practice cross-pollination of ideas and code.

Next stuff I want to read and possibly implement or transplant parts of: dgryski/go-trigram, armon/go-radix, especially as used in the dgryski/carbonmem server to search through Graphite metrics. Other fun stuff by dgryski: an implementation of the ARC caching algorithm and bloom filters. (you might want to get used to reading Wikipedia pages also). And mreiferson/wal, a write ahead log by one of the nsqd authors, which looks like it'll become the successor of the beloved diskqueue code.

Go forth and transplant!

Also posted on the Raintank blog

02 Sep 2015 4:25pm GMT

01 Sep 2015

feedPlanet Grep

Bert de Bruijn: A use case for exporting and importing distributed vswitches

In a recent VMware project, an existing environment of vSphere ESXi hosts had to be split off to a new instance of vCenter. These hosts were member of a distributed virtual switch, an object that saves its configuration in the vCenter database. This information would be lost after the move to the new vCenter, and the hosts would be left with "orphaned" distributed vswitch configurations.

Thanks to the export/import function now available in vSphere 5.5 and 6.x, we can now move the full distributed vswitch configuration to the new vCenter:

What used to be orphaned configurations on the host, are now valid member switches of the distributed switch you just imported!

01 Sep 2015 7:31pm GMT

Mattias Geniar: CentOS 7 NetworkManager Keeps Overwriting /etc/resolv.conf

The post CentOS 7 NetworkManager Keeps Overwriting /etc/resolv.conf appeared first on ma.ttias.be.

In CentOS or Red Hat Enterprise Linux (RHEL) 7, you can find your /etc/resolv.conf file, which holds all nameserver configurations for your server, to be overwritten by the NetworkManager.

If you check the content of /etc/resolv.conf, it may look like this.

$ cat /etc/resolv.conf
# Generated by NetworkManager
search mydomain.tld

The NetworkManager will assume it has the rights to control /etc/resolv.conf, if it finds a DNS related configuration in your interface configuration file.

$ grep DNS /etc/sysconfig/network-scripts/ifcfg-*

To prevent Network Manager to overwrite your resolv.conf changes, remove the DNS1, DNS2, ... lines from /etc/sysconfig/network-scripts/ifcfg-*.

Now, you can manually change the /etc/resolv.conf file again, and you should be good to go. NetworkManager will no longer overwrite your DNS nameserver configurations.

The post CentOS 7 NetworkManager Keeps Overwriting /etc/resolv.conf appeared first on ma.ttias.be.

Related posts:

  1. How To Add Secondary IP / Alias On Network Interface in RHEL / CentOS 7 This guide will show you how to add an extra...
  2. RHEL 6/ CentOS 6: Slow SSH logins with DNS timeouts If you're experiencing slow logins via SSH on a Red...
  3. Re-enabling IPv6 support on CentOS kernels after update A Kernel update on one box led to the following...

01 Sep 2015 7:00pm GMT

31 Aug 2015

feedPlanet Grep

Dries Buytaert: Giving back and making money

It's not easy to build an Open Source software company.

Canonical recently has made a change to its intellectual property policy. The new policy prevents developers from distributing altered binary versions of Ubuntu. Users are still allowed to distribute unaltered Ubuntu freely, but if they make changes to Ubuntu, Canonical wants developers to either go through a review process or remove all references to Canonical trademarks, Canonical logos, and proprietary software and recompile the Ubuntu archive without any of those.

This change has caused friction with the Open Source community; many are not happy with these restrictions as it goes against the culture of Open Source sharing and collaboration. After all, Ubuntu itself is built on top of the work of hundreds of thousands of Open Source developers, and now Ubuntu is making it difficult for others to do the same.

Canonical's stated intention is to protect its trademarks and reputation; they don't want anyone to call something "Ubuntu" when it's not actually "Ubuntu". I understand that. That aside, many understand that the unstated goal is to make money from licensing deals. The changes affect organizations that base their custom distributions on Ubuntu; it's easier to buy a license from Canonical than to figure how to remove all the trademarks, proprietary software, logos, etc.

Jono Bacon, Canonical's former community manager, wrote a balanced post about the situation.

My thoughts? I understand Canonical has to find ways to make money. Most companies are downright greedy, but not Canonical or Mark Shuttleworth. I find the Open Source community "penny wise and pound foolish" about the situation.

I can relate because Canonical, like Acquia, is among a small group of Open Source companies that try to do good and do well at scale. We invest millions of dollars each year contributing to Open Source: from engineering, to marketing, to sponsoring community events and initiatives. It is not easy to build a software company on Open Source, and we all struggle to find the right balance between giving back and making money. This is further complicated when competitors choose to give back less or don't give back at all. Companies like Canonical and Acquia are good for Open Source, and helping them find that balance is key. Don't forget to support those that give back.

31 Aug 2015 4:29pm GMT

26 Aug 2015

feedPlanet Grep

Les Jeudis du Libre: Arlon, le 3 septembre, S02E01 : la VoIP avec Asterisk

Logo AsteriskC'est bientôt la rentrée, et comme toutes les bonnes choses, les Jeudis du Libre d'Arlon reviennent pour une nouvelle saison. Pour cette première édition, nous vous proposons une présentation du logiciel Asterisk et de la VoIP.

Informations pratiques

Présentation de l'intervenant

Thierry PIERRAT a 47 ans. Il est issu d'une formation d'ingénieur technicien en Productique et matériaux nouveaux et développe des logiciels de gestion de production. Il a commencé l'informatique sur Apple 2 en 83 en programmant en assembleur. En 1994, il découvre GNU/Linux.

Il est co-fondateur de Allied Data Sys. SA (ADS), avec Pascal BELLARD en 2003. L'objectif étant de réaliser des solutions de Téléphonie IP basées au départ sur le projet Bayonne / CT server puis sur Asterisk. Aujourd'hui, il s'agit de proposer une solution globale packagée Open Source reposant sur GNU/SliTaz, Asterisk et Odoo pour gérer l'infrastructure informatique, téléphonique et administrative des sociétés.

Il est intervenant au niveau BAC +3 et Bac +5 pour les formations Réseau et téléphonie IP.

26 Aug 2015 5:09am GMT

Frank Goossens: Music from Our Tube: new Floating Points track!

Floating Points just released this on his YouTube channel;

YouTube Video
Watch this video on YouTube or on Easy Youtube.

Possibly related twitterless twaddle:

26 Aug 2015 4:20am GMT

25 Aug 2015

feedPlanet Grep

Mattias Geniar: Pretty git log in one line

The post Pretty git log in one line appeared first on ma.ttias.be.

If you type git log to see the commit history in a git repository, the standard output isn't very terminal-friendly. It's a lot of text, with very little information displayed on your screen. You can, however, change the output of your git log to be more condensed and show more output on the same screen size.

By default, a git log looks like this.

$ git log

commit 3396763626316124388f76be662bd941df591118
Author: Mattias Geniar 
Date:   Fri Aug 21 09:16:26 2015 +0200

    Add twitter link

commit c73bbc98b5f55e5a4dbfee8e0297e4e1652a0687
Author: Mattias Geniar 
Date:   Wed Aug 19 09:19:37 2015 +0200

    add facebook link

Each commit, with the date and author + the commit message. But boy, it takes up a lot of screen space.

A simple fix is to pass the --pretty=oneline parameter, which makes it all fit on a single line.

$ git log --pretty=oneline

3396763626316124388f76be662bd941df591118 Add twitter link
c73bbc98b5f55e5a4dbfee8e0297e4e1652a0687 add facebook link

It's taking up less space, but missing crucial information like the date of the commit.

There are longer versions of that same --pretty parameter. In fact, it allows you to specify all the fields you want in the output.

$ git log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit

* 3396763 - (HEAD, origin/master, master) Add twitter link (4 days ago) 
* c73bbc9 - add facebook link (6 days ago) 
* cb555df - More random values (6 days ago) 
*   60e7bbf - Merge pull request #1 from TSchuermans/patch-1 (7 days ago) 
| * 8044a8f - Typo fix (7 days ago) 

The output is indented to show branch-points and merges. In colour, it looks like this.


To make life easier, you can can add a git alias so you don't have to remember the entire syntax.

$ git config --global alias.logline "log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit"
$ git logline

More data on the same screen real estate!

The post Pretty git log in one line appeared first on ma.ttias.be.

Related posts:

  1. Xapian Bindings 1.2.7/8 in PHP: undefined symbol: zend_error_noreturn in Unknown on line Slightly frustrated blogpost on Xapian versions 1.2.7 or higher when...
  2. Why We're Still Seeing PHP 5.3 In The Wild (Or: PHP Versions, A History) WordPress offers an API that can list the PHP versions...
  3. Life Without Ops There's a reason why the --noop mode doesn't actually do...

25 Aug 2015 7:00pm GMT

Dries Buytaert: Digital Distributors vs Open Web: who will win?

I've spent a fair amount of time thinking about how to win back the Open Web, but in the case of digital distributors (e.g. closed aggregators like Facebook, Google, Apple, Amazon, Flipboard) superior, push-based user experiences have won the hearts and minds of end users, and enabled them to attract and retain audience in ways that individual publishers on the Open Web currently can't.

In today's world, there is a clear role for both digital distributors and Open Web publishers. Each needs the other to thrive. The Open Web provides distributors content to aggregate, curate and deliver to its users, and distributors provide the Open Web reach in return. The user benefits from this symbiosis, because it's easier to discover relevant content.

As I see it, there are two important observations. First, digital distributors have out-innovated the Open Web in terms of conveniently delivering relevant content; the usability gap between these closed distributors and the Open Web is wide, and won't be overcome without a new disruptive technology. Second, the digital distributors haven't provided the pure profit motives for individual publishers to divest their websites and fully embrace distributors.

However, it begs some interesting questions for the future of the web. What does the rise of digital distributors mean for the Open Web? If distributors become successful in enabling publishers to monetize their content, is there a point at which distributors create enough value for publishers to stop having their own websites? If distributors are capturing market share because of a superior user experience, is there a future technology that could disrupt them? And the ultimate question: who will win, digital distributors or the Open Web?

I see three distinct scenarios that could play out over the next few years, which I'll explore in this post.

Digital Distributors vs Open Web: who will win?

This image summarizes different scenarios for the future of the web. Each scenario has a label in the top-left corner which I'll refer to in this blog post. A larger version of this image can be found at http://buytaert.net/sites/buytaert.net/files/images/blog/digital-distrib....

Scenario 1: Digital distributors provide commercial value to publishers (A1 → A3/B3)

Digital distributors provide publishers reach, but without tangible commercial benefits, they risk being perceived as diluting or even destroying value for publishers rather than adding it. Right now, digital distributors are in early, experimental phases of enabling publishers to monetize their content. Facebook's Instant Articles currently lets publishers retain 100 percent of revenue from the ad inventory they sell. Flipboard, in efforts to stave off rivals like Apple News, has experimented with everything from publisher paywalls to native advertising as revenue models. Expect much more experimentation with different monetization models and dealmaking between the publishers and digital distributors.

If digital distributors like Facebook succeed in delivering substantial commercial value to the publisher they may fully embrace the distributor model and even divest their own websites' front-end, especially if the publishers could make the vast majority of their revenue from Facebook rather than from their own websites. I'd be interested to see someone model out a business case for that tipping point. I can imagine a future upstart media company either divesting its website completely or starting from scratch to serve content directly to distributors (and being profitable in the process). This would be unfortunate news for the Open Web and would mean that content management systems need to focus primarily on multi-channel publishing, and less on their own presentation layer.

As we have seen from other industries, decoupling production from consumption in the supply-chain can redefine industries. We also know that introduces major risks as it puts a lot of power and control in the hands of a few.

Scenario 2: The Open Web's disruptive innovation happens (A1 → C1/C2)

For the Open Web to win, the next disruptive innovation must focus on narrowing the usability gap with distributors. I've written about a concept called a Personal Information Broker (PIM) in a past post, which could serve as a way to responsibly use customer data to engineer similar personal, contextually relevant experiences on the Open Web. Think of this as unbundling Facebook where you separate the personal information management system from their content aggregation and curation platform, and make that available for everyone on the web to use. First, it would help us to close the user experience gap because you could broker your personal information with every website you visit, and every website could instantly provide you a contextual experience regardless of prior knowledge about you. Second, it would enable the creation of more distributors. I like the idea of a PIM making the era of handful of closed distributors as short as possible. In fact, it's hard to imagine the future of the web without some sort of PIM. In a future post, I'll explore in more detail why the web needs a PIM, and what it may look like.

Scenario 3: Coexistence (A1 → A2/B1/B2)

Finally, in a third combined scenario, neither publishers nor distributors dominate, and both continue to coexist. The Open Web serves as both a content hub for distributors, and successfully uses contextualization to improve the user experience on individual websites.


Right now, since distributors are out-innovating on relevance and discovery, publishers are somewhat at their mercy for traffic. However, a significant enough profit motive to divest websites completely remains to be seen. I can imagine that we'll continue in a coexistence phase for some time, since it's unreasonable to expect either the Open Web or digital distributors to fail. If we work on the next disruptive technology for the Open Web, it's possible that we can shift the pendulum in favor of "open" and narrow the usability gap that exists today. If I were to guess, I'd say that we'll see a move from A1 to B2 in the next 5 years, followed by a move from B2 to C2 over the next 5 to 10 years. Time will tell!

25 Aug 2015 12:25pm GMT

24 Aug 2015

feedPlanet Grep

Mattias Geniar: MySQL Back-up: Take a mysqldump with each database in its own SQL File

The post MySQL Back-up: Take a mysqldump with each database in its own SQL File appeared first on ma.ttias.be.

If's often very useful to have a couple of MySQL oneliners nearby. This guide will show you how to take a mysqldump of all databases on your server, and write each database to its own SQL file. As a bonus, I'll show you how to compress the data and import it again -- if you ever need to restore from those files.

Take a mysqldump back-up to separate files

To take a back-up, run the mysqldump tool on each available database.

$ mysql -N -e 'show databases' | while read dbname; do mysqldump --complete-insert --single-transaction "$dbname" > "$dbname".sql; done

The result is a list of all your database files, in your current working directory, suffixed with the .sql file extension.

$ ls -alh *.sql

-rw-r--r-- 1 root root  44M Aug 24 22:39 db1.sql
-rw-r--r-- 1 root root  44M Aug 24 22:39 db2.sql

If you want to write to a particular directory, like /var/dump/databases/, you can change the output of the command like this.

$ mysql -N -e 'show databases' | while read dbname; do mysqldump --complete-insert --single-transaction "$dbname" > /var/dump/databases/"$dbname".sql; done

Mysqldump each database and compress the SQL file

If you want to compress the files, as you're taking them, you can run either gzip or bzip on the resulting SQL file.

$ mysql -N -e 'show databases' | while read dbname; do mysqldump --complete-insert --single-transaction "$dbname" > "$dbname".sql; [[ $? -eq 0 ]] && gzip "$dbname".sql; done

The result is again a list of all your databases, but gzip'd to save diskspace.

$ ls -alh *.gz

-rw-r--r--  1 root root  30K Aug 24 22:42 db1.sql.gz
-rw-r--r--  1 root root 1.6K Aug 24 22:42 db1.sql.gz

This can significantly save you on diskspace at the cost of additional CPU cycles while taking the back-up.

Import files to mysql from each .SQL file

Now that you have a directory full of database files, with the database name in the SQL file, how can you import them all again?

The following for-loop will read all files, strip the ".sql" part from the filename and import to that database.

Warning: this overwrites your databases, without prompting for confirmation. Use with caution!

$ for sql in *.sql; do dbname=${sql/\.sql/}; echo -n "Now importing $dbname ... "; mysql $dbname < $sql; echo " done."; done

The output will tell you which database has been imported already.

$ for sql in *.sql; do dbname=${sql/\.sql/}; echo -n "Now importing $dbname ... "; mysql $dbname < $sql; echo " done."; done 

Now importing db1 ...  done.
Now importing db2 ...  done.

These are very simple one-liners that come in handy when you're migrating from server-to-server.

The post MySQL Back-up: Take a mysqldump with each database in its own SQL File appeared first on ma.ttias.be.

Related posts:

  1. Easiest Way To Create A MySQL Database Copy Here's just about the easiest way of copying a MySQL...
  2. MySQL Upgrade To 5.1: Database Name Prefix #mysql50# If you've upgraded from a MySQL version prior to 5.1,...
  3. MySQL: table is read-only You can get the following error in your Apache 's...

24 Aug 2015 8:52pm GMT

Xavier Mertens: Sending Windows Event Logs to Logstash

[The post Sending Windows Event Logs to Logstash has been first published on /dev/random]

Eventlog to LogstashThis topic is not brand new, there exists plenty of solutions to forward Windows event logs to Logstash (OSSEC, Snare or NXlog amongst many others). They perform a decent job to collect events on running systems but they need to deploy extra piece of software on the target operating systems. For a specific case, I was looking for a solution to quickly transfer event logs from a live system without having to install extra software.

The latest versions of the Microsoft Windows come with Powershell installed by default. Powershell is, as defined by Wikipedia, a task automation and configuration management framework. PowerShell 3 introduced nice cmdlets to convert data from/to JSON which is a format natively supported by Logstash. The goal is to have a standalone Powershell script executed from a share or a read-only USB-stick that will process Windows event logs and send them to a remote preconfigured Logstash server on a specific TCP port.

The first step is to prepare our Logstash environment to receive new events. Let's create a new input and store received events to a dedicated index (it will be easier to investigate the collected data):

input {
    tcp {
        port => 5001
        type => 'eventlog'
        codec => json {
            charset => 'UTF-8'

filter {
    if [type == 'eventlog' {
        grok {
            match => [ 'TimeCreated', "Date\(%{NUMBER:timestamp}\)" ]
        date {
            match => [ 'timestamp', 'UNIX_MS' ]
output {
    if [type == 'eventlog' {
        elasticsearch {
            host => 'localhost'
            port => 9300
            node_name => 'forensics'
            cluster => 'forensics-cluster'
            index => 'logstash-evenlog-%{+YYYY.MM.dd}'

The Powershell script collects event logs via the cmdled Get-WinEvent and convert them in JSON format with ConvertTo-Json. The fact that Logstash expects one event per line, data received by Get-WinEvent are converted to an array and processed in a loop. Before sending the event via a TCP session, '\r' and '\n' are removed. Edit the script, change the destination IP/port and just execute the script to send a copy of all the event logs to your Logstash (take care, it could overload your server). A few minutes later (depending on the amount of data to index), you'll be able to investigate the events from your favourite Kibana session:

Events in Logstash

(Click to enlarge)

Some remarks:

The script is available in my github.com repository.

[The post Sending Windows Event Logs to Logstash has been first published on /dev/random]

24 Aug 2015 12:30pm GMT

Les Jeudis du Libre: Mons, le 24 septembre – Le logiciel libre pour favoriser l’interopérabilité en milieu hospitalier : l’exemple d’Orthanc en imagerie médicale

Logo de OrthancCe jeudi 24 septembre 2015 à 19h se déroulera la 41ème séance montoise des Jeudis du Libre de Belgique.

Le sujet de cette séance : Le logiciel libre pour favoriser l'interopérabilité en milieu hospitalier : l'exemple d'Orthanc en imagerie médicale

Thématique : Santé

Public : Tout public

L'animateur conférencier : Sébastien Jodogne (Orthanc & CHU Liège)

N.B. : Sébastien Jodogne, le logiciel Orthanc et le CHU de Liège ont été récompensés cette année :

Lieu de cette séance : CHU Ambroise Paré, Auditoire Leburton. Boulevard Kennedy 2 à 7000 Mons. Cf. accès et sur la carte OSM.

La participation sera gratuite et ne nécessitera que votre inscription nominative, de préférence préalable, ou à l'entrée de la séance. Merci d'indiquer votre intention en vous inscrivant via la page http://jeudisdulibre.fikket.com/. La séance sera suivie d'un verre de l'amitié.

Les Jeudis du Libre à Mons bénéficient aussi du soutien de nos partenaires : CETIC, Normation, OpenSides, MeaWeb, NextLab, Phonoid et Creative Monkeys.

Si vous êtes intéressé(e) par ce cycle mensuel, n'hésitez pas à consulter l'agenda et à vous inscrire sur la liste de diffusion afin de recevoir systématiquement les annonces.

Pour rappel, les Jeudis du Libre se veulent des espaces d'échanges autour de thématiques des Logiciels Libres. Les rencontres montoises se déroulent chaque troisième jeudi du mois, et sont organisées dans des locaux et en collaboration avec des Hautes Écoles et Facultés Universitaires montoises impliquées dans les formations d'informaticiens (UMONS, HEH et Condorcet), et avec le concours de l'A.S.B.L. LoLiGrUB, active dans la promotion des logiciels libres.

Description : Au cours des vingt dernières années, l'essor puis la démocratisation des nouvelles technologies d'imagerie médicale ont mené à de profondes révolutions dans la prise en charge clinique de nombreuses pathologies, comme le cancer ou l'insuffisance cardiaque. Le volume sans cesse croissant d'images auquel tout hôpital est actuellement confronté n'est pas sans créer de nombreuses difficultés informatiques : acheminement automatique des images depuis les dispositifs d'acquisition jusque dans les logiciels d'analyse d'images, échanges inter- et extra-hospitaliers, anonymisation des données…

Face à ces besoins impérieux de la collectivité et face au manque d'offres commerciales adaptées, le Département de Physique Médicale du CHU de Liège a décidé de concevoir un produit informatique innovant et de qualité industrielle. Ce logiciel, nommé Orthanc, est un serveur d'imagerie médicale léger, robuste et versatile. Orthanc a la particularité d'être un logiciel libre : tous les hôpitaux du monde peuvent donc l'utiliser librement, selon une démarche universitaire, collaborative et ouverte.

L'exemple d'Orthanc montre que le logiciel libre permet d'amener une meilleure indépendance technologique des services médicaux face à leurs fournisseurs. De manière plus générale, les standards ouverts du domaine de la santé (tels que FHIR et DICOM) et leurs implémentations libres de référence sont des outils essentiels pour créer une interopérabilité maximale entre écosystèmes propriétaires dans le milieu médical, au bénéfice de notre système de soins de santé. En outre, bénéficier d'une telle interopérabilité est une énorme opportunité pour stimuler notre tissu industriel en santé électronique, ainsi que pour garantir les libertés numériques des patients.

24 Aug 2015 7:55am GMT

23 Aug 2015

feedPlanet Grep

Mattias Geniar: Install Go 1.5 On CentOS 6 and 7

The post Install Go 1.5 On CentOS 6 and 7 appeared first on ma.ttias.be.

This is a really quick guide on how to install the recently released Go 1.5 on a CentOS 6 or 7 server.

Start by grabbing the latest 1.5 release from the download pages.

$ cd /tmp
$ wget https://storage.googleapis.com/golang/go1.5.linux-amd64.tar.gz

Extract the binary files to /usr/local/go.

$ tar -C /usr/local -xzf /tmp/go1.5.linux-amd64.tar.gz

For easy access, symlink your installed binaries in /usr/local/go to /use/local/bin, which should be in your default $PATH in your shell.

$ echo $PATH

$ ln -s /usr/local/go/bin/go /usr/local/bin/go
$ ln -s /usr/local/go/bin/godoc /usr/local/bin/godoc
$ ln -s /usr/local/go/bin/gofmt /usr/local/bin/gofmt

Alternatively, add the /usr/local/go/bin directory to your $PATH. Add the following line to your ~/.profile file.

export PATH=$PATH:/usr/local/go/bin

You now have the working go binary for version 1.5

$ go version
go version go1.5 linux/amd64

The post Install Go 1.5 On CentOS 6 and 7 appeared first on ma.ttias.be.

23 Aug 2015 8:29pm GMT

21 Aug 2015

feedPlanet Grep

Mattias Geniar: Foreman 1.9: ERROR: column hosts.last_freshcheck does not exist

The post Foreman 1.9: ERROR: column hosts.last_freshcheck does not exist appeared first on ma.ttias.be.

If you've recently upgraded your Foreman 1.8 setup to 1.9, you may see the following error in your dashboard when navigating to a particular host.

# Oops, we're sorry but something went wrong

PGError: ERROR: column hosts.last_freshcheck does not exist LINE 1: ..."name" AS t1_r1, "hosts"."last_compile" AS t1_r2, "hosts"."l... ^ : SELECT "reports"."id" AS t0_r0, "reports"."host_id" AS t0_r1, "reports"."reported_at" AS t0_r2, ...

The upgrade steps will tell you to execute the following database migrations:

$ cd /usr/share/foreman
$ foreman-rake db:migrate
$ foreman-rake db:seed

You can check if the migrations were all executed correctly, by running the following command.

$ cd /usr/share/foreman
$ foreman-rake db:migrate:status
   up     20150618093433  Remove unused fields from hosts
   up     20150622090115  Change reported at
   up     20150714140850  Remove new from compute attributes

If the output shows "up", it means that particular database migration script is up-to-date and was executed. Nothing to do here.

The error concerning the hosts.last_freshcheck column is a result of a cleanup issue where obsoleted columns have been removed. The resulting code is found in pull request 2471.

If your database migrations are all completed and you still see the error, restart your Apache server. The Ruby/Passengers processes keep a cache in memory of the database structure, that isn't refreshed when running the foreman-rake db:migrate commands.

The post Foreman 1.9: ERROR: column hosts.last_freshcheck does not exist appeared first on ma.ttias.be.

Related posts:

  1. Clearing All Data From PuppetDB (exported resources) I had an annoying problem in my test environment when...
  2. Puppet: Error: Could not retrieve catalog from remote server: Error 400 on SERVER: stack level too deep on node something.pp As a Puppet user, you can run into the following...
  3. Puppet: Error 400 on SERVER ArgumentError: malformed format string - %S at … Here's an error to screw with your debugging skills. ~$...

21 Aug 2015 6:00pm GMT