26 Nov 2014

feedPlanet Grep

Mattias Geniar: Varnish FetchError: Gunzip+ESI Failed at the very end

This error can occur in a Varnish setup in multiple forms, but they commonly include the Gunzip + ESI error line. FetchError c Gunzip+ESI Failed at the very end ... FetchError c TestGunzip error at the very end The most

Read more ›

The post Varnish FetchError: Gunzip+ESI Failed at the very end appeared first on ma.ttias.be.

Related posts:

  1. Apache + PHP on Ubuntu: apache2filter does not support gzip If you're running Apache2 + PHP in a configuration with...
  2. Varnish: filter by Source IP using Varnishlog (in Varnish 2.x and 3.x) This is a small follow-up for the varnishlog-oneliners post, on...
  3. Varnish 4.0.0 released together with configuration templates Good news! Today, Varnish 4.0.0 has been released!. Among the...

26 Nov 2014 8:00pm GMT

Frank Goossens: Music from Our Tube; John Dummer’s Famous Music Band – Nine by nine

Heard it on Radio Nova a couple of times already, this (John Dummer's Famous Music Band - Nine by nine) appeals to my inner folkie;

YouTube Video
Watch this video on YouTube or on Easy Youtube.

(clearly not really live and terrible video quality, but still more interesting then watching a still or seeing a record spinning)

26 Nov 2014 4:07pm GMT

25 Nov 2014

feedPlanet Grep

Xavier Mertens: Detecting Suspicious Devices On-The-Fly

RadarJust a link to my guest diary posted today on isc.sans.edu. I briefly introduced a method to perform permanent vulnerability scanning of newly detected hosts. The solution is based on OSSEC, ArpWatch and Nmap.

The article is here.

25 Nov 2014 7:47pm GMT

Dries Buytaert: The power of self-managed teams in Drupal


The concept of official initiatives came out of lessons learned from the Drupal 7 development. We learned a lot from that and in a recent blog post about Drupal initiative leads, I recognized that we need to evolve our tools, our processes, and our organizational design. Others like Nathaniel Catchpole, Larry Garfield and Gábor Hojtsy have shared some of their thoughts already. One of the things I'm most proud of is that the Drupal community is always looking to improve and reinvent itself. Evolving is an important part of our culture. Each time it will get better, but still won't be perfect.

For me, one of the biggest take-aways (but not the only one) is that for an initiative to succeed, it needs to be supported by a team. An initiative needs to carry out a technical vision, plan the work, communicate with all stakeholders, mobilize volunteers, raise funding, organize sprints, and more. It can easily be more than one person can handle -- especially if it isn't your full-time job or if your initiative is complex.

More specifically, we have learned that the most successful initiatives appear to be run by teams that are self-managed; the team members collaborate in the development of the initiative, but also share both managerial and operational responsibilities like planning, coordinating, communicating, sprint organizing and more.

Because self-managed teams are both responsible for their outcomes and in control of their decision-making process, members of a self-managing team are usually more motivated than traditional hierarchical teams. This independence and greater responsibility are important in volunteer communities. Self-managed teams also build and maintain institutional knowledge. The outcome of their work is also more easily accepted by other stakeholders (like core committers) because they have already built a lot of consensus.

If I were to be an initiative lead, I'd feel strongly about building my own team rather than being handed a team. My initial assumption was that each initiative lead would build his/her own team. In hindsight, that was a mistake. Team building is not easy. It requires a time investment that can seem to compete with technical priorities. This is an important lesson and something we can do better going forward. Before making an initiative official, we have to make sure that each initiative has a good team and the support to be successful -- either we can help create a team, provide more coaching or formal training around team building, or we shouldn't designate the initiative official until such a team has coalesced.

25 Nov 2014 2:49pm GMT

Mattias Geniar: Getting Started With PHP

I stumbled upon a great resource yesterday called The PHPBridge: Get Started. It's a step-by-step guide to teach you PHP. This seems especially useful if you're coming from another language and want to learn the syntax and methods used by

Read more ›

The post Getting Started With PHP appeared first on ma.ttias.be.

Related posts:

  1. Method Chaining In PHP - Fluent Interface The technical definition of "Fluent Interface", as defined by Wikipedia,...
  2. Nginx returning blank white page on PHP parsed with FastCGI or PHP-FPM I've just spent a while debugging this, was tricky'er than...
  3. HHVM versus PHP-FPM 5.4 vs PHP-FPM 5.5: performance comparison If you haven't heard of HHVM in the last 2...

25 Nov 2014 9:30am GMT

24 Nov 2014

feedPlanet Grep

Mattias Geniar: Debugging Performance Problems With Zabbix Internal Items

Even after all these years, Zabbix remains my monitoring tool of choice. There's plenty of alternatives, but years of investing in the configs, the templates and the automation have kept my love for it. But, it's not always easy to

Read more ›

The post Debugging Performance Problems With Zabbix Internal Items appeared first on ma.ttias.be.

Related posts:

  1. Remove Orphaned Data From Zabbix's MySQL Tables A few years ago, I wrote a couple of SQL...
  2. Zabbix: zabbix_agentd: Can't recreate Zabbix semaphores for IPC key 0x123456 Semaphore ID 123456. Operation not permitted. You can get the following error when you're switching between...
  3. Zabbix: debugging 'Lock wait timeout exceeded; try restarting transaction' on the 'ids' table I recently had fun troubleshooting an issue on a Zabbix...

24 Nov 2014 8:57pm GMT

Mattias Geniar: Remove Orphaned Data From Zabbix’s MySQL Tables

A few years ago, I wrote a couple of SQL queries that I put onto Github to clean up a Zabbix database. It'll take items, triggers, events etc. that are no longer attached to a host, and remove them from

Read more ›

The post Remove Orphaned Data From Zabbix's MySQL Tables appeared first on ma.ttias.be.

Related posts:

  1. MySQL Upgrade To 5.1: Database Name Prefix #mysql50# If you've upgraded from a MySQL version prior to 5.1,...
  2. MySQL purge the binary logs from replication (mysql-bin.xxxx files) There are times when a MySQL replication can hog up...
  3. MySQL: SHOW FUNCTION STATUS WHERE Db = 'name': Cannot load from mysql.proc. The table is probably corrupted If you recently upgraded from a MySQL 5.0 or 5.1...

24 Nov 2014 6:00pm GMT

Mattias Geniar: Snakes On A Keyboard

Now this is a very cool hardware mod. You have had this keyboard for all of 24 hours now. The thing has a bunch of LEDs and some arrow keys. I'm disappointed that you haven't got Snake running on it

Read more ›

The post Snakes On A Keyboard appeared first on ma.ttias.be.

Related posts:

  1. Define Happyness: The Act Of Making Someone Else Happy I don't often write something personal (and before you remove...
  2. Bandwidth limitations in Belgium: oh sigh. I ranted about this in 2008, and things have improved....
  3. How We Save The Day, But Receive No Glory For It It's amazing how many good things are achieved on daily...

24 Nov 2014 4:23pm GMT

Fabian Arrotin: Switching from Ethernet to Infiniband for Gluster access (or why we had to …)

As explained in my previous (small) blog post, I had to migrate a Gluster setup we have within CentOS.org Infra. As said in that previous blog post too, Gluster is really easy to install, and sometimes it can even "smells" too easy to be true. One thing to keep in mind when dealing with Gluster is that it's a "file-level" storage solution, so don't try to compare it with "block-level" solutions (so typically a NAS vs SAN comparison, even if "SAN" itself is wrong for such discussion, as SAN is what's *between* your nodes and the storage itself, just a reminder.)

Within CentOS.org infra, we have a multiple nodes Gluster setup, that we use for multiple things at the same time. The Gluster volumes are used to store some files, but also to host (different gluster volumes with different settings/ACLs) KVM virtual-disks (qcow2). People knowing me will say : "hey, but for performances reasons, it's faster to just dedicate for example a partition , or a Logical Volume instead of using qcow2 images sitting on top a filesystem for Virtual Machines, right ?" and that's true. But with our limited amount of machines, and a need to "move" Virtual Machine without a proper shared storage solution (and because in our setup, those physical nodes *are* both glusterd and hypervisors), Gluster was an easy to use solution to :

It was working, but not that fast ... I then heard about the fact that (obviously) accessing those qcow2 images file through fuse wasn't efficient at all, but that Gluster had libgfapi that could be used to "talk" directly to the gluster daemons, bypassing completely the need to mount your gluster volumes locally through fuse. Thankfully, qemu-kvm from CentOS 6 is built against libgfapi so can use that directly (and that's the reason why it's automatically installed when you install KVM hypervisor components). Results ? better , but still not was I/we was/were expecting ...

When trying to find the issue, I discussed with some folks in the #gluster irc channel (irc.freenode.net) and suddenly I understood something that it's *not* so obvious for Gluster in distributed+replicated mode : for people having dealt with storage solutions at the hardware level (or people using DRBD, which I did too in the past, and that I also liked a lot ..) in the past, we expect the replication to happens automatically at the storage/server side, but that's not true for Gluster : in fact Glusterd just exposes metadata to gluster clients, which then know where to read/write (being "redirected" to correct gluster nodes). That means so than replication happens at the *client* side : in replicated mode, the clients will write itself twice the same data : once on each server ...

So back to our example, as our nodes have 2*1Gb/s Ethernet card, and that one is a bridge used by the Virtual Machines, and the other one "dedicated" to gluster, and that each node is itself a glusterd/gluster client, I let you think about the max perf we could get : for a write operation : 1Gbit/s , divided by two (because of the replication) so ~ 125MB / 2 => in theory ~ 62 MB/sec (and then remove tcp/gluster/overhead and that drops to ~ 55MB/s)

How to solve that ? well, I tested that theory and confirmed directly that it was the case, when in distributed mode only, write performances were automatically doubled. So yes, running Gluster on Gigabit Ethernet suddenly was the bottleneck. Upgrading to 10Gb wasn't something we could do, but , thanks to Justin Clift (and some other Gluster folks), we were able to find some "second hand" Infiniband hardware (10Gbps HCAs and switch)

While Gluster has native/builtin rdma/Infiniband capabilities (see "tranport" option in the "gluster create volume" command), we had in our case to migrate existing Gluster volumes from plain TCP/Ethernet to Infiniband, while trying to get the downtime as small as possible. That is/was my first experience with Infiniband, but it's not as hard as it seems, especially when you discover IPoIB (IP over Infiniband). So from a Syadmin POV, it's just "yet another network interface", but a 10Gbps one now :)

The Gluster volume migration then goes like this : (schedule a - obvious - downtime for this) :

On all gluster nodes (assuming that we start from machines installed only with @core group, so minimal ones) :

yum groupinstall "Infiniband Support"

chkconfig rdma on

<stop your clients or other apps accessing gluster volumes, as they will be stopped>

service glusterd stop && chkconfig glusterd off && init 0

Install then the hardware in each server, connect all Infiniband cards to the IB switch (previously configured) and power back on all servers. When machines are back online, you have "just" to configure the ib interfaces. As in my cases, machines were "remote nodes" and not having a look at how they were configured, I had to use some IB tools to see which port was connected (a tool like "ibv_devinfo" showed me which port was active/connected, while "ibdiagnet" shows you the topology and other nodes/devices). In our case it was port 2, so let's create the ifcfg-ib{0,1} devices (and ib1 being the one we'll use) :


The interesting part here is the "CONNECTED_MODE=yes" : for people who already uses iscsi, you know that Jumbo frames are really important if you have a dedicated VLAN (and that the Ethernet switch support Jumbo frames too). As stated in the IPoIB kernel doc , you can have two operation mode : datagram (default 2044 bytes MTU) or Connected (up to 65520 bytes MTU). It's up to you to decide which one to use, but if you understood the Jumbo frames thing for iscsi, you get the point already.

An "ifup ib1" on all nodes will bring the interfaces up and you can verify that everything works by pinging each other node, including with larger mtu values :

ping -s 16384 <other-node-on-the-infiniband-network>

If everything's fine, you can then decide to start gluster *but* don't forget that gluster uses FQDN (at least I hope that's how you configured initially your gluster setup, already on a dedicated segment, and using different FQDN for the storage vlan). You just have to update your local resolver (internal DNS, local hosts files, whatever you want) to be sure that gluster will then use the new IP subnet on the Infiniband network. (If you haven't previously defined different hostnames for your gluster setup, you can "just" update that in the different /var/lib/glusterd/peers/* and /var/lib/glusterd/vols/*/*.vol)

Restart the whole gluster stack (on all gluster nodes) and verify that it works fine :

service glusterd start

gluster peer status

gluster volume status

# and if you're happy with the results :

chkconfig glusterd on

So, in a short summary:

24 Nov 2014 10:37am GMT

Mattias Geniar: Remote Code Execution via ‘less’ on Linux Boxes

Mondays, gotta love'm. Many Linux distributions ship with the 'less' command automagically interfaced to 'lesspipe'-type scripts, usually invoked via LESSOPEN. This is certainly the case for CentOS and Ubuntu. Unfortunately, many of these scripts appear to call a rather large

Read more ›

The post Remote Code Execution via 'less' on Linux Boxes appeared first on ma.ttias.be.

Related posts:

  1. PHP execution via passthru(), exec(), shell_exec(), …: sh: /php: No such file or directory If you're running PHP scripts through the command line (crontab...
  2. MySQL - myisamchk: error: myisam_sort_buffer_size is too small If you're trying to repair large MyISAM tables in MySQL,...
  3. Setting up a catch-all e-mail account on Linux using Postfix You may want to install a catch-all e-mail account on...

24 Nov 2014 8:36am GMT

23 Nov 2014

feedPlanet Grep

Mattias Geniar: Presentation: DNSSEC, The Good, The Bad & The Secure

Another set of slides I found that never got published, it seems. The presentation was actually never given, but was prepared for several conferences. It stops abruptly and was never completed, but still contains a lot of useful material (at

Read more ›

The post Presentation: DNSSEC, The Good, The Bad & The Secure appeared first on ma.ttias.be.

Related posts:

  1. Implementing & Maintaining DNSSEC On Bind9 Nameservers I won't be going into detail what DNSSEC is, and...
  2. DNSSEC: NSEC3 iterations too big for weakest DNSKEY strength When configuring DNSSEC, it's common you will run into the...
  3. Presentation: Mobile Zabbix, Why Mobile Matters (MoZBX) Going through some old files, I found a presentation I...

23 Nov 2014 9:33pm GMT

Mattias Geniar: Presentation: Mobile Zabbix, Why Mobile Matters (MoZBX)

Going through some old files, I found a presentation I gave in Riga on the Zabbix Conference in 2012, that I never posted online. Better late than never! The slides are about a Mobile WebUI I made for the Zabbix

Read more ›

The post Presentation: Mobile Zabbix, Why Mobile Matters (MoZBX) appeared first on ma.ttias.be.

Related posts:

  1. MoZBX: The Mobile Zabbix Client I'm a pretty big fan of Zabbix in general, the...
  2. This is what Open Source is about: Mobile Zabbix used on Hospital Ship A few months ago I released a Mobile interface for...
  3. Zabbix: monitor a TCP port with the Zabbix Agent If you want to monitor a remote host from the...

23 Nov 2014 9:23pm GMT

Mattias Geniar: CPU Flame Graphs

I've only heard of CPU Flame Graphs since the article on NodeJS performance issues at Netflix. ... given a performance problem, observability is of the utmost importance. Flame graphs gave us tremendous insight into where our app was spending most

Read more ›

The post CPU Flame Graphs appeared first on ma.ttias.be.

Related posts:

  1. MoZBX: The Mobile Zabbix Client I'm a pretty big fan of Zabbix in general, the...
  2. Reinstall the Linux Kernel on CentOS or RHEL One would expect that yum's reinstall command would do the...
  3. Compile a (CentOS) Kernel And IPTables With TPROXY Support A default (CentOS) kernel doesn't have TPROXY support, which is...

23 Nov 2014 9:03pm GMT

Mattias Geniar: Enable MySQL’s slow query log without a restart

You're debugging a MySQL server and want to enable the Slow Query, you can do so via the MySQL CLI. There's no need to make changes to the my.cnf file and restart your MySQL service -- even though that would

Read more ›

The post Enable MySQL's slow query log without a restart appeared first on ma.ttias.be.

Related posts:

  1. Fixing MySQL master-slave replication upon query error When you run a MySQL master/slave replication, it can happen...
  2. MySQL purge the binary logs from replication (mysql-bin.xxxx files) There are times when a MySQL replication can hog up...
  3. MySQL: table is read-only You can get the following error in your Apache 's...

23 Nov 2014 6:00pm GMT

21 Nov 2014

feedPlanet Grep

Xavier Mertens: NoSuchCon Wrap-Up Day #3

NoSuchCon VenueHere we go with a review of the last day. As usual, the social event had huge impacts on some attendees but after coffee everything was almost back to normal. The day started with Braden Thomas who presented "Reverse engineering MSP 430 device" or reverse engineering a real-estate lock box.

In US/Canada, such devices are used by real-estate agencies to store the keys of homes for sale. They allow to access the key when the owner is not present. Why focus on such devices? First, because they are used by many people and, usually, they tend to store crypto secrets into the flash. It's cheap and easy but not necessarily nice. There is a legacy key using cell radio but more and more users use the eKey (an IOS/Android app). Braden explained with many details all the steps he performed to be able to access the firmware and then to extract the crypto key. Guess what? The presentation ended with a live demo: Braden just successfully unlock a lock. During the presentation, he explained the different attacks that are available and a special one (that was successful) called "Paparazzi" attack: the goal is to use the flash from a camera against a decap chip to make it behave differently.

The Paparazzi Attack

Then, Peter Hlavaty talked about "Attack on the core". This talk went in the same direction as the one presented yesterday about bypassing security controls in Windows 8.1. On most operating systems, the kernel is the nice place to place malicious code. Why? Because modern o operating systems are more and more protected by implementing multiple controls. The talk focused on CLP3 to CPL0 ("Current Privilege Level"). Level 3 being the user mode and level 0 the kernel mode. Peter not only focussed on Windows but also on Linux and Android. That's clear: the kernel is the new target!

After a welcomed coffee break, Jean-Philippe Aumasson, renowned cryptographer, talked about… cryptography with a talk called "Cryptographic backdooring". Usually, cryptography means a lot of formulas, etc but Jean-Philippe's talk was very didactic! Why speak about backdoors? Because they are present in many crypto implementation and there is no official research paper on this topic. A backdoor can be used for surveillance, deception, … and also terrorists! There are also more and more backdoors in products and applications today.

Jean-Philippe on Stage

Jean-Philippe explained what is a backdoor. His definition is:

A feature or defect that allows surreptitious access to data

Based on weakened a algorithms or covert channels. But what is a good backdoor? It must be:

  • Undetectable
  • Principle of "NOBUS" (No One But Us, NSA term)
  • Reusable and unmodifiable
  • Simple

Then, he reviewed examples of backdoor and how they have been implemented. A very nice talk!

There was no lunch break for me because I attended a workshop about RF hardware: "Fun with RF remotes" prepared by Damien Cauquil. The goal of the workshop was to build a … RF door bell brute forcer. After an introduction to the RF technology and some demos to capture and analyse signals, it was a hands-on session. All participants received a door bell pack (a remote controller + door bell). The challenge was to hack the remote and make Damien's doorbell rings. It was a premiere for me. After soldering some components and some stress, it worked! Very nice workshop!

Fun wih RF Remote

And the last half-day started with Guillaume Valadon and Nicolas Vivet who presented "Detecting BGP hijacks in 2014". I arrived a bit late due to the hardware workshop. The first part was a recap about BGP, how it works, what are the features, etc. BGP hijacks are not new but they can have a dramatic effect! An hijack is a conflicting BGP announcement. It means that your packets are sent across not authorised networks (from a BGP point of view). The next part of the talk focussed on detected the hijacks. This is a critical step for ISP's. Guillaume and Nicolas explained in details the platform deployed worldwide to collect BGP messages and store them, then they are processed by OCAml. They can emulate a BGP router via some Python code. By putting all the components together, they are able to analyse the BGP announces and detect issues. But this is offline and consumes a lot of data. They also presented a real-time detection mechanism. A nice presentation with many details. I recommend to read the slides if you're working with BGP. Their conclusions are that such attacks are a real risk and that traffic must be encrypted and authenticated to prevent it to be read by 3rd parties.

Alex Ionescu came with a "surprise talk". The title was "Unreal mode: Breaking the protected process". It was a surprised talk because he received a last minute green light from Microsoft. Windows Vista introduced new protections at kernel level. In Windows 8.1, that model was extended to protect key processes even from admin and to mitigate attacks like pass-the-hash. Alex explained how digital signatues are working with the new versions of the OS. He also explained how process protection works (even with admin rights some processes can't be killed or accessed by debuggers. A mass of interesting information if you're working with Windows security models.

Alex on Stage

And to close the conference, a keynote was presented by Anthony Zboralski: "No Such Security". Anthony defines himself as "a bank robber". When he was young he played with many computers and quickly started to break stuff. After some issues with the Justice, he switched to security consultancy. His keynote was a suite of reflexion about the security that is implemented today by companies but also recommended by consultancy companies.

Anthony on Stage

The second edition of NoSuchCon is over! It is a great event with highly technical and nice presentation. I also met lot of new or old friends. The talks have already been published here: http://www.nosuchcon.org/talks/2014/.

21 Nov 2014 9:09pm GMT

Frank Goossens: Ik ben niet gelovig, maar …

… voor deze uitspraak laat ik Paus Franciscus hier wel graag aan het woord;

Men vindt altijd geld om oorlog te voeren, wapens te kopen en zonder scrupules financiële operaties te leiden maar er ontbreekt altijd geld om jobs te creëren, te investeren in kennis en om het leefmilieu te beschermen.
(Paus Franciscus in een videoboodschap op een congres over de sociale leer van de Kerk 21/11/2014 om 19:47:00).

Bron: deredactie.be (weliswaar zonder de contextuele links)

21 Nov 2014 8:14pm GMT