24 Apr 2014 10:11pm GMT

24 Apr 2014 9:18pm GMT

24 Apr 2014 8:54pm GMT

I've just released Morepath 0.2! Morepath is your friendly neighborhood Python web microframework with super powers.

Major changes include:

• Python 3 support thanks to a generous contribution by Alec Munro.
• Documented security infrastructure.

And various smaller tweaks. You can also read the changelog.

What makes Morepath special? You can read about its superpowers and read how it compares to other web frameworks, but here are some highlights:

• Route to model, then look up views. This leads to better link generation, better generic views, better reusable code.
• Have apps that can extend other apps, combine apps together. Extend and override anything, even Morepath behavior itself, but don't affect other apps in the same runtime.

It's lots of power contained in a small codebase.

(If you followed the Morepath link above, ignore the 0.1 in the title bar there: those are the 0.2 docs, not 0.1. I've now fixed the doc building configuration to include the version number from the package itself.)

24 Apr 2014 1:22pm GMT

The following text is in German, since we're announcing a regional user group meeting in Düsseldorf, Germany.

Ankündigung

Das nächste Python Meeting Düsseldorf findet an folgendem Termin statt:

Dienstag, 24.09.2014, 19:00 Uhr
Raum 1, 2.OG im Bürgerhaus Stadtteilzentrum Bilk
Düsseldorfer Arcaden, Bachstr. 145, 40217 Düsseldorf

Neuigkeiten

Bereits angemeldete Vorträge

Charlie Clark
"Status openpyxl, bzw. Lösung neuer Probleme"
"IndexedList - eine Liste optimiert für "in" Abfragen"
"Bericht von der PyCon 2014 in Montreal"

Marc-Andre Lemburg
"Python Code mit lib2to3 modernisieren"
"DDOS Attacken mit Python bekämpfen"
"Bericht von der FOSDEM 2014"

Wir suchen noch weitere Vorträge. Bei Interesse, bitte unter info@pyddf.de melden.

Geänderte Startzeit

Dieses Mal treffen wir uns erst um 19:00 Uhr im Bürgerhaus in den Düsseldorfer Arcaden, da wir keinen Termin für 18 Uhr bekommen haben. Hier eine kurze Beschreibung:

Das Bürgerhaus teilt sich den Eingang mit dem Schwimmbad und befindet sich an der Seite der Tiefgarageneinfahrt der Düsseldorfer Arcaden.

Über dem Eingang steht ein großes "Schwimm'in Bilk" Logo. Hinter der Tür direkt links zu den zwei Aufzügen, dann in den 2. Stock hochfahren. Der Eingang zum Raum 1 liegt direkt links, wenn man aus dem Aufzug kommt.

>>> Eingang in Google Street View

Einleitung

Das Python Meeting Düsseldorf ist eine regelmäßige Veranstaltung in Düsseldorf, die sich an Python Begeisterte aus der Region wendet.

Einen guten Überblick über die Vorträge bietet unser PyDDF YouTube-Kanal, auf dem wir Videos der Vorträge nach den Meetings veröffentlichen.

Veranstaltet wird das Meeting von der eGenix.com GmbH, Langenfeld, in Zusammenarbeit mit Clark Consulting & Research, Düsseldorf:

Programm

Das Python Meeting Düsseldorf nutzt eine Mischung aus Open Space und Lightning Talks, wobei die Gewitter bei uns auch schon mal 20 Minuten dauern können :-)

Lightning Talks können vorher angemeldet werden, oder auch spontan während des Treffens eingebracht werden. Ein Beamer mit XGA Auflösung steht zur Verfügung. Folien bitte als PDF auf USB Stick mitbringen.

Lightning Talk Anmeldung bitte formlos per EMail an info@pyddf.de

Kostenbeteiligung

Das Python Meeting Düsseldorf wird von Python Nutzern für Python Nutzer veranstaltet.

Da Tagungsraum, Beamer, Internet und Getränke Kosten produzieren, bitten wir die Teilnehmer um einen Beitrag in Höhe von EUR 10,00 inkl. 19% Mwst. Schüler und Studenten zahlen EUR 5,00 inkl. 19% Mwst.

Wir möchten alle Teilnehmer bitten, den Betrag in bar mitzubringen.

Anmeldung

Da wir nur für ca. 20 Personen Sitzplätze haben, möchten wir bitten, sich per EMail anzumelden. Damit wird keine Verpflichtung eingegangen. Es erleichtert uns allerdings die Planung.

Meeting Anmeldung bitte formlos per EMail an info@pyddf.de

Weitere Informationen

Weitere Informationen finden Sie auf der Webseite des Meetings:

http://pyddf.de/

Viel Spaß !

Marc-Andre Lemburg, eGenix.com

24 Apr 2014 9:00am GMT

Two OpenStax team members were featured guests on the Floss Weekly podcast this morning. Floss Weekly covers open source software for the TWIT podcast network. Kathi Fletcher and Ross Reedstrom told the history of the project along with a discussion of the work we are currently doing on OpenStax CNX. The podcast is available to view or download from TWIT.

23 Apr 2014 9:10pm GMT

Plone developers constantly search for more efficient ways of Plone performance. Dexterity is a new platform for content types in Plone and will be used instead of Archetypes in Plone 5. As a result there is necessity to create custom forms using Dexterity.
Quintagroup offers new Plone product - collective.easyform that generates web forms that save or mail form input. Easyform provides a Plone form builder through-the-web using fields, widgets, actions and validators.

How to use:

• Select Easyform from the Add new drop-down menu. Choose form title, description and other settings.
• Add fields or fieldsets to create a unique form that will meet your particular requirements. There are enough basic field types to satisfy any demands: File Upload, Text line (String), Integer, Yes/No, Date, Date/Time, Floating-point number, Choice, Rich Text, Image, Multiple Choice, Text, Password, ReСaptcha field.
• Continue to customize form by setting the order of fields, defining required and hidden ones, choosing validator, if necessary, and other field type specific settings.

Check out our video tutorial on collective.easyform

Try it yourself!

Collective.easyform is compatible with Plone 4.3.2. It is distributed as a Python egg and can easily be installed into your buildout similarly to other Plone packages. Visit the following pages to find more information about collective.easyform:

• Quintagroup.com page
• Collective.easyform on Plone.org

• Readthedocs documentation

• Collective.easyform on wiki
• Github repository

23 Apr 2014 2:51pm GMT

Ansible is a great tool. We've been using it at my job with a fair amount of success. When it was chosen, we didn't have a requirement for supporting Auto scaling groups in AWS. This offers a unique problem - we need machines to be able to essentially provision themselves when AWS brings them up. This has interesting implications outside of AWS as well. This article covers using the Ansible API to build just enough of a custom playbook runner to target a single machine at a time, and discusses how to wire it up to knockd, a "port knocking" server and client, and finally how to use user data in AWS to execute this at boot - or any reboot.

Ansible - A "Push" Model

Ansible is a configuration management tool used in orchestration of large pieces of infrastructure. It's structured as a simple layer above SSH - but it's a very sophisticated piece of software. Bottom line, it uses SSH to "push" configuration out to remote servers - this differs from some other popular approaches (like Chef, Puppet and CFEngine) where an agent is run on each machine, and a centralized server manages communication with the agents. Check out How Ansible Works for a bit more detail.

Every approach has it's advantages and disadvantages - discussing the nuances is beyond the scope of this article, but the primary disadvantage that Ansible has is one of it's strongest advantages: it's decentralized and doesn't require agent installation. The problem arises when you don't know your inventory (Ansible-speak for "list of all your machines") beforehand. This can be mitigated with inventory plugins. However, when you have to configure machines that are being spun up dynamically, that need to be configured quickly, the push model starts to break down.

Luckily, Ansible is highly compatible with automation, and provides a very useful python API for specialized cases.

Port Knocking For Fun And Profit

Port knocking is a novel way of invoking code. It involves listening to the network at a very low level, and listening for attempted connections to a specific sequence of ports. No ports are opened. It has its roots in network security, where it's used to temporarily open up firewalls. You knock, then you connect, then you knock again to close the door behind you. It's very cool tech.

The standard implementation of port knocking is knockd, included with most major linux distributions. It's extremely light weight, and uses a simple configuration file. It supports some interesting features, such as limiting the number of times a client can invoke the knock sequence, by commenting out lines in a flat file.

User Data In EC2

EC2 has a really cool feature called user data, that allows you to add some information to an instance upon boot. It works with cloud-init (installed on most AMIs) to perform tasks and run scripts when the machine is first booted, or rebooted.

Auto Scalling

EC2 provides a mechanism for spinning up instances based on need (or really any arbitrary event). The AWS documentation gives a detailed overview of how it works. It's useful for responding to sudden spikes in demand, or contracting your running instances during low-demand periods.

Ansilbe + Knockd = Centralized, On-Demand Configuration

As mentioned earlier, Ansible provides a fairly robust API for use in your own scripts. Knockd can be used to invoke any shell command. Here's how I tied the two together.

Prerequisites

All of my experimentation was done in EC2, using the Ubuntu 12.04 LTS AMI.

To get the machine running ansible configured, I ran the following commands:

$ sudo apt-get update
$ sudo apt-get install python-dev python-pip knockd
$ sudo pip install ansible

Note: its important that you install the python-dev package before you install ansible. This will provide the proper headers so that the c-based SSH library will be compiled, which is faster than the pure-python version installed when the headers are not available.

You'll notice some information from the knockd package regarding how to enable it. Take note of this for final deployment, but we'll be running knockd manually during this proof-of-concept exercise.

On the "client" machine, the one who is asking to be configured, you need only install knockd. Again, the service isn't enabled by default, but the package provides the knock command.

EC2 Setup

We require a few things to be done in the EC2 console for this all to work.

First, I created a keypair for use by the tool. I called "bootstrap". I downloaded it onto a freshly set up instance I designated for this purpose.

NOTE: It's important to set the permissions of the private key correctly. They must be set to 0600.

I then needed to create a special security group. The point of the group is to allow all ports from within the current subnet. This gives us maximum flexibility when assigning port knock sequences.

Here's what it looks like:

Depending on our circumstances, we would need to also open up UDP traffic as well (port knocks can be TCP or UDP based, or a combination within a sequence).

For the sake of security, a limited range of a specific type of connection is advised, but since we're only communicating over our internal subnet, the risk here is minimal.

Note that I've also opened SSH traffic to the world. This is not advisable as standard practice, but it's necessary for me since I do not have a fixed IP address on my connection.

Making It Work

I wrote a simple python script that runs a given playbook against a given IP address:

"""
Script to run a given playbook against a specific host
"""

import ansible.playbook
from ansible import callbacks
from ansible import utils

import argparse
import os, sys

parser = argparse.ArgumentParser(
    description="Run an ansible playbook against a specific host."
)

parser.add_argument(
    'host',
    help="The IP address or hostname of the machine to run the playbook against."
)

parser.add_argument(
    "-p",
    "--playbook",
    default="default.yml",
    metavar="PLAY_BOOK",
    help="Specify path to a specific playbook to run."
)

parser.add_argument(
    "-c",
    "--config_file",
    metavar="CONFIG_FILE",
    default="./config.ini",
    help="Specify path to a config file. Defaults to %(default)s."
)

def run_playbook(host, playbook, user, key_file):
    """
    Run a given playbook against a specific host, with the given username
    and private key file.
    """
    stats = callbacks.AggregateStats()
    playbook_cb = callbacks.PlaybookCallbacks(verbose=utils.VERBOSITY)
    runner_cb = callbacks.PlaybookRunnerCallbacks(stats, verbose=utils.VERBOSITY)

    pb = ansible.playbook.PlayBook(
        host_list=[host,],
        playbook=playbook,
        forks=1,
        remote_user=user,
        private_key_file=key_file,
        runner_callbacks=runner_cb,
        callbacks=playbook_cb,
        stats=stats
    )

    pb.run()

options = parser.parse_args()

playbook = os.path.abspath("./playbooks/%s" % options.playbook)

run_playbook(options.host, playbook, 'ubuntu', "./bootstrap.pem")

Most of the script is user-interface code, using argparse to bring in configuration options. One unimplemented feature is using an INI file to specify things like the default playbook, pem key, user, etc. These things are just hard coded in the call to run_playbook for this proof-of-concept implementation.

The real heart of the script is the run_playbook function. Given a host (IP or hostname), a path to a playbook file (assumed to be relative to a "plabooks" directory), a user and a private key, it uses the Ansible API to run the playbook.

This function represents the bare-minimum code required to apply a playbook to one or more hosts. It's surprisingly simple - and I've only scratched the surface here of what can be done. With custom callbacks, instead of the ones used by the ansible-playbook runner, we can fine tune how we collect information about each run.

The playbook I used for testing this implementation is very simplistic (see the Ansible playbook documentation for an explaination of the playbook syntax):

---
- hosts: all
  sudo: yes
  tasks:
  - name: ensure apache is at the latest version
    apt: update_cache=yes pkg=apache2 state=latest
  - name: drop an arbitrary file just so we know something happened
    copy: src=it_ran.txt dest=/tmp/ mode=0777

It just installs and starts apache, does an apt-get update, and drops a file into /tmp to give me a clue that it ran.

Note that the hosts: setting is set to "all" - this means that this playbook will run regardless of the role or class of the machine. This is essential, since, again, the machines are unknown when they invoke this script.

For the sake of simplicity, and to set a necessary environment variable, I wrapped the call to my script in a shell script:

#!/bin/bash
export ANSIBLE_HOST_KEY_CHECKING=False
cd /home/ubuntu
/usr/bin/python /home/ubuntu/run_playbook.py $1 >> $1.log 2>&1

The $ANSIBLE_HOST_KEY_CHECKING environment variable here is necessary, short of futzing with the ssh configuration for the ubuntu user, to tell Ansible to not bother verifying host keys. This is required in this situation because the machines it talks to are unknown to it, since the script will be used to configure newly launched machines. We're also running the playbook unattended, so there's no one to say "yes" to accepting a new key.

The script also does some very rudimentary logging of all output from the playbook run - it creates logs for each host that it services, for easy debugging.

Finally, the following configuration in knockd.conf makes it all work:

[options]
        UseSyslog

[ansible]
        sequence    = 9000, 9999
        seq_timeout = 5
        Command     = /home/ubuntu/run.sh %IP%

The first configuration section [options], is special to knockd - its used to configure the server. Here we're just asking knockd to log message to the system log (e.g. /var/log/messages).

The [ansible] section sets up the knock sequence for an machine that wants Ansible to configure it. The sequence set here (it can be anything - any port number and any number of ports >= 2) is 9000, 9999. There's a 5 second timeout - in the event that the client doing the knocking takes longer than 5 seconds to complete the sequence, nothing happens.

Finally, the command to run is specified. The special %IP% variable is replaced when the command is executed by the IP address of the machine that knocked.

At this point, we can test the setup by running knockd. We can use the -vD options to output lots of useful information.

We just need to then do the knocking from a machine that's been provisioned with the bootstrap keypair.

Here's what it looks like (these are all Ubuntu 12.04 LTS instances):

On the "server" machine, the one with the ansible script:

$  sudo knockd -vD
config: new section: 'options'
config: usesyslog
config: new section: 'ansible'
config: ansible: sequence: 9000:tcp,9999:tcp
config: ansible: seq_timeout: 5
config: ansible: start_command: /home/ubuntu/run.sh %IP%
ethernet interface detected
Local IP: 172.31.31.48
listening on eth0...

On the "client" machine, the one that wants to be provisioned:

$ knock 172.31.31.48 9000 9999

Back on the server machine, we'll see some output upon successful knock:

2014-03-23 10:32:02: tcp: 172.31.24.211:44362 -> 172.31.31.48:9000 74 bytes
172.31.24.211: ansible: Stage 1
2014-03-23 10:32:02: tcp: 172.31.24.211:55882 -> 172.31.31.48:9999 74 bytes
172.31.24.211: ansible: Stage 2
172.31.24.211: ansible: OPEN SESAME
ansible: running command: /home/ubuntu/run.sh 172.31.24.211

Making It Automatic With User Data

Now that we have a way to configure machines on demand - the knock could happen at any time, from a cron job, executed via a distributed SSH client (like fabric), etc - we can use the user data feature of EC2 with cloud-init to do the knock at boot, and every reboot.

Here is the user data that I used, which is technically cloud config code (more examples here):

#cloud-config
packages:
 - knockd

runcmd:
 - knock 172.31.31.48 9000 9999

User data can be edited at any time as long as an EC2 instance is in the "stopped" state. When launching a new instance, the field is hidden in Step 3, under "Advanced Details":

Once this is established, you can use the "launch more like this" feature of the AWS console to replicate the user data.

This is also a prime use case for writing your own provisioning scripts (using something like boto) or using something a bit higher level, like CloudFormation.

Auto Scaling And User Data

Auto Scaling is controlled via "auto scaling groups" and "launch configuration". If you're not familiar these can sound like foreign concepts, but they're quite simple.

Auto Scaling Groups define how many instances will be maintained, and set up the events to scale up or down the number of instances in the group.

Launch Configurations are nearly identical to the basic settings used when launching an EC2 instance, including user data. In fact, user data is entered in on Step 3 of the process, in the "Advanced Details" section, just like when spinning up a new EC2 instance.

In this way, we can automatically configure machines that come up via auto scaling.

Conclusions And Next Steps

This proof of concept presents an exciting opportunity for people who use Ansible and have use cases that benefit from a "pull" model - without really changing anything about their setup.

Here are a few miscellaneous notes, and some things to consider:

• There are many implementations of port knocking, beyond knockd. There is a huge amount of information available to dig into the concept itself, and it's various implementations.
• The way the script is implemented, it's possible to have different knock sequences execute different playbooks. A "poor-man's" method of differentiating hosts.
• The Ansible script could be coupled the AWS API to get more information about the particular host it's servicing. Imagine using a tag to set the "class" or "role" of the machine. The API could be used to look up that information about the host, and apply playbooks accordingly. This could also be done with variables - the values that are "punched in" when a playbook is run. This means one source of truth for configuration - just add the relevant bits to the right tags, and it just works.
• I tested this approach with an auto scaling group, but I've used a trivial playbook and only launched 10 machines at a time - it would be a good idea to test this approach with hundreds of machines and more complex plays - my "free tier" t1.micro instance handled this "stampeding herd" without a blink, but it's unclear how this really scales. If anyone gives this a try, please let me know how it went.
  
• Custom callbacks could be used to enhance the script to send notifications when machines were launched, as well as more detailed logging.

23 Apr 2014 11:47am GMT

A summary of DocSprint Munich.

22 Apr 2014 8:10pm GMT

Since I've started this site, I've been using Quills for blogging functionality in Plone. So far, it's been perfectly fine for my relatively simple needs and I haven't found a reason to change. Unfortunately, except for some work to port Quills to Plone 4, development and bugfixes on Quills has completely dried up. No new releases were made for more than a year and the mailing list was almost dead. I only realised this when I fixed some bugs while porting my site to Plone 4 and couldn't find anybody to help making a new Plone 4 compatible release.

Eventually I got hold of Tim Hicks, one of the original maintainers of Quills, and I offered to help out with maintaining Quills and to make the new releases.

So, I'm happy to announce that new Plone 4 compatible releases of the Quills packages have been made. Notably Products.Quills, Products.QuillsEnabled and quills.app (all now at version 1.8a1). This new release is labeled alpha, as I'm still very new to the Quills codebase, but I'm already using it on this blog without any major problems.

If there is anyone who is still using Quills on a legacy Plone site and would like to migrate to Plone 4, just pin your Products.Quills (and Products.QuillsEnabled) version to 1.8a1 in your versions.cfg or bulidout.cfg.

Hopefully everything goes well (did I mention it's alpha ;) but if not, feel free to let me know and I'll try to help out.

21 Apr 2014 9:41am GMT

collective.contact.core is a Plone add-on that helps to manage organizations and staff in Plone (main developers are Vincent Fretin and Cedric Messiant). This product provides directory that can contain contact information for different content types: organizations/sub-organizations, persons,and positions. Contact info option depends on for which content types you set the IContactDetails behavior so it can cover many different uses.

Easy in use:

1. Add directory to your website and insert all the additional information required. You'll need to specify types of positions and organizations that will be used (e.g. Faculty/Staff/Students for universities). Don't worry about filling out the form, since it can be edited at any time later.

   

2. Create organization(s) in the directory. Depending on the hierarchy, add other organizations (they may correspond to units, divisions, departments, etc.). An organization can contain position (e.g Dean, secretary, SEO) that will be connected with person (a physical person). Choose Organization/Position from the Add new drop-down menu or click on Create contact to divaricate your directory.

   

A person content type can hold one or more positions or be member of one or more organizations. All contact types have optional fields with variety of contact information, including phone, cell phone, fax, email, address, zip code, etc. Such data management is very suitable for universities.

collective.contact.core can be useful for all kinds of organizations, despite their size, number of employees or subdivisions. Created directory is easy to manipulate and can be branched or edited at any time.

collective.contact.core adds new content types, but preserves Plone functionality, especially concerning users' rights. Every 'organization' content type is similar to folder, thus you can specify in the Sharing tab what rights users have . Moreover, default Plone search is very efficient when you want to search for a specific person or position on all the website.

Use collective.contact.core to arrange your organization and contact information.

Contributors:

• Gauthier Bastien, IMIO
• Vincent Fretin, Ecreall
• Stéphan Geulette, IMIO
• Cédric Messiant, Ecreall
• Frédéric Peters, Entr'ouvert
• Thomas Desvenain, Ecreall

More information:

• Plone.org
• Pypi
• Github

18 Apr 2014 1:07pm GMT

We are pleased to be offering a two day training session at the 2014 Plone Symposium Midwest this year. The two-day course will cover Pyramid development topics, aimed at Plone developers.

For details, please see the training page.

17 Apr 2014 7:44pm GMT

Thanks to an awesome contribution by Alec Munro, Morepath, your friendly neighborhood Python micro framework with super powers, has just gained Python 3 support!

Developing something new while juggling the complexities of Python 2 and Python 3 in my head at the same time was not something I wanted to do -- I wanted to focus on my actual goals, which was to create a great web framework.

So then I had to pick one version of Python or the other. Since my direct customer use cases involves integrating it with Python 2 code, picking Python 2 was the obvious choice.

But now that Morepath has taken shape, taking on the extra complexity of supporting Python 3 is doable. The Morepath test coverage is quite comprehensive, and I had already configured tox (so I could test it with PyPy). Adding Python 3.4 meant patiently going through all the code and adjusting it, which is what Alec did. Thank you Alec, this is great!

Morepath's dependencies (such as WebOb) already had Python 3 support, so credit goes to their maintainers too (thanks Chris McDonough in particular!). This includes the Reg library, which I polyglotted to support Python 3 myself a few months ago.

All this doesn't take away from my opinion that we need to do more to support the large Python 2 application codebases. They are much harder to transition to Python 3 than well-tested libraries and frameworks, for which the path was cleared in the last 5 years or so.

[update: this is still in git; the Morepath 0.1 release is Python 2 only. But it will be included in the upcoming Morepath 0.2 release]

17 Apr 2014 2:25pm GMT

I was introduced to Codio.com by +Rok Garbas. It turns out to be a very nice platform for developing Plone projects. So far what I like is that every Codio box pretty much ships with all the Plone dependencies while at the same time having a full suite of Node based tools (important for modern Javascript development), this is a great time saver on new projects. These are still early days so I

17 Apr 2014 3:21am GMT

The plone.org website is safe from the Heartbleed bug and, as such, plone.org passwords have not been disclosed.

14 Apr 2014 9:01pm GMT