21 Sep 2016

feedPlanet Plone

Martijn Faassen: Is Morepath Fast Yet?

Morepath is a Python web framework. But is it fast enough for your purposes?

Does performance matter?

Performance is one of the least important criteria you should use when you pick a Python web framework. You're using Python for web development after all; you are already compromising performance for ease of use.

But performance makes things seem easy. It boils down the whole choice between web frameworks to a single seemingly easy to understand variable: how fast is this thing? All web frameworks are the same anyway, right? (Wrong). We don't want the speed of our application be dragged down by the web framework. So we should just pick the one that is fastest. Makes total sense.

It makes total sense until you take a few minutes to think about it. Performance, sure, but performance doing what? Performance is notoriously difficult to measure. Sending a single "hello world" response? Parsing complex URLs with multiple variables in them? HTML template rendering? JSON serialization? Link generation? What aspect of performance matters to you depends on the application you're building. Why do we worry so much about performance and not about features, anyway?

Choosing a web framework based on performance makes no sense for most people. For most applications, application code dominates the CPU time spent. Pulling stuff out of a database can take vastly more time than rendering a web response.

What matters

So it makes sense to look at other factors when picking a web framework. Is there documentation? Can it do what I need it to do? Will it grow with me over time? Is it flexible? Is it being maintained? What's the community like? Does it have a cool logo?

Okay, I'm starting to sound like someone who doesn't want to admit the web framework I work on, Morepath, is atrociously slow. I'm giving you all kinds of reasons why you should use it despite its performance, which you would guess is pretty atrocious. It's true that the primary selling point of Morepath isn't performance -- it's flexibility. It's a micro-framework that is easy to learn but that doesn't let you down when your requirements become more complex.

Morepath performance

I maintain a very simple benchmark tool that measures just one aspect of performance: how fast a web framework at the Python WSGI level can generate simple "hello world" responses.


I run it against Morepath once every while to see how we're doing with performance. I actually care more about what the framework is doing when Morepath generates the response than I care about the actual requests per second it can generate. I want Morepath's underlying complexity to be relatively simple. But since performance is so easy to think about I take advantage of that as a shortcut. I treat performance as an approximation of implementation complexity. Plus it's cool when your framework is fast, I admit it.

The current Morepath development version takes about 5200 ms per 100,000 requests, which translates to about 19200 requests per second. Let's see how that compares to some of the friendly competition:

                ms     rps  tcalls  funcs
django       10992    9098     190     85
flask        15854    6308     270    125
morepath      5204   19218     109     80

So at this silly benchmark, Morepath is more than twice as fast as Django and more than three times faster than Flask!

Let me highlight that for marketing purposes and trick those who aren't reading carefully:

Morepath is more than 2 times faster than Django and more than 3 times faster than Flask

Yay! End of story. Well, I gave a bit of a selective view just now. Here are some other web frameworks:

                ms     rps  tcalls  funcs
bottle        2172   46030      53     31
falcon        1539   64961      26     24
morepath      5204   19218     109     80
pyramid       3920   25509      72     57
wheezy.web    1201   83247      25     23

I'm not going to highlight that Bottle is more than two times faster at this silly benchmark nor that Falcon is more than three times faster. Let's not even think about wheezy.web.

I think this performance comparison actually highlights my point that in practice web framework performance is usually irrelevant. People aren't flocking to wheezy.web just because it's so fast. People aren't ignoring Flask because it's comparatively slow. I suspect many are surprised are surprised Flask is one of the slowest frameworks in this benchmark, as it's a lightweight framework.

Flask's relatively slow performance hasn't hurt its popularity. This demonstrates my point that web framework performance isn't that important overall. I don't fully understand why Flask is relatively slow, but I know part of the reason is werkzeug, its request/response implementation. Morepath is actually doing a lot more sophisticated stuff underneath than Flask and it's still faster. That Pyramid is faster than Morepath is impressive, as what it needs to do during runtime is similar to Morepath.

Let's look at the tcalls column: how many function calls get executed during a request. There is a strong correlation between how many Python function calls there are during a request and requests per second. This is why performance is a decent approximation of implementation complexity. It's also a clear sign we're using an interpreted language.

How Morepath performance has changed

So how has Morepath's performance evolved over time? Here's a nice graph:

Morepath performance over time

So what does this chart tell us? Before its 0.1 release when it still used werkzeug, Morepath was actually about as slow as Flask. After we switched to webob, Morepath became faster than Flask, but was still slower than Django.

By release 0.4.1 a bunch of minor improvements had pushed performance slightly beyond Django's -- but I don't have a clear idea of the details. I also don't understand exactly why there's a performance bump for 0.7, though I suspect it has to do with a refactor of application mounting behavior I did around that time -- while that code isn't exercised in this benchmark, it's possible it simplified a critical path.

I do know what caused the huge bump in performance in 0.8. This marked the switch to Reg 0.9, which is a dispatch library that is used heavily by Morepath. Reg 0.9 got faster, as this is when Reg switched to a more flexible and efficient predicate dispatch approach.

Performance was stable again until version 0.11, when it went down again. In 0.11 we introduced a measure to make the request object sanitize potentially dangerous input, and this cost us some performance. I'm not sure what caused the slight performance drop in 0.14.

And then there's a vast performance increase in current master. What explains this? Two things:

  • We've made some huge improvements to Reg again. Morepath benefits because it uses Reg heavily.
  • I cheated. That is, I found work that could be skipped in the case no URL parameters are in play, as in this benchmark.

Skipping unnecessary work was a legitimate improvement of Morepath. The code now avoids accessing the relatively expensive GET attribute on the webob request, and also avoids a for loop through an empty list and a few if statements. In Python, performance is sensitive to even a few extra lines of code on the critical path.

But when you do have URL parameters, Morepath's feature that lets you convert and validate them automatically is pretty nice -- in almost all circumstances you should be glad to pay the slight performance penalty for the convenience. Features are usually worth their performance cost.

So is Morepath fast yet?

Is Morepath fast yet? Probably. Does it matter? It depends. What benchmark? But for those just skimming this article, I'll do another sneaky highlight:

Morepath is fast. Morepath outperforms the most popular Python web frameworks while offering a lot more flexibility.

21 Sep 2016 11:36am GMT

19 Sep 2016

feedPlanet Plone

Benoît Suttor: collective.preventactions

See source code on https://github.com/collective/collective.preventactions

Sometimes we have some folders or some documents which should have not been deleted.

But our customers/users have deleted them and it occurs some errors on application.

So we decided to create a new package to prevent delete or/and move actions.

How it works

We add marker interface to objects which can not be deleted or renamed (= moved).

We subscribe all IItem objects to OFS.interfaces.IObjectWillBeRemovedEvent and OFS.interfaces.IObjectWillBeMovedEvent

When one of these events is received, and object is marked as not deleted or not renamed, we raised an exception and object is not deleted or moved.

In the futur, we expect to add a dashboard to have a view of all contents with these markers interfaces to easily use it.

You can also set some contents not deleteable (for example) as this in your setuphandler :

from collective.preventactions.interfaces import IPreventDelete
from plone import api
from zope.interface import alsoProvides

def post_install(context):
obj = api.content.get('/Plone/content-not-deleteable')
alsoProvides(obj, IPreventDelete)

Now you can have a look at source code of package and try it.

19 Sep 2016 8:56pm GMT

16 Sep 2016

feedPlanet Plone

Andreas Jung: Press release: XML-Director 2.0 released

XML-Director is our Plone-based integration platform for building high-quality enterprise-grade publishing solutions.

16 Sep 2016 1:07pm GMT

13 Sep 2016

feedPlanet Plone

Andreas Jung: Don't use "docker" in Github repo names or as Twitter handles

The strict trademark guidelines of Docker Inc. can easily be turned into a case against any open-source programmer or project using "docker" as a repository name or for example as a Twitter handle. Act now!

13 Sep 2016 12:26pm GMT

07 Sep 2016

feedPlanet Plone

Benoît Suttor: New package: collective.geo.faceted

We use collective.geo suite for geolocatisation on some of our projects. We also use eea.facetednavigation for easily find content into our website or application. So we decided to add a map view for eea.facetednavigation.

Why did we create collective.geo.faceted ?

We use collective.geo suite for geolocatisation for some of our projects. We also use eea.facetednavigation to easily find content into our website or application.

So we decided to add a map view for eea.facetednavigation and we created collective.geo.faceted.

How it works


We prefer to use Leaflet than OpenLayers, because it seems easier to use for us.

So we decided to use collective.geo.leaflet machinery for map creation.


We use geojson standard to add points on map. It is a famous standard used to geo content.

The view created for "faceted" will simply update the geojson and this geojson will also update the map. For the generation of geojson, we extended collective.geo.json view.


Map is added into a viewlet dedicated to the faceted view. We choose to use a viewlet out of 'content-core slot' (content-core slot is used by faceted to update automatically the contents). Indeed each technology (faceted and map) uses singular javascript , it seems better to not mix both technology.

Plone objects are on map and they are updated thanks to these lines of code:

jQuery(document).ready(function() {

This code fetches "faceted" events when "faceted" has modifed its criterias and used update_map javascript function to update new geojson on map.

Image is better than words:


This package is tested for Plone 4 with plone.app.contenttypes used as default Plone content types.

Maybe in future we should add a profile like plone5 e.g. example.p4p5 or create a branch plone4 on github and make master branch used to plone5.

07 Sep 2016 7:34pm GMT

Martijn Faassen: Introducing Bob Strongpinion

I posted an article about programming yesterday. (punctuated equilibrium in software development) In it I try to share some insights I've had about software development, and illustrate it with a library I work on that I think is interesting. I posted it to reddit /r/programming, obviously hoping to get a bit of attention with it. I think the topic is interesting to other developers, and writing it took a bit of time. Besides, I'm human enough to want positive attention.

My reddit post quickly sank from sight with nary an upvote. That got me thinking about what kind of posts I could make that would draw more attention. I joked a bit that having strong opinions boldly spoken gets more attention -- but I should blame the topic and my writing more than anything else. To blame the outside world makes you stop learning from it, and I want to learn.

In engineering there are always trade offs, and it's important to be aware of them. It is also nicer to be respectful of works that people have put a lot of time and effort in. And if you disregard them you are also less likely to learn and grow. So I won't go out there and say Morepath, the Python web framework I work on, is better than, say, Flask or Django or Pyramid. It depends on what you're doing. When I compare Morepath to other web frameworks, I probably don't excite you; it may in fact be sleep inducing.

I think Morepath is great. I think it solves problems that many Flask and Django developers don't even fully realize they actually have. That's in fact part of the problem I have in explaining it. But I also recognize that in many circumstances other frameworks are the better choice, for a variety of reasons.

But sometimes I wish I was more bold in expressing my opinions than I usually am. Sometimes I wish I was more like Bob Strongpinion.

Who is Bob Strongpinion?

Bob Strongpinion has strong opinions and conviction. He would blog saying all other web frameworks suck compared to Morepath. He'd have a lot blog posts with eye-catching titles:

Django was yesterday. The future belongs to Morepath.

10 Reasons Morepath is just plain better.

You're doing configuration wrong. This library gets it right. (Dectate)

Single dispatch is over. Get ready for predicate dispatch. (Reg)

Why routing to the model instead of the view is winning.

Bob Strongpinion posts a lot of articles with lists, and they're all 10 items explaining that what he likes is plain better. Bob Strongpinion doesn't believe in trade offs in engineering. There's better, and there's worse, and what he likes is just plain better. Bob Strongpinion knows there's ONE tool that's right for EVERY job.

If someone doesn't agree with Bob Strongpinion's choice of tools, they're either stupid, or more likely, evil. Bob Strongpinion may not have more experience in a problem domain than you do, but it's the RIGHT experience so he's still right.

When Bob Strongpinion makes a pronouncement in a comment thread, the case is closed. Disagree with him and incur his righteous wrath.

Engineering projects with Bob Strongpinion on it always succeed as he picks the right tools. And when they don't, it's someone else's fault. When Bob Strongpinion doesn't get to use his favorite tools it's no wonder the project failed.

When Bob Strongpinion uses an operating system, it's because it's the best one for everyone, unless they're too wimpy, so he's still elite. Bob Strongpinion definitely believes systemd is a great conspiracy to destroy Linux.


I think Bob Strongpinion would get a lot more upvotes on reddit. He'd get attention; some of it negative, but a lot of it would be positive.

"Yeah, Strongpinion! You tell the truth about software development!"

And he's always right. Obvious comparisons to certain public figures come to mind. You can see why his mindset would be comfortable, so it's understandable why Bob Strongpinion lives among us.

I sometimes wish I could be more like Bob Strongpinion when I promote my own work. As you can see from the above, I can channel him pretty well. I snuck him in while rejecting him, how sneaky! The Strongpinion must be strong in me. I've even done a list of 10 bullet points before. It got some upvotes.

But in the end I keep choosing not to be him.

07 Sep 2016 12:37pm GMT

06 Sep 2016

feedPlanet Plone

Martijn Faassen: Punctuated Equilibrium in Software

Punctuated equilibrium

Punctuated equilibrium is a concept in evolution theory. It was developed to explain a feature of the fossil record: biological species appear quite suddenly and then tend to be in relative stasis for a long period, only undergoing very gradual changes. The species is in equilibrium with its environment. Then suddenly this stasis is punctuated: there is a relatively brief period where a large series of changes occur. This results in the evolution of a new species. The rapid changes can be brought on by changes in the environment, or by a lucky mutation in a single individual that opens up a whole set of possibilities for subsequent changes.

I've noticed that software too can evolve with a pattern of punctuated equilibrium. I'll illustrate this using a Python library that I work on: Reg. To explain how it evolved I need to go into some detail about it. Luckily, Reg is quite an interesting little library.

Reg is predicate dispatch implementation for Python. It didn't start out that way, but that's what it is now. The Morepath web framework, which I also work on, uses Reg to enable some powerful features. I'll refer to Morepath a few places in this article as it provides use cases for Reg.

Reg ancestry

The ancestor of Reg is the zope.interface library, which was created around the year 2002 by Jim Fulton. It is still in very active use by large projects like the Pyramid web framework and the Plone CMS.

zope.interface lets you define interfaces for Python objects. It's similar to the Python abc module, though zope.interface came earlier and is more general.

zope.interface also implements a registry that maps interface (or type) to interface to support an adapter lookup pattern: you can adapt an object of an interface (or type) to an object with an interface you want.

In a web application you could for instance look up a HTML view interface (API) for a given content object such as a document or a someone's address, or whatever other type of content object you may have in your system. We'll give an example of this in code when we get to Reg.

The genesis of zope.interface took a few years and involved a predecessor project. Like Reg itself it underwent evolution by punctuated equilibrium in its early years. I describe this a bit in the Reg history document.

I try to keep history documents describing the evolution of various projects I work on, as I think they can provide insight into a project beyond what the rest of the documentation can bring. If you like software history, see Morepath history, Dectate history and Reg history. (The Reg history overlaps with this article, so if you're curious to learn more, do check it out later.)

After 2002 zope.interface became stable: its API hasn't changed much, and neither has its implementation. There were a few small tweaks here and there, in particular to add Python 3 compatibility, but that's it. At some point around 2009 I made some proposals to improve its API, but got nowhere. That's when I started playing around with the idea to reimplement it for myself.

The genesis of Reg

It often takes a period of experimentation and play to create something new. It's important during this phase not to think too much about immediate practical goals. Focus on a few core features that interest you; don't worry about it covering everything. Banish any thoughts about backwards compatibility and how to upgrade existing large code bases; that would be detrimental to the spirit of playfulness.

"Why are you reimplementing zope.interface, Martijn?"

"Just for fun, I don't expect anyone to use this."

After a few years of occasional play with various ideas I had concerning zope.interface, they finally started to come together in 2013. The goal of Reg at the time was straightforward: it was to be like zope.interface, but with an implementation I could understand, and with a streamlined API.

I'm going to show sample code now. Be aware that the sample code in this article may be somewhat fictional for educational purposes.

Reg initially worked like this:

# the view API
class IView(reg.Interface):
    def __call__(self):
        "If you call this, you get a web representation."

# register implementation of the view API for Document and
# HTTP Request classes
@IView.register(Document, Request)
class DocumentToViewAdapter:
    def __init__(self, doc, request):
        self.doc = doc
        self.request = request

    def __call__(self):
        return "<p>%s</p>" % self.doc.content

# can register other implementations, for example for Address and
# Request

# create instances we can look up for
doc = Document()
request = Request()

# look up the view adapter for a specific object, in this case a document
# The specific implementation you find depends on the class of doc and
# request arguments
view = IView.adapt(doc, request)
# and get the representation
html = view()

Here we define an IView interface. You can then register adapters for this view that take parameters (the object and a HTTP request) and turn it into an object that can create a HTML representation of the parameters.

Major transition to generic functions

I worked with the zope.interface version of Reg for a while in the context of the Morepath web framework. This gave me some practical experience. I also talked about Morepath and Reg in public and got some feedback. Even minimal feedback is great; it sets thoughts into motion. I quickly realized that Reg's API could be simplified if I centered it around generic functions with multiple dispatch instead of interfaces and adapters. Something like this:

# generic function definition instead of interface
def view(obj, request):
    """"If you call this, you get a web representation."""
    raise NotImplemented

# an implementation for Document and Request
@view.register(Document, Request)
def document_view(obj, request):
    return "<p>%s</p>" % obj.content

# get representation for document by calling view()
html = view(doc, request)

This looks simpler. The interface definition is gone, the adapter class is gone. We just have a function that dispatches on the type of its arguments.

Reg worked like this for over a year. It was stable. I didn't foresee any more large changes.

Major transition to predicate dispatch


I created a predicate registry implementation. This lived inside of a module in Reg, but it could as well have been in a totally different library: the code was unrelated to the rest of Reg.

The use case for this predicate registry was Morepath view lookup. The predicate system let you register objects by a selection of keys. You could look up a view based on the request_method attribute (HTTP GET, POST, etc) of the request object, for instance, not just by type.

Two things came together in the fall of 2014:

  • I realized that it was annoying that the multiple dispatch system automatically dispatched on all arguments to the function -- in many cases that wasn't required.
  • I needed the predicate system to understand about types and inheritance. The multiple dispatch system in Reg understood types but not predicates, and the predicate dispatch system understood predicates but not types.

Then I realized that if I generalized Reg and turned it into a predicate dispatch system, I could actually unify the two systems. The dialectic (thesis, antithesis, synthesis) is a strong force for creativity in software development

With predicate dispatch you can dispatch on any aspect of the arguments to a function; not just its class but also any attribute of it. You can still do multiple dispatch as before: dispatch on the type of an argument is now just be a special case. Since arguments now needed a description of what predicate they dispatch on, I could also have arguments that are ignored by the dispatch system altogether.

This is when I finally understood some of the reasoning behind the PEAK-Rules library, which is a Python predicate dispatch implementation that predates Reg by many years. Almost everything has been implemented before, but with reimplementation you gain understanding.

With that insight, the equilibrium was punctuated, and Reg underwent rapid change again. Now it looked like this:

# dispatch function instead of generic function
# note how we explicitly name the arguments we want to match on
# (obj, request) in the predicates, and how we match on the
# request_method attribute. match_instance still matches on type.
                            lambda request: request.request_method))
def view(obj, request):
    raise NotImplemented

# an implementation for GET requests of documents
@view.register(obj=Document, request_method='GET')
def document_view(obj, request):
    return "<p>%s</p>" % obj.content

# get representation for document by calling view()
html = view(doc, request)

When we define a dispatch function, we can now precisely describe on what aspects of the arguments we want to do dispatch. When we register an implementation, we can use more than just the types of the arguments. We can also have arguments that do not play a role in dispatch at all.

This system allows Morepath to have views looked up by the type of the instance being represented, the last name in the path (/edit, /details), the HTTP request method (GET, POST, etc), and the type of object in the POST body, if any.

I succeeded in making predicates participate in the cache that I already had to speed up multiple dispatch, so this change both simplified and increased performance.

Major transition to dispatch methods

After this huge change, Reg was stable again for almost 2 years. I didn't think it needed further major changes. I was wrong.

The trigger was clear this time, as it was a person. Stefano Taschini, who started contributing to the Morepath web framework project. Stefano's a very smart guy, so I'm doing my best to learn from him. Listen hard, even if your impulse, like mine, is to defend your design decisions. I was lucky that Stefano started to think about Reg. So while Reg seemed outwardly stable, the pressure for change was slowly building up.

In the summer of 2016 Stefano and I had a lot of discussions and created a few branches of Reg and Morepath. All that work has now landed in master of Reg and Morepath. The implementation of Reg is simpler and more explicit, and its performance has been increased. Yet again we had a major punctuation in the evolution of Reg.

I mentioned before how the code samples in this article are somewhat fictional. One fiction is the way you register implementations in Reg. It didn't actually work this way until now. Instead of this:

@view.register(obj=Document, request_method='GET')
def document_view(obj, request):
    return "<p>%s</p>" % obj.content

until very recently, you'd write something like this:

r = reg.Registry()

def document_view(obj, request):
    return "<p>%s</p>" % obj.content

r.register(view, document_view, obj=Document, request_method='GET')

So, we used to have an explicit registry object in Reg. This was there because of a use case of Reg that I haven't told you about yet: we need it to support separate dispatch contexts in the same run-time. Morepath uses this let you compose a larger web application out of multiple smaller ones, each with their own context.

To control which context Reg used you could pass in a special magic lookup parameter to each dispatch function call:

view(doc, request, lookup=registry)

Dispatch implementations needed access to the context too. They could get to it by defining a magic lookup argument in their signature:

def document_view(obj, request, lookup):
    return "<p>%s</p>" % process_content(obj.content, lookup=lookup)

If you didn't specify the lookup, an implicit thread-local lookup was used.

All this wasn't ideal. During the creation of Reg I was fully aware of Python's mantra "explicit is better than implicit", but I didn't know a better way to make context-specific dispatch calls work. I tried my best to at least isolate the implicit behavior in a well-controlled portion of Reg, and to allow a fully explicit option with lookup arguments, but the machinery to support all this was more complex than I'd wish.

When Stefano and I discussed this we came up with the following ideas:

  • Remove multiple registries. Instead allow simple registration on dispatch functions as we've already seen in the examples above. Each function keeps its own private registry. Stefano pushed hard for this while I was resistant, but he was right.
  • To control context, introduce the notion of dispatch methods. Build dispatch methods on dispatch functions.

A dispatch method is associated with a context class:

class Context:
    def foo(self, obj):
        raise NotImplementedError()

You can register implementations with the method:

def implementation(self, obj):

When you call the dispatch method you call it in its context:

c = Context()

Each subclass of the context class creates a new context, with a fresh set of registrations:

# a completely new and clean context
class NewContext(Context):

Instead of a magic lookup argument to call a generic function in a particular context, you simply use self as the instance of the context. This fits Python a lot better and is faster as well. Magical lookup arguments were gone. Thread-local implicit context was gone too. All is well. With Stefano on board now, Reg's bus factor has doubled too.

A new period of stasis?

Large changes create room for further changes. We've already seen a lot of follow-on changes, especially in the area of performance, and I think we haven't seen the end of this yet. I am starting to understand now why PEAK-Rules has AST manipulation code. We may not quite have reached a point of equilibrium yet.

But after that performance engineering, surely Reg won't need any further drastic changes anymore? I can't think of any. But I've been here several times before. zope.interface is assumed to be done; Reg isn't. If you assume a project is done that could become a self-fulling prophecy and cause the project to stagnate before its time.

Dare to change

Reg is a piece of software that sits at the lower levels of our software stack. Morepath is on top of it, and applications built with it are on top of that. I've been impressed by how much of the underlying codebase of Morepath we've been able to change without breaking Morepath applications much.

Of course the amount of code written with Morepath is insignificant compared to that written with web frameworks like Django or Flask or Pyramid, so we can still afford to be be bold -- now, when the community is still small, before many more people join us, is the time to make changes. That is why we can play with a cool technique like predicate dispatch that while not new, is still unfamiliar to many. It is also a creative challenge to make the unfamiliar approachable.

If you're interested in any of this and want to talk to us, the Morepath devs are one click away.

Self-serving mercenary statement: if you need a developer and like what you hear, talk to me -- I'm on the lookout for interesting projects.

06 Sep 2016 10:58am GMT

01 Sep 2016

feedPlanet Plone

Makina Corpus: The world's simplest Python template engine

A template engine is a component able to render some data using a given string template.

We use it extensively in web development (that's not surprising because web development is mainly about reading some data and render them as strings, that's what I say when someone ask me about my job).

So there are already a lot of Python based template engines (Jinja2, Mako, Chameleon, ...).

The purpose of this article is to build a very basic template engine based only on the Python standard string class formatting capacities.

You probably know...

You probably know .format() replaces the old "%" based formatting solution:

>>> "My name is %s and I am %d" % ("Eric", 41)
'My name is Eric and I am 41'
>>> "My name is {name} and I am {age}".format(name="Eric", age=41)
'My name is Eric and I am 41'

It allows to perform all the typical formatting features, for instance:

>>> '{:^20}'.format('centered')
' centered '
>>> '{:^20}'.format('align right')
' align right '
>>> '{:>20}'.format('align right')
' align right'
>>> '{:<20}'.format('align left')
'align left '

and many other things (have a look to https://pyformat.info/).

Did you know?

format() is also able to access attributes or items of the parameters.

For instance with a dictionary, we can do this:

>>> 'Name: {person[name]}, age: {person[age]}'.format(person={'name': 'Eric', 'age': 41})
'Name: Eric, age: 41'

And the same goes with attributes:

>>> class Song(object):
... title = 'Where is my mind'
>>> 'My favorite song is: {song.title}'.format(song=Song())
'My favorite song is: Where is my mind'

That's really cool. It starts looking like a template engine, right?

Ok, but few things are missing

What we usually expect from a template engine is to be able to:

Let's see how we can handle that.

Calling methods

A method is an attribute of an object, we can call attribute, why couldn't wecall a method? Let's try:

>>> 'My name is {name.upper}'.format(name='eric')
'My name is <built-in method upper of str object at 0x7f67dc8d1630>'

Yeah, not exactly what we expected...

An interesting feature of format() is the format specification: instead of just inserting a field with {field}, we can specify a format like this: {field:spec}.

That's exactly what we do with float for instance:

>>> "{:.3}".format(3.14159)

Well, it is actually very easy to implement our own spec by derivating the Formatter class. So let's implement a ':call' spec in charge of calling the current field:

class SuperFormatter(string.Formatter):

    def format_field(self, value, spec):
        if spec == 'call':
            return value()
            return super(SuperFormatter, self).format_field(value, spec)

We can use it that way:

>>> sf.format('My name is {name.upper:call}', name="eric")
'My name is ERIC'



Similarly, we can implement a :repeat spec,

class SuperFormatter(string.Formatter):

    def format_field(self, value, spec):
        if spec.startswith('repeat'):
            template = spec.partition(':')[-1]
            if type(value) is dict:
                value = value.items()
            return ''.join([template.format(item=item) for item in value])
            return super(SuperFormatter, self).format_field(value, spec)

Here, we pass a parameter to the spec to provide the template to use when we iterate on the loop items. So the resulting format is: <field>:repeat:<template>.

This subtemplate is a regular format() template where we escape the curly brackets by doubling them, and where the only field is the loop variable (named item).

So we can use it like this:

>>> sf.format('''Table of contents:
... {chapters:repeat:Chapter {{item}}
... }''', chapters=["I", "II", "III", "IV"])
'''Table of contents:
Chapter I
Chapter II
Chapter III
Chapter IV


Let's also implement a :if spec to test the field value, and then display or not the subtemplate:

class SuperFormatter(string.Formatter):

    def format_field(self, value, spec):
        if spec.startswith('if'):
            return (value and spec.partition(':')[-1]) or ''
            return super(SuperFormatter, self).format_field(value, spec)

At first, it seems stupid, because it looks like it will only be able to conditionnally display static portions, like this:

>>> sf.format('Action: Back / Logout {manager:if:/ Delete}', manager=False)
'Action: Back / Logout '
>>> sf.format('Action: Back / Logout {manager:if:/ Delete}', manager=True)
'Action: Back / Logout / Delete'

What if we want to render conditionnally a portion of template containing fields, like this:

>>> sf.format('Action: Back / Logout {manager:if:/ Delete {id}}', manager=False, id=34)
'Action: Back / Logout '
>>> sf.format('Action: Back / Logout {manager:if:/ Delete {id}}', manager=True, id=34)
'Action: Back / Logout / Delete 34'

Hey that works! That was unexpected. To be honest, I first wrote the test exactly like this, and I expected it to fail, but it was not! Why is that?

Because here the curly brackets are not escaped, so they are processed by the main format() call. So cool!

Here we go

That's it, we have a pretty nice and complete template engine.

The full code of our world's simplest Python template engine is here, the implementation itself is 10 lines long, Python is a really powerful language!

If you have any funny idea to improve it (remember, I want to keep it short), pull requests are welcome.

The objective is mostly to have fun with Python and demonstrate what the standard Formatter class is able to do, but, having a second though, I might intregate it in Rapido :).

01 Sep 2016 12:00am GMT

Makina Corpus: Paypal tracking with Rapido

Paypal Instant Payment Notification (IPN)

If we put a Paypal button somewhere in a website, it is nice to know if the user got through the payment process properly and did not just canceled in the middle.

To achieve this, Paypal provides an asynchronous messaging service named Instant Payment Notification (IPN). Here is the process:

How to do it with Rapido

Creating the Paypal button is easy, we just copy/paste the core provided by Paypal in a block (see the Rapido documentation to learn how to create blocks) and we add our "custom" variable:


type: BASIC


def trackingid(context):
currentUser = context.api.user.get_current()
return currentUser.getUserName() # or anything else relevant


<form action="https://www.sandbox.paypal.com/cgi-bin/webscr" method="post" target="_top">
<input type="hidden" name="custom" value="{trackingid}"/>
<input type="hidden" name="cmd" value="_s-xclick">
<input type="hidden" name="hosted_button_id" value="XXXXXXXXXX">
<input type="image" src="https://www.sandbox.paypal.com/fr_FR/FR/i/btn/btn_paynowCC_LG.gif" border="0" name="submit" alt="PayPal, le réflexe sécurité pour payer en ligne">
<img alt="" border="0" src="https://www.sandbox.paypal.com/fr_FR/i/scr/pixel.gif" width="1" height="1">

Note: make sure we do not insert the full block form but just its children with our Diazo rule because we would get a form into a form, so the rule will be like that:

<after css:content="h1.documentFirstHeading">
<include href="@@rapido/myapp/block/pay" css:content-children="form[name='pay']"/>

Now let's see how to implement the listener that will get the Paypal notification. At some point, we will need to create a POST requests, so we need an extra library for that. We will use the well-known Python requests library, so we need to declare it safe for Rapido.

Hence we add the following somewhere in our custom theme module, for instance in __init__.py (or anywhere else):

import requests
from rapido.core import app
app.safe_modules.requests = requests

Warning: allowing to use a module like Requests in Rapido can have consequences regarding security, so if you do not control who is allowed to create Rapido apps (by default, only managers), do not do it.

And now, we create a block named ipn, containing an element named listener, the IPN URL we will declare to Paypal will be:


Here is the implementation:


type: BASIC


def listener(context):
trackingid = context.request.form['custom']
paypal_params = {'cmd': '_notify-validate'}
# collect all the Paypal params
for key in context.request.form:
paypal_params[key] = context.request.form[key]
# acknowledge with a POST
req = context.modules.requests.post(
# check if everthing is ok
if req.text == "VERIFIED":
# return an empty answer to the Paypal original request
return ''

That's it, that's just 10 lines of code (of course we will need to implement mark_payment_has_done() depending on our case inb order to persist in a record that the payment is done and verified).

01 Sep 2016 12:00am GMT

29 Aug 2016

feedPlanet Plone

Reinout van Rees: Simple qgis plugin repo

At my company, we sometimes build QGIS plugins. You can install those by hand by unzipping a zipfile in the correct directory, but there's a nicer way.

You can add custom plugin "registries" to QGIS and QGIS will then treat your plugins just like regular ones. Here's an example registry: https://plugins.lizard.net/ . Simple, right? Just a directory on our webserver. The URL you have to configure inside QGIS as registry is that of the plugins.xml file: https://plugins.lizard.net/plugins.xml .

The plugins.xml has a specific format:

<?xml version="1.0"?>

  <pyqgis_plugin name="GGMN lizard integration" version="1.6">
       <description>Download GGMN data from lizard, interpolate and add new points</description>
       <author_name>Reinout van Rees, Nelen &amp; Schuurmans</author_name>

   .... more plugins ...


As you see, the format is reasonably simple. There's one directory on the webserver that I "scp" the zipfiles with the plugins to. I then run this script on the directory. That script extracts the (mandatory) metadata.txt from all zipfiles and creates a plugins.xml file out of it.

A gotcha regarding the zipfiles: they should contain the version number, but, in contrast to python packages, the version should be prefixed by a dot instead of a dash. So no myplugin-1.0.zip but myplugin.1.0.zip. It took me a while before I figured that one out!

About the metadata.txt: QGIS has a "plugin builder" plugin that generates a basic plugin structure for you. This structure includes a metadata.txt, so that's easy.

(In case you want to use zest.releaser to release your plugin, you can extend zest.releaser to understand the metadata.txt format by adding https://github.com/nens/qgispluginreleaser . It also generates a correctly-named zipfile.)

29 Aug 2016 12:37pm GMT

26 Aug 2016

feedPlanet Plone

Andreas Jung: XML-Director 2.0 update

XML-Director is our Plone-based integration platform for building high-quality enterprise-grade publishing solutions.

26 Aug 2016 5:18pm GMT

Eric Steele: Building Boats

I recently read a quote attributed to Antoine de Saint-Exupéry:

"If you want to build a ship, don't drum up the men and women to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea."

The managing of open source projects is often described as "cat herding", but I'm hard pressed to think of a single instance where standing behind and attempting to goad contributors into moving towards a specific destination has ever worked. The best work we've done has come from storytelling, from saying "beyond the horizon is a thing and it is wonderful and will you take us there?"

Quote Investigator believes the Saint-Exupéry quote is really a modernization of a statement from his "Citadelle", and while certainly not as succinct, this original version feels far more appropriate:

"One will weave the canvas; another will fell a tree by the light of his ax. Yet another will forge nails, and there will be others who observe the stars to learn how to navigate. And yet all will be as one. Building a boat isn't about weaving canvas, forging nails, or reading the sky. It's about giving a shared taste for the sea, by the light of which you will see nothing contradictory but rather a community of love."

Or maybe the cat metaphor works if you just use a laser pointer.

26 Aug 2016 2:47am GMT

20 Aug 2016

feedPlanet Plone

Vikas Parashar: Google Summer of Code 2016 — Wrap Up

Welcome to the last GSoC report.

Continue reading on »

20 Aug 2016 8:52pm GMT

17 Aug 2016

feedPlanet Plone

Encolpe: Try-except versus if-else in Python 3.5

Today I ran again in the question why to use if-else when try-except is shorter?

There is a semantic part that we loose information on a try-except as we don't know what the developer did expect in the most case and there is a performance part. I found a test for python 2 (Try / Except Performance in Python: A Simple Test) with simple assumptions: we try to get a value in a one keyed dictionary and make iterations to get a statistic on this access. I made it a little more complex with a dictionary with 10,001 keys.

Results analysis:

Then the if-else has a predictable comportment and help the maintainer.

I'm agree that the semantic part of the if-else can be replace by a comment but i still not recommend to try-except if it is not to raise an exception.

The results:

The case where the key does not exist: 
1,000 iterations: 
with_try (0.250 ms) 
with_try_exc (0.291 ms) 
without_try (0.119 ms) 
without_try_not (0.120 ms) 
1,000,000 iterations: 
with_try (231.647 ms) 
with_try_exc (263.633 ms) 
without_try (119.238 ms) 
without_try_not (118.602 ms) 
1,000,000,000 iterations: 
with_try (224659.381 ms) 
with_try_exc (260333.897 ms) 
without_try (109796.531 ms) 
without_try_not (111871.690 ms) 
The case where the key does exist: 
1,000 iterations: 
exists_with_try (0.066 ms) 
exists_with_try_exc (0.070 ms) 
exists_without_try (0.166 ms) 
exists_without_try_not (0.180 ms) 
1,000,000 iterations: 
exists_with_try (57.661 ms) 
exists_with_try_exc (56.909 ms) 
exists_without_try (113.633 ms) 
exists_without_try_not (116.340 ms) 
1,000,000,000 iterations: 
exists_with_try (57650.440 ms) 
exists_with_try_exc (57395.376 ms) 
exists_without_try (114659.023 ms) 
exists_without_try_not (117646.034 ms)

The code:

#!/usr/bin/env python3

import time

d = dict.fromkeys(range(0, 10000), 0)
d['somekey'] = 1

ITERATIONS = (1000, 1000*1000, 1000*1000*1000)

def time_me(function):
    def wrap(*arg):
        start = time.time()
        r = function(*arg)
        end = time.time()
        print("%s (%0.3f ms)" % (function.__name__, (end-start)*1000))
        return r
    return wrap

# Not Existing
def with_try(iterations):
    for i in range(0, iterations):
            get = d['notexist']

def with_try_exc(iterations):
    for i in range(0, iterations):
            get = d['notexist']
        except Exception:

def without_try(iterations):
    for i in range(0, iterations):
        if d.get('notexist'):

def without_try_not(iterations):
    for i in range(0, iterations):
        if not d.get('notexist'):

# Existing
def exists_with_try(iterations):
    for i in range(0, iterations):
            get = d['somekey']

def exists_with_try_exc(iterations):
    for i in range(0, iterations):
            get = d['somekey']
        except Exception:

def exists_without_try(iterations):
    for i in range(0, iterations):
        if d.get('somekey'):

def exists_without_try_not(iterations):
    for i in range(0, iterations):
        if not d.get('somekey'):

print ("\n\nThe case where the key does not exist:")
for iteration in ITERATIONS:
    print ("\n%d iterations:" % iteration)

print ("\n\nThe case where the key does exist:")
for iteration in ITERATIONS:
    print ("\n%d iterations:" % iteration)

17 Aug 2016 9:34am GMT

Encolpe: When UnicodeDecodeError become irrational check $LANG

I spent hours this week trying to understand how an installation script can fail on some installations.

In input we have an utf-8 encoded file and we add some xml files, also 'utf-8' encoded. These are parsed with Markdown.

python -m lom2mlr.markdown -l -c rationale.md

It is really simple but sometimes we ran into a strange error:

Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/edegoute/Projects/lom2mlr/lom2mlr/lom2mlr/markdown/__main__.py", line 3, in <module>
File "lom2mlr/markdown/__init__.py", line 55, in compile
File "/home/edegoute/Projects/lom2mlr/local/lib/python2.7/site-packages/markdown/__init__.py", line 529, in markdownFromFile
kwargs.get('encoding', None))
File "/home/edegoute/Projects/lom2mlr/local/lib/python2.7/site-packages/markdown/__init__.py", line 441, in convertFile
html = self.convert(text)
File "/home/edegoute/Projects/lom2mlr/local/lib/python2.7/site-packages/markdown/__init__.py", line 375, in convert
newRoot = treeprocessor.run(root)
File "lom2mlr/markdown/test_mlr.py", line 76, in run
print(" " * int(element.tag[1]) + element.text)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xc9' in position 4: ordinal not in range(128)

First, it was difficult to understand how an unicode error can raise an iso-8859-1 problem on utf-8 files. Going deep I found some known problems with 'codecs.open' in python2.7 but no solution. I tried to force Markown to treat these files as 'iso-8859-1' files, then it ran an utf-8 unicode error at the same line not in the opening. It was sounding too many magic for me.

At that point, I checked again the installation was identical: same python version, same pip version, same eggs versions. I tried some egg upgrade without any success. And finally came the idea to check environment variables. Bingo! On all systems with failing installations we have no localization ($LANG=C). The fix was so simple:

export LANG=en_US.UTF-8

That's it!

I still don't understand the magic in the codecs python module. Why it computes a different encoding when the function call already asks for one? The workaround is simple for programmers.

17 Aug 2016 9:19am GMT