fidzu : Plone

Introduction

Why muck around with HAProxy and Varnish when you can have nginx do it all for you. The setup is easy and it's a lot easier to maintain.

Installing nginx with buildout

You can setup nginx fairly easily with buildout. The only fancy part of our setup is that we're going to include the nginx cache purge module.

1. Add the cache purge part to your buildout
   

   [ngx_cache_purge]
   recipe = hexagonit.recipe.download
   url = http://labs.frickle.com/files/ngx_cache_purge-1.1.tar.gz
   strip-top-level-dir = true
   

2. I needed the pcre source to compile also
   

   [pcre-source]
   recipe = hexagonit.recipe.download
   url = ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.00.tar.gz
   strip-top-level-dir = true
   

3. and the nginx part
   

   [nginx-build]
   recipe = hexagonit.recipe.cmmi
   url = http://nginx.org/download/nginx-0.8.45.tar.gz
   configure-options =
       --with-http_stub_status_module
       --conf-path=${buildout:directory}/settings/nginx.conf
       --error-log-path=${buildout:directory}/var/log/nginx-error.log
       --pid-path=${buildout:directory}/var/nginx.pid
       --lock-path=${buildout:directory}/var/nginx.lock
       --with-pcre=${pcre-source:location}
       --with-http_ssl_module
       --add-module=${ngx_cache_purge:location}
   

4. and finally, add it all to the parts directive
   

   parts =
       ...
       pcre-source
       ngx_cache_purge
       nginx-build
       ...
   
   

5. After you re-run buildout, you'll be able to run nginx by issuing a command like this:
   

   ./parts/nginx/sbin/nginx -c /path/to/configuration/nginx.conf
   

nginx configuration

Here is a simple sample configuration for nginx configured with load balancing and caching. It can obvious get as complicated as you want it, but I definitely think this is easier than managing haproxy, varnish and nginx.

pid /path/to/buildout/var/nginx.pid;
lock_file /path/to/buildout/var/nginx.lock;

worker_processes 2;
daemon off;

events {
    worker_connections 1024;
}

error_log /path/to/buildout/var/log/nginx-error.log warn;

# HTTP server

http {
    server_names_hash_bucket_size 64;

    # this is how you do simple round robin load balancing with nginx.
        # you can define as many backup servers as you'd like here.
    upstream plone {
        server 127.0.0.1:8080;
        server 127.0.0.1:8081;
    }
    
    access_log /path/to/buildout/var/log/main-access.log;


        # can specify multiple cache paths for different resources/paths/proxies
    # if needed..
    # the levels=1:2 just means it'll store the cache'd files 2 levels down in
    # the folder structure
    proxy_cache_path  /var/www/cache  levels=1:2 keys_zone=thecache:100m max_size=1000m inactive=600m;
    proxy_temp_path /var/www/cache/tmp;

    # Here is the caching purge handling. Purge request come in here
    server {
        listen 8089;
        server_name www.example.com;

        access_log /path/to/buildout/var/log/purge.log;

        location / {
          allow                 127.0.0.1;
          deny                  all;
          proxy_cache_purge thecache $scheme$proxy_host$request_uri;
        }
    }
    
    server {
        listen 80;
        server_name www.example.com;


                # log for cache hits.
                log_format cache '***$time_local '
                                             '$upstream_cache_status '
                                             'Cache-Control: $upstream_http_cache_control '
                                             'Expires: $upstream_http_expires '
                                             '"$request" ($status) '
                                             '"$http_user_agent" ';

                access_log /path/to/buildout/var/log/cache.log cache;

        # Enable gzip compression
        gzip             on;
        gzip_min_length  1000;
        gzip_proxied     any;
        gzip_types       text/xml text/plain application/xml;

        # Show status information on /_main-status
        location = /_main_status_ {
            stub_status on;
            allow 127.0.0.1;
            deny all;
        }

                # do not cache when users are logged in..
        proxy_cache_bypass $cookie___ac;

            location / {
                proxy_redirect                  off;
                        proxy_set_header                Host $host;
                        proxy_set_header                X-Real-IP $remote_addr;
                        proxy_set_header                X-Forwarded-For $proxy_add_x_forwarded_for;
                        client_max_body_size            0;
                        client_body_buffer_size         128k;
                        proxy_send_timeout              120;
                        proxy_buffer_size               4k;
                        proxy_buffers                   4 32k;
                        proxy_busy_buffers_size         64k;
                        proxy_temp_file_write_size      64k;
                        proxy_connect_timeout           75;
                        proxy_read_timeout              205;   
                http://plone/VirtualHostBase/http/www.example.com:80/Plone      /VirtualHostRoot/;

                proxy_cache_bypass $cookie___ac;
                proxy_cache thecache;
                proxy_cache_key $scheme$proxy_host$request_uri;
            }
    }
}

31 Jul 2010 4:11am GMT

This little tutorial explains how to increase size of existing .vdi virtual hard drive file once the guest OS is already installed on it. I've taken a KISS approach describing each end every single step.

30 Jul 2010 1:40pm GMT

[edit (2010-07-30): add link to full iPython extension script which doesn't require login]The Plone 3 Products Development Cookbook shows a way to make your Plone 3 buildout create a Zope-aware iPython script. With a slight modification, the script can be used for Plone 4 as well.buildout.cfg:parts += ipzope[ipzope]# an IPython Shell for interactive use with zope running.# You also need to put

30 Jul 2010 8:25am GMT

30 Jul 2010 8:24am GMT

Easier configuration and Python 2.7 support

29 Jul 2010 5:00pm GMT

I JUST heard about Adobe's acquisition of Day Software and have to admit my first reaction was total disappointment. I always admired Day's commitment to architecture and standards. Day is one of the few upper upper tier web content management companies to stay focused on the web - not just as a place [...] Related posts:

1. New ECM Interoperability Standard Proposal on AIIM There is a new proposal for a standards based...
2. Whatever happened to the URL? Even back when I was developing websites in 1998,...
3. iECM: Interoperable Enterprise Content Management iECM is a new standard being developed through AIIM...

Related posts brought to you by Yet Another Related Posts Plugin.

28 Jul 2010 1:39pm GMT

Trying to fix #10441 (the single remaining blocker for Plone 4.0 rc1)

28 Jul 2010 9:30am GMT

Since I couldn't comment on last Martin Aspeli's blog post (couldn't login with openID, maybe its only me, but i think something is not working right there) I decided to write my response here. There are many arguments that don't hold and a lot more is not being covered and should be mentioned.

As first I'm excited that Github is becoming so popular. I'm git user and just having git in the name of Github makes me happy. No joke, really. But all of the talk Martin did in his post should go in direction of "social coding portals" like: github, bitbucket, ... and probably some more ... not only Github itself, but this you probably know.

Lets go line by line...

from Martin's blog:
Someone else forks my repository and tinkers a bit. They then email me to ask me to pull their changes into my repository. If I have the time and inclination (and read my email), that may bring us back to one canonical version. However, in time, there is a single point of failure on me as the maintainer. At some point, I may stop caring or get too busy to follow up.

I agree with this, but the same problem you will get with most of the projects out there. In Plone world we have collective where almost everybody has commit rights, so we kinda got spoiled that you can simply commit to some repository. With all other projects you still need to contact the author to get your patched in. Its been like this since ages. Just with git this is done with one or two commands, while applying patch takes a little more.

from Martin's blog:
Here, Git has an answer: anyone can fork my repository and take over ownership. This is powerful and adds fault tolerance to the "single maintainer" model. But socially, it is dubious. Who really controls development? Who has a right to publish a new release? What if I come back after a few months and decide I wanted to own the project all along? We now need to reconcile the two forks. This may be tricky. Perhaps we don't do it. People now have to figure out which one of two versions of the software to "bet" on. This is onerous, and likely to deter other contributors.

Normally you check pypi page or home page of software you want to use, there you get direction where to start looking for source code. So once you found a place, then here is where Github shines. All forks which are done through the web are considered friendly forks, since its easy to commit back. Its even possible to look the history of forks and where they split. And now the "porn" part, you can actually diff different forks in Github, to see the differences. You hate/love me already? :)

from Martin's blog:
Let's say we bring the main line of development back to "my" repository. Once again, I have to commit a lot of time to reviewing and pulling in changes. Maybe I fall behind. So another user comes around and wants to work a bit. Which one of the two trees does he fork? Which ones does he pull in updates for?

Look the link above. With Github is easy to look friendly forks and compare changes. Its all just few clicks away.

from Martin's blog:
Perhaps I decide to relinquish control to another developer. I let my GitHub fork die, and expect people to use another one. But how do I manage this? How do I hand over ownership to someone else's account without confusing people who've been using my repository? Or people who stumble upon it in the future?

Now for every little project you start and you start your own project in some of the tools. its kinda silly, makes its hard to maintain. since you will probably use it only you. but if the project overgrows you then best way is to create new github user as you would do with any other project oriented system out there.

from Martin's blog:
Here's a cautionary tale that happened to me today: I've been using Soaplib, a Python library for building SOAP servers. There was a release on PyPI, but it had a bug that meant I had to get a checkout. The PyPI page listed a GitHub URL, so I used that to clone the repository. A few weeks later, all our builds suddenly broke. The remote repository was gone, deleted from GitHub. Turns out, the original maintainer had given owner responsibility to another person, who has his own GitHub repository. Development had gone on there for months, unbeknownst to me. One day, the original maintainer decided to delete his repository to not confuse people. Noble, but rather inconvenient.

This can be easily solved by you friendly forking Soaplib, for the time of development. Then in production you of-course wont use any url dependent resources, right? Then even if the project is moved you can easily friendly fork it again or just use the version. I never ran into a product that was discontinued while developing. But as also Martin said, this is rather minor thing (inconvenient).

from Martin's blog:
Again, there are lots of solutions to this problem. However, I think that fundamentally, as a user and potential contributor to a library, I want to find "the" repository and commit my changes there. A personal fork is a good idea until I can get access, but there has to be a path for me to get repository access and become a recognised, trusted contributor. Open source projects have used this model for years, as a way to encourage, recognise and empower contributors and build a shared sense of ownership around the project. I expect the project to be bigger than any one contributor or owner, and I expect the infrastructure to be able to outlive their involvement.

I think this is a feature of Github that you missed out. Organisations. For quick example look at MongoDB.

from Martin's blog:
This is why, for things that are a bit bigger than one person, and a bit smaller than a major open source project, I've got mixed feelings about GitHub, and even mixed feelings about Git itself. They are great tools. I find myself wanting to use them. But I also worry that they are too flexible for their own good, and that the most obvious way of using GitHub is not a good one for open source projects.

I dont see such problems there. Since others can work under your github account or you create its own project github account. Now saying that Git is "a little to much" for small projects. You can also give your contributors possibility to use Hg or Svn ... AT THE SAME TIME and commiting to the same codebase. (more in next comments)

from Martin's blog:
I think Hanno Schlichting put it well: With Subversion, you get a development model, which, whilst not perfect, is easy to understand and has worked out very well in practice for numerous projects. With Git, you're expected to make up your own model. I don't think people are very good at that, and rarely plan ahead for when their code outgrows them.

I know it's possible to use Git like Subversion, with a central repository. But I'm also not seeing a great many people doing that. We have to remember that Git was first built for the Linux kernel, which has a development model unlike most other projects, where it really is up to a set of core maintainers to review every patch and selectively pull it in. They need the kind of flexibility and power that Git offers. For many other projects, I'm not sure this flexibility is always a good thing.

28 Jul 2010 1:45am GMT

The Charlottesville, VA Plone Users Group will hold our inaugural meeting on July 29th at 4 - 5 PM in the Claude Moore Health Sciences Library at the UVA Health System in Charlottesville - Room 1326.

27 Jul 2010 11:29pm GMT

Why you shouldn't host things under your own name if you care about others using it

Git is definitely the flavour of the month when it comes to version control systems. In a relatively short space of time, it's attained quite substantial mindshare. I think this has a lot to do with GitHub. And I think GitHub's success has a lot to do with its friendly, inviting user interface. It's a great service, and it's managed to become a de-facto home for Git repositories for a lot of people.

The great strength of Git - enhanced by GitHub - is that it allows people to put up their own repositories very easily. As a developer, I have a natural home for all manner of personal projects, and a great tool chain around them, including a Wiki and an issue tracker.

So what's the problem? Consider this URL:

http://github.com/optilude/some-project

Notice that my (user)name is in the URL.

Once the project outgrows my own tinkering, I may have some contributors, who want to tinker on their own. Git and GitHub encourage this kind of thing. They can fork my project with just one click. But now there are multiple versions of the repository (and the wiki and the issue tracker), and no-one can really be too sure which one to use.

There are lots of potential problems with this, with lots of potential solutions. Let's think about a few scenarios.

• Someone else forks my repository and tinkers a bit. They then email me to ask me to pull their changes into my repository. If I have the time and inclination (and read my email), that may bring us back to one canonical version. However, in time, there is a single point of failure on me as the maintainer. At some point, I may stop caring or get too busy to follow up.
• Here, Git has an answer: anyone can fork my repository and take over ownership. This is powerful and adds fault tolerance to the "single maintainer" model. But socially, it is dubious. Who really controls development? Who has a right to publish a new release? What if I come back after a few months and decide I wanted to own the project all along? We now need to reconcile the two forks. This may be tricky. Perhaps we don't do it. People now have to figure out which one of two versions of the software to "bet" on. This is onerous, and likely to deter other contributors.
• Let's say we bring the main line of development back to "my" repository. Once again, I have to commit a lot of time to reviewing and pulling in changes. Maybe I fall behind. So another user comes around and wants to work a bit. Which one of the two trees does he fork? Which ones does he pull in updates for?
• Perhaps I decide to relinquish control to another developer. I let my GitHub fork die, and expect people to use another one. But how do I manage this? How do I hand over ownership to someone else's account without confusing people who've been using my repository? Or people who stumble upon it in the future?

Sooner or later, some order has to be imposed on this model. Here, unfortunately, Git and GitHub offer only a handwave. There are a multitude of models that we could adopt. Perhaps we move the project to a more "project-centric" hosting service such as Google Code or Launchpad. Perhaps we set up an "organisation" user on GitHub (but what organisation would that be? for a small project, there may not be an obvious one) with shared ownership, and move our code there. Perhaps we continue with a highly decentralised model, where pulling changes is always an ad-hoc task, and no-one has a stronger claim to ownership than anyone else. There are pros and cons to all of these options, of course, but none seem especially good for a project that has a few users, a well-populated wiki and a bunch of issues in its tracker.

Here's a cautionary tale that happened to me today: I've been using Soaplib, a Python library for building SOAP servers. There was a release on PyPI, but it had a bug that meant I had to get a checkout. The PyPI page listed a GitHub URL, so I used that to clone the repository. A few weeks later, all our builds suddenly broke. The remote repository was gone, deleted from GitHub. Turns out, the original maintainer had given owner responsibility to another person, who has his own GitHub repository. Development had gone on there for months, unbeknownst to me. One day, the original maintainer decided to delete his repository to not confuse people. Noble, but rather inconvenient.

Again, there are lots of solutions to this problem. However, I think that fundamentally, as a user and potential contributor to a library, I want to find "the" repository and commit my changes there. A personal fork is a good idea until I can get access, but there has to be a path for me to get repository access and become a recognised, trusted contributor. Open source projects have used this model for years, as a way to encourage, recognise and empower contributors and build a shared sense of ownership around the project. I expect the project to be bigger than any one contributor or owner, and I expect the infrastructure to be able to outlive their involvement.

This is why, for things that are a bit bigger than one person, and a bit smaller than a major open source project, I've got mixed feelings about GitHub, and even mixed feelings about Git itself. They are great tools. I find myself wanting to use them. But I also worry that they are too flexible for their own good, and that the most obvious way of using GitHub is not a good one for open source projects.

I think Hanno Schlichting put it well: With Subversion, you get a development model, which, whilst not perfect, is easy to understand and has worked out very well in practice for numerous projects. With Git, you're expected to make up your own model. I don't think people are very good at that, and rarely plan ahead for when their code outgrows them.

I know it's possible to use Git like Subversion, with a central repository. But I'm also not seeing a great many people doing that. We have to remember that Git was first built for the Linux kernel, which has a development model unlike most other projects, where it really is up to a set of core maintainers to review every patch and selectively pull it in. They need the kind of flexibility and power that Git offers. For many other projects, I'm not sure this flexibility is always a good thing.

Flame away.

27 Jul 2010 1:16pm GMT

An extension to collective.portletpage and a "remote dashboard administration" tool

27 Jul 2010 11:45am GMT

An extension to collective.portletpage and a "remote dashboard administration" tool

27 Jul 2010 11:45am GMT

If you haven't heard, Kiwi PyCon 2010 is to be held in the beautiful Bay of Islands, just outside the Waitangi Treaty grounds at the Copthorne Hotel on the weekend of November 20-21.
So far we've had 6 financial sponsors commit to the event and 6 schwag sponsors, which puts us in line for an excellent event with a substantial grab bag of goodies the likes of which any IT professional would drool over.

On the talk content side of things, Jacob Kaplan-Moss, co-founder of the Django project, will give Saturday's keynote, Amazon Web Services will be presenting and Weta Digital have also got a couple of their team coming along, just to name a few. If you'd like to have your name alongside this impressive lineup, the due date for talk submissions is this coming Monday, August 2nd so head on over to http://nz.pycon.org/submit-talk today and get submitting and help contribute to the success of Kiwi PyCon 2010!
Alternatively if you'd like to come along and soak up the python goodness - further pricing information can be found here and the registration form is available here. Early bird registrations close September 1 so don't delay!

27 Jul 2010 11:34am GMT

Please mark your calendars for the weekend of July 31, 2010 for a plone.org upgrade - and possible downtime.

25 Jul 2010 8:58pm GMT

This blog entry was inspired by the discussion I just had with Michael on IRC, as he just added plugins in unittest2.

Setuptools' entry points feature is hated and loved by developers. If you are not familiar with them, you can read this post from Armin.

Hated because when you install a project that contains entry points (let's call them plugins), they can be used in another application without letting you know. So basically if a plugin sucks, it can break another application just by being installed in your Python. And it's not easy to have an overview of what plugins are installed and potentially active.The worst is that projects that provide entry points, usually provide many other things. But if you want to deactivate the plugin, you have to remove the whole project… Note that plugins are not loaded at Python startup. What happens is that any application can iterate over the metadata of installed projects, looking for plugins, and eventually loading them if wanted.

Loved, because from a developer point of view you can have a new feature added in a program with no extra configuration at all. Take Nose. Thanks to entry points, it's dead easy to create a plugin for this test runner, and tell people to pip-install this new project. Zero config. Nice. Another great thing is that it's global to Python. Any application can consume any entry point. Entry points are implicit plugins I guess.

Distutils has a plugin system as well: you can add new commands by adding in distutils.cfg the path to the Python package containing the command. That's an explicit plugin system since the end-user has to configure it manually so Distutils uses it. Mercurial uses the same technique: activating a plugin is done in .hgrc. I would call these explicit plugins.

I think we can get the benefits of entry points without their caveats really simply. And provide a generic plugin system for all. Let's summarize what we want:

• being able to list all installed plugins for every Python application
• being able to remove a plugin or deactivate it. Without being forced to uninstall the project that provided it
• have a plugin automatically installed and activated when the project that provides it is installed

Here's how we can do. That's a brain dump, please give me some feedback !

Global plugin registry

Let's have a .python-plugins.cfg file in the user's home (and one global to Python. The user cfg is merged with the global one at startup, and overrides the values - thanks Mongoose_Q for mentioning this on Twitter). It's a simple ini-like file like .hgrc, where each section represents a python application and a group name. A group is just a family of plugins. For instance 'commands' can be a group for the 'distutils' application. In this section, each line is a plugin, represented by a pointer to the module or class, followed by a label as the value.

Here's an example for a distutils 'i18n' command. It's a MyClass class, located in the foo package, in the bar module:

  [distutils:commands]
  foo.bar:MyClass = i18n

The link to the code comes first because some plugins could have no name:

  [app:group]
  package.module:Class =

Accessing the registry

distutils can provide an API to read the file, iterate and load the plugins:

    >>> from distutils2 import plugins
    >>> plugins.get('distutils', 'command')
    <iterator>
    >>> plugins.get('distutils', 'command').next()
    <Plugin "i18n" at foo.bar:MyClass>

    >>> plugin = plugins.get('distutils', 'command').next()
    >>> plugin.load()    # gets the code and loads it
    <MyClass Instance>

Installing the plugins

Last, distutils could provide a mechanism to automatically register a plugin.

Projects could describe their plugins in their setup.cfg:

  [plugins]
  distutils.commands.i18n =  foo.bar:MyClass

Then distutils would automatically inject them at installation time in .python-plugins.cfg only if the end user agrees:

  $ python setup.py install
  distutils has detected a "i18n" plugin for distutils:commands. Do you want to activate it (Y/n) ?

25 Jul 2010 10:14am GMT