fidzu : Plone

09 Nov 2011 9:30pm GMT

This just in from the production department: use Varnish. (And please forgive the heavily meme-laden approach to describing these techniques .)

Cache ALL the hosts

Our ability to use Varnish in production is no secret by now, or at least it shouldn't be. What is often less clear is exactly how to use it. One way I like[1], is to run Varnish on your public IP port 80 and make Apache listen on your private IP port 80. Then proxy from Varnish to Apache and enjoy easy caching goodness on all your virtual hosts in Apache.

Configuration

This should require less than five minutes of down time to implement. First, configure the appropriate settings. (Well, first install Apache and Varnish if you haven't already: `aptitude install varnish apache2` on Ubuntu Linux[0].)

Varnish

To modify the listen IP address and port, we typically edit a file like /etc/default/varnish (in Ubuntu). However you do it, configure the equivalent of the following on your system:

DAEMON_OPTS="-a 174.143.252.11:80 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -s malloc,256m"

This environment variable is then passed to varnishd on the command line. Next, pass traffic to Apache like so (in /etc/varnish/default.vcl on Ubuntu):

backend default {
 .host = "127.0.0.1";
 .port = "80";
 }

Now on to Apache.

Please note that the syntax above is for Varnish 3.x and the syntax has (annoyingly) changed from 2.x to 3.x.

Apache

The Apache part is a bit simpler. You just need to change the listen port (on Ubuntu this is done in /etc/apache2/ports.conf), typically from something like:

Listen *:80

to:

Listen 127.0.0.1:80

Restart ALL the services

Now restart both services. If all goes well you shouldn't notice any difference, except better performance, and when you make a website change and need to clear the cache[2]. For this, I rely on telnetting to the varnish port and issuing the `ban.url` command (formerly `url.purge` in 2.x):

$ telnet localhost 6082
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
200 205     
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,2.6.35.4-rscloud,x86_64,-smalloc,-smalloc,-hcritbit

Type 'help' for command list.
Type 'quit' to close CLI session.

ban.url /
200 0

Cache ALL the disks

This site has Varnish and Apache configured as described in this article. It also has disk caching in Apache enabled, thanks to Elizabeth Leddy's article:

• http://plonechix.blogspot.com/2011/08/10-minute-caching-with-apache.html

As a result, it's PEPPY AS THE DICKENS™ on a 512MB "slice" (Cloud server) from Rackspace Cloud. And now you know yet another "Plone secret". Now go make your Plone sites faster, and let me know how it goes in the comments section below.

Notes

[0] Using the latest distribution, "oneric".

[1] I first saw this technique at NASA when NASA Science was powered by Plone; I found it odd at the time but years later it makes perfect sense.

[2] Ideally you'd configure this in p.a.caching, but I've not been able to stomach this yet.

09 Nov 2011 5:50pm GMT

Just add the method below to the RedirectionTool and call it from the browser as:

http://localhost:8080/site/portal_redirection/getAllRedirects

Assuming that the site is running at loaclhost:8080 that is :-S

That will show a list of redirects that can be imported into plone 4.x


security.declareProtected(View, 'getAllRedirects')
def getAllRedirects(self):
"get'm'all"
result = []
reference_tool = getToolByName(self, 'reference_catalog')
for k,uuid in self._redirectionmap.items():
obj = reference_tool.lookupObject(uuid)
if obj is None:
print 'could not find redirect from: %s to %s' % (k, uuid)
else:
path = '/'.join(('',)+obj.getPhysicalPath()[2:])
result.append( '%s,%s' % (k,path) )
return '\n'.join(result)


08 Nov 2011 2:58pm GMT

Connexions is now on Google+! Add us to your circles to get the latest news and updates. Google+ is another way to stay informed about everything Connexions just like Twitter, Facebook and our blog.

08 Nov 2011 2:26pm GMT

Download the example theme from the Diazo theming tutorial I presented at Plone Conference 2011, San Francisco. I'll post a link to the video once it's up.

07 Nov 2011 11:39pm GMT

This blog post is about sauna.reload which is a Python package adding an automatic reload feature to Plone CMS.

The short history of reloading

The web developer community takes reload-code-on-development as granted. You edit your source file, hit refresh and poof: your changed are there. It is a good way and the only real web development way and one of the sane things PHP community has taught to us. Things could be different: for example if you are developing an embedded code on mobile platforms you might need to build the ROM image for hours before you see your changed source code line in the action.

Side note: To check your code changes it is often faster the hit enter on the browsee address bar line as this does not reload CSS and JS files for the HTTP request

For PHP the reloading is easy as the software stack parses every PHP file again on every request. There is no reload - only load. In fact there is a whole "optimizer" open source business model spawned around PHP just to make it faster. PHP processes are stateless.

For Python the things are not so simple. Python web frameworks have separate process lifespan and HTTP request handling lifespans. Often a server process is started, it spawns N threads and each thread handles one HTTP request at a time. Python processes are stateful.

As being stateful, Python processes may start slowly. You need to parse all source code and create initial memory objects, etc. Frameworks do not optimize for fast start-up because it really doesn't matter on the production service as you spawn Python processes only once, when the server is booted.

Plone CMS, coming with choking 250 MB of source code (Linux kernel has 400-500 MB) is the extreme of this slow initialization. Plone CMS is the biggest open source Python project out there (does anyone dare to challenge my clain?). When Plone starts it loads most of source code to memory and parsing and initializing Python itself, plus various XML files, is quite an achievement.

What Tornado, Django, Paster etc. Python frameworks do that when they reload they simply zap the process dead and reinitialize 100% virgin process. This is ok as these low level frameworks have little overhead. Though it becomes painful slow too when your Django service grows for many applications and the code starts piling up…

For Plone 100% start up this is no go. Plone people care about the start up time, but there is not much they can do about it… even the smallest sites have start up times of dozens of seconds.

Python and module reload

Plone used to have (has) a helper package called plone.reload. It works by reloading Python modules. Python interpreter allows you to reload a module again in the memory. The code is actually based on xreload.py written by Guido van Rossum himself.

However, there is a catch. Naive code reload does not work. As xreload.py comments state there are issues. Global stateful objects, other initialization specific code paths, etc. are ignored. So with plone.reload you ended up with a broken Python process as many times you ended up with a successful reload. It's so frustrating to use that you don't actually want to use it.

From Finland with Love

sauna.reload was initially created in Sauna Sprint 2011 event, arranged by EESTEC and Plone Community. The event name comes from the abundance of sauna in Finland and the package name comes from the fact that a lot of sauna was put into the effort of creating it.

Going trident

A fork. (The orignal image)

A spoon (The orignal image)

sauna.reload takes a different strategy of reloading

The idea behind sauna.reload goes back to the days as I was working with Python for Nokia Series 60 phones. Symbian seriously sucks, peace on its memory (though not sure if it's dead yet or just a walking corpse). If you did any Symbian development you saw how it could never fly - such a monster it was.

One of the issues was start-up time of the apps, especially Python start up time. Symbian IO was slow and simple fact to running the application to the point where all imports had been run took several seconds. For all of this time you could just show some happy loading screen for the user. It was very difficult to deliver a Python application which would have acceptable user experience on Symbian.

Far back in the day I chatted with Jukka Laurila who gave me some tips. Emacs used to have unexec() command which simply dumped the application state to the disk and resumed later. This works nicely if your application is an isolated binary and does not have any references to DLLs, so it doesn't work so nicely on any contemporary platform.

On the other hand, Maemo folks had something called PyLauncher. PyLauncher loaded a daemon to memory and this daemon had Python interpreter and pygtk library loaded. When Python script had to be run, the PyLauncher daemon simply fork()'ed a new child and this new child inherited already loaded Python interpreter and pygtk for free. Fork() is a copy-on-write operation on modern operating systems.

This gave me a cunning idea.

Taking a shortcut

When you develop a web application and you want to reload changes you usually don't want to reload the whole stack. In fact, your custom code tends to sit up on the top of the framework stack and unless you are a framework core developers yourself, you never poke into those files.

What if we run Plone initialization process to the point it is just about to load your custom code, freeze the process and then just always go back to this point when we want to load our (changed) code?

Plone process is in fact a series of Python egg loads. Your custom eggs are loaded last.

With sofware only your imagination is the limit

So I presented the idea to Asko Soukka and Esa-Matti Suuronen from University of Jyväskylä (one of the largest Plone users in Finland) in Sauna Sprint 2011. If I recall correctly their first reaction was "wow dude that definitely is not going to work".

But at the end of the tunnel was the freedom of Plone web developers - being released from the chains of Zope start-up times forever. Even if it had turned out sauna.reload was not possible we would have been stranded in the middle of the forest having nothing to do. So the guys decided to give it a crack. All the time they spent to develop sauna.reload would be paid back later with gracious multiplier.

sauna.reload is using awesome cross-platform Watchdog Python package to monitor the source code files. Plone is frozen in the start-up process before any custom source code eggs from src/ folder are being loaded. When there is a file-system change event sauna.reload kills the forked child process and reforks a new child from the frozen parent process, effectively continuing the loading from the clean table where none of your source code files were present.

Going uphill

San Francisco, where we are hosting Plone Conference 2011 and sprints, is a city of hills. They feel quite steep for a person who comes from a place where maps don't have countours.

With Zope 2, the technology the ancients left to us, things were not that straightforward.

ZODB, the database used in Plone, happens in-process for the development instances (single process mode). When you go back to the orignal frozen copy of Plone process you'll also travel back in append-only database time.

The sauna.reload devs managed to fix this. However this code is prone to ZODB internal changes as happened with Plone 4.1.

Another funky issues were how to monkey-patch Zope loading process early enough and reordering egg loading process so that we hold loading the custom code till very end.

For The Win

sauna.reload works on all Python packages which declare z3c.autoinclude loading in setup.py. Just go and use it. This, of course, is based on the assumption you use an operating system which supports fork() operation and unfortunately Windows is not privileged for this.

Note: OSX users might need to use trunk version of Watchdog until a bug fix is released

Your Plone restart time goes down to 1 second from 40 seconds. Also, the restart happens automatically every time you save your files. It works with Grok, Zope component architecture, everything… ZCML is reloaded too. You should feel an unsurmountable boost in your productivity.

We still have some quirks in sauna.reload. If you try hard enough you rarely might be able to corrupt database. We don't know how, so keep trying and report your findings pack to GitHub issue tracker. Also as sauna.reload is meant for the development only you shouldn't really worry about breaking things. We also have an old issue tracker which was not migrated to collective - how one can migrate a issue tracker on Github?

The Future

Who knows? Maybe someone gets inspiration of this work and ports it to other Python frameworks.

Our plan is to release a Plone developer distribution which comes with sauna.reload and other goodies by default. This way new Plone developers could instantly start hacking their code by just copy-pasting examples and keep hitting refresh.

Subscribe to this blog in a reader Follow me on Twitter

07 Nov 2011 10:53pm GMT

Plone IDE is an ACE Javascript editor based effort to provide easy and sane Plone development environment aimed for newcomers (though power users will probably enjoy it too). It started as a proof-of-concept. In Plone Conference 2011 people are sprinting to bring it to the level of usability that it can be reliably used in everyday development. Plone IDE effort was bootstrapped by Franco Pellegrini who got chipped in to the conference by the awesome Plone community.

After various stakeholders had a chance to come together around a round table a vision of Plone IDE was formed. Here are my draft slides about Plone IDE and its relationship to other on-going Plone community efforts. Please see my slides of Easy Way which try to explain the background of bringing Plone closer to an everyday web developer from Plone's current Python expert only mode.

The problem: Current Plone development workflow is optimized for expert level Python developers with powerful tools providing high probability of cutting off your thumbs accidentally. Before getting the taste of Hello World, the newcomer developers must fight they way of lot of unrelated matters and hope have the thumbs intact in the end of the day.

• See brief Plone IDE introduction video

Solution: Provide a single (zip) package which contains everything you need to get started. Just start Plone, open the editor and copy-paste in the code from the tutorial. The development workflow must match the actual work you need to be able to change anything in Plone: it is not limited to changes you can do in the site database.

If you are interested in this project please contact me or Franco through Twitter. We also lurk in IRC on #plone channels.

BTW Most of the actual IDE code is not Plone specific, so this might turn out as a drop-in editor for any Python web development project

Subscribe to this blog in a reader Follow me on Twitter

07 Nov 2011 7:46pm GMT

Worst.package.name.ever... inentionally! :-) But seriously, read the warnings and heed them. If I have to tell you to read the warnings, you shouldn't use this package.

We in Plone land have been long frustrated about how easily a poorly behaved add-on could break a Plone site. Simply installing the add-on and removing it can leave your site inaccessible, even the ZMI! Often times add-ons that provide only minimal additional object behavior can break the entire object when removed.

Now this isn't actually the ZODB's fault at all. The ZODB actually has quite robust handling of broken object and it works quite well. I'm afraid that fault here lies in zope.interface and zope.component. They're simply both too central to almost all ZODB applications to fail to provide graceful handling of broken objects. So I've taken advantage of the sprints following the Plone Conference here in San Francisco to work on this.

The experimental.broken package provides patches for zope.interface and zope.component to make the services they provide more tolerant of broken objects in the ZODB and allows them to function as if the interfaces and components aren't there when the add-on has been removed. Even better, if the add-on is restored, the full behavior of the interfaces and components is restored.

That said, this is all highly experimental so don't use it. If you do use it, never use it in production and only use it on a copy of your ZODB. And if you do use it, definitely report your experience. If you do have a broken site I'd love to hear how it works for you. Just add experimental.broken to your instance eggs.

If this approach tests well and the core Zope developers think it's the right approach I'll be merging this into zope.interface and zope.component.

07 Nov 2011 7:54am GMT

Hi,

Most power users mays already know this. But this is a real enlightenment for newbies who are stuck with the command line pdb utility. Even for experienced developper as I am… supposed to be, this is a real advantage to have on screen at the same time the source code, with breakpoints in the margin of it, the inspector for your global and local variables, the access to the call stack and execution frames.

Start reading the fine manual here and come back reading how to get this in Zope 2.

Of course I assume you have a recent Eclipse (Indigo) and Pydev (2.2.4)

Find in your Eclipse software directory the full path of the directory that contains "pydevd.py". In other words :

$ cd /your/eclipse/root
$ find . -name 'pydevd.py'

In my MacBook the file is found in "/Developer/eclipse/plugins/org.python.pydev.debug_2.2.4.2011110216/pysrc"

(note that upgrades of PyDev may change this path to something else)

Edit your development buildout config file and add this in your plone.recipe.zope2instance part:

[instance]
# recipe = plone.recipe.zope2instance
...
extra-paths =
    /Developer/eclipse/plugins/org.python.pydev.debug_2.2.4.2011110216/pysrc

Of course re-run your buildout. Open the debug perspective in Eclipse and start the PyDev debugger server (see link above).

You can now add anywhere you want (need) the line "import pydevd;pydevd.set_trace()", or use the shortcut "pydevd" to insert your first hard breakpoint.

Start your Zope instance and make the necessary clicks to execute the line that has the hard breakpoint. Wow, great : the next line of Python to be executed is hilited from there, you cand step, step over, go to the next return, and even add soft breakpoint double clicking in the left margin anywhere in your source code.

All other things you may need to know about debugging with PyDev are here and usable "as is" in a Zope server.

Another great feature for those who develop with Windows : when debugging with Eclipse/Pydev you can start your development instance as a service, when before you needed to run the sloooooow "instance fg" and benefit of Eclipse remote debugging.

07 Nov 2011 12:02am GMT

Yesterday, Calvin Hendryx-Parker gave a very interesting talk about selling Plone. How do you talk to potential customers? By listening rather than talking.

The guidelines in a nutshell:

WHAT NOT DO:

use jargon - just keep it simple and understandable

show irrelevant features - it only looks more difficult, even if you thinks it's cool to show of

talk about yourself - it is not about you

talk bad about other technologies - Plone is good, but there is more out there

WHAT TO DO:

listen - find out what it is the other person is looking for

know who you are talking to - find out how well the audience is informed and if necessary, adjust the presentation

build a relationship - do not start a conversation with a sales pitch. Find common interests to begin with

BE PREPARED:

have everything installed and tested - If you fail to plan, you plan to fail. So make sure all the equipment is working

have a folder with demo content of many types - if a demo is not working, have something else for backup

run it local - we like to believe there is internet everywhere, but sometimes there is not. So run your demo local. At least have a backup stand by

06 Nov 2011 10:10pm GMT

Recently, the Plone community bought plone.com. Now we have to turn it into a powerful yet simple platform that promotes Plone to end-users. According to Mark Corum, we need to launch it within four months from today.

There is Plone.org, a place for developers, where you can find new features and where developers can share ideas. It is often difficult to understand by end users. That is why we will launch Plone.com. The first and most important thing? Office managers, decision makers and also your grand mother should be able to understand the content.

Companies and organisations who are looking for a new CMS, will find all the answers about Plone on Plone.com. In the next months, a group of people within the community will write new content.

Speaking of Plone.org:

- at this moment, 60% of the visitors don't find what they are looking for and leave

- the site contains about 10.000 pages and 28.000 elements

- Some information is outdated

It migh be an idea to update some of the information on plone.org

06 Nov 2011 6:36pm GMT

The new Plone Foundation Board had a meeting on day 3 to answer questions. What are the most important things to do in the near future?

some topics:

- One of the important things at hand will be marketing. There should be more news and announcements on Plone.org. Sometimes events happen and almost nobody knows about it. Also, we hear stories about countries where important companies started to use Plone. That is news and should be on the site.

- Another thing: working on a guideline about how to behave at Plone events. There are examples about sexual remarks at events. And although it was not a Plone-event, this board wants to make a statement; incorrect behaviour within the community will not be tolerated. Especially since we want more woman to get involved.

- This board wants more transparancy, more communication and interaction with the community.

05 Nov 2011 10:55pm GMT

It is day 3 of the conference here in SF, but there is still one blog to be published about yesterday. It is the 'why are we here' blog. Calvin Hendryx-Parker and Alexander Limi discussed it during their presentation about the state of Plone. "We are here because of the community."

We have people from Europe, South America, Canada and even from Africa and Asia. The Plone community is great because we care for each other. When someone needs money to go to a conference, people donate. And there is more. People in the Plone community assume that other developers mean well, despite of differences in opinion.

Limi and Hendryx-Parker told the story about Dorneles Treméa, a guy who was a key Plone developer. He died this year in a car accident near his home in Brazil, only 31 years of age. He was a very positive guy; he came to Norway for a Plone event showing up in summer clothes. But he did not care. He kept smiling, just happy to be there. Even though he is not here anymore, we can still learn from him.

For people who are new to the Plone Community, stories like this are inspiring. It helps to understand that the community is about more than just developing. There is is no need to fly across the globe for just that. People attend to meet, work together and to feel that they are part of something special.

05 Nov 2011 7:06pm GMT

This is my five minutes lighting talk presentation in Plone Confrence 2011. It's a crash course how to write Javascript by following the modern best practices and not in that crappy way when you where still studying web development circa HTML3.

The presentation was originally aimed for Python developers, but suits for anyone with some programming experience and not much Javascript experience.

Discussed topics include: require.js, ECMAScript 5, strict mode, JSLint, this scoping and bind(), binding event handlers using jQuery

Ps. Always when searching Javascript related information in Google et. al. use MDN (Mozilla Developer Network) prefix as otherwise you'll get misinformation and spammy results. E.g. search "MDN javascript bind" instead of "javascript bind".

Pps. Put w3cschool.com domain on the black list in your search engine. It's bad.

Ppps. PDF is available here.

Subscribe to this blog in a reader Follow me on Twitter

05 Nov 2011 5:37pm GMT

Lightning talk at the Plone Conference 2011.

05 Nov 2011 9:57am GMT