22 Oct 2014

feedPlanet Plone

Davide Moro: Sqlite array type and Python SQLAlchemy

I need to write up things just for remembering how I solved a particular issue if occurs in the future.

Sqlite (with http://sqlitebrowser.org) is great for rapid prototypes development but it lacks some useful implementations provided by Postgresql (for example the sqlalchemy.dialects.postgresql.ARRAY type).

I solved implementing a SQLAlchemy TypeDecorator with a json serialization:

Here it is the self-explaining code:

from sqlalchemy.schema import Column
from sqlalchemy.types import (
Integer,
String,
TypeDecorator,
)
from sqlalchemy import Sequence
from pyramid_sqlalchemy import BaseObject as Base
import json

class ArrayType(TypeDecorator):
""" Sqlite-like does not support arrays.
Let's use a custom type decorator.

See http://docs.sqlalchemy.org/en/latest/core/types.html#sqlalchemy.types.TypeDecorator
"""
impl = String

def process_bind_param(self, value, dialect):
return json.dumps(value)

def process_result_value(self, value, dialect):
return json.loads(value)

def copy(self):
return ArrayType(self.impl.length)

class Element(Base):
__tablename__ = 'elements'

id = Column(Integer(),
Sequence('element_id_seq'),
primary_key = True)
# ...
myarray = Column(ArrayType())

If you are not using Pyramid just replace the pyramid_sqlalchemy's Base wrapper import with:

from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()

22 Oct 2014 10:22pm GMT

21 Oct 2014

feedPlanet Plone

Starzel.de: Extended Mastering Plone Training in Munich in November

When we prepare a training, we always create more material than what we can walk through in the short amount of time we have available.

We decided that we want to offer you more by extending the training period to 5 days. In 5 days we can cover the material deeper have more exercises and additional topics.

The additional topics include:

We would love to have you with us.

More information is available on our dedicated training page.

21 Oct 2014 1:05pm GMT

19 Oct 2014

feedPlanet Plone

Davide Moro: Pyramid starter seed template powered by Yeoman (part 1)

Book of the month I'm reading this summer: Pylons/Pyramid (http://docs.pylonsproject.org/en/latest).


Pyramid (http://www.pylonsproject.org) is a minimal Python-based web development framework that let you "start small and finish big".

It stole a lot of (good) ideas and concepts from other mature Python web frameworks and it is build with the pluggable and extensible concepts in mind. Read: no need to fork applications.

Furthermore Pyramid is database and template engine agnostic: you are free.

From the very beginning Pyramid allows you to become productive quickly. So why not start with something of useful?

Pyramid + Yeoman

The goal of this experiment is integrate yeoman with Pyramid (or other frameworks like NodeJs/Express with AngularJS or Plone as already did), preserving the yeoman's workflow.

UPDATE 20140926: here you can see a Plone + AngularJS + Yeoman article (collective.angularstarter)

In this article I'll talk about what are the benefits you get integrating your Pyramid app with Yeoman, in future posts I'll discuss how they work under the hood with additional technical details omitted here (each used component deserves an entire blog post).

Yeoman

You might wonder why? Because of the importance of tooling. Since it is very important build an effective developer tooling ecosystem, I want to integrate the simple starter demo app with commonly used tools to help you stay productive. So this simple application prototype it is just an experiment that should help you to integrate with modern web development tools provided by the yeoman workflow stack (http://yeoman.io).

Choosing the right tools is very important for the best develop experience and I cannot work anymore without Yeoman, especially when coding with Javascript.

Grunt

Yeoman it is internally based on three important components (nodejs powered):

Bower


So with the yeoman's tools you can just code, avoid annoying repetitive tasks and don't worry about:

So let's see together what happened to our pyramid starter demo template created with pcreate -t starter integrated with a yeoman's generator-webapp project.

The result will be a Pyramid starter seed project integrated with modern non Python-based web development tools.

Goals

Management of third party assets

You no longer have to manually download and manage your scripts with the Bower package manager.

From http://bower.io:

"""Bower works by fetching and installing packages from all over, taking care of hunting, finding, downloading, and saving the stuff you're looking for."""

So just type something like: bower install angular-translate --save and you'll get the rigth resource with pinning support.

Tasks automation

Automation, automation, automation.

From http://gruntjs.com:

"""Why use a task runner? In one word: automation. The less work you have to do when performing repetitive tasks like minification, compilation, unit testing, linting, etc, the easier your job becomes. After you've configured it, a task runner can do most of that mundane work for you-and your team-with basically zero effort."""

Examples:

Jslint

No more deploy Javascript code with bad indentation, syntax errors or bad code practices.

All syntax errors or bad practise will be found.

Image minification

The build process will detect and minify automatically all your asset images.

Uncss task

Modern (and heavy) UI frameworks like Twitter Bootstrap provide an excellent solution for prototyping your initial project, but most of the times you are using a very minimal subset of their functionalities.

https://twitter.com/davidemoroThis inspiring Addy Osmani's blog post helps you to remove unused css in your pages with a grunt task named grunt-uncss (https://github.com/addyosmani/grunt-uncss):

The original not-minified bootstrap.css weights in at 120 kB before removing unused rule.

Css concat and minification

You can split your css code into different files and then the build process will concat and minify them creating a unique app.css file. This way you write modular and better readable css files, reducing the number of browser requests.

The theme.css file is quite small but in real projects you can save more. In this case:

The configured build pipeline is concat, uncss and cssmin. 122.85 kB (original bootstrap.css) -> 4.64 kB (uncss) -> 3.45 kB (minification)

Automatic CDN-ification

It is handy using unminified versions of third party javascript libraries during development and switch to CDN versions in production mode with well known benefits for your website.

Don't worry: the cdnify task will take care about this boring issue. Automatically.

You save a boring manual and error-prone configuration.

Composable bootstrap.js version

The Pyramid starter project is based on Twitter Bootstrap.

Twitter Bootstrap

Depending on your project you can load the whole Twitter Bootstrap Javascript code at once or including individual plugins.

As you can see the Javascript component of Twitter Bootstrap is very modular: http://getbootstrap.com/javascript. So if you don't use a particular feature, just don't include it.

This way in development mode you will have all individual plugins splitted in different files, in production it will served a unique concatenated and minified Javascript file built automatically.

So if you just need alert.js and dropdown.js you can get a 2.79 kB plugins.js:

The concatenation of alert.js and dropdown.js produces a 7.06 kB, that weight in at 2.79 kB after minification instead of the 8.9 kB (gzipped) bootstrap-min.js corresponding to not gzipped 27.2 kB.

Html (template) minification

Since the ZPT/Chameleon templating language is an extension of HTML with xml syntax,

Brower are able to display unrendered ZPT/Chameleon templates

theorically it can play well with html minificators.

I know, template minification can lead to potential unexpected problems due to minification issues on template files... but this is my personal playground, so let me play please!

So... why not switch to a pre-compiled minified template of your ZPT/Chameleon files when you are in "production mode"?

Obviously during development you will use the original template files.

The interesting side of this approach is that there is no overhead at response time, since the minification task runs just one time before deploying your application. It might be an option if you want just your html minified and you cannot feasibly add to your site or project additional optimization tools at web server level.

Anyway I have tried this mad experiment and... if you don't use too aggressive minification params, it seems to work fine with good results. Try it at your own risk or just disable it. Here you can the effects on the generated index.html used in production:

Template minified (7.62 kB -> 4.16 kB)

Result: a lighter Pyramid

Same results but a lighter Pyramid app:

Let's see how it behave the standard Pyramid starter project:

Standard Pyramid starter project (production.ini)

And the Pyramid starter seed:

Pyramid starter seed (production.ini)

As you can see the seed version is ~38 Kb smaller and more performant.

Useful links

That's all?

No, you can do more, for example:

Let me know what you think about that, please. Hope soon I will manage to write the second part of this blog post explaining how I did it. In the mean time you can:

Links

19 Oct 2014 9:07pm GMT

16 Oct 2014

feedPlanet Plone

UW Oshkosh How-To's: How to change versions of Plone in your Vagrant install

New info below! Thanks to Steve McMahon :)

The Vagrant installer is a very nice, easy way to set up Plone.

If you recently used the Vagrant install but found Plone 5.0a2 instead of the Plone 4.3.3 you expected, here's how to change it.

Edit the Vagrantfile and change the line


UI_URL = "https://launchpad.net/plone/5.0/5.0a2/+download/Plone-5.0a2-UnifiedInstaller.tgz"

to


UI_URL = "https://launchpad.net/plone/4.3/4.3.3/+download/Plone-4.3.3-UnifiedInstaller.tgz"

Save the file.

Run these commands. Some of them may not be necessary, depending on whether you've run vagrant recently.

vagrant halt
vagrant destroy
vagrant up

Alternatively go to Github.com

Steve McMahon suggests going to Github.com and downloading this directly:

https://github.com/plone/plonedev.vagrant/archive/4.3-maintenance.zip

and that this is available via the "Branch" dropdown at

https://github.com/plone/plonedev.vagrant/

which takes you to

https://github.com/plone/plonedev.vagrant/branches

16 Oct 2014 9:12pm GMT

UW Oshkosh How-To's: Tracking down pesky buildout version pins or requirements

Here are some ways to figure out the cause of conflicting version errors like this when you run buildout:

While:
  Installing client1.
Error: There is a version conflict.
We already have: Products.MimetypesRegistry 2.0.2
but Products.Archetypes 1.7.14 requires 'Products.MimetypesRegistry>=2.0.3'.

Buildout cfg files

First, check any version pins you might have in your *.cfg files.

In the above example, I would run this:

grep Products.MimetypesRegistry *.cfg

in my zinstance or zeocluster directory. This will tell me which files refer to the offending add-on.

Extends

If the only references to the add-on correctly show the version you want (and you can't see where the wrong version is being pinned), next check if your buildout.cfg "extends" values are correctly ordered.

If your "known good sets" ("KGS") come *after* your local version pins (in versions.cfg in the example below), you may get conflicts.

The following shows the correct ordering:

extends =
    http://dist.plone.org/release/4.1.6/versions.cfg
    http://good-py.appspot.com/release/dexterity/1.2.1
    versions.cfg

This is exactly what happened to me... I had a KGS for an old version of Dexterity, and it was that KGS that had a version pin that conflicted with my local pins in my versions.cfg.

Buildout can help

The way I found out that it was a KGS that was conflicting is that I ran "buildout annotate" so that it told me where its various version pins were coming from:

bin/buildout annotate

which provided this output:

...
    ...
Products.MimetypesRegistry= 2.0.2
    http://good-py.appspot.com/release/dexterity/1.0.1?plone=4.1
...
    ...

16 Oct 2014 8:51pm GMT

15 Oct 2014

feedPlanet Plone

Tom Gross: Porting tests to plone.app.testing for Plone 5

A major version of a piece of software always means to leave behind some burdon. Plone ships with two testing framworks since Plone 4. Now it is time to get rid of one of them: PloneTestCase. With the newer plone.app.testing framework it is possible to specify layers to encapsulate testing scenarios and dependencies. I don't want to compete the excelent plone.app.testing documentation here but provide some tipps for porting your addons from PloneTestCase to plone.app.testing.

First: Look at some examples! Most of the Plone core packages are already ported to plone.app.testing. If you have a lot of packages with one namespace and a similar setup it probably makes sense to start with a ZopeSkel or mr.bob template for your testing base class.

Second: Use a Testing base class! Define one or two base classes for all your testing needs in one product. This makes migrations a lot easier. With the help of the PloneTestCase class of plone.app.testing.bbb half the work is done. All you need is a layer which installs your addon and does the other things (create content, etc.) you need for testing.

Third: Doctests Porting doctests is a little bit more tricky. The way doctests are run changed a little bit with plone.app.testing. You no longer pass a testing class to the test but add a layer. This can be easily done with the layered helper function found in plone.testing. You just pass in the layer you defined for your unittests and you can access it in your doctests. Because there is no test class there is no self.<whatever-method> supported in doctests. Greping 'self' in all doctests and replacing it with layer specific code is usually the way to go.

>>> self.setRoles(['Manager])

would turn into

>>> from plone.app.testing import TEST_USER_ID, setRoles
>>> setRoles(layer['portal'], TEST_USER_ID, ['Manager'])

And you don't need to use any Zope based variant of doctest. Just use plain python doctest and suite.

Forth: Functionaltests Basically all tests inherited from plone.app.testing.bbb.PloneTestCase are functional tests and support the publish method to publish an object in a testing environment. Sometimes I found kind of unpredictable behaviour using this method. Usually it can be avoided using zope.testbrowser (see next point). You need to use it if you are testing alternate publishing methods (like WebDAV or VirtualHostMonster) which rarly be the case.

One thing I could track down is a cookie reset if diazo is turned on. This is because of a subrequest which is issued during traversal. You can disable diazo during testing:

>>> response = self.publish(docpath, basic_auth, env={'diazo.off': "1"})

To debug functional testing you need the following patch in your (failing) test.

def raising(self, info):
    import traceback
    traceback.print_tb(info[2])
    print info[1]

from Products.SiteErrorLog.SiteErrorLog import SiteErrorLog
SiteErrorLog.raising = raising

Fifth: Testbrowser Using zope.testbrowser is supported with plone.app.testing too. There are two main differences: a browser instance is initiated with the application: like this:

>>> browser = Browser(self.layer['app'])   # in functional test cases

or

>>> browser = Browser(layer['app'])    # in doctests

You need to commit changes before! you initiate the browser.

>>> from transaction import commit
>>> commit()

Sixth: plone.protect

If you are using a view, which uses CSRF protection via plone.protect you may want to disable this feature in tests temporarily. You can call your view by injecting a CSRF token into the request like this:

>>> from plone.protect import createToken
>>> request.form['_authenticator'] = createToken()

The original idea I found in this blog.

Seventh: Functional doctests In functional doctest sometimes a http function is found. This is the doctest analog of the functional test publish method. Currently it fails with plone.app.testing. I am investigating this and keep you posted, if I found something ...

And now happy porting to plone.app.testing of your addons. BTW the porting of some products is left for core Plone. If you want to give it a try ... go ahead. :)

See you on the Plone Conference in Bristol, Tom

15 Oct 2014 8:00pm GMT

Six Feet Up: 1.9.0 release brings new features and documentation to collective.recipe.plonesite

plone.pngThe 1.9.0 release of collective.recipe.plonesite is now available on PyPi. This release brings several new features and some improvements to the documentation.

The biggest new feature is the integration with collective.upgrade. This allows for the plonesite recipe to automatically upgrade the Plone portal and run any upgrade steps for installed packages in the site. Credit goes to Ross Patterson for adding this feature.

A couple more features were also added to the latest release. Fabio Rauber contributed a feature that provides the ability to have the plonesite recipe automatically add mount points. See the documentation for an example of how this works. The last new feature is the ability to run the plonesite part using sudo. This allows the plonesite part to be run when the Zope instance and ZEO server are run under different UIDs. Toni Mueller provided this new option.

Lastly, the documentation was cleaned up in the source and for the end user. The documentation on PyPi is now grouped by option type so that it is easier to find what you are looking for. The docs were also cleaned up in source control so that it is easier to contribute.

15 Oct 2014 12:00pm GMT

Andreas Jung: Results of the Plone Developer Survey

15 Oct 2014 6:58am GMT

14 Oct 2014

feedPlanet Plone

Four Digits: Announcing the Anniversary Sprint 2015

Four Digits will be 10 years old on the 23rd of June 2015, so what better way to celebrate then a Plone Sprint! So after the succes of the Living Statues Sprint 2010, 2011, the Plone Conf 2012 and Arnhem Sprint 2013 Four Digits will be hosting another sprint in 2015! The sprint will be from Monday to Friday and will finish with a big party on Friday evening.

anniversary-sprint-logo.png

When and Where

22 - 26 June 2015, Jansbinnensingel 26, 6811AL Arnhem, The Netherlands

Topics

Topics will be announced.

Organizer

Four Digits (+31 26 4422700)

Travel info

Travelling to Arnhem after a plainride can be done easiest by train. The office is located at walking distance of the Arnhem train station.

From Schiphol there is a direct train to Arnhem which takes about an hour.

From Weeze Airport there is a bus(platform 5) from Nijmegen take a train to Arnhem (about 8 an hour).
Alternative From Weeze, take the bus direct to Arnhem Gelredome(EUR 18,-). From there take line 5 or 7 (or any other bus) to the city and get out at "Jansbinnensingel". From here you can see our office

From Eindhoven, take bus 401 to Eindhoven CS (Railway station). From there take one of the following trains:
- Train to Hertogenbosch('s), from there a train to Arnhem
- Train to Nijmegen, from there a train to Arnhem

If you have any questions or want any please don't hesitate to contact us!

Location

The sprint will be held at the Four Digits Office, located in the city centre of Arnhem. We have room for more than enough sprinters. There are a lot of hotels/B&B's close to the office. If you need any help finding a suitable hotel let us know and we'll gladly assist you.

Internet

Wireless network on a fiber connection.

Sign up

If you would like to join us, please sign up for this project, if you want to stay a part of the sprint, please let us know. If you have any questions, don't hesitate to contact us.

Food

Everyone can join us for live cooking at the office and breakfast and lunch are included as well for all sprinters.

Hotels

"Hotel Arnhem Centraal", "Hotel Haarhuis", "Hotel Molendal", "Hotel Old Dutch", "Hotel Blanc", "Bordeaux B&B" and "Hotel Rembrandt" are all within walking distance of our office (Jansbinnensingel 26, Arnhem). If you can't find a suitable hotel let us know and we'll help you find one

More information on the CoActivate page

14 Oct 2014 6:17pm GMT

Plone.org: Announcing the Anniversary Sprint 2015 in Arnhem, June 22-26

Four Digits will be 10 years old on the 23rd of June 2015, so what better way to celebrate then a Plone Sprint! So after the succes of the Living Statues Sprint 2010, 2011, the Plone Conf 2012 and Arnhem Sprint 2013 Four Digits will be hosting another sprint in 2015! The sprint will be from Monday to Friday and will finish with a big party on Friday evening.

14 Oct 2014 3:59pm GMT

Six Feet Up: Learning Plone as a SysAdmin

Plone Code BadgeI recently started learning to develop for the Python-based CMS, Plone. This is my perspective on learning to be a Plone developer with years of experience as a SysAdmin and Python developer.

More Powerful Than I Thought

Getting to know Plone, I can really see its appeal for someone who needs a CMS. It can handle anything you want it to. I've been really impressed with the granularity of:

It's impressive to me that you can even create your own content types. Plone is packed with features for large organizations and let's you customize it to your requirements.

There's no Free Lunch

The flip side of the ability to do whatever you want is that it has a steep learning curve. Figuring out where parts of Plone start and end can be difficult:

This is further complicated by the fact that almost everything has an "old way" and a "new way".

Make sure you understand vanilla Plone before you dive into anything custom on an existing site. There are also many versions of Plone in the wild that are still supported from 2 to 4.3, and 5 is just around the corner. Make sure to learn on the version your organization is already using or plans to use.

How I would learn Plone

If you need to learn how to develop for Plone, I recommend to start you:

Learn Buildout First

With Plone you don't install packages like you traditionally would with `pip install`. You list all the packages in a file that Buildout uses to install everything at once and track their location. You should also understand the difference between a Unified Installer buildout or a manual buildout.

ZODB and Plone

Get to know the commands and sequence required to start the ZODB and Plone and shut them down. Then you can get familiar with the command scripts that helps take care of this for you.

Learn How to Get Into the Debugger

Next I would get familiar with the Plone debugger which will let you inspect and interact with live objects.
http://docs.plone.org/develop/plone/getstarted/debug_mode.html


Explore the Admin Interface

Once you are comfortable with getting Plone installed, running, and debugged, you can start getting familiar with the user's admin interface inside Plone. Take a look around and get familiar with creating content like blog posts as well as changing system settings.

Understand the Zope Management Interface (ZMI)

Once you've played with settings and content in Plone, you can hop over into the ZMI and see how the changes you made in the Plone interface modifies and creates data in Zope.
http://docs.zope.org/zope2/zope2book/UsingZope.html

At this point you should be familiar enough with Plone to start looking at how to build the site you want. Then things get real interesting depending on where you go with this powerful CMS.

Did you enjoy this post? I'll continue to blog on my perspective of Plone and how to modify and manage it.

14 Oct 2014 12:00pm GMT

11 Oct 2014

feedPlanet Plone

UW Oshkosh How-To's: How to hide the title and/or description on a Plone 4 page

Based on information at http://plone.org/documentation/kb/how-to-write-templates-for-plone-4

Create the view template

If you want to hide the title and description on a Plone page, you can create a new template that does that.

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"
      xmlns:tal="http://xml.zope.org/namespaces/tal"
      xmlns:metal="http://xml.zope.org/namespaces/metal"
      xmlns:i18n="http://xml.zope.org/namespaces/i18n"
      lang="en"
      metal:use-macro="context/main_template/macros/master"
      i18n:domain="plone">
<body>

<metal:content-title fill-slot="content-title">
    <metal:content-title define-macro="content-title">
     </metal:content-title>
</metal:content-title>

<metal:content-description fill-slot="content-description">
    <metal:content-description define-macro="content-description">
     </metal:content-description>
</metal:content-description>

<metal:content-core fill-slot="content-core">
    <metal:content-core define-macro="content-core">
        <metal:field use-macro="python:context.widget('text', mode='view')">
            Body text
        </metal:field>
    </metal:content-core>
</metal:content-core>

</body>
</html>

The difference between the above and the default view template is that both the content-title and the content-description macros no longer do anything.

Next, you have to enable this new customized view template for pages ("Documents").

Add the new view template to the page/Document type

Using the new view template

11 Oct 2014 4:53am GMT

10 Oct 2014

feedPlanet Plone

Espen Moe-Nilssen: A different approach to mobile theming.

Goals

Installation

Setup

How does i work?

The same site visited via two different domains

The slider part

(this part is still a bit experimental)

Troubleshooting

10 Oct 2014 1:55pm GMT

09 Oct 2014

feedPlanet Plone

UW Oshkosh How-To's: How to export and import Plone users

I found this very useful information which shows how to copy the usernames and passwords from one Plone site to another, by exporting the credentials into a file then reading the file into another site:

http://play.pixelblaster.ro/blog/archive/2011/03/09/export-import-users-in-and-out-of-plone

import cPickle

def export(self):
    pas = self.acl_users
    users = pas.source_users
    passwords = users._user_passwords
    result = dict(passwords)

    f = open('/tmp/out.blob', 'w')
    cPickle.dump(result, f)
    f.close()

    return "done"

def import_users(self):
    pas = self.acl_users
    users = pas.source_users
    f = open('/tmp/out.blob')
    res = cPickle.load(f)
    f.close()

    for uid, pwd in res.items():
        users.addUser(uid, uid, pwd)

    return "done"

However, I didn't want to use a file since I have a small number of users to copy. Instead, I used the ZEO client in "debug" mode to copy the username & password dictionary.

Getting the credentials from the source site

On the source site I did started the instance in debug mode (you could do this with a ZEO client too if you have a ZEO setup by using bin/client1 for example):

$ bin/instance debug

>>> mysite=app.MYSITEID
>>> pas=mysite.acl_users
>>> users=pas.source_users
>>> passwords=users._user_passwords
>>> dict(passwords) {'user1': '{SSHA}hashedpw1', 'user2': '{SSHA}hashpw2'}

I copied the last line (the passwords dictionary contents) to my terminal clipboard.

Applying the credentials on the destination site

Then on the destination site I launched the instance in debug mode and navigated back down to the same spot.

Then I defined a variable called "passwords" and set it to the text I'd copied into my clipboard above. That's the line that starts with "passwords =".

Then, using that new "passwords" variable, I made a for loop and called the users.addUser() method with each set of credentials in the passwords dictionary.

Finally I committed the ZODB transaction to make the new users persist in the database.

$ bin/instance debug

>>> newsite=app.NEWSITEID
>>> pas=newsite.acl_users
>>> users=pas.source_users
>>> passwords = dict({'user1': '{SSHA}hashedpw1', 'user2': '{SSHA}hashpw2'})
>>> for u in passwords:
... users.addUser(u, u, passwords[u])
...
>>> import transaction
>>> transaction.commit()
>>>

See Also

This documentation explains "Exporting and importing member passwords":

http://docs.plone.org/develop/plone/members/member_basics.html?highlight=password#exporting-and-importing-member-passwords

09 Oct 2014 4:10pm GMT

Plone.org: Plone Foundation Welcomes New Members

Johannes Raggam, Leonardo J. Caballero G. and Maarten Kling are the Foundation's newest members.

09 Oct 2014 3:20pm GMT