31 Oct 2014

feedPlanet Plone

Maurits van Rees: Matt Hamilton closing remarks

Thank you to the hotel, video and sound guys, our sponsors. Thank you Netsight team. Astra has arranged all the logistics and organized me. Thank you Scott or Wifi. Thank you Vic, my wife, who felt very welcome.

Audience: "Thank you Matt!"

Thank you Plone community.

Not the end. Sprints tomorrow.

Looking forward to next year. What about 2015?

Alex Ghica. http://eaudeweb.ro. We will do the next Plone conference in Bucharest in Romania. Part of Europe, EU, NATO, so all is safe. :-) Venue is a five star hotel, city center, well connected. International airport. Ticket from London if you book it now is 20 euro. 12 to 18 October 2015. Thank you Elena Melis. Thank you Maurizio Delmonte as Sprint manager. Thank you Plone Foundation.

31 Oct 2014 5:38pm GMT

Mikko Ohtamaa: The peak of Plone T-shirt collection

Below is the family photo of my Plone T-shirts, before their utter destruction. Just to keep memory around for the future generations. Some misc. T-shirts mixed in.

1-IMG_6185

From left to right, top to bottom

Though Plone may be sunsetting technology, as Pokemons would say, friends forever.

See you on the other side.

koala

Subscribe to RSS feed Follow me on Twitter Follow me on Facebook Follow me Google+

31 Oct 2014 4:01pm GMT

Maurits van Rees: Plone Lightning talks Friday

Ben Cole and Matt SS: netsight.cloudstorage

Add-on designed to offload large data to cloud storage. Transcoding of videos to web-compatible format. Control panel to configure things. You can also stream it.

We will make a release of that soon. For now, see https://github.com/netsight/netsight.cloudstorage

Manabu Terada: Plone behind CDN

Oud customer requested highest available system, lots of requests, large requests, more than 384 MBps. CDN: Content Delivery Network. Using CloudFront. nginx, varnish, Plone 1-3. With some configuration nginx will first get the content normally from Plone, later requests will be handled by the CDN. Using this for various universities, including single sign-on for students, some for local government.

Please join Plone Symposium Tokyo group.

Eric Steele: Hey, when is Plone x.x.x going to be released?

We check Jenkins, making sure all tests pass, check for new packages, make sure changelog are updated, check crazy commits, update versions.cfg, check p.a.upgrade, ask for locales, update metadata.xml, make pending release, unified change log, notify installers team, probably forgetting a few.

You make some progress releasing, then someone does more changes, hard to keep up and keep it stable. Ticket 13283: merge several packages into one, may help.

Thank you to Zest for creating zest.releaser, keeping me sane.

Alexander Loechel: security

Studies on security. Take at least fifteen minutes per day per system to check for updates. Drupal had some problems recently, unless you updated your system within seven hours of releasing the hotfix.

Check these packages:

  • plone.app.vulnerabilities
  • plone.hud, plone.app.hud
  • Check for vulnerability checks: plone.vulnerabilitychecks

See https://github.com/loechel/plone.vulnerabilitychecks.core

Timo Stollenwerk: Plone Testing and Continuous Integration Team

Keeping a green build since 2011. I am the guy who shouts at you when you break the plone testing build, that is why I now sound like I sound.

Continuous integration rules: http://buildoutcoredev.readthedocs.org/en/latest/continous-integration.html

We did some sprinting this year. I went to the Jenkins User Conference Europe conference in June 2014, presenting how Plone uses it.

CI infrastructure. Sven did a really great job of creating an infrastructure with nodes, multiple builds and test jobs. Ramon created mr.roboto to report results back. Asko improved acceptance testing, with robotframework. Testing cleanup, moving from PloneTestCase to plone.app.testing.

Future plans: maybe Docker setup.

Team page: http://plone.org/community/teams/cit

Gil forcada: are you ready for sprinting?

Have you ever not finished what you are sprinting on?

Use https://github.com/collective/sprints

Add info on your sprint topic: titel, what, who, code status. Update it at the end of the day.

Fred van Dijk: Hierarchy and/or Relational

Another look at how you structure information on your site.

Too much hierarchy is deadly for customer centric. Plone has a hierarchical folder structure, that always shines through, even when you add relational items or tags. Conflict: chaos or relational.

We want to use our hierarchical Plone hammer and see nails everywhere. Designer had other ideas, maybe because of Drupal background. They wanted 25 different content types, just so they could have relations between themes, projects, etcetera. Oh, there was no Folderish type.

But couldn't we use mostly simple Folders and Pages? The functionality is not actually different, just the label.

Landing pages around themes. collective.cover. Theme page has links to reports and other stuff.

Two part labeling: categories and themes. collective.classifiers dexterity behavior.

You can search in collections on these classifications: give me all 'themes' on category 'water'.

Similar solution may be collective.taxonomy.

Jens Klein: Alpine City Sprint 2015

This winter we will at last again have a sprint in Austria, in Innsbruck, capitol of Tirol. Lots of (olympic) ski areas. Very nice view from high restaurant that we will be visiting. Your family is welcome too. Guided tour in quantum physics institute.

Sprinting on probably Plone 5. Stabilize what we have, for doing daily business with Plone without headaches but with fun. plone.app.multilingual needs some love. You are invited to work and have some fun and maybe stay a few days longer and enjoy Innsbruck.

Find us on Coactivate.

Pawel Lewicki: ADFS

ADFS: Active Directory. Single Sign-on.

Using dm.zope.saml2, extending it with some functionality.

Julia Pasov: Plone, Virtuoso, Elasticsearch

Purpose: link data from multiple sources. Fast search for semantic data (rdf). Index data from an endpoint (Virtuoso) into Elasticsearch.

You need an RDF marshaller for the Plone website. Via Virtuoso. RDF River plugin for Elasticsearch.

But you can use the plugin to get the rdf from Plone directly into Elasticsearch, without Virtuoso in the middle.

Asko Soukka: Mosaic

Mosaic is a rendering engine for Plone. Flexible custom layout of the page. You can make prototype layouts that editors can use.

Christian Ledermann: Founders4Schools

We connect teachers with entrepreneurs. Get them to speak to school children. See http://Founders4Schools.org.uk

We have an Appathon:

  1. Think of an app. Work in teams to think of an idea for an app.
  2. Create an app prototype. Go into universities to get students with code skills.
  3. Winners announced.

Guest house is in Mountain View. Companies like Google, Facebook, Tesla. Consolation prizes of Raspberry Pis.

Please host an Appathon yourself, very nice. Contact us for ideas and support.

Nejc Zupan: Plone on Heroku

We have it running. You need a credit card currently, but we will work on that.

Click the button here:

https://github.com/niteoweb/heroku-buildpack-plone

In your own buildout you need to change a few things, enable RelStorage, all documented on the above repository. Our company website is running for a couple of months on Heroku.

Eric Brehault: nodeenv

nodeenv is a virtualenv for Node. easy_install nodeenv, activate it, use npm install ....

See https://pypi.python.org/pypi/nodeenv

31 Oct 2014 2:54pm GMT

Maurits van Rees: Plone Foundation meeting

Present: Alexander, Andy, Cris, Érico, Paul, Steve from the board, plus lots of foundation members.

Paul opens the meeting. Everyone is welcome, only members can vote.

Items on the agenda: the old board leaves, new board enters, annual report.

Thank you ambassadors. We had new members, let's give them an applause. Thank you sponsors. Thank you sprint organisers and sponsors. The Foundation sponsors sprints, but that is only a beginning amount to help start things off.

We made a slight loss, we are not super rich, but we are secure. We have seen that the more money we spend the more money comes in, so I recommend to continue that.

Steve moves acceptance of the annual report, Philip seconds. Motion passes unanimously.

New board: seven nominations for seven positions, so no elections needed. If you prefer elections, run for the board, we love it.

Steve moves to appoint the board. Érico seconds. Motion passes unanimously.

Thank you to the outgoing board members, Érico, Andy, Spanky.

Welcome new board members: T. Kim Nguyen, Chrissy Wainwright, and Carol Ganz, who join Paul Roeland, Alexander Loechel, Steve McMahon, and Cris Ewing.

Officers will be appointed in the board meeting next week.

Chris C. moves to adjourn the meeting, several people second. Unanimously approved. Meeting is adjourned until next year. Place is hardly a secret, but will be announced later today.

More info on the Plone Foundation can be found at https://plone.org/foundation

31 Oct 2014 12:28pm GMT

Maurits van Rees: Open space on Plone 2020

Led by Philip Bauer and Martin Aspeli.

Roadmap on Plone 2020 was started in Brasilia last year at the conference. Was mostly focused on underlying technology, maybe moving from Pyramid to Zope. Sparked a discussion. Finding common ground. Where do we want to be? Which Python version? Which audience? What are our key strategies to be better?

Take ten minutes, write one or two items on a sticky pad: why are we here? Why are we using Plone? Maybe because it pays the bills, maybe because it does something that no one else does.

[Maurits: 1. Professional and friendly community. 2. It is my job (that I love).]

Martin summarizes.

So: if we are going to change Plone, can we please not mess the above up?

Critical mass: lots of people use Plone, so customers do not depend on only you: other companies can take over or assist, if needed. That is good.

Now look to the future. Write down on sticky notes: one or two things that you want to change in Plone in the next five years. Be specific, concrete.

[Maurits: less ways to do stuff, one canonical way, for example plone.api. Confusing for new (and old) developers that you can add something in a skin layer or a browser view or a utility or a tool. The move to storing every config option in the configuration registry is a good example of progress. Have a stripped down Plone with the old ways removed, though still addable as option if you include some extra packages and know what you are doing.]

Martin summarizes and we have a discussion. No one said: use Pyramid.

Plone Open Garden in Sorrento next year may be a good time to have a strategic planning summit. The Plone Foundation may be able to help people get there, with travel stipends. Also: it is good to pay the foundation so they can continue to do that.

Let's continue discussion on mailing lists too. Python and JSON api seem clear winners to work on. Consensus on removing things, but there are risks. Continue the conversation on UX improvements.

This right here is the roadmap team, not some secret group. This is where Plone happens.

Can we put this somewhere in documentation? [Feel free to copy this somewhere and improve it; Maurits.] Communicate it, add a news item. Can we do that? Yes, Eric will sit down with Christina.

Can we somewhere let people vote on ideas? Not just developers here, but users everywhere, without needing to subscribe to a mailing list.

31 Oct 2014 11:56am GMT

30 Oct 2014

feedPlanet Plone

Maurits van Rees: Keynote Michael Johnson: Kickstarting the Personal Space Age

I am the founder of http://pocketspacecraft.com

I am a physicist. Working in meteorology and ocean ventilation models and tools. Voyager and ROSAT analysis. Att Imperial College. Software developer, computer scientist. Aerospace. Co created the KickSat project.

Founded in 2010, operations in China, Isle of Man, UK, US.

Space is bug and relatively unexplored. Missions are expensive and infrequent, funding is tight, missions are risk averse, it is scary to try really new ideas, processors are often ten years old because we know all the bugs there.

Goal: before I retire (say in the next 25 years) I want to send a spacecraft to orbit and/or land on the surface of 'every' body in the solar system. Every is too much, but maybe one million objects?

We need a pocket space craft, that individuals can buy. Personal space age: everyone can get involved.

We want to explore. Consumer projects: how are you going to support that, on that scale? We want instant gratification for people, but it normally takes years to get something up in the air. Work together.

Help different communities involved. If you do something good for the science community, your project may be able to get on board a mission to Mars. Lots of legal hurdles, insurance. Small space crafts, but big problems.

Video games are a good metric for this. People are prepared to pay 50 dollars for that. Lots of challenges there.

What helps, is open source. An Open Source Space System is being worked on. This is done by individuals in their spare time, but also by big names in space. Bring the International CubeSat Consortium in the mix.

Using open source actually helps against some of the legal hurdles, making you exempt from them.

The CubeSat standard defines a standard space craft. One unit is en by ten by ten centimeter, using 1 Watt, 1 kilogram. You can combine a few. About 50k dollar to launch one. That was started ten years ago. About 70 made in that time, and about 70 in the last six months. So it is being picked up. Much much more planned for next year. Launched from the international space station.

How is it possible? Moore's law. A KickSat, much smaller than CubeSat, already has more computer power than Voyager had, so don't knock it.

Standard 3 unit CubeSat launch is available today. Not tied to any launch vehicle or nation. We need to get it close to where we want to get (moon, planet). Then we deploy and do whatever we want (except introducing bacteria, so there are rules).

Pocket Spacecraft prototype, 32-96 millimeter diameter, less than 50 micrometer thick, 10-100 milligram, 5-100 MIPS, up to 100 GB storage, optical communications. Can be manually produced.

Long term goal: print space craft in space. Design your space craft, send the instructions in the direction of the printer in orbit around Mars, wait 20 to 40 minutes depending on time of year, and you can launch your space craft.

There is now Open Mission Control software, and Pocket Mission Control for your Android.

You need to be able to talk to these space craft, via ground stations. We can do that amateur radio based. See http://mygroundstations.com

But with a credit card and some convincing you can rent NASA communications by the hour and use the same stuff that still talks to Voyager, at 80 light hours distance.

But there also is the LOFAR network, for radio astronomy, which you could use to pinpoint your space craft. Lots of data, several Peta byte for 15 minutes. Expensive, but prices will drop: Moore's law again. There is an awful lot to do, but that is fine.

Where do you want to explore today?

Standards and info:

Spacecraft parts:

Questions?

50 percent of CubeSats fail. But 50 percent of large space projects fail as well. Interesting, isn't it?

Space junk? We are responsible about that, making sure we do no harm, otherwise we may no longer be allowed in space.

30 Oct 2014 6:17pm GMT

Maurits van Rees: Calvin Hendryx-Parker: Hands on with Multisite Management using Lineage

I am CTO at Six Feet Up. Doing Plone since 2003. Lineage was one of my first contributions.

Lineage is an add-on for Plone. Enabled by the pieces that are built into Plone, like INavigationRoot, IPossibleSite, together as IChildSite. So we are using the ZCA (Zope Component Architecture).

With Lineage you create one site and lots of sub sites for departments. Advantages compared to having lots of Plone sites:

Disadvantages:

There are various add-ons/extensions that build on Lineage, for example lineage.themeselection for giving sub sites a different theme.

Add collective.lineage to the eggs in your buildout. Activate it in the add-ons control panel. Go to a folder, to the actions drop down and enable this folder as a sub site. Within this folder, the home will be this folder. You can add sub sites within sub sites if you want. You can exclude a sub site from navigation if you want to.

I want to show you some cool tricks you can do.

You can use custom site types. As long as they are IFolderish types, you can enable a sub site on it. At Penn State we used that to ease creating a new course.

Some add-ons:

With Resonate it is possible to move content between sub sites, via the workflow, giving control to the content managers of the sub sites.

With Resonate you can syndicate content to various sub sites, still using standard Plone workflow machinery. A teaser of a news item can then be shown on the other sub sites, where they point to the original item, so that one remains the canonical one, which is good for SEO. The editor can see which sub sites accepted and rejected the share.

How does it perform? I set up two Zope instances, one with 100 Plone Sites, one with 1 Plone Site and 100 sub sites. Advantage of Lineage, with blank sites: about 9 MB against about 130 MB. Startup memory did not differ much, slightly better for Lineage. (Audience: may be influenced by the ZODB cache size too.)

Lineage is actually Plone 5 compatible. Version 2.0 released last night.

Code: https://pypi.python.org/pypi/collective.lineage

Example usage: http://www.la.psu.edu

For migration from the old separate sites to sub sites we used transmogrifier.

30 Oct 2014 5:06pm GMT

Maurits van Rees: Franco Pellegrini and Johannes Raggam: Ready, set... Mockup!

This is about Plone's new front-end library. Mockup is a framework for adding Javascript functionality from other libraries to Plone. Will be part of Plone 5.

For example the 'moment' pattern:

<span class="pat-moment">some date</span>

That will output a nicer date using the client browser language.

Currently, on Plone 4.3, we are still developing js as if it were 2004. 41 Javascripts registered in a default site. The resource registries have a packer functionality, with last commit from 2009. Maybe not so good anymore. No tests.

History of mockup. Based on Patternslib, created by Cornelis Kolbach and Wichert Akkerman and others in 2011. Forked by Rok Garbas in 2012, split up in Mockup and Mockup core in December 2013. plone.widgets and plone.app.widgets by Nathan van Gheem in November 2011, an attempt to bring more modern UI to Plone.

Build environment:

[Showing some code.]

We made a generator for mockup:

npm install -g generator-plonemockup
yo plonemockup

Then you answer a few questions and you get a base pattern.

Now do some coding and call make and it will minify, create a bundle, etc.

There is a training that you can follow online:

http://mockup-training.readthedocs.org

Alternatives:

Our opinion about Mockup: great framework, big improvement to current situation, nice workflow, but uncertainty about future, maybe Angular could be better. Open space tomorrow at 11:00.

Audience: mockup (or patternslib) is an abstraction layer, Angular is something really different. Don't use Angular as base for the Plone JS. We have to get the whole community into thinking 'Javascriptish', really push it into the head of more developers. It is actually pretty easy to change a Mockup pattern into a Patternslib pattern or the other way around. Changing a date picker pattern into a different one? Not so difficult either. Not enough people are currently feeling secure developing with Mockup, but that will come.

30 Oct 2014 3:51pm GMT

Maurits van Rees: Érico Andrei: The case for Plone Distributions

In some cases there are many sites that are almost the same, for example universities using Plone EDU. Different look and feel, different content, same base.

At Simples Consultoria we have been successful in dealing with this market.

Marketing Plone is very hard. It can do so much, is so flexible, how are you going to market that to a specific audience.

Marketing a solution is easier. If a client wants an intranet, I am not going to tell him that we made a portal for the Brazilian government.

Customizing Plone should not be hard. Distributions should be first class citizens. Offer a download specifically for educations or for an intranet on plone.org. The code we use to make the standard installers, VMs, packages, should be there for distributions too.

We need documentation, tests. We would love to have help with Jenkins and stuff.

Talk the customer's language, know their needs. For example PloneGov and local initiatives.

Something like the Plone Intranet Consortium is the very best thing that happened to Plone in a long time. We need to work like this. Companies should act together. Plone and Intranet are a perfect match. Bring new people to Plone. Companies will save Plone. We love Plone, but Plone needs customers and companies.

30 Oct 2014 2:32pm GMT

Maurits van Rees: Eric Bréhault: Running a Plone product on Substance D

Why should you want to create or run a Plone product on Substance D? Because it is fun. It might be a good experience for the future of Plone.

Substance D has all the good things from Pyramid. Plus stores data in a ZODB. It is a CMS.

Rapido is the next Plomino version. Plomino started in 2006, still based on Archetypes, stores data into CMF objects. Uses extensively ZCatalog and PythonScript.

I turned it into Rapido. Plone 5. Based on dexterity.

[Demo of Plone part and Substance D part.]

So how is this done?

My experience with Substance D. Pros:

Cons:

Conclusions for me about Plone future.

30 Oct 2014 12:51pm GMT

Maurits van Rees: Laurence Rowe: Layering web applications on web services with JSON Linked Data

This talk is about the ENCODE portal, an encyclopedia of DNA elements.

I want to talk here about a pattern, not really about the specific technologies involved.

I work at the data coordination center, generating and organizing data. We get metadata submissions, store it in a metadata database. It is really a knowledge management system. It could have been built in Plone, but it would not be a great fit. I started from scratch based on Pyramid and ReactJS. Nowadays services more and more have a Javascript UI, where Javascript talks to the backend.

Embrace Javascript. There has been progressive enhancement. Single page web apps really need it. For building a portal, I have been looking at isomorphic web applications. Originally from the Render framework. Quickly loading pages are important. The exit rate for visitors goes up with the loading time.

Json is the lowest common denominator for data. xml is more flexible, but more complex. In Python it is much easier to use json.

JSON-LD: json link data. Adopted recently by the w3c. It is partly about semantic data, but we are not using that yet.

At first we needed to duplicate the routing information on the server and the client side. JSON-LD allows us to express type information which avoid the duplication.

You can have JSON-LD like this [in "pseudo json" here, just for the idea, Maurits]:

{
  @context: context/jsonld
  @id: ...
  @type: [biosample, item]
  @submitted_by: {
    @id: /users/lrowe
    @type: [user, item]
    name: Laurence Rowe
    }
}

Defined using JSON Schema, which is an IETF draft. Schema version, field definitions, types, validators. It is an extensible format.

All our normalized data is stored in Postgres as json. JSON-LD has the concept of framing, which objects should be embedded (mapping to a view in Plone terms), possibly doing some elasticsearch. Above it is server rendering (with NodeJs) that is creating html, and sends it to the browser. After the first page load, the browser uses Javascript to talk the the JSON-LD backend directly, instead of via the NodeJS server, letting the ReactJS Javascript do its own rendering.

The browser opens a page. Gets html back. Then it does a query for json data and gets that back and shows it on the page.

Indexing linked data. Updating one item may need a reindex of other items that link to it. We have written code for that. Using elasticsearch as a cache.

Come work with us, we are hiring.

See code at https://github.com/ENCODE-DCC

30 Oct 2014 12:10pm GMT

Maurits van Rees: Guido Stevens: The Plone Intranet Consortium

This is a big picture presentation. But don't worry, there will be a demo at the end.

United we stand, divided we fall. Plone is in a rough spot. Let's design and build some solutions. Yes, Wordpress is eating our lunch and our cake, but we should not sit back.

Code: https://github.com/ploneintranet/ploneintranet.suite

I have my own company, Cosent.

The Plone community is good. The backend code is good. The user interface was good, but has fallen back. Good enough is not good enough anymore. We need to do better. Only excellent user experience will win us customers.

When as a Plone company you want to sell Plone as an Intranet, you have a good starting point, but you are missing lots of pieces. You would have to build that yourself and let one customer pay for it. That is not going to happen, or at least it is a hard sale.

In the Plone community, the commits per month is rising, and the amount of committers per month is also rising. Doing pretty well.

So what is wrong? We need to involve or we will die, like dinosaurs. Plone Core currently is Web 1.0. We need a Plone Social, web 2.0: read/write, social networking, activity stream, time centric, personal perspectives, bottom-up sharing.

Roadmap: http://cosent.nl/roadmap

In 2010 I had the choice to ditch Plone or fix Plone. I chose to put all my eggs in one basket: Plone. Is Plone Social good enough? No. Look at Jive, that is some serious competition. We do not need to beat such a SAAS product, but we need to be close. Then customers will still prefer the local company that is closer to them and more attuned to their needs.

I think the "spare time" model of development is broken. It has brought us this far, but it is not enough anymore. Stuff is nearly finished on sprints, and then lags too long.

We need a new model. We have the talent, the technology. We can do it. We need to invest in a high quality, out-of-the-box product baseline. Low purchase cost, immediate sales appeal, fast delivery, shared maintenance across the community, new community ethos in collaborating together.

As Plone Intranet Consortium we band together and want to work on this. We had a meeting after the Plone conference last year. Every company there had their own workspace solution, everyone was maintaining their own stack. No resources enough to generalize it for the community.

Design first. It is not about eye candy. You start with a decent vision of what your strategy is, what your project is trying to solve.

Roadmap-driven Scrum development. Normal working day, in company time. Legitimate leadership serves the community. The consortium board funds the roadmap. Investment per company: 1000 euro per month, plus one developer day per week. Cash is used to hire people to help with the design. Sprint every Wednesday.

It is 100 percent open source. It is not a product that we will make money on. We will make money on the service we deliver. We want to move the license to the Plone Foundation, we will talk about that.

What we are developing, are add-ons, not a core fork. Plone 5 compatible. Will port to mockup. You are welcome to join the consortium.

Cornelis has made a patternslib-based clickable prototype, that needs no backend to operate.

Demo by Alexander Pilz.

User experience sells. We showed this demo to a client last week and he thought it was an impressive preview of social functions in future Plone.

Roadmap. Phase one: activity streams, team spaces, dashboards, document structures/wiki. Phase two: calendaring, search, news hub.

We are pioneering a new business model for open source.

  1. Dream a vision.
  2. Combine investment.
  3. Design first! Use dedicated designers.
  4. Develop and release.
  5. (or really 3.1 already) Win customers.

We can boldly go where no one has gone before. We are Plone, we can do anything.

We have an open space tomorrow. Welcome! Sprint on Saturday and Sunday.

Code: https://github.com/ploneintranet/ploneintranet.suite

30 Oct 2014 10:46am GMT

Reinout van Rees: Ubuntu PPA madness

I'm going flipping insane. In ye olde days, when I was programming with the python CMS Plone, my dependencies were limited to python and PIL. Perhaps lxml. LXML was a pain to install sometimes, but there were ways around it.

Working on OSX was no problem. Server setup? Ubuntu. The only thing you really had to watch in those days was your python version. Does this old site still depends on python 2.4 or is it fine to use 2.6? Plone had its own Zope database, so you didn't even need database bindings.

Now I'm working on Django sites. No problem with Django, btw! But... the sites we build with it are pretty elaborate geographical websites with lots of dependencies. Mapnik, matplotlib, numpy, scipy, gdal, spatialite, postgis. And that's not the full list. So developing on OSX is no fun anymore, using a virtual machine (virtualbox or vmware) is a necessity. So: Ubuntu.

But... ubuntu 12.04, which we still use on most of the servers, has too-old versions of several of those packages. We need a newer gdal, for instance. And a newer spatialite. The common solution is to use a PPA for that, like ubuntugis-stable.

Now for some random things that can go wrong:

  • We absolutely need a newer gdal, so we add the ubuntugis-stable PPA. This has nice new versions for lots of geo-related packages, for instance the "proj" projection library.

  • It doesn't include the "python-pyproj" package, though, which means that the ubuntu-installed python-pyproj package is compiled against a different proj library. Which means your django site segfaults. Digging deep with strace was needed to discover the problem.

  • Of course, if you need that latest gdal for your site, you add the PPA to the server. Everything runs fine.

  • A month later, the server has to be rebooted. Now the three other sites on that same server fail to start due to the pyproj-segfault. Nobody bothered to check the other sites on the server, of course. (This happened three times on different servers. This is the sort of stuff that makes you cast a doubtful eye on our quite liberal "sudo" policy...)

  • Pinning pyproj to 1.9.3 helped, as 1.9.3 worked around the issue by bundling the proj library instead of relying on the OS-packaged one.

  • Ubuntugis-stable sounds stable, but they're of course focused on getting the latest geo packages into ubuntu. So they switched from gdal 1.9 to 1.10 somewhere around june. So /usr/lib/libgdal1.so became /usr/lib/libgdal1h.so and suddenly "apt-get update/upgrade" took down many sites.

    See this travis-ci issue for some background.

  • The solution for this PPA problem was another PPA: the postgres one. That includes gdal 1.9 instead of the too-new 1.10.

  • Possible problem: the postgres PPA also uses 1.9 on the new ubuntu 14.04. 14.04 contains gdal 1.10, so using the postgres PPA downgrades gdal. That cannot but break a lot of things for us.

  • I just discovered a site that couldn't possibly work. It needs the ubuntugis-stable PPA as it needs a recent spatialite. But it also needs the postgres PPA for the 1.9 gdal! And those two don't match.

  • It still works, though. I'm not totally sure why. On a compilation machine where we build a custom debian package for one of the components, the postgres PPA was installed manually outside of the automatic build scripts. And a jenkins server where we test it still has the ubuntugis PPA, but somehow it still has the old 1.9 gdal. Probably someone pinned it?

  • Another reason is probably that one of the components was compiled before the 1.9/1.10 gdal change and didn't need re-compilation yet. Once that must be done we're probably in deep shit.

  • If I look at some ansible scripts that are used to set up some of our servers, I see the ubuntugis PPA, the mapnik/v2.2.0 PPA and the redis PPA. Oh, how can that ever work? The software on those servers needs the 1.9 gdal, right?

  • I asked a colleague. Apparently the servers were all created before june and they haven't done an "apt-get upgrade" since. That's why they still work.

Personally, I think the best way forward is to use ubuntu 14.04 LTS with its recent versions. And to stick to the base ubuntu as much as possible. And if one or two packages are needed in more recent versions, try to somehow make a custom package for it without breaking the rest. I did something like that for mapnik, where we somehow needed the ancient 0.7 version on some servers.

If a PPA equates to never being able to do "apt-get update", I don't really think it is the best way forward for servers that really have to stay up.

Does someone have other thoughts? Other solutions? And no, I don't think docker containers are the solution as throwing around PPAs doesn't get more stable once you isolate it in a container. You don't break anything else, true, but the container itself can be broken by an update just fine.

30 Oct 2014 10:10am GMT

Maurits van Rees: Jens W. Klein: Big Fat Fast Plone - Scale Up, Speed Up.

I am owner of Klein & Partner, member of Blue Alliance, Austria. Doing Plone since version 1.0.

Default Plone is not so fast. Scales great horizontally (so adding machines), but there are still bottlenecks, primarily loading stuff from zodb.

First customer Noeku, over 30 Plone sites, hi availability, low to mid budget, self hosted on hardware, VMs. The pipeline is: nginx, varnish, pound, several Plone instances, databases (zodb, mysql, samba).

Second customer Zumtobel, brand specific international product portals, customer extranets, b2b e-shop, hosting on dedicated machines.

Third customer HTU Graz, one Plone site with several subsites (with lineage), lots of students looking at it, so we have a peak load.

Main Plone database is ZEO or PostgreSQL, plus blobstorage (NFS, NAS). Load balancer: haproxy or pound. Caching proxy: varnish (don't use squid please). Webserver: nginx (better not use apache).

The Plone instances and the client connection pool (between Plones and database) can use memcached (maybe multiple), LDAP, other third party services.

If you want to improve things, you must measure it: use munin, everywhere you can. fio is a simple but powerful tool to get measures on your io. Read up on how Linux manager disk/ram. Know your hardware and your VMs (if any).

Database level

  • Noeku: zeo server, blobstorage, both replicated with drdb
  • Zumtobel: RelStorage on PostgreSQL, blobs from NAS over NFS.
  • HTU Graz: RelStorage on PostgreSQL, all on one machine.

First things first: never store blobs in ZODB: use blobstorage. Standard Plone 4.3 images from news items are stored in zodb. You can change that. Check your code and add-ons.

ZEO server plus blobstorage: ensure a fast IO to harddisk or RAM, and have enough RAM for disk buffering.

Blobstorage on NFS/NAS: shared blobs and mount them on each node. Mount read-only on web server node and use collective.xsendfile (X-HTTP-Accel) to make it faster.

RelStorage plus blobstorage: never store blobs in the SQL database (same as zodb). No MySQL if you can avoid it. Configure your SQL database.

Connections pool: ZEO vs RelStorage. ZEO server pushes invalidations to client. RelStorage: ZEO client polls for invalidated objects. Disk cache of pickled objects per zope instance. On RelStorage size you can use memcached, which is a big advantage, reducing load on the database.

  • Noeku, ZEO.
  • Zumtobel: RelStorage, history free, 2 VMs, 16 instances plus some worker instances for asynchronous work, each 2 or 4 threads. RAM cache 30,000 or 100,000 objects, memcached as shared connection cache. If packing takes too long, try relstorage_packer.
  • HTU: RelStorage, history free. 6 instances, each 1 thread. RAM cache 30,000 objects. This is something you need to tweak, try out some values and measure the effect. Memcached. Poll interval 120 seconds. Blobstorage: shared folder on same machine.

The above is not specific for Plone. The below is.

Plone

  • Turn off debug mode, logging, deprecation warnings.
  • Configure plone.app.caching, even if you are not using varnish. Browsers cache things too and it can really help.
  • Multiple instances: use memcached instead of standard ram cache.
  • Know plone.memoize and use it.
  • Never calculate search twice. Check your Python and template code to avoid things that boil down to: if expensive_operation(): expensive_operation().
  • Use the catalog.
  • Do not overuse metadata: of you add too many metadata to the catalog brains, they may become bigger than the actual objects, slowing your site down.

Write conflicts:

  • 90% of write conflicts happens in the catalog.
  • To avoid it, try to reduce the time of the transaction. Hard in standard situations, but you may be able to first prepare some data and later commit it to the database.
  • Use collective.solr or collective.indexing. I hope that in Plone 6 we will no longer have our own catalog, but use SOLR.

Lots of objects, hundreds of thousands? Are catalog queries slow? Use a separate mount point for the portal_catalog, with higher cache sizes.

Archetypes versus Dexterity. In AT, you should avoid to wake up the object, ask the catalog instead. With Dexterity, it is sometimes cheaper to wake up the object: if objects are small and you iterate over a folder or subtree, or if adding lots of metadata to the catalog would be needed.

Third party services, like LDAP and other databases, need to be cached. Talking to external systems over the network is slow.

In case of serious trouble: measure! munin, fio, collective.traceview, Products.ZopeProfiler, haufe.requestmonitoring, Products.LongRequestLogger. Change one thing at a time and measure it. Really important!

plone.app.caching: always install it. For custom add-ons with own types and templates, you need extra configuration for each type and template. Do this! Calculate in some time for this task, it is some work, but it is well documented.

On high traffic, introduce a new caching rule for one/two/five minute caches, really helps against peak load.

Load balancer. Pound is stable but old and difficult to get measurements. Haproxy not that simple, newer, nice WebUI for stats. You should point the same request type to the same instance.

Web server: nginx. Set the proxy_* to recommended values.

Read more: on docs.plone.org.

Looking up attributes in Dexterity can be slow, we need to fix that.

30 Oct 2014 9:57am GMT

29 Oct 2014

feedPlanet Plone

Maurits van Rees: Plone Lightning talks Wednesday

Steve McMahon: Plone installation, the future

What is the Installer team doing for Plone 5. You can still use your custom buildout, no problem, so your future may still vary.

Unified Installer will still work. Platform binary installers may not work; possibly Windows; nearly certainly not OS X, as it gets harder and harder to do open source development there.

The hot newness is Cloud or VM kits. We are working on making that happen for Plone too. Our principal tool is going to be Ansible. Build a full-stack server. Vagrant instance to test it before you deploy it. We are not scaring people away from other approaches.

Targets: from 5 to 5000 dollars per month, with configuration. Working on boxes for Digital Ocean, AWS, etc.

Plone 5: no longer ZopeSkel or templar, but mr.bob as tool for bootstrapping a new project. You are free to use the tool you like.

Paul Tax: Anniversary sprint

I am the new guy at Four Digits, Arnhem, The Netherlands. We are organizing a sprint on 22 till 26 June 2015. Hotels near, good wifi, standing desks if you want, agile planning, standups, food and drinks. Party on Friday, because it is our ten year anniversary. No problem if you only want to join for the party. Homebrewn beer.

Paul Roeland: Paragon

Paragon is the search for the very best in Plone add-ons. The hunt is on, especially for add-ons making the lives of integrators easier. No undead add-ons please, no unmaintained. Just good, useful add-ons. You can nominate until November 14. Are they maintained? Are they safe to put on a production site? Go to http://paragon.plone.org and nominate an add-on. Sorry, there are a lot of fields to fill in, but only a few are required. But the more the better, thank you for helping us. Results will be announced in December. We will feature those on the new plone.org.

Asko Soukka: venusianconfiguration

Venusian configuration: no more zcml. Put that configuration in a Python file. It can scan modules, a bit like grok, but explicitly. It will not be released, because I don't want all the hate mail. :-) But maybe incorporate it in Zope configuration.

See https://github.com/datakurre/collective.venusianconfig

Sally, Cris, Jens, Alexander, Andreas: bibliographies

Bibliographies are really important to a certain segment of users, academics specifically. There is funding to make some things better. This is a call for help. I scheduled an open space for this topic on Friday, please come and give your input. Create a plan of action.

Manabu Terada: Recent topics about Plone in Japan

PyCon Japan September 2014, 500 people, one English track. I conducted a session about Plone, about 100 people attended. Number of Python developers is increasing in Japan. We wish to spread Plone usage in Japan. Next year: Plone Symposium Tokyo! May/July 2015. Get interesting sessions, so speakers from overseas are very much invited. Airport, big city: English is no problem, outside may be difficult. Tokyo is very cheap. Safe? Yes, we even hold the Olympics in 2020. See you in Tokyo next year.

Maurizio Delmonte: Plone 5 at Sorrento

Abstract also exists for ten years. Join us at the Sorrento sprint next year. Don't wait until the end to save your room. It is somewhere in April 2015.

Here is a video of last year with people talking about Plone 5.

Bogdan Girman: dexterity base classes

Working at Der Freitag, Berlin. We have a dexterity content type Article. We needed to change the base class? One trick would be to delete the object and add it to the parent again, but that does not work with the event handlers that we also have. Main code to work around that:

parent._delOb(...)
parent._setOb(...)
obj.reindexObject(idxs=['is_folderish', 'object_provides'])

Eric Brehault: Why CMS won't die

All CMS are old. Wordpress is young at 2003.

We can make a no-CMS site quite easily. But a validation workflow and access rights would be nice. Newsletter. Document sharing, with some people. Multilingual. Next thing you know, your no-CMS crashes.

Face it, you need a CMS!

CMS are special. We are CMSistas, and only CMSistas know it. CMS are huge projects. For example: who needs buildout in Python, except us?

Wordpress is about 60 percent of all CMS websites. Huge! But 100 percent tomorrow? No way. I never saw Wordpress at any of my customers.

Nobody promises it will be easy. Is Plone too complex? Every CMS is complex.

29 Oct 2014 6:45pm GMT