01 Jul 2022
Django community aggregator: Community blog posts
Django News - 2022 Campaign Results - Jul 1st 2022
News
PyCharm & DSF Campaign 2022 Results
The sixth annual JetBrains PyCharm promotion in June netted the Django Software Foundation $25,000 this year.
🚨 Django 4.0.6 and 3.2.14 high severity security releases on July 4th
Mariusz Felisiak gave everyone a heads up to look for a Django 4.0.6 and 3.2.14 high severity security update on July 4th.
Sponsored Ad
Django GDPR Cookie Consent
Highly customizable Django app to make your Django website compatible with GDPR Cookie law.
Articles
Forms in Django 4.0+
Django 4.0 made rendering forms more flexible using the template engine. David Smith shows us how to render forms and highlights what features to expect in Django 4.1.
How to set up Webpack and TailwindCSS in a Django Project
A guide to one-time guide setting up Webpack and Tailwind CSS to work with a Django application.
How to Run a Django Migration "By Hand"
Adam Johnson shows us how to apply database migrations by hand using sqlmigrate and dbshell.
Google Summer of Code 2022: One Month Progress
The first post by one of this year's Google Summer of Code recepients, Deepak Dinesh, who is focusing on improving benchmarking within Django.
Videos
Python Web Conf 2022 Talks + Tutorials
All 90 videos from the 2022 Python Web Conf are now available on YouTube.
Securing Django Applications by Gajendra Deshpande
A talk on performing penetration testing on Django web applications as well as strategies and configuration settings for making the source code and Django applications secure.
Make the Most of Django by Paolo Melchiorre
A talk on how to take full advantage of Django while contributing to its success and that of its community.
Sponsored Link
Fit Django perfectly with Kubernetes
Hurricane is an initiative to fit Django perfectly with Kubernetes. It allows you to make use of many django standard features to leverage the capabilities of Kubernetes to its fullest extent.
Projects
Bunny Fonts
Bunny Fonts are GDPR friendly and an alternative to services like Google Fonts.
Salaah01/django-form-creator
A Django app that allows users to dynamically create forms.
strawberry-graphql/strawberry-graphql-django: Strawberry GraphQL Django extension
Strawberry GraphQL Django extension.
Sponsorship
đź“° Sponsor Django News
Want to get your product, service, job, or company in front of over 2,690 Django professionals each week?
We have some summer sponsorship inventory left and would love to feature you. Sponsorship availability, prices, and details are available here.
This RSS feed is published on https://django-news.com/. You can also subscribe via email.
01 Jul 2022 3:00pm GMT
29 Jun 2022
Django community aggregator: Community blog posts
How to Run a Django Migration “By Hand”
Normally your Django project's deploy process runs the migrate command, and that takes care of updating your database as necessary. Especially on smaller databases, Django's migration system can "just do it" for you.
But sometimes it can be necessary to run migrations "by hand" in your database's SQL console. I have found this to be the case with larger, busy databases, and when using tools like pt-online-schema-change to apply schema changes. In this post we'll cover the process for running a migration by hand, and adapting it to reversing migrations.
Run a Migration "By Hand"
Hold onto your butts…
1. Find the SQL to run
We write Django migrations in Python, but they ultimately end up running a series of SQL statements. To run a migration by hand, you need those SQL statements, so you can run them yourself.
You can display the SQL for a migration with Django's sqlmigrate command like:
$ ./manage.py sqlmigrate <app> <prefix>
This outputs each migration operation with a commented header describing what it does and then its actual statements. Replace <app> with the label of the app that the migration lives in. Replace <prefix> with a unique prefix of the migration's name - normally a four digit number like 0003.
For example, to show the SQL for the core app's migration with name starting "0003", you would run:
$ ./manage.py sqlmigrate core 0003 BEGIN; -- -- Add field page_count to book -- ALTER TABLE "core_book" ADD COLUMN "page_count" integer NULL; COMMIT;
You'll only see the bookending with BEGIN and COMMIT on databases that support transactional schema changes (SQLite and PostgreSQL, of Django's built-in backends). These may be disabled per-migration, when necessary.
If any operations cannot be run as SQL, they will have the message THIS OPERATION CANNOT BE WRITTEN AS SQL (from Django 4.1, older versions have slightly different wording). This normally means use of the RunPython operation. For such operations, will need to figure out how to run them by hand separate to the process I'm covering here. You could also consider splitting them into their own migration.
A small note: it's best to run sqlmigrate against your production database. For certain operations, Django queries the database to find names of objects, patricularly index names when migrating older index definitions. Depending on how your various environments' databases were created and migrated over time, these names can be different. Thus, the SQL that Django generates on your staging server may be different to that on production. But on the other hand, using your production settings with an unexecuted migration isn't always easy, so you may just want to beware of this issue and adjust SQL when necessary.
2. Execute the SQL statement-by-statement
Open up your database's SQL shell on the target environment with Django's dbshell command:
$ ./manage.py dbshell
Here you can run the migration's SQL statements from sqlmigrate, one by one. Skip the comments from sqlmigrate (the lines starting --), and make sure you copy whole SQL statements that end with ;.
For example, running the above on PostgreSQL:
$ ./manage.py dbshell psql (14.4, server 13.5 (Debian 13.5-1.pgdg110+1)) Type "help" for help. example=# BEGIN; BEGIN example=*# ALTER TABLE "core_book" ADD COLUMN "page_count" integer NULL; ALTER TABLE example=*# COMMIT; COMMIT example=#
Whilst running the migration, you should keep an eye on your database's key metrics with whatever monitoring tools you use. You may also want to use a second dbshell to run some administrative commands, for example in my recent PostgreSQL post I covered finding and stopping queries that block an ALTER TABLE.
One note for databases supporting transactional schema changes (SQLite, PostgreSQL). If the migration has a single schema-changing statement like ALTER TABLE, you can drop BEGIN and COMMIT. This means less SQL to run, and you'll hold schema locks for slightly less time, reducing the risk of the migration affecting your busy production database.
3. Record migration as executed
After you've run your migration's statements by hand, you need to record the migration as executed. If you don't, the migrate command will try to again execute that migration, which could end catastrophically (but will normally just result in an error like "column already exists").
Django's migration systems keeps a record of executed migrations in a table called django_migrations. You can record a migration as executed with this query template:
INSERT INTO django_migrations (app, name, applied) VALUES (<app>, <name>, NOW());
Replace <app> with the label of the app that the migration lives in. Replace <name> with the full name of the migration, which is the migration's filename without the .py extension. (Be careful not to use just a prefix like 0003!)
For example, to mark our example migration as complete:
example=# INSERT INTO django_migrations (app, name, applied) VALUES ('core', '0003_book_page_count', NOW());
INSERT 0 1
You can check your entry looks right by visually comparing it with others:
SELECT * FROM django_migrations ORDER BY applied DESC;
…and just like that, you're done applying the migration!
Reverse a Migration "By Hand"
You can use the above process to reverse a migration "by hand" with a couple of changes.
In step one, use the --backwards flag to sqlmigrate to generate the SQL statements to undo the migration. Use this template:
$ ./manage.py sqlmigrate --backwards <app> <prefix>
And in step three, you'll want to delete from, rather than insert into, the django_migrations table. Use this template:
DELETE FROM django_migrations WHERE app = <app> AND name = <name>;
Hopefully you don't need to roll back migrations often!
29 Jun 2022 4:00am GMT
24 Jun 2022
Django community aggregator: Community blog posts
Django News - Django 4.1 beta 1 released - Jun 24th 2022
News
Django 4.1 beta 1 released
Django 4.1 beta 1 is now available. It represents the second stage in the 4.1 release cycle and is an opportunity for you to try out the changes coming in Django 4.1.
2022 Python Software Foundation Board Elections
Just a reminder that if you are a registered member of the Python Software Foundation, you may vote in this year's election until June 30th, 2022 AoE.
Get paid to contribute to urllib3
Announcing urllib3's bounty program.
Sponsored Ad
Affordable Heroku-like experience for Django.
Spin up a production-ready Django application, Postgres DB, and Redis in 5 minutes on AWS, DO, or any other provider.
Articles
Customizable Django Admin themes
Arpit shows us how to quickly use django-admin-interface to customize the Django Admin's color scheme and how to load custom themes.
Six things I do every time I start a Django project
Everyone approaches starting a project differently, and this was a fun read to see how Brenton Cleeland does it. This topic might make a fun Django Forum thread too.
How to Find and Stop Running Queries on PostgreSQL
A look at stopping PostgreSQL queries via SQL, techniques for finding problematic queries, and the occasionally useful ability to cancel via operating system tools.
Don't Mock What You Don't Own in 5 Minutes
One of the most common issues programmers have, when they try to test real-world software is how to deal with third-party dependencies.
Why new Macs break your Docker build, and how to fix it
On new Macs, many Python-based Dockerfiles fail in entirely unexpected ways. A look at why this happens and potential fixes in the future.
Podcasts
Django Chat: How to Learn Django (Ep2 Replay)
Carlton and Will discuss how to learn Django whether you are a total beginner or experienced web developer.
Sponsored Link
Fit Django perfectly with Kubernetes
Hurricane is an initiative to fit Django perfectly with Kubernetes. It allows you to make use of many django standard features to leverage the capabilities of Kubernetes to its fullest extent.
Projects
samwillis/nodejs-pypi
If you ever wondered why you couldn't pip install nodejs-bin to install Node.js in your projects, well, now you can.
dozymoe/frozen-django
frozen-django creates static websites from your Django views into HTML files.
This RSS feed is published on https://django-news.com/. You can also subscribe via email.
24 Jun 2022 3:00pm GMT
10 Nov 2011
Planet Python
Terry Jones: Emacs buffer mode histogram
Tonight I noticed that I had over 200 buffers open in emacs. I've been programming a lot in Python recently, so many of them are in Python mode. I wondered how many Python files I had open, and I counted them by hand. About 90. I then wondered how many were in Javascript mode, in RST mode, etc. I wondered what a histogram would look like, for me and for others, at times when I'm programming versus working on documentation, etc.
Because it's emacs, it wasn't hard to write a function to display a buffer mode histogram. Here's mine:
235 buffers open, in 23 distinct modes 91 python +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 47 fundamental +++++++++++++++++++++++++++++++++++++++++++++++ 24 js2 ++++++++++++++++++++++++ 21 dired +++++++++++++++++++++ 16 html ++++++++++++++++ 7 text +++++++ 4 help ++++ 4 emacs-lisp ++++ 3 sh +++ 3 makefile-gmake +++ 2 compilation ++ 2 css ++ 1 Buffer-menu + 1 mail + 1 grep + 1 completion-list + 1 vm + 1 org + 1 comint + 1 apropos + 1 Info + 1 vm-summary + 1 vm-presentation +
Tempting as it is, I'm not going to go on about the heady delights of having a fully programmable editor. You either already know, or you can just drool in slack-jawed wonder.
Unfortunately I'm a terrible emacs lisp programmer. I can barely remember a thing each time I use it. But the interpreter is of course just emacs itself and the elisp documentation is in emacs, so it's a really fun environment to develop in. And because emacs lisp has a ton of support for doing things to itself, code that acts on emacs and your own editing session or buffers is often very succinct. See for example the save-excursion and with-output-to-temp-buffer functions below.
"Display a histogram of emacs buffer modes."
(interactive)
(let* ((totals '())
(buffers (buffer-list()))
(total-buffers (length buffers))
(ht (make-hash-table :test 'equal)))
(save-excursion
(dolist (buffer buffers)
(set-buffer buffer)
(let
((mode-name (symbol-name major-mode)))
(puthash mode-name (1+ (gethash mode-name ht 0)) ht))))
(maphash (lambda (key value)
(setq totals (cons (list key value) totals)))
ht)
(setq totals (sort totals (lambda (x y) (> (cadr x) (cadr y)))))
(with-output-to-temp-buffer "Buffer mode histogram"
(princ (format "%d buffers open, in %d distinct modes\n\n"
total-buffers (length totals)))
(dolist (item totals)
(let
((key (car item))
(count (cadr item)))
(if (equal (substring key -5) "-mode")
(setq key (substring key 0 -5)))
(princ (format "%2d %20s %s\n" count key
(make-string count ?+))))))))
Various things about the formatting could be improved. E.g., not use fixed-width fields for the count and the mode names, and make the + signs indicate more than one buffer mode when there are many.
10 Nov 2011 2:42pm GMT
Terry Jones: Emacs buffer mode histogram
Tonight I noticed that I had over 200 buffers open in emacs. I've been programming a lot in Python recently, so many of them are in Python mode. I wondered how many Python files I had open, and I counted them by hand. About 90. I then wondered how many were in Javascript mode, in RST mode, etc. I wondered what a histogram would look like, for me and for others, at times when I'm programming versus working on documentation, etc.
Because it's emacs, it wasn't hard to write a function to display a buffer mode histogram. Here's mine:
235 buffers open, in 23 distinct modes 91 python +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 47 fundamental +++++++++++++++++++++++++++++++++++++++++++++++ 24 js2 ++++++++++++++++++++++++ 21 dired +++++++++++++++++++++ 16 html ++++++++++++++++ 7 text +++++++ 4 help ++++ 4 emacs-lisp ++++ 3 sh +++ 3 makefile-gmake +++ 2 compilation ++ 2 css ++ 1 Buffer-menu + 1 mail + 1 grep + 1 completion-list + 1 vm + 1 org + 1 comint + 1 apropos + 1 Info + 1 vm-summary + 1 vm-presentation +
Tempting as it is, I'm not going to go on about the heady delights of having a fully programmable editor. You either already know, or you can just drool in slack-jawed wonder.
Unfortunately I'm a terrible emacs lisp programmer. I can barely remember a thing each time I use it. But the interpreter is of course just emacs itself and the elisp documentation is in emacs, so it's a really fun environment to develop in. And because emacs lisp has a ton of support for doing things to itself, code that acts on emacs and your own editing session or buffers is often very succinct. See for example the save-excursion and with-output-to-temp-buffer functions below.
"Display a histogram of emacs buffer modes."
(interactive)
(let* ((totals '())
(buffers (buffer-list()))
(total-buffers (length buffers))
(ht (make-hash-table :test 'equal)))
(save-excursion
(dolist (buffer buffers)
(set-buffer buffer)
(let
((mode-name (symbol-name major-mode)))
(puthash mode-name (1+ (gethash mode-name ht 0)) ht))))
(maphash (lambda (key value)
(setq totals (cons (list key value) totals)))
ht)
(setq totals (sort totals (lambda (x y) (> (cadr x) (cadr y)))))
(with-output-to-temp-buffer "Buffer mode histogram"
(princ (format "%d buffers open, in %d distinct modes\n\n"
total-buffers (length totals)))
(dolist (item totals)
(let
((key (car item))
(count (cadr item)))
(if (equal (substring key -5) "-mode")
(setq key (substring key 0 -5)))
(princ (format "%2d %20s %s\n" count key
(make-string count ?+))))))))
Various things about the formatting could be improved. E.g., not use fixed-width fields for the count and the mode names, and make the + signs indicate more than one buffer mode when there are many.
10 Nov 2011 2:42pm GMT
Mike Driscoll: wxPython: ANN: Namespace Diff Tool
Last night, Andrea Gavana released his new Namespace Diff Tool (NDT) to the world. I got his permission to reprint his announcement here for all those people who don't follow the wxPython mailing list. I think it sounds like a really cool tool. You should check it out and see what you think. Here is the announcement:
Description
===========
The `Namespace Diff Tool` (NDT) is a graphical user interface that can
be used to discover differences between different versions of a library,
or even between different iterations/sub-versions of the same library.
The tool can be used to identify what is missing and still needs to be
implemented, or what is new in a new release, which items do not have
docstrings and so on.
Full description of the original idea by Robin Dunn:
http://svn.wxwidgets.org/viewvc/wx/wxPython/Phoenix/trunk/TODO.txt?vi…
:warning: As most of the widgets in the GUI are owner drawn or custom,
it is highly probable that the interface itself will look messy on other
platforms (Mac, I am talking to you). Please do try and create a patch to
fix any possible issue in this sense.
:note: Please refer to the TODOs section for a list of things that still
need
to be implemented.
Requirements
============
In order to run NDT, these packages need to be installed:
- Python 2.X (where 5 = X = 7);
- wxPython >= 2.8.10;
- SQLAlchemy >= 0.6.4.
More detailed instructions on how to use it, TODO items, list of
libraries/packages I tested NDT against, screenshots and download links can
be found here:
http://xoomer.virgilio.it/infinity77/main/NDT.html
If you stumble upon a bug (which is highly probable), please do let me
know. But most importantly, please do try and make an effort to create a
patch for the bug.
According to the thread, some bugs were already found and fixed.
10 Nov 2011 1:15pm GMT
Mike Driscoll: wxPython: ANN: Namespace Diff Tool
Last night, Andrea Gavana released his new Namespace Diff Tool (NDT) to the world. I got his permission to reprint his announcement here for all those people who don't follow the wxPython mailing list. I think it sounds like a really cool tool. You should check it out and see what you think. Here is the announcement:
Description
===========
The `Namespace Diff Tool` (NDT) is a graphical user interface that can
be used to discover differences between different versions of a library,
or even between different iterations/sub-versions of the same library.
The tool can be used to identify what is missing and still needs to be
implemented, or what is new in a new release, which items do not have
docstrings and so on.
Full description of the original idea by Robin Dunn:
http://svn.wxwidgets.org/viewvc/wx/wxPython/Phoenix/trunk/TODO.txt?vi…
:warning: As most of the widgets in the GUI are owner drawn or custom,
it is highly probable that the interface itself will look messy on other
platforms (Mac, I am talking to you). Please do try and create a patch to
fix any possible issue in this sense.
:note: Please refer to the TODOs section for a list of things that still
need
to be implemented.
Requirements
============
In order to run NDT, these packages need to be installed:
- Python 2.X (where 5 = X = 7);
- wxPython >= 2.8.10;
- SQLAlchemy >= 0.6.4.
More detailed instructions on how to use it, TODO items, list of
libraries/packages I tested NDT against, screenshots and download links can
be found here:
http://xoomer.virgilio.it/infinity77/main/NDT.html
If you stumble upon a bug (which is highly probable), please do let me
know. But most importantly, please do try and make an effort to create a
patch for the bug.
According to the thread, some bugs were already found and fixed.
10 Nov 2011 1:15pm GMT
Python Software Foundation | GSoC'11 Students
Benedict Stein: King Willams Town Bahnhof
Gestern musste ich morgens zur Station nach KWT um unsere Rerservierten Bustickets fĂĽr die Weihnachtsferien in Capetown abzuholen. Der Bahnhof selber ist seit Dezember aus kostengrĂĽnden ohne Zugverbindung - aber Translux und co - die langdistanzbusse haben dort ihre BĂĽros.
Größere Kartenansicht
10 Nov 2011 10:57am GMT
Planet Python
Andy Todd: Extracting a discrete set of values
Today's I love Python moment is bought to you by set types.
I have a file, XML naturally, the contains a series of transactions. Each transaction has a reference number, but the reference number may be repeated. I want to pull the distinct set of reference numbers from this file. The way I learnt to build up a discrete set of items (many years ago) was to use a dict and set default.
>>> ref_nos = {}
>>> for record in records:
>>> ref_nos.setdefault(record.key, 1)
>>> ref_nos.keys()
But Python has had a sets module since 2.3 and the set standard data type since 2.6 so my knowledge is woefully out of date. The latest way to get the unique values from a sequence looks something like this;
>>> ref_nos = set([record.key for record in records])
I think I should get bonus points for using a list comprehension as well.
10 Nov 2011 5:42am GMT
Andy Todd: Extracting a discrete set of values
Today's I love Python moment is bought to you by set types.
I have a file, XML naturally, the contains a series of transactions. Each transaction has a reference number, but the reference number may be repeated. I want to pull the distinct set of reference numbers from this file. The way I learnt to build up a discrete set of items (many years ago) was to use a dict and set default.
>>> ref_nos = {}
>>> for record in records:
>>> ref_nos.setdefault(record.key, 1)
>>> ref_nos.keys()
But Python has had a sets module since 2.3 and the set standard data type since 2.6 so my knowledge is woefully out of date. The latest way to get the unique values from a sequence looks something like this;
>>> ref_nos = set([record.key for record in records])
I think I should get bonus points for using a list comprehension as well.
10 Nov 2011 5:42am GMT
Planet Twisted
Itamar Shtull-Trauring: The Euro crisis, in musical form
PySoy Blog
Arc Riley: Rugby season nearly over, getting back to work
Wow its been a long time.
We wrapped up Google's Summer of Code 2011 in August. The Python Software Foundation did wonderfully overall, for PySoy 6 of our 7 students passed. A great year overall - thanks to all the mentors and students!
Rugby has been a life changer for me. My first game was in September, after floating in and out of practice for years and training pretty heavily since April. No serious injuries, but no shortage of pain; I've frequently needed to sleep in a reclining chair to keep blood from pooling in my shoulders and nurse bruised ribs, dislocated fingers and toes, shin splints, and pulled muscles everywhere. All so worth it.
These guys are like family to me. I know it sounds sappy, but I've come to trust the men in my pack with my life - in a way we all do every time we bind onto each other a scrum. Its not that big of an adjustment culturally though due to the large number of programmers, lawyers, and IT professionals on the team. When you work behind a desk all day its nice to balance it out with a physically intensive training in the evening and games on Saturday.
The climax of the season was Hellfest October 29th in Dallas, TX. Washington Renegades brought our B-side to compete and returned with the 1st place trophy. My teammate Jimbo has more pics on his blog of the tournament, I was wearing #23 as tighthead prop.
We have two more games this season before we settle in for the Winter and indoor off-season training at the gym. A group of us plan to do a 8-week program run by a professional rugby player this Winter to get ready for the Spring season and the Bingham Cup 2012 in Manchester UK next June.
Today the PySoy project was accepted to Google Code-In. We've got a number of student tasks lined up, with many more being worked on for the first batch set to release in less than two weeks. Interested students should hop on Freenode (#PySoy) and get oriented before the program starts so they're ready to jump right into their first task!
10 Nov 2011 1:15am GMT
09 Nov 2011
Planet Twisted
Thomas Vander Stichele: Mach 1.0.0 “Madera” released
Another November, another Fedora. 16 came out, so it was time to update mach again.
And today I thought, is there any reason mach isn't 1.0 yet ? Am I going to do anything more to this piece of code before I want to call it that ?
And the answer is, no. It's the first Python application I've written, and I'm not particularly proud of the code, but I'm happy I've made good use of it for so long, and that it helped push packaging approaches forward and sparked ideas for the Fedora build system.
Since I didn't like the original code for mach2 (there was a version 1 which was Makefile-based), I started a rewrite with unit tests, better code layout, decent classes for abstracting distro-specific stuff, and so on.
The experience of how mock was created based off mach2 was a slightly sour one however, so I wasn't really motivated to finish the mach3 rewrite. Sometimes that's the drawback of open source - sure, forking is specifically allowed, so don't whine about it when it happens. But when it's done gratuitously, with no serious attempt at collaborating, it doesn't feel like it's in the spirit of open source.
Anyway, that was a long time ago. mach2 as it is today, is done. It really only needs updating for newer versions. As long as it works for me, it's unlikely I will continue mach3, but who knows?
Enjoy the release!
09 Nov 2011 10:56pm GMT
Planet Plone
Andreas Jung: Produce & Publish Plone Client Connector released as open-source
09 Nov 2011 9:30pm GMT
Python Software Foundation | GSoC'11 Students
Benedict Stein
Niemand ist besorgt um so was - mit dem Auto fährt man einfach durch, und in der City - nahe Gnobie- "ne das ist erst gefährlich wenn die Feuerwehr da ist" - 30min später auf dem Rückweg war die Feuerwehr da.
09 Nov 2011 8:25pm GMT
Planet Plone
ACLARK.NET, LLC: Plone secrets: Episode 4 – Varnish in front
This just in from the production department: use Varnish. (And please forgive the heavily meme-laden approach to describing these techniques 
.)
Cache ALL the hosts
Our ability to use Varnish in production is no secret by now, or at least it shouldn't be. What is often less clear is exactly how to use it. One way I like[1], is to run Varnish on your public IP port 80 and make Apache listen on your private IP port 80. Then proxy from Varnish to Apache and enjoy easy caching goodness on all your virtual hosts in Apache.
Configuration
This should require less than five minutes of down time to implement. First, configure the appropriate settings. (Well, first install Apache and Varnish if you haven't already: `aptitude install varnish apache2` on Ubuntu Linux[0].)
Varnish
To modify the listen IP address and port, we typically edit a file like /etc/default/varnish (in Ubuntu). However you do it, configure the equivalent of the following on your system:
DAEMON_OPTS="-a 174.143.252.11:80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-s malloc,256m"
This environment variable is then passed to varnishd on the command line. Next, pass traffic to Apache like so (in /etc/varnish/default.vcl on Ubuntu):
backend default {
.host = "127.0.0.1";
.port = "80";
}
Now on to Apache.
Please note that the syntax above is for Varnish 3.x and the syntax has (annoyingly) changed from 2.x to 3.x.
Apache
The Apache part is a bit simpler. You just need to change the listen port (on Ubuntu this is done in /etc/apache2/ports.conf), typically from something like:
Listen *:80
to:
Listen 127.0.0.1:80
Restart ALL the services
Now restart both services. If all goes well you shouldn't notice any difference, except better performance, and when you make a website change and need to clear the cache[2]. For this, I rely on telnetting to the varnish port and issuing the `ban.url` command (formerly `url.purge` in 2.x):
$ telnet localhost 6082 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 200 205 ----------------------------- Varnish Cache CLI 1.0 ----------------------------- Linux,2.6.35.4-rscloud,x86_64,-smalloc,-smalloc,-hcritbit Type 'help' for command list. Type 'quit' to close CLI session. ban.url / 200 0
Cache ALL the disks
This site has Varnish and Apache configured as described in this article. It also has disk caching in Apache enabled, thanks to Elizabeth Leddy's article:
As a result, it's PEPPY AS THE DICKENS™ on a 512MB "slice" (Cloud server) from Rackspace Cloud. And now you know yet another "Plone secret". Now go make your Plone sites faster, and let me know how it goes in the comments section below.
Notes
[0] Using the latest distribution, "oneric".
[1] I first saw this technique at NASA when NASA Science was powered by Plone; I found it odd at the time but years later it makes perfect sense.
[2] Ideally you'd configure this in p.a.caching, but I've not been able to stomach this yet.
09 Nov 2011 5:50pm GMT
Planet Zope.org
Updated MiniPlanet, now with meta-feed
My MiniPlanet Zope product has been working steady and stable for some years, when suddenly a user request came along. Would it be possible to get a feed of all the items in a miniplanet? With this update it became possible. MiniPlanet is an old-styl...
09 Nov 2011 9:41am GMT
08 Nov 2011
Planet Plone
Max M: How to export all redirects from portal_redirection in an older Plone site
Just add the method below to the RedirectionTool and call it from the browser as:
http://localhost:8080/site/portal_redirection/getAllRedirects
Assuming that the site is running at loaclhost:8080 that is :-S
That will show a list of redirects that can be imported into plone 4.x
security.declareProtected(View, 'getAllRedirects')
def getAllRedirects(self):
"get'm'all"
result = []
reference_tool = getToolByName(self, 'reference_catalog')
for k,uuid in self._redirectionmap.items():
obj = reference_tool.lookupObject(uuid)
if obj is None:
print 'could not find redirect from: %s to %s' % (k, uuid)
else:
path = '/'.join(('',)+obj.getPhysicalPath()[2:])
result.append( '%s,%s' % (k,path) )
return '\n'.join(result)
08 Nov 2011 2:58pm GMT
Python Software Foundation | GSoC'11 Students
Benedict Stein: Brai Party
Brai = Grillabend o.ä.
Die möchte gern Techniker beim Flicken ihrer SpeakOn / Klinke Stecker Verzweigungen...
Die Damen "Mamas" der Siedlung bei der offiziellen Eröffnungsrede
Auch wenn weniger Leute da waren als erwartet, Laute Musik und viele Leute ...
Und natĂĽrlich ein Feuer mit echtem Holz zum Grillen.
08 Nov 2011 2:30pm GMT
07 Nov 2011
Planet Zope.org
Welcome to Betabug Sirius
It has been quite some time that I announced_ that I'd be working as a freelancer. Lots of stuff had to be done in that time, but finally things are ready. I've founded my own little company and set up a small website: Welcome to Betabug Sirius!
07 Nov 2011 9:26am GMT
05 Nov 2011
Planet Twisted
Glyph Lefkowitz: Blocking vs. Running
I've heard tell of some confusion lately around what the term "non-blocking" means. This isn't the first time I've tried to explain it, and it certainly won't be the last, but blogging is easier than the job Sisyphus got, so I can't complain.
A thread is blocking when it is performing an input or output operation that may take an unknown amount of time. Crucially, a blocking thread is doing no useful work. It is stuck, consuming resources - in particular, its thread stack, and its process table entry. It is sucking up resources and getting nothing done. These are resources that one can most definitely run out of, and are in fact artificially limited on most operating systems, because if one has too many of them, the system bogs down and becomes unusable.
A thread may also be "stuck" doing some computationally intensive work; performing a complex computation, and sucking up CPU cycles. There is a very important distinction here, though. If that thread is burning up CPU, it is getting work done. It is computing. This is why we have computers: to compute things.
It is of course possible for a program to have a bug where a program goes into an infinite loop, or otherwise performs work on the CPU without actually getting anything useful to the user done, but if that's happening then the program is just buggy, or inefficient. But such a program is not blocking: it might be "thrashing" or "stuck" or "broken", but "blocking" means something more specific: that the program is sitting around, doing nothing, while it is waiting for some other thing to get work done, and not doing any of its own.
A program written in an event-driven style may be busy as long as it needs to be, but that does not mean it is blocking. Hence, event-driven and non-blocking are synonyms.
Furthermore, non-blocking doesn't necessarily mean single-process. Twisted is non-blocking, for example, but it has a sophisticated facility for starting, controlling and stopping other processes. Information about changes to those processes is represented as plain old events, making it reasonably easy to fold the results of computation in another process back into the main one.
If you need to perform a lengthy computation in an event-driven program, that does not mean you need to stop the world in order to do it. It doesn't mean that you need to give up on the relatively simple execution model of an event loop for a mess of threads, either. Just ask another process to do the work, and handle the result of that work as just another event.
05 Nov 2011 1:16am GMT
03 Nov 2011
Planet Zope.org
Assertion helper for zope.testbrowser and unittest
zope.testbrowser is a valuable tool for integration tests. Historically, the Zope community used to write quite a lot of doctests, but we at gocept have found them to be rather clumsy and too often yielding neither good tests nor good documentation. That's why we don't use doctest much anymore, and prefer plain unittest.TestCases instead. However, doctest has one very nice feature, ellipsis matching, that is really helpful for checking HTML output, since you can only make assertions about the parts that interest you. For example, given this kind of page:
>>> print browser.contents
<html>
<head>
<title>Simple Page</title>
</head>
<body>
<h1>Simple Page</h1>
</body>
</html>
If all you're interested in is that the <h1> is rendered properly, you can simply say:
>>> print browser.contents <...<h1>Simple Page</h1>...
We've now ported this functionality to unittest, as assertEllipsis, in gocept.testing. Some examples:
self.assertEllipsis('...bar...', 'foo bar qux')
# -> nothing happens
self.assertEllipsis('foo', 'bar')
# -> AssertionError: Differences (ndiff with -expected +actual):
- foo
+ bar
self.assertNotEllipsis('foo', 'foo')
# -> AssertionError: "Value unexpectedly matches expression 'foo'."
To use it, inherit from gocept.testing.assertion.Ellipsis in addition to unittest.TestCase.
03 Nov 2011 7:19am GMT
23 Oct 2011
Planet TurboGears
Cliff Wells: FreeSWITCH on Scientific Linux 6.1
SL 6.1 (and I assume RHEL and CentOS 6.1 as well) has introduced an issue for building and running FreeSWITCH. Apparently a lot of stuff now relies on dynamically linking to libnss3. libnss3, in turn, depends on libnspr4.so, which depends on libplds4.so. Seemingly, this should not be an issue (stuff depends on chained shared objects all over the place), but somehow it is.
What happens is first you can't compile FreeSWITCH. You get complaints about unresolved symbols in /usr/lib64/libnss3.so. The solution is to run the following commands:
yum install nspr-devel env LIBS="-lnss3 -lnspr4 -lplds4" ./configure make && make install
This will get you a compiled version of FreeSWITCH. However, when you actually run it, you'll find that several modules won't load at runtime (including the ODBC driver, should you happen to be using it). The solution for this is similar. Assuming you are using an init script to launch FreeSWITCH, you can add the following line to the top of /etc/init.d/freeswitch:
export LD_PRELOAD="/usr/lib64/libnss3.so /usr/lib64/libnspr4.so /usr/lib64/libplds4.so"
Voila. Everything works. Hopefully the FreeSWITCH devs get on RHEL6 support soon, but meanwhile this should get you by.
23 Oct 2011 3:27pm GMT
18 Oct 2011
Planet TurboGears
Michael Pedersen: Request For Ideas for Hiring Pond
So, a favor to ask of people: I'm working on a web application to help people manage their resumes. As I've gotten further in, I've realized I don't have an actual todo list for it. So, since I'm making this to be used by others, I'll ask everybody here:
What would you want to see? Currently, I've added enough code to allow the program to output something very close to my online resume ( http://www.icelus.org/ ). Next up, I have the following features on my todo list already:
- add docutils to output strings where appropriate (everywhere?)
- add login
- add ability to edit own resume
- add ability to compile new custom resumes
- add ability to output PDF
- add ability to output ODT
- add ability to output DOC
- add generic logo
- add generic photo / head (shadow) shot
- add ability to login using Google, Yahoo, Facebook, etc
- add ability to send resumes directly from Hiring Pond
- add ability to track where resumes have gone
- add ability to track job hunt progress
- add ability to display current employment status (employed, unemployed, contracting)
- add ability to display current employment seeking status (not at all, active, passive)
- add logo to qrcode
- change colors of qrcode
- change color scheme for all of hiring pond to sunset and water
- add fisherman silhouette as hiring pond logo
What else would you all want to see in order to make you want to use this?
18 Oct 2011 8:44pm GMT
04 Oct 2011
Planet TurboGears
Alessandro Molina: TurboGears2 Performance Improvements
As recently some effort has been involved in improving the performances of TurboGears2, I was curious to see how much things improved. As usually, the test isn't really reliable in any way and was just for fun.
All the graphs report the request/sec the application has been able to perform on my computer with only 1 concurrent client. So higher is better.
Here is the comparison between TG2.0 and TG2dev (will be 2.1.4)
I also compared various setups with different template engines on TG2dev
The comparison happened on an application similar to the quickstarted one.
Actually as there is no database involved in this application the template engine impacts a lot and so was a good benchmark for the template engines themselves.
04 Oct 2011 3:35pm GMT
30 Aug 2011
PySoy Blog
Mayank Singh: Post Gsoc musings
It's been a long time since I last blogged. And yes my gsoc got over officially and successfully on 26th. So, I thought of this to be a nice occasion to blog.
I got a really nice oppportunity to work on a game engine still in its nascent stages and observe all the hurdles and problems related to creating such an engine when I got into the gsoc program under python software foundation this april. And yes I experienced all those showstoppers. But due to enormous help from my mentor Arc and google, I was able to surpass them.
To begin with the tale, I was supposed to create a controller for wiimotes to be used with PySoy. But this first involved getting the wiimote communicate with the PC. A wiimote communicates via bluetooth. I googled a lot about this communication and found out that it was almost fully reverse engineered. The best place to visit regarding all this revengs would be wiibrew.org. However, I didn't want to delve into sockets and all considering there were ready-made libraries for wiimote communication. I did a study of all those libraries. I found two libraries that seemed to catch my eye and thus suit for this purpose. One of them was cwiid and the other wiiuse. Cwiid seemed to be the most popular considering its packages in all major linux distros. So, I went forward with cwiid (I would regret this later). Having decided upon the library, next step was to figure out how to use it in the game engine. Since, the backend of our engine(libsoy) was being written in vala and genie (abstractions over C and gobject based), I had to write the vala apis of each non-gobject C library I used. For me, there were two: cwiid and bluez. I spent the first fifteen days understanding the concept of vala apis, and writing vala apis(vapis) for cwiid and bluez. At the end of this period, I was able to write code in Genie(pythonic form of vala) and connect to a wiimote. In the next period that lasted upto the mid-term evaluations, I created various examples to leverage the functions of the library with our game engine. One example allowed to control a cube via the accelerometer though it wasn't calibrated properly. Another allowed to control the same cube via the buttons on the wiimote. However, the technique used to control the cube was to query the wiimote every time for state information and use it to move the cube. As is clear, this wasn't a nice method. Near the end of this period, I started to correct this and write callback functions to act as interrupts for events happening on the wiimote. However, Due to a possible race condition I got stuck on a segmentation fault. Arc suggested to create a separate thread for non-window controller events. I embarked on that. However, in a discussion with Arc later, we discovered that the library cwiid was already using three threads inside it and had a lot of redundant code as well. That's when I had to take tough decision to dump cwiid and do something else.
At this point, Arc started taking great interest in my project and found out that another gsoc student David herrmann was working on writing a bluez driver for wiimote. He even got him on Google+. There we checked out his kernel driver and bluez plugin which were all working great (Kudos to him). The good part about it was that this made the wiimote directly accessible by the x11 xinput2 (something that allows multiple pointers and cursors on the same screen). Also in the meantime, Arc made the major decision to dump Gdk for windowing and is currently using vanilla x11 with egl on top (this is currently in experimental). At this very time, I did my first custom kernel compilations patched with the kernel driver after which numerous more followed due to some mistakes here and there. When I finally got wiimote pairing with my laptop natively, I started looking into the usage of his kernel driver. Turned out, he had switched off accelerometer and IR pointer information reporting due to large usage of bandwidth and you could switch it on by writing to sysfs files created for the wiimote on pairing. I tried googling on a library to edit sysfs files but turned out they did not need any special library but normal file handling to write to them. Also, Arc wasn't happy with sysfs file editing in the game engine so I resolved upon writing an x.org driver for wiimote to handle all this and also map buttons and handle accelerometer and IR data. Currently, the driver isn't yet complete and I am taking a bit of rest post-gsoc. You can check it out at here. I plan to complete it after my college tests. Meanwhile, to test the IR pointer, I had made a makeshift circuit on a breadboard using some IR emitters from certain obstacle detectors I had from an earlier project.
All in all, my gsoc experience was a perfect example of a real-world coding environment filled with all its ups and lows.
30 Aug 2011 8:39pm GMT
16 Aug 2011
PySoy Blog
Anthus Williams: Less than a week
With less than a week the firm "pencils down" date, I'm feeling a little disappointed in where my project is. I'm at the point where mesh morphs are operational (albeit buggy), with what I consider is a solid and simple API.
mesh = soy.models.Mesh()
mesh.size = 6
mesh[0] = face //a soy.atoms.Face object
// repeat for mesh[1] through mesh[5]
clone = mesh.clone()
//clone is a Mesh object that can be rendered in its own right, if it is bound to a body
//change the face and vertex data for clone[0] through clone[5]
target = soy.models.Target(mesh)
morph = mesh.morph(clone,0.5) //mesh.morph(variantMesh,delta) spins off a soy.atoms.Morph object
target.morphs.add(morph)
//now you bind target to a soy.bodies.Body, and when its render() method is called, it will apply all its morphs at their given deltas
Rendering has not been done yet for Mesh, and this process will be complicated on the back-end once we perform optimization. Basically we have to maintain the public vertex ordering while shifting vertices around on the backend so that OpenGL can render faces that have the same material consecutively (having to switch between materials needlessly is costly). This is already done for getters and setters for Mesh, but not honored by clone(), soy.atoms.Morph, or by the Target constructor.
Odds are this process won't even be kind of complete by pencils down, but I would expect something more fully functional by PyCon.
16 Aug 2011 4:47pm GMT
19 Nov 2010
Planet CherryPy
Robert Brewer: logging.statistics
Statistics about program operation are an invaluable monitoring and debugging tool. How many requests are being handled per second, how much of various resources are in use, how long we've been up. Unfortunately, the gathering and reporting of these critical values is usually ad-hoc. It would be nice if we had 1) a centralized place for gathering statistical performance data, 2) a system for extrapolating that data into more useful information, and 3) a method of serving that information to both human investigators and monitoring software. I've got a proposal. Let's examine each of those points in more detail.
Data Gathering
Just as Python's logging module provides a common importable for gathering and sending messages, statistics need a similar mechanism, and one that does not require each package which wishes to collect stats to import a third-party module. Therefore, we choose to re-use the logging module by adding a statistics object to it.
That logging.statistics object is a nested dict:
import logging
if not hasattr(logging, 'statistics'): logging.statistics = {}
It is not a custom class, because that would 1) require apps to import a third-party module in order to participate, 2) inhibit innovation in extrapolation approaches and in reporting tools, and 3) be slow. There are, however, some specifications regarding the structure of the dict.
{
+----"SQLAlchemy": {
| "Inserts": 4389745,
| "Inserts per Second":
| lambda s: s["Inserts"] / (time() - s["Start"]),
| C +---"Table Statistics": {
| o | "widgets": {-----------+
N | l | "Rows": 1.3M, | Record
a | l | "Inserts": 400, |
m | e | },---------------------+
e | c | "froobles": {
s | t | "Rows": 7845,
p | i | "Inserts": 0,
a | o | },
c | n +---},
e | "Slow Queries":
| [{"Query": "SELECT * FROM widgets;",
| "Processing Time": 47.840923343,
| },
| ],
+----},
}
The logging.statistics dict has strictly 4 levels. The topmost level is nothing more than a set of names to introduce modularity. If SQLAlchemy wanted to participate, it might populate the item logging.statistics['SQLAlchemy'], whose value would be a second-layer dict we call a "namespace". Namespaces help multiple emitters to avoid collisions over key names, and make reports easier to read, to boot. The maintainers of SQLAlchemy should feel free to use more than one namespace if needed (such as 'SQLAlchemy ORM').
Each namespace, then, is a dict of named statistical values, such as 'Requests/sec' or 'Uptime'. You should choose names which will look good on a report: spaces and capitalization are just fine.
In addition to scalars, values in a namespace MAY be a (third-layer) dict, or a list, called a "collection". For example, the CherryPy StatsTool keeps track of what each worker thread is doing (or has most recently done) in a 'Worker Threads' collection, where each key is a thread ID; each value in the subdict MUST be a fourth dict (whew!) of statistical data about
each thread. We call each subdict in the collection a "record". Similarly, the StatsTool also keeps a list of slow queries, where each record contains data about each slow query, in order.
Values in a namespace or record may also be functions, which brings us to:
Extrapolation
def extrapolate_statistics(scope):
"""Return an extrapolated copy of the given scope."""
c = {}
for k, v in scope.items():
if isinstance(v, dict):
v = extrapolate_statistics(v)
elif isinstance(v, (list, tuple)):
v = [extrapolate_statistics(record) for record in v]
elif callable(v):
v = v(scope)
c[k] = v
return c
The collection of statistical data needs to be fast, as close to unnoticeable as possible to the host program. That requires us to minimize I/O, for example, but in Python it also means we need to minimize function calls. So when you are designing your namespace and record values, try to insert the most basic scalar values you already have on hand.
When it comes time to report on the gathered data, however, we usually have much more freedom in what we can calculate. Therefore, whenever reporting tools fetch the contents of logging.statistics for reporting, they first call extrapolate_statistics (passing the whole statistics dict as the only argument). This makes a deep copy of the statistics dict so that the reporting tool can both iterate over it and even change it without harming the original. But it also expands any functions in the dict by calling them. For example, you might have a 'Current Time' entry in the namespace with the value "lambda scope: time.time()". The "scope" parameter is the current namespace dict (or record, if we're currently expanding one of those instead), allowing you access to existing static entries. If you're truly evil, you can even modify more than one entry at a time.
However, don't try to calculate an entry and then use its value in further extrapolations; the order in which the functions are called is not guaranteed. This can lead to a certain amount of duplicated work (or a redesign of your schema), but that's better than complicating the spec.
After the whole thing has been extrapolated, it's time for:
Reporting
A reporting tool would grab the logging.statistics dict, extrapolate it all, and then transform it to (for example) HTML for easy viewing, or JSON for processing by Nagios etc (and because JSON will be a popular output format, you should seriously consider using Python's time module for datetimes and arithmetic, not the datetime module). Each namespace might get its own header and attribute table, plus an extra table for each collection. This is NOT part of the statistics specification; other tools can format how they like.
Turning Collection Off
It is recommended each namespace have an "Enabled" item which, if False, stops collection (but not reporting) of statistical data. Applications SHOULD provide controls to pause and resume collection by setting these entries to False or True, if present.
Usage
import logging
# Initialize the repository
if not hasattr(logging, 'statistics'): logging.statistics = {}
# Initialize my namespace
mystats = logging.statistics.setdefault('My Stuff', {})
# Initialize my namespace's scalars and collections
mystats.update({
'Enabled': True,
'Start Time': time.time(),
'Important Events': 0,
'Events/Second': lambda s: (
(s['Important Events'] / (time.time() - s['Start Time']))),
})
...
for event in events:
...
# Collect stats
if mystats.get('Enabled', False):
mystats['Important Events'] += 1
Original post blogged on b2evolution.
19 Nov 2010 7:08am GMT
12 Nov 2010
Planet CherryPy
Kevin Dangoor: Paver is now on GitHub, thanks to Almad
Paver, the project scripting tool for Python, has just moved to GitHub thanks to Almad. Almad has stepped forward and offered to properly bring Paver into the second decade of the 21st century (doesn't have the same ring to it as bringing something into the 21st century, does it? 
Seriously, though, Paver reached the point where it was good enough for me and did what I wanted (and, apparently, a good number of other people wanted as well). Almad has some thoughts and where the project should go next and I'm looking forward to hearing more about them. Sign up for the googlegroup to see where Paver is going next.
12 Nov 2010 3:11am GMT
09 Nov 2010
Planet CherryPy
Kevin Dangoor: Paver: project that works, has users, needs a leader
Paver is a Python project scripting tool that I initially created in 2007 to automate a whole bunch of tasks around projects that I was working on. It knows about setuptools and distutils, it has some ideas on handling documentation with example code. It also has users who occasionally like to send in patches. The latest release has had more than 3700 downloads on PyPI.
Paver hasn't needed a lot of work, because it does what it says on the tin: helps you automate project tasks. Sure, there's always more that one could do. But, there isn't more that's required for it to be a useful tool, day-to-day.
Here's the point of my post: Paver is in danger of being abandoned. At this point, everything significant that I am doing is in JavaScript, not Python. The email and patch traffic is low, but it's still too much for someone that's not even actively using the tool any more.
If you're a Paver user and either:
1. want to take the project in fanciful new directions or,
2. want to keep the project humming along with a new .x release every now and then
09 Nov 2010 7:44pm GMT