13 Dec 2017

feedDjango community aggregator: Community blog posts

Django Quiz 2017

Man feeding pony

Yesterday evening I gave a quiz at the London Django Meetup Group for the second year running. Here it is so you can do it at home (no cheating!). Answers are at the bottom.

Part 1: Trivia

1. What species is Django's unofficial spirit animal?

  1. Pegasus
  2. Unicorn
  3. Pony
  4. Seal
  5. Dolphin
  6. Elephant

2. Djangocon EU this year was in…

  1. Bologna
  2. Genoa
  3. Venice
  4. Florence

3. What does LTS stand for?

  1. Long Tail Support
  2. Long Term Support
  3. Life Time Support
  4. Life Term Support

4. What does WSGI stand for?

  1. Web Socket Gateway Interface
  2. Web Server Gateway Interface
  3. Web Server Gated Interface
  4. WebS GuardIan

5. What does ACID stand for?

  1. Atomicity Consistency Integrity Durability
  2. Atomicity Concurrency Isolation Durability
  3. Atomicity Consistency Isolation Durability
  4. All Carefully Inserted Data

6. When was the first commit on Django?

One point for year, one for month, one for day

7. When was the first commit in Python?

One point for year, one for month, one for day

8. What is the name of the current Django fellow?

One point for first name, one for last

Part 2: Coding with Django

1. What's the import for the new Django 2.0 URL syntax?

  1. from django.paths import url
  2. from django.urls import path
  3. from django.urls import url
  4. from django.urls import fantastic_new_url

2. When you run tests…

  1. settings.DEBUG is forced to True
  2. settings.DEBUG is forced to False
  3. They fail if settings.DEBUG is not True
  4. They fail if settings.DEBUG is not False

3. The email addresses in settings.ADMINS

  1. will be notified of 404 errors and exceptions
  2. will be notified of exceptions
  3. will be the only ones allowed to use the Admin
  4. will be notified of bots crawling the sites

4. Django 1.11 was the first version with a non-optional dependency - what was it on?

Give the PyPI package name.

5. What's the minimum supported version of Python for Django 2.0?

  1. 2.7
  2. 2.8
  3. 2.999999999999999
  4. 3.3
  5. 3.4
  6. 3.5
  7. 3.6

ANSWERS

But first, some vertical space.

A

N

S

W

E

R

S

B

E

L

O

W

Part 1: Trivia

1. What species is Django's unofficial spirit animal?

a) Pegasus

Although called the Django Pony, it's a winged horse, aka pegasus.

2. Djangocon EU this year was in…

d) Florence

See 2017.djangocon.eu. The youtube channel has some talks worth watching.

3. What does LTS stand for?

b) Long Term Support

Though some would like it to last a life time :)

4. What does WSGI stand for?

b) Web Server Gateway Interface

5. What does ACID stand for?

c) Atomicity Consistency Isolation Durability

6. When was the first commit on Django?

2005-07-13

Jacob Kaplan-Moss in SVN (now imported into Git): "Created basic repository structure". See the commit on GitHub.

7. When was the first commit in Python?

1990-08-09

"Initial revision" by Guido van Rossum - see GitHub.

8. What is the name of the current Django fellow?

Tim Graham

Part 2: Coding with Django

1. What's the import for the new Django 2.0 URL syntax?

from django.urls import path

As per the release notes which are worth reading if you haven't yet :)

2. When you run tests…

b) settings.DEBUG is forced to False.

As per the docs

3. The email addresses in settings.ADMINS

b) will be notified of exceptions

As per the docs.

4. Django 1.11 was the first version with a non-optional dependency - what was it on?

pytz

It was optional and highly recommended for many versions before.

5. What's the minimum supported version of Python for Django 2.0?

3.4

Again see the 2.0 release notes!

Fin

Hope you enjoyed doing/reading/skimming this quiz!

13 Dec 2017 6:00am GMT

09 Dec 2017

feedDjango community aggregator: Community blog posts

From MySQL to PostgreSQL

In this article I will guide you through the steps I had to take to migrate Django projects from MySQL to PostgreSQL.

MySQL database has proven to be a good start for small and medium scale projects. It is known and used widely and has good documentation. Also there are great clients for easy management, like phpMyAdmin (web), HeidiSQL (windows), or Sequel Pro (macOS). However, in my professional life there were unfortunate moments, when databases from different projects crashed because of large queries or file system errors. Also I had database integrity errors which appeared in the MySQL databases throughout years because of different bugs at the application level.

When one thinks about scaling a project, they have to choose something more suitable. It should be something that is fast, reliable, and well supports ANSI standards of relational databases. Something that most top Django developers use. And the database of choice for most professionals happens to be PostgreSQL. PostgreSQL enables using several vendor-specific features that were not possible with MySQL, e.g. multidimensional arrays, JSON fields, key-value pair fields, special case-insensitive text fields, range fields, special indexes, full-text search, etc. For a newcomer, the best client that I know - pgAdmin (macOS, linux, windows) - might seem too complex at first, compared with MySQL clients, but as you have Django administration and handy ORM, you probably won't need to inspect the database in raw format too often.

So what does it take to migrate from MySQL to PostgreSQL? We will do that in a few steps and we will be using pgloader to help us with data migration. I learned about this tool from Louise Grandjonc, who was giving a presentation about PostgreSQL query optimization at DjangoCon Europe 2017.

One prerequisite for the migration are passing tests. You need to have functional tests to check if all pages are functioning correctly and unit tests to check at least the most critical or complex classes, methods, and functions.

1. Prepare your MySQL database

Make sure that your production MySQL database migration state is up to date:

(env)$ python manage.py migrate --settings=settings.production

Then create a local copy of your production MySQL database. We are going to use it for the migration.

2. Install pgloader

As I mentioned before, for the database migration we will use a tool called pgloader (version 3.4.1 or higher). This tool was programmed by Dimitri Fontaine and is available as an open source project on GitHub. You can compile the required version from the source. Or if you are using macOS, you can install it with Homebrew:

$ brew update
$ brew install pgloader

Note that PostgreSQL will also be installed as a dependency.

3. Create a new PostgreSQL user and database

Unlike with MySQL, creating new database users and databases with PostgreSQL usually happen in the shell rather than in the database client.

Let's create a user and database with the same name myproject.

$ createuser --createdb --password myproject
$ createdb --username=myproject myproject

The --createdb parameter will enable privilege to create databases. The --password parameter will offer to enter a password. The --username parameter will set the owner of the created database.

4. Create the schema

Link the project to this new PostgreSQL database in the settings, for example:

DATABASES = {
'postgresql': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': get_secret("DATABASE_NAME"),
'USER': get_secret("DATABASE_USER"),
'PASSWORD': get_secret("DATABASE_PASSWORD"),
},
}
DATABASES['default'] = DATABASES['postgresql']

Here the custom get_secret() function returns sensitive information from environment variables or a text file that is not tracked under version control. Its implementation is up to you.

Run the migrations to create tables and foreign key constraints in the new PostgreSQL database:

(env)$ python manage.py migrate --settings=settings.local

5. Create the data migration script

The pgloader uses configuration files with the settings defining how to deal with migrations. Let's create the configuration file myproject.load with the following content:

LOAD DATABASE
FROM mysql://mysql_username:mysql_password@localhost/mysql_dbname
INTO postgresql:///myproject
WITH truncate, data only, disable triggers, preserve index names, include no drop, reset sequences
ALTER SCHEMA 'mysql_dbname' RENAME TO 'public'
;

6. Run data migration

Now it's time to copy the data:

$ pgloader myproject.load

Typically you will get a bunch of warnings about type conversions. These can probably be ignored, because the script will take its best guess how to convert data when importing. If in addition you get errors about duplicated data or tables with foreign keys to missing entries, you will need to fix the issues at MySQL database and then repeat the process. In that case, clean up the MySQL database, update your local copy, recreate PostgreSQL database with dropdb and createdb commands, run Django migrations to create the database schema, and copy the data again.

7. Adapt the code

When the database is successfully migrated, we should run Django project tests and fix all PostgreSQL-specific problems in the project's code. The code running Django ORM will run smoothly, but very likely there will be issues with raw SQL, QuerySet's extra() method, and type conversions.

Typically, these are the differences that you might have to keep in mind:

8. Repeat the process for production

When you are sure that the migration process is fluent and all Django tests pass, you can take your production website down, repeat the migration process locally with the latest production data, copy the migrated local database to production server, update the production code, install new dependencies, and take the website back online.

To create a database dump you can use command:

$ pg_dump --format=c --compress=9 --file=myproject.backup myproject

To restore or create the database from dump use commands:

$ dropdb --username=pgsql myproject
$ createdb --username=myproject myproject
$ pg_restore --dbname=myproject --role=myproject --schema=public myproject.backup

I might probably miss some points and there are some ways to automate the upgrade process for production, but you got the idea.

Conclusion

PostgreSQL is more restrictive than MySQL, but it provides greater performance, more stability, and better compliance with standards. In addition, in PostgreSQL there is a bunch of features that were not available in MySQL. If you are lucky, you can switch your project from MySQL to PostgreSQL in one day.

09 Dec 2017 3:51am GMT

08 Dec 2017

feedDjango community aggregator: Community blog posts

Really simple Django view function timer decorator

I use this sometimes to get insight into how long some view functions take. Perhaps you find it useful too:

def view_function_timer(prefix='', writeto=print):

    def decorator(func):
        @functools.wraps(func)
        def inner(*args, **kwargs):
            try:
                t0 = time.time()
                return func(*args, **kwargs)
            finally:
                t1 = time.time()
                writeto(
                    'View Function',
                    '({})'.format(prefix) if prefix else '',
                    func.__name__,
                    args[1:],
                    'Took',
                    '{:.2f}ms'.format(1000 * (t1 - t0)),
                    args[0].build_absolute_uri(),
                )
        return inner

    return decorator

And to use it:

from wherever import view_function_timer


@view_function_timer()
def homepage(request, thing):
    ...
    return render(request, template, context)

And then it prints something like this:

View Function  homepage ('valueofthing',) Took 23.22ms http://localhost:8000/home/valueofthing

It's useful when you don't want a full-blown solution to measure all view functions with a middleware or something.
It can be useful also to see how a cache decorator might work:

from django.views.decorators.cache import cache_page
from wherever import view_function_timer


@view_function_timer('possibly cached')
@cache_page(60 * 60 * 2)  # two hours cache
@view_function_timer('not cached')
def homepage(request, thing):
    ...
    return render(request, template, context)

That way you can trace that, with tail -f or something, to see how/if the cacheing decorator works.

There are many better solutions that are more robust but might be a bigger investment. For example, I would recommend markus which, if you don't have a statsd server you can configure to logger.info call the timings.

08 Dec 2017 9:07pm GMT