fidzu : Django

The Coding for Entrepreneurs C...

21 Aug 2017 9:13pm GMT

Celery is an asynchronous task queue based on distributed message passing. Task queues are used as a strategy to distribute the workload between threads/machines. In this tutorial I will explain how to install and setup Celery + RabbitMQ to execute asynchronous in a Django application.

To work with Celery, we also need to install RabbitMQ because Celery requires an external solution to send and receive messages. Those solutions are called message brokers. Currently, Celery supports RabbitMQ, Redis, and Amazon SQS as message broker solutions.

Table of Contents

• Why Should I Use Celery?
• Installation
  • Installing RabbitMQ on Ubuntu 16.04
  • Installing RabbitMQ on Mac
  • Installing RabbitMQ on Windows and Other OSs
• Celery Basic Setup
• Creating Our First Celery Task
• Starting The Worker Process
• Managing The Worker Process in Production with Supervisord
• Further Reading

Why Should I Use Celery?

Web applications works with request and response cycles. When the user access a certain URL of your application the Web browser send a request to your server. Django receive this request and do something with it. Usually it involves executing queries in the database, processing data. While Django does his thing and process the request, the user have to wait. When Django finalize its job processing the request, it sends back a response to the user who finally will see something.

Ideally this request and response cycle should be fast, otherwise we would leave the user waiting for way too long. And even worse, our Web server can only serve a certain number of users at a time. So, if this process is slow, it can limit the amount of pages your application can serve at a time.

For the most part we can work around this issue using cache, optimizing database queries, and so on. But there are some cases that theres no other option: the heavy work have to be done. A report page, export of big amount of data, video/image processing are a few examples of cases where you may want to use Celery.

We don't use Celery through the whole project, but only for specific tasks that are time-consuming. The idea here is to respond to the user as quick as possible, and pass the time-consuming tasks to the queue so to be executed in the background, and always keep the server ready to respond to new requests.

Installation

The easiest way to install Celery is using pip:

pip install Celery

Now we have to install RabbitMQ.

Installing RabbitMQ on Ubuntu 16.04

To install it on a newer Ubuntu version is very straightforward:

apt-get install -y erlang
apt-get install rabbitmq-server

Then enable and start the RabbitMQ service:

systemctl enable rabbitmq-server
systemctl start rabbitmq-server

Check the status to make sure everything is running smooth:

systemctl status rabbitmq-server

Installing RabbitMQ on Mac

Homebrew is the most straightforward option:

brew install rabbitmq

The RabbitMQ scripts are installed into /usr/local/sbin. You can add it to your .bash_profile or .profile.

vim ~/.bash_profile

Then add it to the bottom of the file:

export PATH=$PATH:/usr/local/sbin

Restart the terminal to make sure the changes are in effect.

Now you can start the RabbitMQ server using the following command:

rabbitmq-server

Installing RabbitMQ on Windows and Other OSs

Unfortunately I don't have access to a Windows computer to try things out, but you can find the installation guide for Windows on RabbitMQ's Website.

For other operating systems, check the Downloading and Installing RabbitMQ on their Website.

Celery Basic Setup

First, consider the following Django project named mysite with an app named core:

mysite/
 |-- mysite/
 |    |-- core/
 |    |    |-- migrations/
 |    |    |-- templates/
 |    |    |-- apps.py
 |    |    |-- models.py
 |    |    +-- views.py
 |    |-- templates/
 |    |-- __init__.py
 |    |-- settings.py
 |    |-- urls.py
 |    +-- wsgi.py
 |-- manage.py
 +-- requirements.txt

Add the CELERY_BROKER_URL configuration to the settings.py file:

settings.py

CELERY_BROKER_URL = 'amqp://localhost'

Alongside with the settings.py and urls.py files, let's create a new file named celery.py.

celery.py

import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mysite.settings')

app = Celery('mysite')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

Now edit the __init__.py file in the project root:

__init__.py

from .celery import app as celery_app

__all__ = ['celery_app']

This will make sure our Celery app is important every time Django starts.

Creating Our First Celery Task

We can create a file named tasks.py inside a Django app and put all our Celery tasks into this file. The Celery app we created in the project root will collect all tasks defined across all Django apps listed in the INSTALLED_APPS configuration.

Just for testing purpose, let's create a Celery task that generates a number of random User accounts.

core/tasks.py

import string

from django.contrib.auth.models import User
from django.utils.crypto import get_random_string

from celery import shared_task

@shared_task
def create_random_user_accounts(total):
    for i in range(total):
        username = 'user_{}'.format(get_random_string(10, string.ascii_letters))
        email = '{}@example.com'.format(username)
        password = get_random_string(50)
        User.objects.create_user(username=username, email=email, password=password)
    return '{} random users created with success!'.format(total)

The important bits here are:

from celery import shared_task

@shared_task
def name_of_your_function(optional_param):
    pass  # do something heavy

Then I defined a form and a view to process my Celery task:

forms.py

from django import forms
from django.core.validators import MinValueValidator, MaxValueValidator

class GenerateRandomUserForm(forms.Form):
    total = forms.IntegerField(
        validators=[
            MinValueValidator(50),
            MaxValueValidator(500)
        ]
    )

This form expects a positive integer field between 50 and 500. It looks like this:

Then my view:

views.py

from django.contrib.auth.models import User
from django.contrib import messages
from django.views.generic.edit import FormView
from django.shortcuts import redirect

from .forms import GenerateRandomUserForm
from .tasks import create_random_user_accounts

class GenerateRandomUserView(FormView):
    template_name = 'core/generate_random_users.html'
    form_class = GenerateRandomUserForm

    def form_valid(self, form):
        total = form.cleaned_data.get('total')
        create_random_user_accounts.delay(total)
        messages.success(self.request, 'We are generating your random users! Wait a moment and refresh this page.')
        return redirect('users_list')

The important bit is here:

create_random_user_accounts.delay(total)

Instead of calling the create_random_user_accounts directly, I'm calling create_random_user_accounts.delay(). This way we are instructing Celery to execute this function in the background.

Then Django keep processing my view GenerateRandomUserView and returns smoothly to the user.

But before you try it, check the next section to learn how to start the Celery worker process.

Starting The Worker Process

Open a new terminal tab, and run the following command:

celery -A mysite worker -l info

Change mysite to the name of your project. The result is something like this:

Now we can test it. I submitted 500 in my form to create 500 random users.

The response is immediate:

Meanwhile, checking the Celery Worker Process:

[2017-08-20 19:11:17,485: INFO/MainProcess] Received task:
mysite.core.tasks.create_random_user_accounts[8799cfbd-deae-41aa-afac-95ed4cc859b0]

Then after a few seconds, if we refresh the page, the users are there:

If we check the Celery Worker Process again, we can see it completed the execution:

[2017-08-20 19:11:45,721: INFO/ForkPoolWorker-2] Task
mysite.core.tasks.create_random_user_accounts[8799cfbd-deae-41aa-afac-95ed4cc859b0] succeeded in
28.225658523035236s: '500 random users created with success!'

Managing The Worker Process in Production with Supervisord

If you are deploying your application to a VPS like DigitalOcean you will want to run the worker process in the background. In my tutorials I like to use Supervisord to manage the Gunicorn workers, so it's usually a nice fit with Celery.

First install it (on Ubuntu):

sudo apt-get install supervisor

Then create a file named mysite-celery.conf in the folder: /etc/supervisor/conf.d/mysite-celery.conf:

[program:mysite-celery]
command=/home/mysite/bin/celery worker -A web --loglevel=INFO
directory=/home/mysite/mysite
user=nobody
numprocs=1
stdout_logfile=/home/mysite/logs/celery.log
stderr_logfile=/home/mysite/logs/celery.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

stopasgroup=true

; Set Celery priority higher than default (999)
; so, if rabbitmq is supervised, it will start first.
priority=1000

In the example below, I'm considering my Django project is inside a virtual environment. The path to my virtual environment is /home/mysite/.

Now reread the configuration and add the new process:

sudo supervisorctl reread
sudo supervisorctl update

If you are not familiar with deploying Django to a production server and working with Supervisord, maybe this part will make more sense if you check this post from the blog: How to Deploy a Django Application to Digital Ocean.

Further Reading

Those are the basic steps. I hope this helped you to get started with Celery. I will leave here a few useful references to keep learning about Celery:

• Calling Tasks
• Celery Configuration and Defaults
• Celery GitHub Repository

And as usual, the code examples used in this tutorial is available on GitHub:

github.com/sibtc/django-celery-example

Referral Link

If you want to try this setup in a Ubuntu cloud server, you can use this referral link to get a $10 free credit from Digital Ocean.

21 Aug 2017 12:54am GMT

Hello! Today I will guide you through setting up React application with Django! Let's get started!

First thing is where I place my javascript application? Should it be in another repository? Or maybe Django should use webpack to render js?

I decided to use pattern called monorepo. What does it …

20 Aug 2017 1:00pm GMT

Dealing with user input is a very common task in any Web application or Web site. The standard way to do it is through HTML forms, where the user input some data, submit it to the server, and then the server does something with it. Now, the chances are that you might have already heard that quote: "All input is evil!" I don't know who said that first, but it was very well said. Truth is, every input in your application is a door, a potential attack vector. So you better secure all doors! To make your life easier, and to give you some peace of mind, Django offers a very rich, reliable and secure forms API. And you should definitely use it, no matter how simple your HTML form is.

Managing user input, form processing is a fairly complex task, because it involves interacting with many layers of your application. It have to access the database; clean, validate, transform, and guarantee the integrity of the data; sometimes it needs to interact with multiple models, communicate human readable error messages, and then finally it also have to translate all the Python code that represents your models into HTML inputs. In some cases, those HTML inputs may involve JavaScript and CSS code (a custom date picker, or an auto-complete field for example).

The thing is, Django does very well the server-side part. But it doesn't mess much with the client-side part. The HTML forms automatically generated by Django is fully functional and can be used as it is. But it's very crude, it's just plain HTML, no CSS and no JavaScripts. It was done that way so you can have total control on how to present the forms so to match with your application's Web design. On the server-side is a little bit different, as thing are more standardized, so most of the functionalities offered by the forms API works out-of-the-box. And for the special cases, it provide many ways to customize it.

In this tutorial I will show you how to work with the rendering part, using custom CSS and making your forms prettier.

Here is the table of contents of this article:

• Working Example
• Understanding the Rendering Process
• Accessing the Form Fields Individually
• Expanding the Form Fields
• Using Custom HTML Attributes
• Using Django Widget Tweaks
• Rendering Bootstrap 4 Forms
• Reusing Form Components
• Conclusions

Working Example

Throughout the whole tutorial I will be using the following form definition to illustrate the examples:

forms.py

from django import forms

class ContactForm(forms.Form):
    name = forms.CharField(max_length=30)
    email = forms.EmailField(max_length=254)
    message = forms.CharField(
        max_length=2000,
        widget=forms.Textarea(),
        help_text='Write here your message!'
    )
    source = forms.CharField(       # A hidden input for internal use
        max_length=50,              # tell from which page the user sent the message
        widget=forms.HiddenInput()
    )

    def clean(self):
        cleaned_data = super(ContactForm, self).clean()
        name = cleaned_data.get('name')
        email = cleaned_data.get('email')
        message = cleaned_data.get('message')
        if not name and not email and not message:
            raise forms.ValidationError('You have to write something!')

And the following view just to load the form and trigger the validation process so we can have the form in different states:

views.py

from django.shortcuts import render
from .forms import ContactForm

def home(request):
    if request.method == 'POST':
        form = ContactForm(request.POST)
        if form.is_valid():
            pass  # does nothing, just trigger the validation
    else:
        form = ContactForm()
    return render(request, 'home.html', {'form': form})

Understanding the Rendering Process

In many tutorials or in the official Django documentation, it's very common to see form templates like this:

<form method="post" novalidate>
  {% csrf_token %}
  {{ form }}
  <button type="submit">Submit</button>
</form>

It looks like magic, right? Because this particular form may contain 50 fields, and the simple command {{ form }} will render them all in the template.

When we write {{ form }} in a template, it's actually accessing the __str__ method from the BaseForm class. The __str__ method is used to provide a string representation of an object. If you have a look in the source code, you will see that it returns the as_table() method. So, basically {{ form }} and {{ form.as_table }} is the same thing.

The forms API offer three methods to automatically render the HTML form:

• as_table()
• as_ul()
• as_p()

They all work more or less in the same way, the difference is the HTML code that wraps the inputs.

Below is the result of the previous code snippet:

But, if {{ form }} and {{ form.as_table }} is the same thing, the output definitively doesn't look like a table, right? That's because the as_table() and as_ul() doesn't create the <table> and the <ul> tags, so we have to add it by ourselves.

So, the correct way to do it would be:

<form method="post" novalidate>
  {% csrf_token %}
  <table border="1">
    {{ form }}
  </table>
  <button type="submit">Submit</button>
</form>

Now it makes sense right? Without the <table> tag the browser doesn't really know how to render the HTML output, so it just present all the visible fields in line, as we don't have any CSS yet.

If you have a look in the _html_output private method defined in the BaseForm, which is used by all the as_*() methods, you will see that it's a fairly complex method with 76 lines of code and it does lots of things. It's okay because this method is well tested and it's part of the core of the forms API, the underlying mechanics that make things work. When working on your own form rendering logic you won't need to write Python code to do the job. It's much better to do it using the Django Templates engine, as you can achieve a more clean and easier to maintain code.

I'm mentioning the _html_output method here because we can use it to analyze what kind of code it's generating, what it's really doing, so we can mimic it using the template engine. It's also a very good exercise to read the source code and get more comfortable with it. It's a great source of information. Even though Django's documentation is very detailed and extensive, there are always some hidden bits here and there. You also get the chance to see by examples how smart coders solved specific problems. After all, it's an open source project with a mature development process that many have contributed, so the chances are you are reading an optimal code.

Anyway, here it is, in a nutshell, what the _html_output does:

• Aggregate the errors that are not attached to specific fields (non field errors) and errors from hidden fields;
• Place the non field errors and hidden field errors on top of the form;
• Iterate through all the form fields;
• Render the form fields one by one;
  • Render the field name in a label tag;
  • If there are field errors, render a HTML list with errors first;
  • Render the HTML input for the field;
  • If there is a help text, render it after the field.

Here is what the second state of the form looks like, triggering all the validation errors:

Now that we know what it's doing, we can try to mimic the same behavior using the template engine. This way, we will have much more control over the rendering process:

<form method="post" novalidate>
  {% csrf_token %}

  {{ form.non_field_errors }}

  {% for hidden_field in form.hidden_fields %}
    {{ hidden_field.errors }}
    {{ hidden_field }}
  {% endfor %}

  <table border="1">
    {% for field in form.visible_fields %}
      <tr>
        <th>{{ field.label_tag }}</th>
        <td>
          {{ field.errors }}
          {{ field }}
          {{ field.help_text }}
        </td>
      </tr>
    {% endfor %}
  </table>

  <button type="submit">Submit</button>
</form>

You will notice that the result is slightly different, but all the elements are there. The thing is, the automatic generation of the HTML just using the {{ form }} takes advantage of the Python language, so it can play with string concatenation, joining lists (non field errors + hidden field errors), and this sort of things. The template engine is more limited and restrict, but that's not an issue. I like the Django Template engine because it doesn't let you do much code logic in the template.

The only real issue is the random "This field is required" on the top, which refers to the source field. But we can improve that. Let's keep expanding the form rendering, so we can even get more control over it:

<form method="post" novalidate>
  {% csrf_token %}

  {% if form.non_field_errors %}
    <ul>
      {% for error in form.non_field_errors %}
        <li>{{ error }}</li>
      {% endfor %}
    </ul>
  {% endif %}

  {% for hidden_field in form.hidden_fields %}
    {% if hidden_field.errors %}
      <ul>
        {% for error in hidden_field.errors %}
          <li>(Hidden field {{ hidden_field.name }}) {{ error }}</li>
        {% endfor %}
      </ul>
    {% endif %}
    {{ hidden_field }}
  {% endfor %}

  <table border="1">
    {% for field in form.visible_fields %}
      <tr>
        <th>{{ field.label_tag }}</th>
        <td>
          {% if field.errors %}
            <ul>
              {% for error in field.errors %}
                <li>{{ error }}</li>
              {% endfor %}
            </ul>
          {% endif %}
          {{ field }}
          {% if field.help_text %}
            <br />{{ field.help_text }}
          {% endif %}
        </td>
      </tr>
    {% endfor %}
  </table>

  <button type="submit">Submit</button>
</form>

Much closer right?

Now that we know how to "expand" the {{ form }} markup, let's try to make it look prettier. Perhaps using the Bootstrap 4 library.

Accessing the Form Fields Individually

We don't need a for loop to expose the form fields. But it's a very convenient way to do it, specially if you don't have any special requirements for the elements positioning.

Here is how we can refer to the form fields one by one:

<form method="post" novalidate>
  {% csrf_token %}

  {{ form.non_field_errors }}

  {{ form.source.errors }}
  {{ form.source }}

  <table border="1">

      <tr>
        <th>{{ form.name.label_tag }}</th>
        <td>
          {{ form.name.errors }}
          {{ form.name }}
        </td>
      </tr>

      <tr>
        <th>{{ form.email.label_tag }}</th>
        <td>
          {{ form.email.errors }}
          {{ form.email }}
        </td>
      </tr>

      <tr>
        <th>{{ form.message.label_tag }}</th>
        <td>
          {{ form.message.errors }}
          {{ form.message }}
          <br />
          {{ form.message.help_text }}
        </td>
      </tr>

  </table>

  <button type="submit">Submit</button>
</form>

It's not a very DRY solution. But it's good to know how to do it. Sometimes you may have a very specific use case that you will need to position the fields in the HTML by yourself.

Expanding the Form Fields

We can still dig deeper and expand the {{ field }} markup (or if you are doing it individually, it would be the {{ form.name }} or {{ form.email }} fields for example). But now things get a little bit more complex, because we are talking about the widgets. For example, the name field translates into a <input type="text"> tag, while the email field translates into a <input type="email"> tag, and even more problematic, the message field translates into a <textarea></textarea> tag.

At this point, Django makes use of small HTML templates to generate the output HTML of the fields.

So let's see how Django does it. If we open the text.html or the email.html templates from the widgets folder, we will see it simply includes the input.html template file:

{% include "django/forms/widgets/input.html" %}

This suggests the input.html template is probably the most generic one, the specifics of the rendering might be inside it. So, let's have a look:

<input type="{{ widget.type }}"
       name="{{ widget.name }}"
       {% if widget.value != None %} value="{{ widget.value|stringformat:'s' }}"{% endif %}
       {% include "django/forms/widgets/attrs.html" %} />

Basically this small template sets the input type, it's name which is used to access the data in the request object. For example, an input with name "message", if posted to the server, is accessible via request.POST['message'].

Still on the input.html template snippet, it also sets the current value of the field, or leave it empty if there is no data. It's an important bit in the template, because that's what keeps the state of the form after it's submitted and wasn't successfully processed (form was invalid).

Finally, it includes the attrs.html template, which is responsible for setting attributes such as maxlength, required, placeholder, style, or any other HTML attribute. It's highly customizable in the form definition.

If you are curious about the attrs.html, here is what it looks like:

{% for name, value in widget.attrs.items %}
  {% if value is not False %}
    {{ name }}{% if value is not True %}="{{ value|stringformat:'s' }}"{% endif %}
  {% endif %}
{% endfor %}

Now, if you really want to create the inputs by yourself, you can do it like this (just the name field, for brevity):

<input type="text"
       name="name"
       id="id_name"
       {% if form.name.value != None %}value="{{ form.name.value|stringformat:'s' }}"{% endif %}
       maxlength="30"
       required>

Or a little bit better:

<input type="text"
       name="{{ form.name.name }}"
       id="{{ form.name.id_for_label }}"
       {% if form.name.value != None %}value="{{ form.name.value|stringformat:'s' }}"{% endif %}
       maxlength="{{ form.name.field.max_length }}"
       {% if form.name.field.required %}required{% endif %}>

Probably you already figured out that's not the best way to work with forms. And maybe you are also asking yourself why sometimes we refer to a certain attribute as {{ form.name.<something> }} and in other situations we use {{ form.name.field.<something> }}.

I don't want to go into much detail about it right now, but basically form.name is a BoundField (field + data) instance, and then, the form.name.field is the field definition, which is an instance of forms.CharField. That's why some values are available in the bound field instance, and others are in the char field definition.

In any form definition, the form's __iter__ returns a list of BoundField instances, in a similar way, the visible_fields() and hidden_fields() methods also return BoundField instances. Now, if you access the form.fields, it refers to a list of CharField, EmailField, and all other field definitions etc. If that's too much information for you right now, it's okay, you don't have to bother about it right now.

Using Custom HTML Attributes

There are some cases that you only want to add an extra HTML attribute, like a class, a style, or a placeholder. You don't need to expand the input field like we did in the previous example. You can do it directly in the form definition:

forms.py

class ColorfulContactForm(forms.Form):
    name = forms.CharField(
        max_length=30,
        widget=forms.TextInput(
            attrs={
                'style': 'border-color: blue;',
                'placeholder': 'Write your name here'
            }
        )
    )
    email = forms.EmailField(
        max_length=254,
        widget=forms.EmailInput(attrs={'style': 'border-color: green;'})
    )
    message = forms.CharField(
        max_length=2000,
        widget=forms.Textarea(attrs={'style': 'border-color: orange;'}),
        help_text='Write here your message!'
    )

Next, we are going to explore a third-party library that can make your life easier.

Using Django Widget Tweaks

Even though we can control the custom HTML attributes in the form definition, it would be much better if we could set them directly in the template. After all, the HTML attributes refer to the presentation of the inputs.

The django-widget-tweaks library is the right tool for the job. It let you keep the form defaults and just add what you need. It's very convenient, specially when working with ModelForms, as it will reduce the amount of code you have to write to accomplish simple tasks.

I'm not going into much detail about the django-widget-tweaks because I have an article dedicated about it: How to use django-widget-tweaks.

Here's a quick get started guide:

First, install it using pip:

pip install django-widget-tweaks

Add it to the INSTALLED_APPS:

INSTALLED_APPS = [
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'widget_tweaks',
]

Load it in the template:

{% load widget_tweaks %}
<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>Simple is Better Than Complex</title>
</head>
<body>
  ...
</body>

And we are ready to use it! Basically we will use the template tag {% render_field %}. You will see in the next example that we can simply put the attributes just like we would do with raw HTML:

<form method="post" novalidate>
  {% csrf_token %}

  {{ form.non_field_errors }}

  {% for hidden_field in form.hidden_fields %}
    {{ hidden_field.errors }}
    {{ hidden_field }}
  {% endfor %}

  <table border="1">
    {% for field in form.visible_fields %}
      <tr>
        <th>{{ field.label_tag }}</th>
        <td>
          {{ field.errors }}
          {% render_field field style="border: 2px dashed red;" placeholder=field.name %}
          {{ field.help_text }}
        </td>
      </tr>
    {% endfor %}
  </table>

  <button type="submit">Submit</button>
</form>

It's very handy, specially for the cases where you just need to add a CSS class. Which is the case for using the Bootstrap 4 forms templates.

Rendering Bootstrap 4 Forms

Basically to use the Bootstrap 4 library I just included the CDN link they provide in my template:

<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
  <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-beta/css/bootstrap.min.css" integrity="sha384-/Y6pD6FV/Vv2HJnA6t+vslU6fwYXjCFtcEpHbNJ0lyAFsXTsjBbfaDjzALeQsN6M" crossorigin="anonymous">
  <title>Simple is Better Than Complex</title>
</head>

This part of the article will be more to-the-point, as I won't explore the particularities of the Bootstrap 4 implementation. Their documentation is great and rich in examples. If you are not very familiar, you can jump to the Documentation / Components / Forms section for further information.

Let's first focus on the presentation of the inputs, we will get to the errors part later. Here is how we can represent the same form using the Bootstrap 4 tags:

<form method="post" novalidate>
  {% csrf_token %}

  {% for hidden_field in form.hidden_fields %}
    {{ hidden_field }}
  {% endfor %}

  {% for field in form.visible_fields %}
    <div class="form-group">
      {{ field.label_tag }}
      {{ field }}
      {% if field.help_text %}
        <small class="form-text text-muted">{{ field.help_text }}</small>
      {% endif %}
    </div>
  {% endfor %}

  <button type="submit" class="btn btn-primary">Submit</button>
</form>

The input fields looks broken though. That's because the Bootstrap 4 forms expect a CSS class form-control in the HTML inputs. Let's fix it with what we learned in the last section of this article:

{% load widget_tweaks %}

<form method="post" novalidate>
  {% csrf_token %}

  {% for hidden_field in form.hidden_fields %}
    {{ hidden_field }}
  {% endfor %}

  {% for field in form.visible_fields %}
    <div class="form-group">
      {{ field.label_tag }}
      {% render_field field class="form-control" %}
      {% if field.help_text %}
        <small class="form-text text-muted">{{ field.help_text }}</small>
      {% endif %}
    </div>
  {% endfor %}

  <button type="submit" class="btn btn-primary">Submit</button>
</form>

Much better. Now let's see the validation and errors situation. I'm going to use an alert component for the non field errors, and for the fields I will just play with the right CSS classes that Bootstrap 4 provides.

{% load widget_tweaks %}

<form method="post" novalidate>
  {% csrf_token %}

  {% for hidden_field in form.hidden_fields %}
    {{ hidden_field }}
  {% endfor %}

  {% if form.non_field_errors %}
    <div class="alert alert-danger" role="alert">
      {% for error in form.non_field_errors %}
        {{ error }}
      {% endfor %}
    </div>
  {% endif %}

  {% for field in form.visible_fields %}
    <div class="form-group">
      {{ field.label_tag }}

      {% if form.is_bound %}
        {% if field.errors %}
          {% render_field field class="form-control is-invalid" %}
          {% for error in field.errors %}
            <div class="invalid-feedback">
              {{ error }}
            </div>
          {% endfor %}
        {% else %}
          {% render_field field class="form-control is-valid" %}
        {% endif %}
      {% else %}
        {% render_field field class="form-control" %}
      {% endif %}

      {% if field.help_text %}
        <small class="form-text text-muted">{{ field.help_text }}</small>
      {% endif %}
    </div>
  {% endfor %}

  <button type="submit" class="btn btn-primary">Submit</button>
</form>

And here is the result:

It's very cool because it marks with green the fields that passed the validation:

Let's have a close look on what's going on. We can improve the code snippet but I preferred to keep it that way so you can have a better idea about the template rendering logic.

First, I call the form.is_bound method. It tells us if the form have data or not. When we first initialize the form form = ContactForm(), the form.is_bound() method will return False. After a submission, the form.is_bound() will return True. So, we can play with it to know if the validation process already happened or not.

Then, when the validation already occurred, I'm simply marking the field with the CSS class .is-invalid and .is-valid, depending on the case. They are responsible for painting the form components in red or green.

Reusing Form Components

One thing we can do now, is copy the existing code to an external file, and reuse our code snippet for other forms.

includes/bs4_form.html

{% load widget_tweaks %}

{% for hidden_field in form.hidden_fields %}
  {{ hidden_field }}
{% endfor %}

{% if form.non_field_errors %}
  <div class="alert alert-danger" role="alert">
    {% for error in form.non_field_errors %}
      {{ error }}
    {% endfor %}
  </div>
{% endif %}

{% for field in form.visible_fields %}
  <div class="form-group">
    {{ field.label_tag }}

    {% if form.is_bound %}
      {% if field.errors %}
        {% render_field field class="form-control is-invalid" %}
        {% for error in field.errors %}
          <div class="invalid-feedback">
            {{ error }}
          </div>
        {% endfor %}
      {% else %}
        {% render_field field class="form-control is-valid" %}
      {% endif %}
    {% else %}
      {% render_field field class="form-control" %}
    {% endif %}

    {% if field.help_text %}
      <small class="form-text text-muted">{{ field.help_text }}</small>
    {% endif %}
  </div>
{% endfor %}

Then now, our form definition could be as simple as:

<form method="post" novalidate>
  {% csrf_token %}
  {% include 'includes/bs4_form.html' with form=form %}
  <button type="submit" class="btn btn-primary">Submit</button>
</form>

For example, using the code snippet above, we use it to process the UserCreationForm, which is a built-in form that lives inside the django.contrib.auth module. Below, the result:

Conclusions

This article become bigger than I anticipated. I first thought about writing just a quick tutorial about form rendering. Then I remembered that I already had a to-the-point tutorial explaining how to use the django-widget-tweaks. So, instead I decided to dive deep into the details and explore some of the mechanics of the forms API.

I will have a follow-up article focusing on complex forms, rendering all together checkboxes, select fields, date picker and also about developing your own custom widgets.

I hope you learned something new or enjoying reading this article. If you may have any questions or want to discuss further about the topic, please leave a comment below!

As usual, you can find the source code and all the examples on GitHub.

19 Aug 2017 2:00am GMT

As we saw in part 1 of this series, each program reads stuff from input and writes stuff to output. Whenever it reads strings of characters in the input, these strings are encoded in a certain encoding such as UTF-8. The program must decode these strings into an internal representation. When writing to the output, the program encodes strings from its internal representation to an encoding such as UTF-8.

What this internal representation is is usually not our concern. For example, in Python versions earlier than 3.3, the internal representation is either UCS-2 or UCS-4, depending on how Python was compiled. In Python 3.3 or later, the internal representation is more complicated and described in PEP 393. But these details are rarely of interest; what matters is that a program must be able to communicate with other programs via its input/output, and for this to work properly the programs must agree on an external representation. Whenever you see "appétit" rendered as "appétit" it usually means that two programs did not agree on the encoding of the external representation; for example, that one program sent UTF-8 to its output, and another program read this output as its input, thinking it was ISO-8859-1.

So, you have a Django application that displays images, and the browser shows an image, and under the image it renders the caption of the image, which is "Bon appétit!". This caption is stored in a CharField or TextField in a model.

Bon appétit!

The string is stored in a file from where it is retrieved by the RDBMS, travelling through Django, the web server, and the web browser, to be finally painted on the screen by the operating system. On this way, it is encoded and decoded several times. The following figure shows more details of this travel of the string:

In the end what you must remember is that programs store strings in internal representations and encode/decode them when they communicate with other programs. In the next post we will see what you can do to lessen the probability of some error happening along the way.

Was this post useful?

The post Demystifying encodings - part 2 appeared first on Django deployment.

16 Aug 2017 7:24pm GMT

One of my favorite patterns in Django is the combination of "fat" models and cached_property from django.utils.functional.

Fat models are a general MVC concept which encourages pushing logic into methods on your Model layer rather than the Controller ("view" in Django parlance). This has a lot of benefits. It helps maintain the DRY (Don't Repeat Yourself) principle by making common logic easy to find/reuse and makes it easy to break the logic down into small testable units.

One problem with this approach is that as you break down your logic into smaller more reusable units, you may find yourself using them multiple times within a single response. If your methods are particularly resource intensive, it will become an unnecessary performance hit. A common place you find this is in Django tempaltes with patterns like this:

{% if item.top_ten_reviews %}
<ul>
{% for review in item.top_ten_reviews %}
...

This will call the top_ten_reviews method twice. If that method is making database calls, you've now doubled them.

Enter cached_property. This decorator will cache the results of a method for the duration of the request and return it as a property when called again. This technique is known as memoization. Let's look at a simple example:

from django.db import models
from django.utils.functional import cached_property


class Item(models.Model):
    ...
    @cached_property
    def top_ten_reviews(self):
        """Get 10 best reviews"""
        return self.review_set.order_by('-rating')[:10]
    
class Review(models.Model):
    item = models.ForeignKey(Item)
    rating = models.PositiveIntegerField()

In my code, I can lookup an Item object and reference the property item.top_ten_reviews. The first reference will build the queryset, but future references will not, instead recalling the queryset from an in-memory cache.

A couple caveats to cached_property:

1. Like Python's built-in property, it only works for methods without an argument (other than self).
2. It may not be thread-safe. If the object's data isn't changing between threads, it's likely safe. If you are using threads, look to the cached-property module for a thread-safe implemenation.

Wrapping up, cached_property is one of my favorite "easy wins" when working on un-optimized code. It's a quick and relatively safe way to cut out repeated slow tasks without getting into full-blown caching and the ensuing invalidation challenges.

15 Aug 2017 9:02pm GMT

****In Development**** This...

15 Aug 2017 12:21am GMT

Caktus recently held the Q3 2017 ShipIt Day. Each quarter, employees take a step back from business as usual and take advantage of time to work on personal projects or otherwise develop skills. This quarter, we enjoyed fresh crêpes while working on a variety of projects, from coloring books to Alexa skills.

Technology for Linguistics

As both a linguist and a developer, Neil looked at using language technology for a larger project led by Western Carolina University to revitalize Cherokee. This polysynthetic language presents challenges for programming due to its complex word structure.

Using finite state morphology with hfst and Giellatekno, Neil explored defining sounds, a lexicon, and rules to develop a model. In the end, he feels a new framework could help support linguists, and says that Caktus has shown him the value of frameworks and good tooling that could be put to use for this purpose.

Front-end Style Guide Primer

Although design isn't optional in product development, the Agile methodology doesn't address user interface (UI) or user experience (UX) design. We use Agile at Caktus, but we also believe in the importance of solid UX in our projects.

Basia, Calvin, and Kia worked to fill the gap. They started building a front-end style guide, with the intention to supply a tool for Caktus teams to use in future projects. Among style guide components considered during this ShipIt Day were layout, typography, and color palettes. Calvin worked to set up the style guide as a standalone app that serves as a demo and testbed for ongoing style guide work. Kia explored the CSS grid as a flexible layout foundation that makes building pages easier and more efficient while accommodating a range of layout needs. Basia focused on typography, investigating responsive font sizing, modular scale, and vertical rhythm. She also started writing color palettes utilizing colors functions in Stylus.

Front-end style guides have long been advocated by Lean UX. They support modular design, enabling development teams to achieve UI and UX consistency across a project. We look forward to continuing this work and putting our front-end style guide into action!

Command Line Interface for Tequila

Jeff B worked on a command line interface to support our use of Tequila. While we currently use Fabric to execute shell commands, it's not set up to work with Python 3 at the time of writing. Jeff used the Click library to build his project and incorporated difflib from the standard library in order to show a git-style diff of deployment settings. You can dig into the Tequila CLI on the Caktus GitHub account and take a look for yourself!

Wagtail Calendar

Caktus has built several projects using Wagtail CMS, so Charlotte M and Dmitriy looked at adding new functionality. Starting with the goal of incorporating a calendar into the Bakery project, they added an upcoming events button that opens a calendar of events, allowing users to edit and add events.

Charlotte integrated django-scheduler events into Wagtail while Dmitriy focused on integrating the calendar widget onto the EventIndexPage. While they encountered a few challenges which will need further work, they were able to demonstrate a working calendar at the end of ShipIt Day.

Scrum Coloring Book

Charlotte F and Sarah worked together to create a coloring book teaching Scrum information, principles, and diagrams in an easily-digested way. The idea was based on The Scrum Princess. Their story follows Alex, a QA analyst who joins a development team, through the entire process of completing a Scrum project.

Over the course of the day, they came up with the flow of the narrative, formatted the book so that all the images are on separate pages with story text and definitions/image to color. Any illustrators out there who want to help it come to life?

QA Test Case Tools

Gerald joined forces with Robbie, to follow up on Gerald's project from our Q2 2017 ShipIt Day. This quarter, our QA analysts tinkered with QMetry, adding it to JIRA to see whether this could be the tool to take Caktus' QA to the next level.

QMetry creates visibility for test cases related to specific user stories and adds a number of testing functions to JIRA, including the ability to group different scenarios by acceptance criteria and add bugs from within the interface when a test fails. Although there are a few configuration issues to be worked out, they feel that this tool does most of what they want to do without too much back-and-forth.

Wagtail Content Chooser

Phil also took the chance to do some work with Wagtail. Using the built-in page-chooser as a guide, he developed a content-chooser that shows all of the blocks associated with that page's StreamFields. The app can get a content block with its own unique identifier and would enable the admin user to pull that content from other pages into a page being worked on. Next steps will be incorporating a save function.

Publishing an Amazon Alexa Skill

For those seeking inspiring quotes, David authored a skill for Amazon Alexa which would return a random quote from forismatic. An avid fan of swag socks, David came across the opportunity to earn some socks (and an Echo Dot) from Amazon if he submitted an Alexa skill and got it certified. He used the Flask app Flask-Ask to develop the skill rapidly, deployed it to AWS Lambda via Zappa, and is now awaiting certification (and socks). Caktus is an AWS Consulting Partner, so acquiring AWS Alexa development chops would present another service we could offer to clients.

Catching Up on Conferences

Dan caught up on videos of talks from conferences:

• DjangoCon Europe 2017: "Data internationalization in Django" by Raphael Michel covered translating user interface data, different approaches for doing so, and the pros and cons of each.
• PyCon 2017: Constructive Code Review by Eric Rose, also featured in our PyCon Must-See series. He found the advice shared useful beyond the context of code reviews.

He also looked at the possibility of building a new package that preprocesses JavaScript and CSS, but after starting work he realized there's a reason why existing packages are complicated and resolved to revisit this another time.

That's all for now!

Although the ShipIt Day projects represent time away from client work, each project helps our team learn new skills or strengthen existing ones that will eventually contribute toward external work. We see it as a way to invest in both our employees and the company as a whole.

To see some of our skills at work in client projects, check out our Case Studies page.

14 Aug 2017 6:30pm GMT

Django comes with a few optional apps that can easily be installed. One of those apps is the Redirects App, which is particularly useful in the cases where you want to update some existing URLs without compromising your Website SEO or in any case avoid 404 errors.

It basically works by creating a table in the database with two columns, old_path and new_path. Every time your Website raises a 404 error, the Redirects App will intercept the response and check this particular table for a match. If the requested URL is found in the column old_path, instead of raising the 404 error, it will redirect the user to the new_path returning a 301 code (Moved Permanently).

Alright, so let's see how it works in practice.

Installation

The Django Redirects App requires the sites framework to be installed. You can install them by adding the apps to your project's INSTALLED_APPS:

settings.py

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'django.contrib.sites',
    'django.contrib.redirects',
]

Set a SITE_ID so the sites framework works properly.

settings.py

SITE_ID = 1

Now, add the redirects middleware to the MIDDLEWARE configuration:

settings.py

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',

    'django.contrib.redirects.middleware.RedirectFallbackMiddleware',
]

Make sure you run the migrate command to create the required tables:

python manage.py migrate

Usage

The easiest way to use is through Django Admin. If you are not currently using the Django Admin app, and it's a one time thing (for example you are migrating a site from other platform), you can use just the Python API via command line or creating a fixture. If you are not using Django Admin and still wants to add it as a functionality for the site administrator, then you will have to create your own views for it.

Using the Redirects App with Django Admin

It will be automatically added to the Django Admin interface.

You will see, it's very straightforward. Just add the paths and it will do all the hard work for you.

We can test it by typing the old path in the browser and see if redirects correctly. Another way is examining the response body in the terminal. You can easily do it by using curl:

curl --head 127.0.0.1:8000/tips/2017/08/11/django-tip-21.html

HTTP/1.0 301 Moved Permanently
Date: Fri, 11 Aug 2017 15:42:27 GMT
Server: WSGIServer/0.2 CPython/3.6.1
Content-Type: text/html; charset=utf-8
Location: /tips/redirects-app/
X-Frame-Options: SAMEORIGIN
Content-Length: 0

Using the Redirects App with the Python API

You can manually create the redirect records interacting directly with the Redirect model. It lives in django/contrib/redirects/models.py

You can start a Python shell with your project models loaded by running the following command:

python manage.py shell

Here is how you can create a few redirect entries:

from django.contrib.redirects.models import Redirect
from django.contrib.sites.models import Site
from django.conf import settings

site = Site.objects.get(pk=settings.SITE_ID)

Redirect.objects.create(site=site, old_path='/index.html', new_path='/')
Redirect.objects.create(site=site, old_path='/tips/2017/08/11/django-tip-21.html', new_path='/tips/redirects-app/')

Using the Redirects App with Fixtures

Just create a JSON file following the template of the example below:

redirects-fixture.json

[
   {
      "model":"redirects.redirect",
      "fields":{
         "site":1,
         "old_path":"/tips/2017/08/11/django-tip-21.html",
         "new_path":"/tips/redirects-app/"
      }
   },
   {
      "model":"redirects.redirect",
      "fields":{
         "site":1,
         "old_path":"/index.html",
         "new_path":"/"
      }
   }
]

Then you can load it directly to your database by running the command below:

python manage.py loaddata redirects-fixtures.json
Installed 2 object(s) from 1 fixture(s)

And that's it!

11 Aug 2017 9:10pm GMT

This is a solution to the problem I encountered while marrying Django Rest Framework powered API and Axios JS HTTP client: Axios issues GET requests with multi-value parameters in a bit different way than Django expects.

When you create your API with Django Rest Framework, it expects multi-value GET parameters ...

Read now

09 Aug 2017 11:37am GMT

In this article I want to explore some of the basic concepts of Django, setting up a minimal web application to get a deeper understanding of how Django works under the hoods.

An important disclaimer before we start, that's not a tutorial about how to start a Django application, it's more of an exploratory experiment for learning purpose.

Introduction

If you are reading this article, the chances are that you already know that Django is a Web framework written in Python. But that's an abstract definition. In practice, Django is a Python package that lives inside the site-packages directory of your current Python installation. That means it lives alongside with other Python packages, such as Requests, Pillow and NumPy.

A simple way to verify a Django installation is importing it in a Python shell:

>>> import django
>>> print(django.get_version())
1.11.4

But the way we interact with Django is a little bit different than the way we interact with other Python packages. Usually we don't import it directly into our Python programs to make use of its resources.

When we first start learning Django, we are taught that we should start a new project using the django-admin command-line utility by executing the startproject command.

The startproject command creates a basic Django project directory structure with the following files:

• manage.py
• settings.py
• urls.py
• wsgi.py

The contents of the files above are essentially a boilerplate code that's generated using the templates inside the django.conf.project_template folder. It's like a pre-configuration of commonly used resources.

For example, if you are going to develop a Web application, the chances are that you will need a database, handle user authentication, work with sessions, and so on. The boilerplate code from the startproject command makes our life easier. But that's not the only way to start a new project. We could start everything from scratch. And that's sort of what we are going to do in this article.

Also, the names are purely convention, which is a good thing because whenever you browse an existing Django project it's easier to know where to find certain things.

In the next sections we are going to explore the role of each of those files in a Django project.

Running a Project Locally

To run the the development server, we usually execute the following command:

python manage.py runserver

The manage.py script is automatically generated when we start a new Django project by running the command:

django-admin startproject myproject

Basically the django-admin and manage.py does the same things. The main difference is that manage.py adds your project's package in the sys.path and sets the environment variable DJANGO_SETTINGS_MODULE to point to your settings.py file.

That means we could run the development server of my project using the django-admin command-line utility. The difference is that we would need to provide manually the information of the path to my project's package and the settings.py.

So, considering my project is in the following path: /home/projects/myproject, we can start the development server like this:

django-admin runserver --pythonpath='/home/projects/myproject' --settings=myproject.settings

Or if you are currently in the project root in the file system, you could perhaps do this to save typing:

django-admin runserver --pythonpath=. --settings=myproject.settings

As you can see, the settings.py module is the starting point of every Django project. It's the file responsible for putting the pieces together and instructing Django on how to run our project; which apps are installed, what database the project uses (if any), the credentials to connect, where to find the project's templates, the urls, the wsgi module and many other important configurations.

The name settings.py is just a convention. We could use any other name. As far as we tell Django where to find the required information.

The Settings Module

Now that we now the settings module is the central part to run our project, what's the bare minimum configuration we need to feed Django, so to start the development server?

Before we answer that question, know that Django comes pre-configured. Before loading your project's settings module, Django will load the global settings that can be found in the django.conf.global_settings.py module.

The link above takes you to the file directly in the Django's GitHub repository. Take a moment and check it out.

What happen when Django finally loads your project's settings module, is that it will override the default values from the global_settings.py.

The global_settings.py doesn't have all the necessary information. We will need to provide at least a SECRET_KEY.

If you check the Django's global_settings.py file, you will see right at the top of the file that the DEBUG configuration defaults to False. If you don't override it to True, you will also need to configure the ALLOWED_HOSTS variable.

So, the bare minimum configuration just to successfully start the development server could be either defining the SECRET_KEY and DEBUG=True, or SECRET_KEY and ALLOWED_HOSTS.

Starting a Project From Scratch

To try things out, I created a file named tinyapp.py and added the following parameters to it:

DEBUG = True
SECRET_KEY = '4l0ngs3cr3tstr1ngw3lln0ts0l0ngw41tn0w1tsl0ng3n0ugh'

Let's run it:

django-admin runserver --pythonpath=. --settings=tinyapp

It's running. Let's see what happens when we open it in a web browser:

At least something showed up. Let's check the terminal for some further information about the error we got. Checking the terminal that I'm using to run the project, it says:

AttributeError: 'Settings' object has no attribute 'ROOT_URLCONF'

At the moment we happen to have no URL or view. Actually right now Django doesn't even know where to look for URL patterns to see if it have something to process or show.

The ROOT_URLCONF expects the path to a special file that contains a list of URLs so it can match the requested path with the views within our project. This file needs to define a list named urlpatterns. That's correct! We are talking about our well known urls.py module here.

But, so far we don't need a urls.py file. We can tell Django to import the urlpatterns from our tinyapp.py file, so it can also be our ROOT_URLCONF.

DEBUG = True
SECRET_KEY = '4l0ngs3cr3tstr1ngw3lln0ts0l0ngw41tn0w1tsl0ng3n0ugh'
ROOT_URLCONF = __name__

urlpatterns = []

There we go! Our first Django-powered page!

Working With Views

So far so good. But our tiny Django app isn't doing much. Let's add our very first view.

A Django view is just a Python function that receives an HttpRequest object and returns an HttpResponse.

Truth is, our view functions can be defined anywhere in our project. Django won't look for a file named views.py. Again, it's just a convention. Unless you have a good reason for doing it in a different way, stick with the convention.

We will get there later. For now, let's keep writing code inside the tinyapp.py file.

Below, our first Django view named home. It simply returns a text saying "Welcome to the Tinyapp's Homepage!".

from django.http import HttpResponse

DEBUG = True
SECRET_KEY = '4l0ngs3cr3tstr1ngw3lln0ts0l0ngw41tn0w1tsl0ng3n0ugh'
ROOT_URLCONF = __name__

def home(request):
    return HttpResponse('Welcome to the Tinyapp\'s Homepage!')

urlpatterns = []

The next step is instructing Django when to return this view to the visitor. We can do that by adding an entry to the urlpatterns list.

The urlpatterns list expects instances of url() which can be imported from django.conf.urls.

To define a url() you need to at least inform a regex compatible with Python's re module and a view function or the result of as_view() for class-based views.

A basic url routing to the homepage looks like this:

from django.conf.urls import url
from django.http import HttpResponse

DEBUG = True
SECRET_KEY = '4l0ngs3cr3tstr1ngw3lln0ts0l0ngw41tn0w1tsl0ng3n0ugh'
ROOT_URLCONF = __name__

def home(request):
    return HttpResponse('Welcome to the Tinyapp\'s Homepage!')

urlpatterns = [
    url(r'^$', home),
]

The result is:

HTML Templates

Even though we are just returning a string in our first view, the browser tries to render it as if it was an HTML page.

If you want to return text only, you can pass this information along in the HttpResponse so the browser will know it's working with plain text only and won't try to do anything smart with the response body:

from django.conf.urls import url
from django.http import HttpResponse

DEBUG = True
SECRET_KEY = '4l0ngs3cr3tstr1ngw3lln0ts0l0ngw41tn0w1tsl0ng3n0ugh'
ROOT_URLCONF = __name__

def home(request):
    return HttpResponse('Welcome to the Tinyapp\'s Homepage!', content_type='text/plain')

urlpatterns = [
    url(r'^$', home),
]

You can see that the browser renders it a little bit different, as it won't try to parse the content as HTML:

But we know that's not the case in most of the cases while developing Web applications. Let's remove the content_type and add some HTML to our pages.

from django.conf.urls import url
from django.http import HttpResponse

DEBUG = True
SECRET_KEY = '4l0ngs3cr3tstr1ngw3lln0ts0l0ngw41tn0w1tsl0ng3n0ugh'
ROOT_URLCONF = __name__

def home(request):
    return HttpResponse('<h1 style="color:red">Welcome to the Tinyapp\'s Homepage!</h1>')

urlpatterns = [
    url(r'^$', home),
]

We can keep it on and make it a little bit more dynamic:

from django.conf.urls import url
from django.http import HttpResponse

DEBUG = True
SECRET_KEY = '4l0ngs3cr3tstr1ngw3lln0ts0l0ngw41tn0w1tsl0ng3n0ugh'
ROOT_URLCONF = __name__

def home(request):
    color = request.GET.get('color', '')
    return HttpResponse(
        '<h1 style="color:' + color + '">Welcome to the Tinyapp\'s Homepage!</h1>'
    )  # don't use user input like that in real projects!

urlpatterns = [
    url(r'^$', home),
]

We could keep playing with strings and generating HTML on-the-fly:

from django.conf.urls import url
from django.http import HttpResponse

DEBUG = True
SECRET_KEY = '4l0ngs3cr3tstr1ngw3lln0ts0l0ngw41tn0w1tsl0ng3n0ugh'
ROOT_URLCONF = __name__

def home(request):
    color = request.GET.get('color', '')
    return HttpResponse('<h1 style="color:' + color + '">Welcome to the Tinyapp\'s Homepage!</h1>')

def about(request):
    title = 'Tinyapp'
    author = 'Vitor Freitas'
    html = '''<!DOCTYPE html>
    <html>
    <head>
      <title>''' + title + '''</title>
    </head>
    <body>
        <h1>About ''' + title + '''</h1>
        <p>This Website was developed by ''' + author + '''.</p>
    </body>
    </html>'''
    return HttpResponse(html)

urlpatterns = [
    url(r'^$', home),
    url(r'^about/$', about),
]

But hey, there should be a better way to do it. And sure thing there is. That's what the Django's Template Engine is all about.

Before we can use it, we need to tell Django our project makes use of the template engine:

from django.conf.urls import url
from django.http import HttpResponse

DEBUG = True
SECRET_KEY = '4l0ngs3cr3tstr1ngw3lln0ts0l0ngw41tn0w1tsl0ng3n0ugh'
ROOT_URLCONF = __name__
TEMPLATES = [{'BACKEND': 'django.template.backends.django.DjangoTemplates'},]

def home(request):
    # body of the function...

def about(request):
    # body of the function...

urlpatterns = [
    url(r'^$', home),
    url(r'^about/$', about),
]

The Django Template Engine have its own syntax rules. Basically it will read a file (a template), usually a .html file, parse it, process all the special tags like {{ var }} or {% for user in users %} and the final result will be an output string, which normally is a valid HTML document which is returned to the user.

See the example below:

from django.conf.urls import url
from django.http import HttpResponse

DEBUG = True
SECRET_KEY = '4l0ngs3cr3tstr1ngw3lln0ts0l0ngw41tn0w1tsl0ng3n0ugh'
ROOT_URLCONF = __name__
TEMPLATES = [{'BACKEND': 'django.template.backends.django.DjangoTemplates'},]

def home(request):
    color = request.GET.get('color', '')
    return HttpResponse('<h1 style="color:' + color + '">Welcome to the Tinyapp\'s Homepage!</h1>')


from django.template import engines
from django.template.loader import render_to_string

def about(request):
    title = 'Tinyapp'
    author = 'Vitor Freitas'

    about_template = '''<!DOCTYPE html>
    <html>
    <head>
      <title>{{ title }}</title>
    </head>
    <body>
      <h1>About {{ title }}</h1>
      <p>This Website was developed by {{ author }}.</p>
      <p>Now using the Django's Template Engine.</p>
      <p><a href="{% url 'homepage' %}">Return to the homepage</a>.</p>
    </body>
    </html>
    '''

    django_engine = engines['django']
    template = django_engine.from_string(about_template)
    html = template.render({'title': title, 'author': author})

    return HttpResponse(html)

urlpatterns = [
    url(r'^$', home, name='homepage'),
    url(r'^about/$', about, name='aboutpage'),
]

Here we are using the Django's Template Engine programmatically. But what we normally do is storing the HTML templates outside the Python code and telling Django where to look for the templates.

Let's do it step-by-step.

First, create a folder named "templates" alongside our tinyapp.py file.

Now, save the content of the about_template variable inside an HTML file named about.html, inside our recently created templates folder.

templates/about.html

<!DOCTYPE html>
<html>
<head>
  <title>{{ title }}</title>
</head>
<body>
  <h1>About {{ title }}</h1>
  <p>This Website was developed by {{ author }}.</p>
  <p>Now using the Django's Template Engine.</p>
  <p><a href="{% url 'homepage' %}">Return to the homepage</a>.</p>
</body>
</html>

That's what our project looks like now:

myproject/
 |-- templates/
 |    +-- about.html
 +-- tinyapp.py

So, now instead of loading the template from a Python string, we can load it from the filesystem.

First and most importantly, we tell Django where to find the templates:

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [
            '/home/projects/myproject/templates/'
        ],
    },
]

Now we can refactor our about view:

from django.conf.urls import url
from django.http import HttpResponse
from django.template.loader import render_to_string


DEBUG = True
SECRET_KEY = '4l0ngs3cr3tstr1ngw3lln0ts0l0ngw41tn0w1tsl0ng3n0ugh'
ROOT_URLCONF = __name__
TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [
            '/home/projects/myproject/templates/'
        ],
    },
]


def home(request):
    color = request.GET.get('color', '')
    return HttpResponse('<h1 style="color:' + color + '">Welcome to the Tinyapp\'s Homepage!</h1>')

def about(request):
    title = 'Tinyapp'
    author = 'Vitor Freitas'
    html = render_to_string('about.html', {'title': title, 'author': author})
    return HttpResponse(html)


urlpatterns = [
    url(r'^$', home, name='homepage'),
    url(r'^about/$', about, name='aboutpage'),
]

You know the render function from django.shortcuts? That's exactly what it does: use the render_to_string function and returns an HttpResponse instance.

Conclusions

I hope you enjoyed reading this article and that you learned something new today! As I said in the beginning, this was just some sort of experiment. I strongly advise you to get your hands dirty and try things out. It makes you feel more comfortable with Django development, as you get a deeper understanding of its mechanics.

07 Aug 2017 10:00am GMT

I did another couple of benchmarks of different cache backends in Django. This is an extension/update on Fastest cache backend possible for Django published a couple of months ago. This benchmarking isn't as elaborate as the last one. Fewer tests and fewer variables.

I have another app where I use a lot of caching. This web application will run its cache server on the same virtual machine. So no separation of cache server and web head(s). Just one Django server talking to localhost:11211 (memcached's default port) and localhost:6379 (Redis's default port).

Also in this benchmark, the keys were slightly smaller. To simulate my applications "realistic needs" I made the benchmark fall on roughly 80% cache hits and 20% cache misses. The cache keys were 1 to 3 characters long and the cache values lists of strings always 30 items long (e.g. len(['abc', 'def', 'cba', ... , 'cab']) == 30).

Also, in this benchmark I was too lazy to test all different parsers, serializers and compressors that django-redis supports. I only test python-memcached==1.58 versus django-redis==4.8.0 versus django-redis==4.8.0 && msgpack-python==0.4.8.

The results are quite "boring". There's basically not enough difference to matter.

04 Aug 2017 11:32pm GMT

Aviral Tiwari asks:

Is this blog made through Django or some blog engine like WordPress?

Answer

First of all, thanks Aviral for the great question!

Short answer is: Jekyll + Django. Now, if you are interested in a little bit of the history of this blog and the technology stack behind it, keep reading!

This blog is powered by Jekyll, a static site generator written in Ruby. All the pages, all the HTML you see is managed by Jekyll. But at some point this year I had to create a small API using Django to provide a few services and enhance the user experience. So, it's a little bit of a Frankenstein code.

The reason why I picked Jekyll in the beginning was simply because I had no intention to write on regular basis and honestly I did not expect the blog to grow as much as it did. Right now, the blog receives roughly 130,000 visits every month and it have been such a great experience - writing articles, interacting with the readers, reading comments, starting discussions.

Anyway, Jekyll was a very convenient option, because you can host it using GitHub pages. So, all I needed was a domain name and I didn't have to bother about hosting.

It's still holding up pretty well. The pages loads fast enough. After all, it's almost like having a fully cached website. The requests doesn't touch the database, no code being executed, just serving plain HTML pages. The publishing process works pretty well. But as the blog grows, adding more articles and pages, functionalities like "Related posts" and "Read time", it constantly keeps increasing the build time. Nowadays it's taking 15~ seconds to build the source code into the website you see right now.

I wish I had started the blog using WordPress. I mean, it's a great framework. Lot's of websites are using it, it's fairly simple to get started, tons of helping material and tutorials on the Internet. It's a great publishing tool. And after all, it's not about using Python and Django for everything. It's about using the right tools for the right problem.

Many of the things I had to do by hand regarding SEO, organization of the blog content, templates, plugins you would get out-of-the-box using by WordPress.

Essentially I started the blog using Jekyll, hosting on Github. Then at some point I moved it to DigitalOcean so to have more control over the blog. This way I could serve it using https only and add some other features to it.

I really like the DigitalOcean service, I've been using it for more than 3 years now. It's very simple to setup and I find it very inexpensive. I've been running the blog on its tiniest VPS (which costs U$ 5,00 per month) with no problem at all. As you can see the blog runs very smoothly. I've written a blog post about it, if you are interested in knowing more about Django deployment using DigitalOcean you can read it here or if you want to get a U$ 10,00 free credit on DigitalOcean you can sign up using my referral link.

Technology Stack of the Blog

Jekyll

Responsible for generating the static website, converting the posts which are written using Markdown (it's actually kramdown) into HTML pages using the templates I created.

Along with Jekyll, I use the following Ruby gems:

• jekyll-paginate: used in the homepage to paginate the articles
• jekyll-sitemap: automatically generates the sitemap.xml of the blog
• jekyll-time-to-read: automatically calculates the length of the post and generate this widget "X minutes read"
• jekyll-archives: automatically generates the Article Archive page

Django

As I mentioned in the beginning of this post, at some point I had to create a small API using Django to help me handle some of the features of the blog to make it more dynamic and also to help me automate a few tasks, such as its deployment.

This was when Bubbles was born - my Django powered helper. I named it after the monkey that helps King Kai's on the Dragon Ball Z anime.

Among other duties, Bubbles is responsible for:

• Deploying the website when I push new code to a specific branch of my Github repository;
  • It involves waiting for a webhook call from Github;
  • Pulling the new code on my DigitalOcean server;
  • Building the Jekyll website;
  • Testing if the build was successful;
  • Updating the public directory where NGINX serves the website;
• Consuming the Google Analytics API to grab the page views data I display in the posts;
• Consuming the Disqus API to generate a list of last comments displayed in the homepage;
• Processing the questions forms submitted in the Ask a Question page and sending the emails to the person who sent me the question and notifying myself.
• Validating the Google Invisible Recaptcha in some pages.

He lives under the /api/ URL. I instructed NGINX to serve all the requests except those URLS under /api/. Those requests are delegated to my Gunicorn workers that pass the tasks to Bubbles so he can do his thing.

This is what part of my NGINX server block looks like:

server {

    #...

    location /api/ {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_redirect off;
        proxy_pass http://bubbles_web_server;
    }

    location / {
        try_files $uri $uri/ $uri.html =404;
    }
}

Front-End

Back in the days I used to be good with CSS. But nowadays with all the Bootstrap stuff around I got kinda lazy and rusty. I wanted to play around with the CSS a little bit, so I decided to use Skeleton boilerplate and build my own CSS on top of it. The result is what see right now.

Basically what I use in the client-side:

• Skeleton: Responsive CSS Boilerplate
• jQuery
• Font Awesome
• Lunr: I use it to implement a simple client-side search on a JSON file with some articles meta data
• Google Fonts: Custom fonts of the blog

Other Resources

I also use PostgreSQL and Memcached in the web server because some of the APIs I consume have rate limit per hour and also to make the access to the data faster.

Other than that I have a Travis CI setup that builds the blog upon every push, it checks the Jekyll build, search for broken links and check the HTML structure (if there are any broken tag, img tag without the "alt" property and so on).

For writing I use my favorite text editor which is Sublime Text 2. I have a few macros and snippets for creating the code snippet tags I frequently use in the posts and I also have a english spell checker.

Conclusion

This post was not really Django-related, but I hope you enjoyed reading it and finding out more about the underlying technologies that runs this blog.

I intend to move away from the Jekyll at some point. It's great, but as the blog grows it starts to get a little bit challenging to keep developing and writing with it. I thought about moving to WordPress, but I wanted to take the opportunity to create a Django project from scratch, and as I develop it, create a series of posts explaining the whole process.

03 Aug 2017 2:00pm GMT

In this tutorial you will learn how to use the Amazon S3 service to handle static assets and the user uploaded files, that is, the media assets.

First, I will cover the basic concepts, installation and configuration. Then you will find three sections covering:

• Working with static assets only
• Working with static and media assets
• Mixing public assets and private assets

Dependencies

You will need to install two Python libraries:

• boto3
• django-storages

The boto3 library is a public API client to access the Amazon Web Services (AWS) resources, such as the Amazon S3. It's an official distribution maintained by Amazon.

The django-storages is an open-source library to manage storage backends like Dropbox, OneDrive and Amazon S3. It's very convenient, as it plugs in the built-in Django storage backend API. In other words, it will make you life easier, as it won't drastically change how you interact with the static/media assets. We will only need to add a few configuration parameters and it will do all the hard work for us.

Amazon S3 Setup

Before we get to the Django part, let's set up the S3 part. We will need to create a user that have access to manage our S3 resources.

Logged in the AWS web page, find the IAM in the list of services, it's listed under Security, Identity & Compliance:

Go to the Users tab and click in the Add user button:

Give a user name and select the programmatic access option:

Click next to proceed to permissions. At this point we will need to create a new group with the right S3 permissions, and add our new user to it. Follow the wizard and click in the Create group button:

Define a name for the group and search for the built-in policy AmazonS3FullAccess:

Click in the Create group to finalize the group creation process, in the next screen, the recently created group will show up selected, keep it that way and finally click in the button Next: Review:

Review the information, if everything is correct proceed to create the new user. Next, you should see this information:

Take note of all the information: User, Access key ID and the Secret access key. Save them for later.

Click in the Close button and let's proceed. Now, it's time to create our very first bucket.

Bucket is what we call a storage container in S3. We can work with several buckets within the same Django project. But, for the most part you will only need one bucket per website.

Click in the Services menu and search for S3. It's located under Storage. If you see the screen below, you are in the right place.

Click in the + Create bucket to start the flow. Set a DNS-compliant name for your bucket. It will be used to identify your assets. In my case, I choose sibtc-static. So the path to my assets will be something like this: https://sibtc-static.s3.amazonaws.com/static/.

Leave the remaining of the settings as it is, proceed to the next steps just using the defaults and finally hit the Create bucket button. Next you should see the screen below:

Let's leave it like this and let's start working on the Django side.

Installation

The easiest way is to install the libraries using pip:

pip install boto3
pip install django-storages

Now add the storages to your INSTALLED_APPS inside the settings.py module:

settings.py

INSTALLED_APPS = [
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'storages',
]

Working with static assets only

This is the simplest use case. It works out-of-the-box with minimal configuration. All the configuration below goes inside the settings.py module:

settings.py

AWS_ACCESS_KEY_ID = 'AKIAIT2Z5TDYPX3ARJBA'
AWS_SECRET_ACCESS_KEY = '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'
AWS_STORAGE_BUCKET_NAME = 'sibtc-static'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {
    'CacheControl': 'max-age=86400',
}
AWS_LOCATION = 'static'

STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'mysite/static'),
]
STATIC_URL = 'https://%s/%s/' % (AWS_S3_CUSTOM_DOMAIN, AWS_LOCATION)
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

Note that we have some sensitive informations here, such as the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. You should not put it directly to your settings.py file or commit it to a public repository. Instead use environment variables or use the Python library Python Decouple. I have also written a tutorial on how to use Python Decouple.

To illustrate this use case, I created a minimal Django project:

mysite/
 |-- mysite/
 |    |-- static/
 |    |    |-- css/
 |    |    |    +-- app.css
 |    |    +-- img/
 |    |         +-- thumbs-up.png
 |    |-- templates/
 |    |    +-- home.html
 |    |-- __init__.py
 |    |-- settings.py
 |    |-- urls.py
 |    +-- wsgi.py
 +-- manage.py

As you can see, the handling of the static files should go seamlessly:

home.html

{% load static %}<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>S3 Example Static Only</title>
  <link rel="stylesheet" type="text/css" href="{% static 'css/app.css' %}">
</head>
<body>
  <header>
    <h1>S3 Example Static Only</h1>
  </header>
  <main>
    <img src="{% static 'img/thumbs-up.png' %}">
    <h2>It's working!</h2>
  </main>
  <footer>
    <a href="https://simpleisbetterthancomplex.com">www.SimpleIsBetterThanComplex.com</a>
  </footer>
</body>
</html>

Even though we are using our local machine, we will need to run the collectstatic command, since our code will refer to a remote location:

python manage.py collectstatic

You will notice that the copying process will take longer than usual. It's expected. I removed the Django Admin from the INSTALLED_APPS so the example is cleaner. But if you are trying it locally, you will see lot's of files being copied to your S3 bucket.

If we check on the AWS website, we will see our static assets there:

And finally, the result:

As you can see, the storage backend take care to translate the template tag {% static 'img/thumbs-up.png' %} into https://sibtc-static.s3.amazonaws.com/static/img/thumbs-up.png and serve it from the S3 bucket.

In the next example you will learn how to work with both static and media assets.

Working with static and media assets

For this example I created a new bucket named sibtc-assets.

The settings.py configuration will be very similar. Except we will extend the storages.backends.s3boto3.S3Boto3Storage to add a few custom parameters, in order to be able to store the user uploaded files, that is, the media assets in a different location and also to tell S3 to not override files with the same name.

What I usually like to do is create storage_backends.py file in the same directory as my settings.py, and you can define a new Storage Backend like this:

storage_backends.py

from storages.backends.s3boto3 import S3Boto3Storage

class MediaStorage(S3Boto3Storage):
    location = 'media'
    file_overwrite = False

Now on the settings.py, we need to this new backend to the DEFAULT_FILE_STORAGE option:

settings.py

STATICFILES_DIRS = [
    os.path.join(BASE_DIR, 'mysite/static'),
]

AWS_ACCESS_KEY_ID = 'AKIAIT2Z5TDYPX3ARJBA'
AWS_SECRET_ACCESS_KEY = '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'
AWS_STORAGE_BUCKET_NAME = 'sibtc-assets'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME

AWS_S3_OBJECT_PARAMETERS = {
    'CacheControl': 'max-age=86400',
}

AWS_LOCATION = 'static'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, AWS_LOCATION)

DEFAULT_FILE_STORAGE = 'mysite.storage_backends.MediaStorage'  # <-- here is where we reference it

To illustrate a file upload, I created an app named core and defined the following model:

models.py

from django.db import models

class Document(models.Model):
    uploaded_at = models.DateTimeField(auto_now_add=True)
    upload = models.FileField()

Then this is what my view looks like:

views.py

from django.contrib.auth.decorators import login_required
from django.views.generic.edit import CreateView
from django.urls import reverse_lazy

from .models import Document


class DocumentCreateView(CreateView):
    model = Document
    fields = ['upload', ]
    success_url = reverse_lazy('home')

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        documents = Document.objects.all()
        context['documents'] = documents
        return context

The document_form.html template:

<form method="post" enctype="multipart/form-data">
  {% csrf_token %}
  {{ form.as_p }}
  <button type="submit">Submit</button>
</form>

<table>
  <thead>
    <tr>
      <th>Name</th>
      <th>Uploaded at</th>
      <th>Size</th>
    </tr>
  </thead>
  <tbody>
    {% for document in documents %}
      <tr>
        <td><a href="{{ document.upload.url }}" target="_blank">{{ document.upload.name }}</a></td>
        <td>{{ document.uploaded_at }}</td>
        <td>{{ document.upload.size|filesizeformat }}</td>
      </tr>
    {% empty %}
      <tr>
        <td colspan="3">No data.</td>
      </tr>
    {% endfor %}
  </tbody>
</table>

As you can see I'm only using Django's built-in resources in the template. Here is what this template looks like:

I'm not gonna dig into the details about file upload, you can read a comprehensive guide here in the blog (see the Related Posts in the end of this post for more information).

Now, testing the user uploaded files:

I created my template to list the uploaded files, so after a user upload some image or document it will be listed like in the picture above.

Then if we click in the link, which is the usual {{ document.upload.url }}, managed by Django, it will render the image from the S3 bucket:

Now if we check our bucket, we can see that there's a static and a media directory:

Mixing public assets and private assets

Using pretty much the same concepts you define some resources to be privately stored in the S3 bucket. See the configuration below:

storage_backends.py

from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage

class StaticStorage(S3Boto3Storage):
    location = settings.AWS_STATIC_LOCATION

class PublicMediaStorage(S3Boto3Storage):
    location = settings.AWS_PUBLIC_MEDIA_LOCATION
    file_overwrite = False

class PrivateMediaStorage(S3Boto3Storage):
    location = settings.AWS_PRIVATE_MEDIA_LOCATION
    default_acl = 'private'
    file_overwrite = False
    custom_domain = False

settings.py

AWS_ACCESS_KEY_ID = 'AKIAIT2Z5TDYPX3ARJBA'
AWS_SECRET_ACCESS_KEY = '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'
AWS_STORAGE_BUCKET_NAME = 'sibtc-assets'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME

AWS_S3_OBJECT_PARAMETERS = {
    'CacheControl': 'max-age=86400',
}

AWS_STATIC_LOCATION = 'static'
STATICFILES_STORAGE = 'mysite.storage_backends.StaticStorage'
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, AWS_STATIC_LOCATION)

AWS_PUBLIC_MEDIA_LOCATION = 'media/public'
DEFAULT_FILE_STORAGE = 'mysite.storage_backends.PublicMediaStorage'

AWS_PRIVATE_MEDIA_LOCATION = 'media/private'
PRIVATE_FILE_STORAGE = 'mysite.storage_backends.PrivateMediaStorage'

Then we can define this new PrivateMediaStorage directly in the model definition:

models.py

from django.db import models
from django.conf import settings
from django.contrib.auth.models import User

from mysite.storage_backends import PrivateMediaStorage


class Document(models.Model):
    uploaded_at = models.DateTimeField(auto_now_add=True)
    upload = models.FileField()


class PrivateDocument(models.Model):
    uploaded_at = models.DateTimeField(auto_now_add=True)
    upload = models.FileField(storage=PrivateMediaStorage())
    user = models.ForeignKey(User, related_name='documents')

After uploading a private file, if you try to retrieve the URL of the content, the API will generate a long URL that expires after a few minutes:

If you try to access it directly, without the parameters, you will get an error message from AWS:

Conclusions

I hope this tutorial helped to clarify a few concepts of the Amazon S3 and helped you at least get started. Don't be afraid to dig in the official documentation from both boto3 and the django-storages library.

• boto3 documentation
• django-storages documentation

I have also prepared three fully functional examples (the ones that I used in this tutorial), so you can explore and find out more on how I implemented it.

github.com/sibtc/simple-s3-setup

In this repository you will find three Django projects, one for each use case:

• s3-example-public-and-private
• s3-example-static-and-media
• s3-example-static-only

Don't forget to add your own credentials to make it work! I Left them empty on purpose.

01 Aug 2017 2:00pm GMT

Do you like riding a bike? If ...

31 Jul 2017 10:42pm GMT

We use `.gitignore` files ofte...

31 Jul 2017 6:19pm GMT