19 Nov 2019

feedDjango community aggregator: Community blog posts

One Team’s Development Patterns With Vue.js and Django REST Framework

Within the past year, my development team at Caktus worked on a project that required a front-end framework to build a fast, easy-to-use product for a client. After a discussion of frameworks such as React, Vue, and Angular, and their approaches, our team settled on using Vue.js, along with a Django back-end with Django REST Framework (DRF). Initially, we chose Vue because we were more familiar with it, rather than its current competitor React, but as we worked on the product, we ended up having a number of team discussions about how to organize our code well and avoid extra code debt. This blog outlines some of the development patterns we chose as we worked through a number of issues, such as simplifying a multitude of almost identical Vuex mutations, finding a consistent way of holding temporary state, and working with nested objects on the front-end and back-end.

Note: this blog post assumes familiarity either with Vue.js, or a similar front-end framework, like React or AngularJS.

Issue 1: Almost Identical Mutations

In our use of Vue.js, we chose to use the Vuex library for state management in a store, as recommended by the Vue documentation. Following Vuex documentation, we created a number of mutations to alter the state of objects inside of our store. For example, a user in our store (in this blog post's examples, we'll focus on this user object) could look like:

{
  'first_name': '',
  'middle_name': '',
  'last_name': '',
  'email': '',
  'date_of_birth': '',
}

And, so following the Vuex docs, we created mutations for updating each of the user's properties in the store, like this:

updateUserFirstName (state, firstName) {
  state.user.first_name = firstName
},
updateUserMiddleName (state, middleName) {
  state.user.middle_name = middleName
},
updateUserLastName (state, lastName) {
  state.user.last_name = lastName
},
updateUserEmail (state, email) {
  state.user.email = email
},
updateUserDateOfBirth (state, dateOfBirth) {
  state.user.date_of_birth = dateOfBirth
}

This worked great when having only a few objects in the store, but this pattern became increasingly repetitive when both the number of objects in the store (user, company, address, etc.) and the number of properties for each object (first name, middle name, last name, gender, date of birth, etc.) increased. It became clear that having a mutation for each property (updateUserFirstName, updateUserMiddleName, updateUserLastName, updateUserDateOfBirth, etc.) led to redundant lines of mutation code. As a result, we wanted to find a less repetitive, and a more DRY (Don't Repeat Yourself) way of doing so, in order to keep our codebase more readable and maintainable. After some discussion and coding, we created a generic and flexible mutation that would allow us to update any property of the user object:

updateUserAttributes (state, updatesObject) {
  /* Update specific attributes of the user in the store. */
  state.user = Object.assign(deepcopy(state.user), updatesObject)
},

This way, we could update the user's first name and last name by calling

this.$store.commit(
  'updateUserAttributes',
  { 'first_name': 'Newfirstname', 'last_name': 'Newlastname' }
)

from any of our Vue components, which is both less lines of code (less code debt), and easier to write.

Note: The code above uses a deepcopy() function, which is a utility function we wrote for copying an object, and all of its nested fields and objects so that the copied object's nested data no longer points to the original nested data, using lodash's cloneDeep() method. As we found out earlier, simply using Object.assign() creates a new object with nested fields that point to the original object's nested fields, which becomes problematic if the new object gets committed to the store, but then the original object's nested field gets changed, surprisingly leading to the same field change on the new object.

Issue 2: Using Store Or Components For Temporary State

Another issue we discussed at length had to do with how to manage the state of what the user was doing on a particular page. For example, if the user is editing their personal information on a page, we could hold that data in the store, and end up updating the data in the store as the user types on their keyboard. Alternatively, we could let the Vue component hold onto the data, and save them in the back-end when the user presses 'Save,' and wait for the next page load to load it into the store. Or, we could let the Vue component hold onto the data and update it in the store and the back-end when the user clicks 'Save.' Here's a breakdown of pros and cons to each approach:

only refer to store update store and backend on 'Save' update backend on 'Save'
only 1 place to manage state multiple places to manage state (store for permanent state and components for temporary state) multiple places to manage state permanent state and components for temporary state)
user sees changes immediately user sees changes on when clicking 'Save' user sees changes in other components on page load
user sees that clicking 'Save' changes the page user may not want to see changes in multiple components while typing user may not want to see changes in multiple components while typing
forcing the user to click 'Save' gives a clearer indication of what the user is doing forcing the user to click 'Save' gives a clearer indication of what the user is doing

Though making the changes instantly in the store took away the need to manage data in the Vue component, we ultimately decided that it would be more reusable and useful to save the changes both in the front-end store and in the back-end API when the user clicked a 'Save' button. Moreover, having the user click a 'Save' button seemed a more clear indication that they want to save a piece of data than just typing a value in a field. As a result, we ended up utilizing Vue components' data to hold the local state of the objects, and sent those objects to both the back-end and the store when the user clicked 'Save.' In such a pattern, a user detail page could look something like this:

<template>
...
</template>

<script>
export default {
  name: 'UserDetailPage',
  data () {
    return {
      localUser: {}
    }
  },
methods: {
  saveUser () {
    /* Make an API call to the backend, and update the store. */
    ...
    }
  }
}
</script>

So the data that the user was editing would be held in the localUser object, and when the user clicks 'Save,' the data would be sent to both the back-end and the store.

Issue 3: Nested Objects

A third issue we had a number of discussions about was how to manage an object's relations to other objects. From our user example above, a user likely has an address (which is its own object), and the user could have a number of siblings, who have their own addresses, and the user could work at a company, which also has an address, which could have employees, who have their own addresses. Very quickly, the user object could have many layers of nested data, which can become rather unmanageable:

store = {
  user: {
    'id': 100,
    'first_name': '',
    'middle_name': '',
    'last_name': '',
    'email': '',
    'date_of_birth': '',
    'address': {
      'id': 123,
      'street_line_1': '',
      'street_line_2': '',
      'city': '',
      ...
    },
    'siblings':
      [
        {
          'id': 200,
          'first_name': '',
          ...
      },
      {
        'id': 2,
        'first_name': '',
        ...
      },
      ...
    ]
  }
}

One solution for avoiding unmanageable nesting would be to manage each object's nested relations with only the related object's ID in the nested fields:

store = {
  user: {
    'first_name': '',
    'middle_name': '',
    'last_name': '',
    'email': '',
    'date_of_birth': '',
    'address': 123,
    'siblings': [200, 2],
  },
  addresses:
    [
      {
        'id': 123,
        'street_line_1': '',
        ...
      }
  ]
}

This makes each object's data much smaller in the store, and decreases repetition of data (for example, when a user's address matches their sibling's address, we don't need to have multiple places with the same data; only referring to the object's ID should be enough).

However, this approach makes communication between the front-end and the back-end more complex. For instance, the user may want to edit their own information on their detail page to:

  • change their email
  • change their address
  • add a child

In such a case, the front-end may have to make three API calls (one to create an address, one to create a child, and one to update user.email, user.address, and user.children), which could take significantly longer than just making one API call to update everything related to the user. As an aside, learn more about creating an API endpoint in this post.

Seeing the pros and cons of each approach, we had some discussion about which way to develop, and after working through several API endpoints, we ended up with the nested approach, though we also tried to limit the amount of nesting that the back-end would be forced to handle. Django REST Framework does support editing an object's relations to other objects by writing custom update() or create() serializer methods (read more here) and we were able to write these methods for some of our serializers. As a result, we were left with some nested data, but tried to be cognizant of how many nested relations we were using, and to limit them when possible.

Team Discussions Led to Sustainable Decisions

Throughout the course of the project, we continued to have conversations about the development patterns we were forming and following, and we attempted to make decisions that would lead to a more maintainable app with less code debt. Through these team discussions, we were able to come up with solutions that worked for us in terms of handling very similar Vuex mutations, handling temporary state, and working with nested data. As a consequence, we were able to deliver a product that satisfied our client's needs, and is maintainable for our future work. We have such conversations as we continue to build apps that are well-built, maintainable, and work well for our users.

19 Nov 2019 12:44am GMT

14 Nov 2019

feedDjango community aggregator: Community blog posts

Kenneth Love

14 Nov 2019 12:00am GMT

13 Nov 2019

feedDjango community aggregator: Community blog posts

Django – NGINX: deploy your Django project on a production server

Django - NGINX is a popular and well tested combination used to deploy web applications in production. In this post I will explain the steps needed to have your Django project deployed on a production server using Ubuntu 18.04.

To have Django - NGINX deployed on your production server follow these simple steps.

1. Install required packages using apt

sudo apt install nginx uwsgi uwsgi-plugin-python3

Why do you need uWSGI? In very simple terms NGINX on its own cannot run a Python process to host your application, for this you'll need a so called application server that will host a Python process running your Django project. NGINX and uWSGI will "talk" each other using the uwsgi protocol.

2. Create directories for your static and media files

Static files are "not-python" files needed by your Django project, for example Javascript, CSS and images. Media files will be the files uploaded by the users of your application. Not every application will let users upload files, but it's a very common scenario. Django will not serve static and media files by itself. We'll leverage NGINX to serve them.

First of all you have to create the directories. Here I assume that you are currently using the user ubuntu with the default home directory /home/ubuntu:

mkdir -p /home/ubuntu/static /home/ubuntu/media
sudo chown www-data.www-data /home/ubuntu/media

The second command will make the user named www-data the owner of the /home/ubuntu/media directory. www-data will be the user running your Python process in uWSGI, and that user should be able to write in the media directory to correctly save user uploaded files.

3. Setup your Django project and install requirements

This step really depends on your particular Django application, for the purpose of this tutorial I will assume that your Django project is installed in the directory /home/ubuntu/django_project/ with the following structure:

/home/ubuntu/django_project/
├── app1
│   ├── admin.py
│   ├── __init__.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests.py
│   ├── views.py
├── manage.py
└── project
    ├── __init__.py
    ├── settings
    │   └── __init__.py
    │   └── base.py
    │   └── production.py
    ├── urls.py
    ├── wsgi.py

Also I will assume that you installed all your Python requirements, for example using apt or pip.

I always follow a best practice when starting a new Django project, by splitting the monolithic settings.py file in different files, one for each deploy environment (local, test, production, …). You can read a more in depth explanation of this approach if you aren't used to it.

In our case Django will use the module project/settings/production.py for his settings. Here we set the STATIC_ROOT and MEDIA_ROOT variables to the directories we created at step 2:

from .base import *

ALLOWED_HOSTS = [ 'www.example.com' ] # customize with your domain name

DATABASES = {
    'default': { ... } # here the configuration for your database
}

STATIC_ROOT = '/home/ubuntu/static'
MEDIA_ROOT = '/home/ubuntu/media'

4. Collect static files

Run the following command to collect all static files for your Django project:

./manage.py collectstatic

This command will copy all static files (Javascript, CSS, images) for all your Django apps in the STATIC_ROOT directory configured in production.py. For instance /home/ubuntu/static.

5. Configure uWSGI to host your Django project

Create a file named django.ini in the /etc/uwsgi/apps-enabled/ directory. The content of the file should be something like this:

[uwsgi]
chdir = /home/ubuntu/django_project # customize with your django installation directory
env = DJANGO_SETTINGS_MODULE=project.settings.production # customize with your settings module
wsgi-file = project/wsgi.py # customize with the relative path to your wsgi.py file
workers = 1

Restart uWSGI with:

service uwsgi restart

You should find the uWSGI logs in /var/log/uwsgi/apps/django.log. Therefore you can check them to see if the Python process started correctly or there are issues.

6. Configure NGINX to serve your application

Create a file named django in the /etc/nginx/sites-enabled/ directory. The content of the file should be something like this:

server {
    listen 80;
    server_name www.example.com; # customize with your domain name

    location / {
        # django running in uWSGI
        uwsgi_pass unix:///run/uwsgi/app/django/socket;
        include uwsgi_params;
        uwsgi_read_timeout 300s;
        client_max_body_size 32m;
    }

    location /static/ {
       # static files
       alias /home/ubuntu/static/; # ending slash is required
    }

    location /media/ {
        # media files, uploaded by users
        alias /home/ubuntu/media/; # ending slash is required
    }
}

Restart NGINX with:

service nginx restart

7. Enjoy your Django application

Point the browser to your domain, and you should see your Django application in all of its glory!

8. Extra step: automate all these steps with Ansible!

If you have to manage many different Django projects on many different servers, certainly you'll find that automating stuff is always a good idea.

Read my post on How to deploy a Django project in 15 minutes with Ansible to automate all the steps described here.

In conclusion, I hope that this post helped you with configuring Django - NGINX to deploy your Django project on a production server. Please let me know if you have questions leaving a comment in the area below or contacting me by email or social account.

The post Django - NGINX: deploy your Django project on a production server appeared first on guguweb.com.

13 Nov 2019 3:46pm GMT

12 Nov 2019

feedDjango community aggregator: Community blog posts

Django DetailView - podstawowy widok generyczny

DetailView w Django to podstawowy widok generyczny. Poznaj go koniecznie!

12 Nov 2019 6:09pm GMT

06 Nov 2019

feedDjango community aggregator: Community blog posts

Channels

06 Nov 2019 11:55pm GMT

Start Learning Web Development

There is a LOT of good information out there. Tutorials, videos, paid courses, free courses… so much good stuff. When I decided I wanted to learn to code,…

06 Nov 2019 12:54am GMT

31 Oct 2019

feedDjango community aggregator: Community blog posts

Django Slug Tutorial

How to add slugs to a Django DetailView.

31 Oct 2019 3:28pm GMT

30 Oct 2019

feedDjango community aggregator: Community blog posts

How Django Experts Think

30 Oct 2019 10:00pm GMT

29 Oct 2019

feedDjango community aggregator: Community blog posts

How to make Django request.is_ajax() work with JS fetch()

Django's request object has a nifty little method is_ajax. It allows determining whether a request comes from a JS framework (aka old-school ajax). While it works fine with some JS libraries, including the venerable jQuery, it won't work out of the box with modern JS native fetch API ...

Read now

29 Oct 2019 9:15am GMT

24 Oct 2019

feedDjango community aggregator: Community blog posts

Things I want to remember about SSH

SSH, short for Secure Shell, is a protocol for secure network communications. It is widely used for executing commands on remote servers, and for file uploads or downloads. If you are working with Django, use Git version control, or administrate servers, you surely are using SSH. In this post, I want to share some technical details about it.

Secure Shell is using private and public key pairs. You can either use automatically generated private and public keys combined with a password, or manually generated private and public keys. In the latter case, you need to keep your private key on your computer and upload the public key to the remote server.

Creating a pair of SSH keys manually

If you are using GitHub, Bitbucket, DigitalOcean, or some other service, you might have seen the possibility to upload public SSH keys for direct access to remote servers.

Here is how you usually create the SSH keys on the computer from which you want to establish a secure connection (your local machine or one of your servers that has access to other servers or services). In the Terminal you would execute these commands:

$ ssh-keygen
$ ssh-agent /usr/local/bin/bash
$ ssh-add ~/.ssh/id_rsa

The id_rsa is the name of the default SSH private key. The public key would be id_rsa.pub. And by default they both will be located under ~/.ssh/.

When running ssh-keygen you can choose different key names and even add a passphrase. For instance, you could have github_id_rsa and github_id_rsa.pub keys for communication with GitHub. My recommendation would be for each new service to create a new private-public key pair so that in case you need to transfer your computer's data to a different machine, you could selectively transfer the access to the remote servers.

Also, if you are not using the passphrase for the SSH key pair, I would recommend having your disk encrypted and a secure user password for your computer. If your laptop gets stolen, the thief wouldn't be able to get to your remote servers without knowing your computer's password.

Creating an access to a remote server by SSH key

In the case of GitHub, Bitbucket, and other online services with SSH communication, you usually have to copy the contents of the public key into a text field in a web form.

If you want to create a secure communication by manually generated private-public keys with a server where your Django project is deployed, you should append the contents of the public key to the ~/.ssh/authorized_keys file on the remote server.

To get the content of the public key in the Terminal, you can use:

$ cat ~/.ssh/id_rsa.pub

Then copy the output to the clipboard.

Or on macOS you can run pbcopy as follows:

$ pbcopy < ~/.ssh/id_rsa.pub 

To append the contents of the public key to the remote server, you can do this:

$  echo "...pasted public key...">>~/.ssh/authorized_keys

Creating authorization at a remote server by password

If you want to establish an SSH connection with a password and automatically generated private-public keys, you would need to edit /etc/ssh/sshd_config and ensure these two settings:

PasswordAuthentication yes
PermitEmptyPasswords no

After the change, you would restart the ssh server with the following command:

$ sudo service ssh restart

Also, make sure that the user you are connecting with has a password:

$ sudo passwd the_user

Connecting to a remote server

The default way to connect via SSH to a remote server with a password is executing the following in the Terminal:

$ ssh the_user@example.com

To connect with a private key, you would execute this:

$ ssh -i ~/.ssh/examplecom_id_rsa the_user@example.com

Next, let's see how we can simplify this using some local SSH configuration.

Configuring local SSH client

Edit ~/.ssh/config and add the following lines for each SSH connection that you want to define:

Host examplecom
HostName example.com
User the_user
IdentityFile ~/.ssh/examplecom_id_rsa

If the domain of the website is not yet pointing to the IP address of the server, you can also connect by IP address:

Host examplecom
HostName 1.2.3.4
User the_user
IdentityFile ~/.ssh/examplecom_id_rsa

The following allows you to login to your remote servers by manually generated private-public key with just these lines:

$ ssh examplecom

To request for password instead of using the manually generated keys, you would need to modify the snippet as follows:

Host examplecom
HostName example.com
User the_user
PubkeyAuthentication=no

When you connect via SSH and wait don't type anything for 30 minutes or so, the connection gets lost. But you can require your client to connect to the server every 4 minutes or so by adding the following lines to the beginning of the ~/.ssh/config on your local computer:

Host *
ServerAliveInterval 240

Uploading and downloading files using SSH connection

Typically, Secure Shell allows you to execute terminal commands on the remote server using bash, zsh, sh, or another shell. But very often, you also need to transfer files securely to and from the server. For that, you have these options: scp command, rsync command, or FTP client with SFTP support.

scp

The scp stands for Secure Copy.

This is how you would copy the secrets.json file from the remote server to your local development environment:

$ scp the_user@example.com:~/src/myproject/myproject/settings/secrets.json ./myproject/settings/secrets.json

Here is an example of the same, but with custom ~/.ssh/config configuration:

$ scp examplecom:~/src/myproject/myproject/settings/secrets.json ./myproject/settings/secrets.json

To copy the file from the local computer to the remote server, you would switch the places of source and target:

$ scp ./myproject/settings/secrets.json examplecom:~/src/myproject/myproject/settings/secrets.json

rsync

To synchronize directories on the server and locally, you can use the rsync command. This is how to do it for downloading the media/ directory (note that the trailing slashes matter):

$ rsync --archive --compress --partial --progress the_user@example.com:~/src/myproject/myproject/media/ ./myproject/media/

Here is an example of the same with a custom ~/.ssh/config configuration:

$ rsync --archive --compress --partial --progress examplecom:~/src/myproject/myproject/media/ ./myproject/media/

To upload the media/ directory to the remote server, you would again switch places for the source and target:

$ rsync --archive --compress --partial --progress ./myproject/media/ examplecom:~/src/myproject/myproject/media/

sftp

FTP clients like Transmit allow you to have SFTP connections either by username and password or by username and private key. You can even generate the private-public keys directly in the app there.

SFTP works like FTP, but all communication is encrypted there.

The final words

Use only encrypted connections for your network communications, encrypt your hard disk if you use manually generated private-public keys, and use strong passwords.

Be safe!


Cover photo by Jason D.

24 Oct 2019 8:09am GMT

23 Oct 2019

feedDjango community aggregator: Community blog posts

Middleware

23 Oct 2019 10:00pm GMT

22 Oct 2019

feedDjango community aggregator: Community blog posts

Testing Cloudflare workers

I like Cloudflare workers. They allow you to execute arbitrary Javascript as close to your users as possible. For simple use cases your scripts probably don't need to be tested, but once they grow in size and complexity you probably want to be confident that they don't break production. Something I dislike about the serverless trend is that most platforms are hard to test and debug. Testing is poorly documented and examples are far and few between. I wrote this blog post after implementing a non-trivial Cloudflare worker script for a client. The script used the Caching API, communicated with a backend server and manipulated HTTP headers. It required correctness tests before we rolled it out in production. I dug through the Cloudflare documentation and related blog posts to find testing examples, but there wasn't enough material out there so I figured I'd consolidate the knowledge I have on testing Cloudflare workers. The starting point for testing workers is in the [Cloudflare](https://developers.cloudflare.com/workers/archive/deploying-workers/unit-testing/) documentation. However I struggled get the examples to work and it didn't cover common Node.js testing techniques such as stubbing, mocking and running debuggers. This post will cover common testing techniques and scenarios I use to test Cloudflare workers. [TOC] ### Introducing Cloudworker The engineering team at [Dollar Shave Club](https://engineering.dollarshaveclub.com/) developed the [Cloudworker](https://github.com/dollarshaveclub/cloudworker) project to provide a development and testing platform for Cloudflare workers. It's a [simulated execution environment](https://github.com/dollarshaveclub/cloudworker#cloudflare-worker-compatibility), very similar to the environment Cloudflare uses in production. > Cloudworker strives to be as similar to the Cloudflare Worker runtime as > possible. A script should behave the same when executed by Cloudworker and > when run within Cloudflare Workers. Please file an issue for scenarios in > which Cloudworker behaves differently. As behavior differences are found, this > package will be updated to match the Cloudflare Worker runtime. This may > result in breakage if scripts depended on those behavior differences. Cloudworker allows you to both run a local server to run worker scripts and to run the environments programmatically, such as in Mocha test suites. We'll cover programmatic use of Cloudworker in this post. ### Setup To run through these examples you'll need a few libraries: - [Mocha.js](https://mochajs.org/) as the test runner - [Chai.js](https://www.chaijs.com/) for assertions - [Cloudworker](https://github.com/dollarshaveclub/cloudworker) to run a Cloudflare worker environment - [Sinon.js](https://sinonjs.org/) for spying and mocking - [Express.js](https://expressjs.com/) or Nock for a HTTP backend - [Axios](https://github.com/axios/axios) for HTTP testing - [NDB](https://github.com/GoogleChromeLabs/ndb) for debugging ### Testing #### Unit testing requests and responses Cloudworker allows us to unit test requests and responses by injecting requests to the worker script and observing the generated response. ![Cloudworker.dispatch()](https://i.imgur.com/WpMS9Xz.png) Suppose we are building a simple worker script that returns a JSON response. Cloudflare workers implement the [Service worker API](https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API). In practise this means that they receive a 'fetch' event containing a [request](https://developer.mozilla.org/en-US/docs/Web/API/Request/Request) and respond to the request by returning a [response](https://developer.mozilla.org/en-US/docs/Web/API/Response/Response). You have probably dealt with the request and response interfaces before, because they are the same objects as the ones used in the web [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API). Here is our example worker: ```javascript async function handleRequest(event) { const body = {message: 'Hello mocha!'}; return new Response(JSON.stringify(body), { status: 200 }) } // eslint-disable-next-line no-restricted-globals addEventListener('fetch', event => { event.respondWith(handleRequest(event)); }); ``` Now let's write a unit test which injects a request and runs it through the worker to return a response. The test looks like a standard mocha test. The trick is that we'll use the Cloudworker library to simulate a worker execution environment. When creating the Cloudworker object we read our worker file and pass it as a string to the Cloudworker constructor. It then spawns a separate VM where the worker runs and exposes methods to dispatch requests to the worker. The dispatch method takes a [request](https://developer.mozilla.org/en-US/docs/Web/API/Request/Request) and returns a promise containing the [response](https://developer.mozilla.org/en-US/docs/Web/API/Response/Response). Using Mocha and Chai we can assert that the properties of the response object, such as HTTP body and status code are correct. Example unit test: ```javascript hl_lines="20" const fs = require('fs'); const path = require('path'); const Cloudworker = require('@dollarshaveclub/cloudworker'); const { expect } = require('chai'); const workerScript = fs.readFileSync(path.resolve(__dirname, '../worker.js'), 'utf8'); describe('worker unit test', function () { this.timeout(60000); let worker; beforeEach(() => { // Create the simulated execution environment worker = new Cloudworker(workerScript); }); it('tests requests and responses', async () => { const request = new Cloudworker.Request('https://mysite.com/api') // Pass the request through Cloudworker to simulate script exectuion const response = await worker.dispatch(request); const body = await response.json(); expect(response.status).to.eql(200); expect(body).to.eql({message: 'Hello mocha!'}); }); }); ``` #### HTTP integration testing with an upstream server Cloudflare workers usually communicate with an upstream backend. It could be a Rails, Django or Express server. We want to test the interaction between the worker script and the upstream backend. Since the worker uses HTTP to interact with the backend in production, we can create a simple HTTP API for testing purposes. ![Image of Cloudworker.dispatch + HTTP upstream](https://i.imgur.com/jSJTndX.png) If you've written your backend in Node.js you could technically import it as a library and use it in tests. If not you can use Express or Nock to build a testing API that returns identical responses to your production backend. ```javascript hl_lines="4" async function handleRequest(event) { const { request } = event; // Fetch the response from the backend const response = await fetch(request); // The response is originally immutable, so we have to clone it in order to // set the cache headers. // https://community.cloudflare.com/t/how-can-we-remove-cookies-from-request-to-avoid-being-sent-to-origin/35239/2 const newResponse = new Response(response.body, response); newResponse.headers.set('my-header', 'some token'); return newResponse; } // eslint-disable-next-line no-restricted-globals addEventListener('fetch', event => { event.respondWith(handleRequest(event)); }); ``` The worker will issue actual HTTP requests to the backend server running on localhost. In this case the worker adds an additional HTTP header to the response before returning it to the client. We will test that the new header is indeed added to the response. In our tests we'll create an [Express](https://expressjs.com/) testing server. The testing server will listen on localhost and respond to the worker over HTTP. ```javascript const fs = require('fs'); const path = require('path'); const Cloudworker = require('@dollarshaveclub/cloudworker'); const { expect } = require('chai'); const express = require('express'); const workerScript = fs.readFileSync(path.resolve(__dirname, '../upstream-worker.js'), 'utf8'); function createApp() { const app = express(); app.get('/', function (req, res) { res.json({message: 'Hello from express!'}); }); return app; } describe('worker unit test', function () { this.timeout(60000); let serverAddress; let worker; beforeEach(() => { const upstream = createApp().listen(); serverAddress = `http://localhost:${upstream.address().port}` worker = new Cloudworker(workerScript); }); it('tests requests and responses', async () => { const req = new Cloudworker.Request(serverAddress); const res = await worker.dispatch(req); const body = await res.json(); expect(res.headers.get('my-header')).to.eql('some token'); expect(body).to.eql({message: 'Hello from express!'}); }); }); ``` #### Using nock for HTTP mocking It's also possible to use [github.com/nock/nock](https://github.com/nock/nock) to simplify the creation of the testing server. ```javascript hl_lines="19 20 21" const fs = require('fs'); const path = require('path'); const Cloudworker = require('@dollarshaveclub/cloudworker'); const { expect } = require('chai'); const nock = require('nock'); const workerScript = fs.readFileSync(path.resolve(__dirname, '../upstream-worker.js'), 'utf8'); describe('upstream server test', function () { this.timeout(60000); let worker; beforeEach(() => { worker = new Cloudworker(workerScript); }); it('uses Nock upstream server', async () => { const url = 'http://my-api.test'; nock(url) .get('/') .reply(200, {message: 'Hello from Nock!'}); const request = new Cloudworker.Request(url); const response = await worker.dispatch(request); const body = await response.json(); expect(body).to.eql({message: 'Hello from Nock!'}); }); }); ``` #### Running a Cloudworker server So far we've used the Cloudworker `dispatch()` method to inject Request objects and return Response objects. But what if we want to send real HTTP requests to the worker just as we would in production? Luckily Cloudworker can run as a standalone HTTP server on localhost, similar to how Cloudflare runs the worker in production. The `.listen()` method starts a HTTP server that binds to a random port on localhost. ![Image of http client + HTTP upstream](https://i.imgur.com/GAVJ49g.png) We can now use our favourite HTTP client library such as [axios](https://github.com/axios/axios) or [request](https://github.com/request/request) to send HTTP requests to the Cloudworker. ```javascript hl_lines="15" const fs = require('fs'); const path = require('path'); const Cloudworker = require('@dollarshaveclub/cloudworker'); const { expect } = require('chai'); const axios = require('axios'); const workerScript = fs.readFileSync(path.resolve(__dirname, '../simple-worker.js'), 'utf8'); describe('http client test', function () { this.timeout(60000); let serverAddress; beforeEach(() => { const worker = new Cloudworker(workerScript); const server = worker.listen(); serverAddress = `http://localhost:${server.address().port}` }); it('uses axios', async () => { const response = await axios.get(serverAddress); expect(response.status).to.eql(200); expect(response.data).to.eql({message: 'Hello mocha!'}); }); }); ``` #### Spying and mocking with Sinon.js Sometimes you want to test the internals of your worker script. For example, if your worker uses [caching](https://developers.cloudflare.com/workers/reference/apis/cache/) you may want to test that it returns cached responses rather than making upstream HTTP requests. For this reason we may want to spy on the `fetch` HTTP client the worker uses. This allows us to assert on all the method calls made with fetch. The Cloudworker library was written to simulate the production execution at Cloudflare at closely as possible and therefore each worker runs in a separate [VM sandbox](https://github.com/dollarshaveclub/cloudworker/issues/109#issuecomment-519117984). This makes mocking difficult because we can't access or modify variables defined inside of the worker. The library instead allows you to create bindings at creation time to replace Service worker primitives with our own mock variants. We can replace the built-in `fetch` HTTP client with a mock version to check how many times and with what arguments fetch was called. ```javascript worker = new Cloudworker(workerScript, { bindings: { fetch: fetchMock } }); ``` Here we'll use Sinon.js to mock the fetch API inside of the worker and assert it was called once with the expected request. ```javascript hl_lines="18 19 20 21 36" const fs = require('fs'); const path = require('path'); const Cloudworker = require('@dollarshaveclub/cloudworker'); const fetch = require('@dollarshaveclub/node-fetch'); const sinon = require('sinon'); const nock = require('nock'); const workerScript = fs.readFileSync(path.resolve(__dirname, '../upstream-worker.js'), 'utf8'); describe('unit test', function () { this.timeout(60000); let worker; let fetchMock; beforeEach(() => { fetchMock = sinon.fake(fetch); worker = new Cloudworker(workerScript, { // Inject our mocked fetch into the worker script bindings: { fetch: fetchMock } } ); }); it('uses Sinon.js spies to assert calls', async () => { const url = 'http://my-api.test'; nock(url) .get('/') .reply(200, {message: 'Hello from Nock!'}); const request = new Cloudworker.Request(url) await worker.dispatch(request); const expected = new Cloudworker.Request(url); sinon.assert.calledWith(fetchMock, expected); }); }); ``` ### Tips & tricks #### Debugging with NDB NDB is a Node.js debugger that runs in Chrome. It allows us to stop the execution of a worker script and inspect the call stack or the scope variables. The caveat when running a debugger in Cloudworker is that breakpoints within NDB do not work because the worker script [is `eval()`ed at runtime](https://github.com/dollarshaveclub/cloudworker/issues/109). However we can workaround this by adding `debugger;` statements to the source code. For example: ```javascript async function handleRequest(event) { const body = {message: 'Hello mocha!'}; debugger; return new Response(JSON.stringify(body), { status: 200 }) } // eslint-disable-next-line no-restricted-globals addEventListener('fetch', event => { event.respondWith(handleRequest(event)); }); ``` Running ndb allows us to stop and inspect a worker during execution. ```bash ndb mocha test/unit.test.js ``` ![Image debugger](https://i.imgur.com/5kGBLV8.png) #### Debugging 500 errors By default Cloudworker will return a 500 error as a repose anytime things go wrong. This is frustrating to debug as you have to guess where the error happened and inject a debugger statement. I like to add a helper function to automatically debug exceptions that arise in the worker process. In production the debugging code isn't executed, but in tests and development it is. The helper function wraps the main handler and catches any errors that happen. ```javascript async function handleRequest(event) { const body = {message: 'Hello mocha!'}; return new Response(JSON.stringify(body), { status: 200 }) } // A wrapper function which only debugs errors the DEBUG_ERRORS variable is set async function handle(event) { // If we're in production the DEBUG_ERRORS variable will not be set if (typeof DEBUG_ERRORS === 'undefined' || !DEBUG_ERRORS) { return handleRequest(event); } // Debug crashes in test and development try { const res = await handleRequest(event); return res; } catch(err) { console.log(err); debugger; } } // eslint-disable-next-line no-restricted-globals addEventListener('fetch', event => { event.respondWith(handle(event)); }); ``` We use Cloudworker bindings to inject the variable `DEBUG_ERRORS` into the global state of the worker. ```javascript hl_lines="15" const fs = require('fs'); const path = require('path'); const Cloudworker = require('@dollarshaveclub/cloudworker'); const workerScript = fs.readFileSync(path.resolve(__dirname, '../worker-debug-errors.js'), 'utf8'); describe('unit test', function () { this.timeout(60000); let worker; beforeEach(() => { worker = new Cloudworker(workerScript, { bindings: { // Add a global variable that enables error debugging DEBUG_ERRORS: true, } } ); }); it('uses worker.dispatch() to test requests', async () => { const req = new Cloudworker.Request('https://mysite.com/api') await worker.dispatch(req); }); }); ``` Running mocha with ndb enabled will now ensure the debugger stops anytime unexpected errors happen. ```bash ndb mocha test/ ``` #### Enabling the Cache API By default Cloudworker does not enable the caching, presumably because it's still an experimental feature. To test caching you need to manually enable it when creating the Cloudworker object. ```javascript const worker = new Cloudworker(script, { enableCache: true }); ``` ### Examples The example project can be found on [Github](https://github.com/danihodovic/testing-cloudflare-workers).

22 Oct 2019 5:00am GMT

Testing Cloudflare workers

I like Cloudflare workers. They allow you to execute arbitrary Javascript as close to your users as possible. For simple use cases your scripts probably don't need to be tested, but once they grow in size and complexity you probably want to be confident that they don't break production. Something I dislike about the serverless trend is that most platforms are hard to test and debug. Testing is poorly documented and examples are far and few between. I wrote this blog post after implementing a non-trivial Cloudflare worker script for a client. The script used the Caching API, communicated with a backend server and manipulated HTTP headers. It required correctness tests before we rolled it out in production. I dug through the Cloudflare documentation and related blog posts to find testing examples, but there wasn't enough material out there so I figured I'd consolidate the knowledge I have on testing Cloudflare workers. The starting point for testing workers is in the [Cloudflare](https://developers.cloudflare.com/workers/archive/deploying-workers/unit-testing/) documentation. However I struggled get the examples to work and it didn't cover common Node.js testing techniques such as stubbing, mocking and running debuggers. This post will cover common testing techniques and scenarios I use to test Cloudflare workers. [TOC] ### Introducing Cloudworker The engineering team at [Dollar Shave Club](https://engineering.dollarshaveclub.com/) developed the [Cloudworker](https://github.com/dollarshaveclub/cloudworker) project to provide a development and testing platform for Cloudflare workers. It's a [simulated execution environment](https://github.com/dollarshaveclub/cloudworker#cloudflare-worker-compatibility), very similar to the environment Cloudflare uses in production. > Cloudworker strives to be as similar to the Cloudflare Worker runtime as > possible. A script should behave the same when executed by Cloudworker and > when run within Cloudflare Workers. Please file an issue for scenarios in > which Cloudworker behaves differently. As behavior differences are found, this > package will be updated to match the Cloudflare Worker runtime. This may > result in breakage if scripts depended on those behavior differences. Cloudworker allows you to both run a local server to run worker scripts and to run the environments programmatically, such as in Mocha test suites. We'll cover programmatic use of Cloudworker in this post. ### Setup To run through these examples you'll need a few libraries: - [Mocha.js](https://mochajs.org/) as the test runner - [Chai.js](https://www.chaijs.com/) for assertions - [Cloudworker](https://github.com/dollarshaveclub/cloudworker) to run a Cloudflare worker environment - [Sinon.js](https://sinonjs.org/) for spying and mocking - [Express.js](https://expressjs.com/) or Nock for a HTTP backend - [Axios](https://github.com/axios/axios) for HTTP testing - [NDB](https://github.com/GoogleChromeLabs/ndb) for debugging ### Testing #### Unit testing requests and responses Cloudworker allows us to unit test requests and responses by injecting requests to the worker script and observing the generated response. ![Cloudworker.dispatch()](https://i.imgur.com/WpMS9Xz.png) Suppose we are building a simple worker script that returns a JSON response. Cloudflare workers implement the [Service worker API](https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API). In practise this means that they receive a 'fetch' event containing a [request](https://developer.mozilla.org/en-US/docs/Web/API/Request/Request) and respond to the request by returning a [response](https://developer.mozilla.org/en-US/docs/Web/API/Response/Response). You have probably dealt with the request and response interfaces before, because they are the same objects as the ones used in the web [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API). Here is our example worker: ```javascript async function handleRequest(event) { const body = {message: 'Hello mocha!'}; return new Response(JSON.stringify(body), { status: 200 }) } // eslint-disable-next-line no-restricted-globals addEventListener('fetch', event => { event.respondWith(handleRequest(event)); }); ``` Now let's write a unit test which injects a request and runs it through the worker to return a response. The test looks like a standard mocha test. The trick is that we'll use the Cloudworker library to simulate a worker execution environment. When creating the Cloudworker object we read our worker file and pass it as a string to the Cloudworker constructor. It then spawns a separate VM where the worker runs and exposes methods to dispatch requests to the worker. The dispatch method takes a [request](https://developer.mozilla.org/en-US/docs/Web/API/Request/Request) and returns a promise containing the [response](https://developer.mozilla.org/en-US/docs/Web/API/Response/Response). Using Mocha and Chai we can assert that the properties of the response object, such as HTTP body and status code are correct. Example unit test: ```javascript hl_lines="20" const fs = require('fs'); const path = require('path'); const Cloudworker = require('@dollarshaveclub/cloudworker'); const { expect } = require('chai'); const workerScript = fs.readFileSync(path.resolve(__dirname, '../worker.js'), 'utf8'); describe('worker unit test', function () { this.timeout(60000); let worker; beforeEach(() => { // Create the simulated execution environment worker = new Cloudworker(workerScript); }); it('tests requests and responses', async () => { const request = new Cloudworker.Request('https://mysite.com/api') // Pass the request through Cloudworker to simulate script exectuion const response = await worker.dispatch(request); const body = await response.json(); expect(response.status).to.eql(200); expect(body).to.eql({message: 'Hello mocha!'}); }); }); ``` #### HTTP integration testing with an upstream server Cloudflare workers usually communicate with an upstream backend. It could be a Rails, Django or Express server. We want to test the interaction between the worker script and the upstream backend. Since the worker uses HTTP to interact with the backend in production, we can create a simple HTTP API for testing purposes. ![Image of Cloudworker.dispatch + HTTP upstream](https://i.imgur.com/jSJTndX.png) If you've written your backend in Node.js you could technically import it as a library and use it in tests. If not you can use Express or Nock to build a testing API that returns identical responses to your production backend. ```javascript hl_lines="4" async function handleRequest(event) { const { request } = event; // Fetch the response from the backend const response = await fetch(request); // The response is originally immutable, so we have to clone it in order to // set the cache headers. // https://community.cloudflare.com/t/how-can-we-remove-cookies-from-request-to-avoid-being-sent-to-origin/35239/2 const newResponse = new Response(response.body, response); newResponse.headers.set('my-header', 'some token'); return newResponse; } // eslint-disable-next-line no-restricted-globals addEventListener('fetch', event => { event.respondWith(handleRequest(event)); }); ``` The worker will issue actual HTTP requests to the backend server running on localhost. In this case the worker adds an additional HTTP header to the response before returning it to the client. We will test that the new header is indeed added to the response. In our tests we'll create an [Express](https://expressjs.com/) testing server. The testing server will listen on localhost and respond to the worker over HTTP. ```javascript const fs = require('fs'); const path = require('path'); const Cloudworker = require('@dollarshaveclub/cloudworker'); const { expect } = require('chai'); const express = require('express'); const workerScript = fs.readFileSync(path.resolve(__dirname, '../upstream-worker.js'), 'utf8'); function createApp() { const app = express(); app.get('/', function (req, res) { res.json({message: 'Hello from express!'}); }); return app; } describe('worker unit test', function () { this.timeout(60000); let serverAddress; let worker; beforeEach(() => { const upstream = createApp().listen(); serverAddress = `http://localhost:${upstream.address().port}` worker = new Cloudworker(workerScript); }); it('tests requests and responses', async () => { const req = new Cloudworker.Request(serverAddress); const res = await worker.dispatch(req); const body = await res.json(); expect(res.headers.get('my-header')).to.eql('some token'); expect(body).to.eql({message: 'Hello from express!'}); }); }); ``` #### Using nock for HTTP mocking It's also possible to use [github.com/nock/nock](https://github.com/nock/nock) to simplify the creation of the testing server. ```javascript hl_lines="19 20 21" const fs = require('fs'); const path = require('path'); const Cloudworker = require('@dollarshaveclub/cloudworker'); const { expect } = require('chai'); const nock = require('nock'); const workerScript = fs.readFileSync(path.resolve(__dirname, '../upstream-worker.js'), 'utf8'); describe('upstream server test', function () { this.timeout(60000); let worker; beforeEach(() => { worker = new Cloudworker(workerScript); }); it('uses Nock upstream server', async () => { const url = 'http://my-api.test'; nock(url) .get('/') .reply(200, {message: 'Hello from Nock!'}); const request = new Cloudworker.Request(url); const response = await worker.dispatch(request); const body = await response.json(); expect(body).to.eql({message: 'Hello from Nock!'}); }); }); ``` #### Running a Cloudworker server So far we've used the Cloudworker `dispatch()` method to inject Request objects and return Response objects. But what if we want to send real HTTP requests to the worker just as we would in production? Luckily Cloudworker can run as a standalone HTTP server on localhost, similar to how Cloudflare runs the worker in production. The `.listen()` method starts a HTTP server that binds to a random port on localhost. ![Image of http client + HTTP upstream](https://i.imgur.com/GAVJ49g.png) We can now use our favourite HTTP client library such as [axios](https://github.com/axios/axios) or [request](https://github.com/request/request) to send HTTP requests to the Cloudworker. ```javascript hl_lines="15" const fs = require('fs'); const path = require('path'); const Cloudworker = require('@dollarshaveclub/cloudworker'); const { expect } = require('chai'); const axios = require('axios'); const workerScript = fs.readFileSync(path.resolve(__dirname, '../simple-worker.js'), 'utf8'); describe('http client test', function () { this.timeout(60000); let serverAddress; beforeEach(() => { const worker = new Cloudworker(workerScript); const server = worker.listen(); serverAddress = `http://localhost:${server.address().port}` }); it('uses axios', async () => { const response = await axios.get(serverAddress); expect(response.status).to.eql(200); expect(response.data).to.eql({message: 'Hello mocha!'}); }); }); ``` #### Spying and mocking with Sinon.js Sometimes you want to test the internals of your worker script. For example, if your worker uses [caching](https://developers.cloudflare.com/workers/reference/apis/cache/) you may want to test that it returns cached responses rather than making upstream HTTP requests. For this reason we may want to spy on the `fetch` HTTP client the worker uses. This allows us to assert on all the method calls made with fetch. The Cloudworker library was written to simulate the production execution at Cloudflare at closely as possible and therefore each worker runs in a separate [VM sandbox](https://github.com/dollarshaveclub/cloudworker/issues/109#issuecomment-519117984). This makes mocking difficult because we can't access or modify variables defined inside of the worker. The library instead allows you to create bindings at creation time to replace Service worker primitives with our own mock variants. We can replace the built-in `fetch` HTTP client with a mock version to check how many times and with what arguments fetch was called. ```javascript worker = new Cloudworker(workerScript, { bindings: { fetch: fetchMock } }); ``` Here we'll use Sinon.js to mock the fetch API inside of the worker and assert it was called once with the expected request. ```javascript hl_lines="18 19 20 21 36" const fs = require('fs'); const path = require('path'); const Cloudworker = require('@dollarshaveclub/cloudworker'); const fetch = require('@dollarshaveclub/node-fetch'); const sinon = require('sinon'); const nock = require('nock'); const workerScript = fs.readFileSync(path.resolve(__dirname, '../upstream-worker.js'), 'utf8'); describe('unit test', function () { this.timeout(60000); let worker; let fetchMock; beforeEach(() => { fetchMock = sinon.fake(fetch); worker = new Cloudworker(workerScript, { // Inject our mocked fetch into the worker script bindings: { fetch: fetchMock } } ); }); it('uses Sinon.js spies to assert calls', async () => { const url = 'http://my-api.test'; nock(url) .get('/') .reply(200, {message: 'Hello from Nock!'}); const request = new Cloudworker.Request(url) await worker.dispatch(request); const expected = new Cloudworker.Request(url); sinon.assert.calledWith(fetchMock, expected); }); }); ``` ### Tips & tricks #### Debugging with NDB NDB is a Node.js debugger that runs in Chrome. It allows us to stop the execution of a worker script and inspect the call stack or the scope variables. The caveat when running a debugger in Cloudworker is that breakpoints within NDB do not work because the worker script [is `eval()`ed at runtime](https://github.com/dollarshaveclub/cloudworker/issues/109). However we can workaround this by adding `debugger;` statements to the source code. For example: ```javascript async function handleRequest(event) { const body = {message: 'Hello mocha!'}; debugger; return new Response(JSON.stringify(body), { status: 200 }) } // eslint-disable-next-line no-restricted-globals addEventListener('fetch', event => { event.respondWith(handleRequest(event)); }); ``` Running ndb allows us to stop and inspect a worker during execution. ```bash ndb mocha test/unit.test.js ``` ![Image debugger](https://i.imgur.com/5kGBLV8.png) #### Debugging 500 errors By default Cloudworker will return a 500 error as a repose anytime things go wrong. This is frustrating to debug as you have to guess where the error happened and inject a debugger statement. I like to add a helper function to automatically debug exceptions that arise in the worker process. In production the debugging code isn't executed, but in tests and development it is. The helper function wraps the main handler and catches any errors that happen. ```javascript hl_lines="7 8 9 10 11 12 13 14 15 16 17 18 19 20 21" async function handleRequest(event) { const body = {message: 'Hello mocha!'}; return new Response(JSON.stringify(body), { status: 200 }) } // A wrapper function which only debugs errors the DEBUG_ERRORS variable is set async function handle(event) { // If we're in production the DEBUG_ERRORS variable will not be set if (typeof DEBUG_ERRORS === 'undefined' || !DEBUG_ERRORS) { return handleRequest(event); } // Debug crashes in test and development try { const res = await handleRequest(event); return res; } catch(err) { console.log(err); debugger; } } // eslint-disable-next-line no-restricted-globals addEventListener('fetch', event => { event.respondWith(handle(event)); }); ``` We use Cloudworker bindings to inject the variable `DEBUG_ERRORS` into the global state of the worker. ```javascript hl_lines="15" const fs = require('fs'); const path = require('path'); const Cloudworker = require('@dollarshaveclub/cloudworker'); const workerScript = fs.readFileSync(path.resolve(__dirname, '../worker-debug-errors.js'), 'utf8'); describe('unit test', function () { this.timeout(60000); let worker; beforeEach(() => { worker = new Cloudworker(workerScript, { bindings: { // Add a global variable that enables error debugging DEBUG_ERRORS: true, } } ); }); it('uses worker.dispatch() to test requests', async () => { const req = new Cloudworker.Request('https://mysite.com/api') await worker.dispatch(req); }); }); ``` Running mocha with ndb enabled will now ensure the debugger stops anytime unexpected errors happen. ```bash ndb mocha test/ ``` #### Enabling the Cache API By default Cloudworker does not enable the caching, presumably because it's still an experimental feature. To test caching you need to manually enable it when creating the Cloudworker object. ```javascript const worker = new Cloudworker(script, { enableCache: true }); ``` ### Examples The example project can be found on [Github](https://github.com/danihodovic/testing-cloudflare-workers).

22 Oct 2019 5:00am GMT

16 Oct 2019

feedDjango community aggregator: Community blog posts

Signals

16 Oct 2019 10:01pm GMT

09 Oct 2019

feedDjango community aggregator: Community blog posts

Migrating from Python 2 to 3 at EdX - David Ormsbee & Nimisha Asthagiri

09 Oct 2019 10:00pm GMT

Multipage Forms in Django

Introduction

Most online forms fit on a single page. Think of a "join our forum" or "contact us" form into which the user enters a name, email address, and maybe a few other pieces of information. If you're building this kind of functionality into a Django site, you can take advantage of Django's built-in form classes. These are especially handy when dealing with model forms, where the form fields correspond to the fields on a model that will be recorded in your database.

But what if you need a form that spans more than one page? Like a multipage job application where your personal details are on page 1, your relevant experience is on page 2, and so on? There are third-party libraries to help with this, but if you do it yourself you can sidestep a dependency and maybe become more familiar with Django's form handling.

So let's do that. In the post below we'll go step-by-step through the creation of a multipage job application form. We'll start with the simplest functionality and then make it (slightly) more sophisticated. The most important modules ("models.py", "forms.py", and "views.py") will be reproduced here, but a working, standalone project is available from our GitHub account.

Feel free to clone the repo, get the demo up and running, and use or modify it to your own purposes. Or just follow along here in the post.

Disclaimer: The approach below is one that I personally have used to create multipage forms on a few different websites. I'm sure that there are other approaches too, but this is my personal take on the problem.

Now let's jump in!

Requirements

The Model

We'll be working with a model form, so the first thing we need is a model that will represent a submitted job application. One thing to keep in mind: spreading a form over multiple pages means that model instances must be savable to the database before they are complete. After all, the fields on page 2 will not have values yet when page 1 is submitted. Therefore some of the fields on your model must be defined with null=True and/or blank=True, even if you would not normally want to allow this. Don't worry - we'll still be able to require that the user submit a non-blank value, but we'll be doing it at the form level, not at the model (database) level.

So here's our job application model. Apparently this company isn't asking for much information -- just a name, job experience, and a promise of accuracy -- but for the purposes of this demonstration, it will do:

models.py

import hashlib, random, sys
from django.db import models
from . import constants

def create_session_hash():
  hash = hashlib.sha1()
  hash.update(str(random.randint(0,sys.maxsize)).encode('utf-8'))
  return hash.hexdigest()

class JobApplication(models.Model):
  # operational
  session_hash = models.CharField(max_length=40, unique=True)
  stage = models.CharField(max_length=10, default=constants.STAGE_1)
  # stage 1 fields
  first_name = models.CharField(max_length=20, blank=True)
  last_name = models.CharField(max_length=20, blank=True)
  # stage 2 fields

  prior_experience = models.TextField(blank=True)
  # stage 3 fields
  all_is_accurate = models.BooleanField(default=False)

  def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)
    if not self.session_hash:
      while True:
        session_hash = create_session_hash()
        if JobApplication.objects.filter(session_hash=session_hash).count() == 0:
          self.session_hash = session_hash
          break

  @staticmethod
  def get_fields_by_stage(stage):
    fields = ['stage']  # Must always be present
    if stage == constants.STAGE_1:
      fields.extend(['first_name', 'last_name'])
    elif stage == constants.STAGE_2:
      fields.extend(['prior_experience'])
    elif stage == constants.STAGE_3:
      fields.extend(['all_is_accurate'])
    return fields

(Note that this module refers to a module called "constants", which defines values for "stage" and is used both here and in "views.py". This module is not reproduced in this blog post, but if you download the complete project from Github, you will find it there.)

One field essential to this multipage form is stage, which lets us determine which subset of fields to render on page 1 of the form, which on page 2, and so on. Then the data fields (first_name, last_name, prior_experience, and all_is_accurate) will handle the values submitted by the user.

But let's talk about the session_hash field. The key to making a multipage form work is that when the data from page 2 (for example) is submitted, you save it to the same model that you used for page 1. Since html is inherently stateless, we store a hash code in the session to tie the separate GET and POST requests together. Each model instance gets its own unique SHA-1 hash, which will be saved to the model on the first valid POST request. Later requests from the user's browser will include this hash, allowing us to retrieve the correct model instance.

The create_session_hash() and __init__() methods on the model support this. Note the use of the while loop to guard against the vanishingly tiny possibility that we would randomly generate a hash that already exists on a model. Since there are 2^160 different 40-character hexadecimal hash codes, we won't get stuck in that loop for long (and almost certainly not at all).

Finally, we need something to separate the model fields into groups that will be rendered on page 1, page 2, and page 3 of the form. The get_fields_by_stage() method does this for us. The method does not require a JobApplication instance to work, so I've made it a static method.

The Form

The fields on a typical Django form are hard coded. One might look like:

class MyForm(ModelForm):

  foo = forms.IntegerField()

  class Meta:
    model = MyModel
    fields = "__all__"

But a multipage form is a dynamic form. The fields need to be determined at runtime, depending on the state a particular instance is in. Here's the very minimal "forms.py" for our project:

forms.py

from django.forms.models import ModelForm

class BaseApplicationForm(ModelForm):
  pass

Note that our BaseApplicationForm class doesn't have hard-coded fields like the typical example. In fact it doesn't have anything except what it inherits from "ModelForm". Later on we'll add more, but this is all we need to start.

The View

Here's the last big piece of this project: "views.py":

views.py

from django.forms import modelform_factory
from django.shortcuts import redirect
from django.urls import reverse
from django.views.generic import FormView
from . import constants
from .forms import BaseApplicationForm
from .models import JobApplication

def get_job_application_from_hash(session_hash):
  # Find and return a not-yet-completed JobApplication with a matching
  # session_hash, or None if no such object exists.
  return JobApplication.objects.filter(
    session_hash=session_hash,
  ).exclude(
    stage=constants.COMPLETE
  ).first()

class JobApplicationView(FormView):
  template_name = 'job_application/job_application.html'
  job_application = None
  form_class = None

  def dispatch(self, request, *args, **kwargs):
    session_hash = request.session.get("session_hash", None)
    # Get the job application for this session. It could be None.
    self.job_application = get_job_application_from_hash(session_hash)
    # Attach the request to "self" so "form_valid()" can access it below.
    self.request = request
    return super().dispatch(request, *args, **kwargs)

  def form_valid(self, form):
    # This data is valid, so set this form's session hash in the session.
    self.request.session["session_hash"] = form.instance.session_hash
    current_stage = form.cleaned_data.get("stage")
    # Get the next stage after this one.
    new_stage = constants.STAGE_ORDER[constants.STAGE_ORDER.index(current_stage)+1]
    form.instance.stage = new_stage
    form.save()  # This will save the underlying instance.
    if new_stage == constants.COMPLETE:
      return redirect(reverse("job_application:thank_you"))
    # else
    return redirect(reverse("job_application:job_application"))

  def get_form_class(self):
    # If we found a job application that matches the session hash, look at
    # its "stage" attribute to decide which stage of the application we're
    # on. Otherwise assume we're on stage 1.
    stage = self.job_application.stage if self.job_application else constants.STAGE_1
    # Get the form fields appropriate to that stage.
    fields = JobApplication.get_fields_by_stage(stage)
    # Use those fields to dynamically create a form with "modelform_factory"
    return modelform_factory(JobApplication, BaseApplicationForm, fields)

  def get_form_kwargs(self):
    # Make sure Django uses the same JobApplication instance we've already
    # been working on.
    kwargs = super().get_form_kwargs()
    kwargs["instance"] = self.job_application
    return kwargs

(Note that this module refers to the template "job_application.html" and a second view called "thank_you". Its use of the reverse() method also implies the existence of a "urls.py". These elements are not reproduced in this blog post, but if you download the complete project from Github, you will find them there.)

Details on the various methods of this view are in the comments above, but to summarize briefly:

Required Fields and Validation

At this point our multipage form is working, but there are still some odd things about it. For one thing, we are not validating the user's input at all. We also have no way to make a field required. Oddest of all, the "stage" field can be changed by the user when they submit the form!

So let's address these issues now:

Validating Input

This part is no different from any Django form. You can add clean_<fieldname>() methods for the fields you want to validate or a general clean() method as usual. For example, we could change our "BaseApplicationForm" class, which currently only has a pass statement, so that it looks like this:

forms.py

class BaseApplicationForm(ModelForm):

  def clean_first_name(self):
    first_name = self.cleaned_data.get("first_name", "")
    if "e" in first_name:
      raise ValidationError("People with 'e' in their first name need not apply.")
    # else
    return first_name

Required Fields and Hidden Inputs

We need the "stage" field to be a hidden field, but since "stage" is a CharField on the JobApplication model, Django defaults to using a TextInput for the corresponding field on the Form. Let's also suppose that we would like to make the "first_name", "last_name", and "all_is_accurate" fields be required. We need a way to tell our dynamic form that certain fields should be required and that other fields should be rendered as hidden inputs.

First let's add a couple of new lines to our model. Maybe put them right above the defintion of the __init__() method:

models.py

  ...

  hidden_fields = ['stage']
  required_fields = ['first_name', 'last_name', 'all_is_accurate']

  def __init__(self, *args, **kwargs):
    ...

Notice those new variables are defined at the class level. They will be the same for every instance of our model, but that's all right because we're only ever going to read from them.

And again we'll modify the BaseApplicationForm class in "forms.py". Add an __init__() method so that it looks like this:

forms.py

class BaseApplicationForm(ModelForm):

  def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)
    required_fields = self.instance.required_fields
    hidden_fields = self.instance.hidden_fields
    for field in self.fields:
      if field in required_fields:
        self.fields.get(field).required = True
      if field in hidden_fields:
        self.fields.get(field).widget = HiddenInput()

  def clean_first_name(self):
    first_name = self.cleaned_data.get("first_name", "")
    if "e" in first_name:
      raise ValidationError("People with 'e' in their first name need not apply.")
    # else
    return first_name

Now the "stage" field should be hidden and the "first_name", "last_name", and "all_is_accurate" fields are all required at the form level. We now have a working multipage form.

Conclusion

This brings us to the end of our discussion of how to create a multipage form in Django. We have created a JobApplication model and an accompanying form with three pages complete with input validation, required fields, and hidden fields.

Again, for a working example, you can clone the codebase from GitHub. The working example contains all the missing modules and templates referred to above, and also lets you see submitted JobApplication objects using Django's built-in admin.

Of course, there are many ways in which our little job application app could be improved for use in a production site. For example, we could add "created" and "modified" fields to the JobApplication model, which will allow us to forcibly expire JobApplication instances that were left in an incomplete state for too long.

models.py

  ...
  created = models.DateTimeField(auto_now_add=True)
  modified = models.DateTimeField(auto_now=True)
  ...

(The working example in GitHub implements this feature also).

We could also add an in-form "back" button to allow the user to revisit pages they have already submitted, and a "forward" button to allow them to quickly return to where they were before they used the "back" button. Maybe some fields should only be required if other fields are given values. And maybe the user's progression from page 1 of the form to the end should not be the same for all users. We could implement pages of the form that are only rendered if the user has entered certain values earlier on.

Perhaps these topics can be the subject of a follow-up post. Until then, happy coding!

09 Oct 2019 9:20pm GMT