framework

24 Apr 2024

Django community aggregator: Community blog posts

Workbench, the Django-based agency software

I get the impression that there's a lot of interesting but unknown software in Django land. I don't know if there's any interest in some of the packages I have been working on; if not this blog post is for myself only.

(Hi)story time

As people may know I work at Feinheit, an agency which specializes in digital communication services for SMEs, campaigns for referendums, and website and webapp development. At the time of writing we are a team of about 20-25 communication experts, graphic designers, programmers and project managers.

We have many different clients and are working on many different projects at the same time and are billing by the hour¹. Last year my own work has been billed to more than 50 different customers. In the early days we used a shared file server with spreadsheet files to track our working hours. Luckily we didn't often overwrite the edits others made but that was definitely something which happened from time to time.

We knew of another agency who had the same problems and used a FileMaker-based software. Their solution had several problems, among them the fact that it became hard to evolve and that it got slower and slower as more and more data was entered into it over the years. They had the accounting know how and we had the software engineering know how so we wrote a webapp based on the Django framework. As always, it was much more work than the initial estimate, but if we as programmers didn't underestimate the effort needed we wouldn't have started many of the great projects we're now getting much value and/or enjoyment from, hopefully both. The product of that work was Metronom. The first release happened a little bit later than harvest but it already came with full time tracking including an annual working time calculator, absence management, offers, invoices including PDF generation etc, so it was quite a bit more versatile while still being easier to use than "real" business software solutions.

I personally was of the opinion that the product was good enough to try selling it, but for a variety of reasons (which I don't want to go into here) this never happened and we decided that we didn't want to be involved anymore.

However, this meant that we were dead-end street with a software that didn't belong to us anymore, which wasn't evolving to our changing requirements. I also didn't enjoy working on it anymore. Over the years I have tried replacing it several times but that never came to pass until some time after the introduction of Holacracy at our company. I noticed that I didn't have to persuade everyone but that I, as the responsible person for this particular decision, could "just"² move ahead with a broad interpretation of the purpose and accountabilities of one of my roles.

Workbench

Screenshot

Workbench is the product of a few long nights of hacking. The project was started as an experiment in 2015 and was used for sending invoices but that wasn't really the intended purpose. After long periods of lying dormant I have brought the project to a good enough state and switched Feinheit away from Metronom in 2019.

I have thought long and hard about switching to one of the off-the-shelf products and it could very well be that one of them would work well for us. Also, we wouldn't have to pay (in form of working hours) for the maintenance and for enhancements ourselves. On the other hand, we can use a tool which is tailored to our needs. Is it worth the effort? That's always hard to answer. The tool certainly works well for the few companies which are using it right now, so there's no reason to agonize over that.

At the time of writing, Workbench offers projects and services, offers and invoices incl. PDF generation, recurring invoices, an address book, a logbook for rendered services, annual working time reports, an acquisition funnel, a stupid project planning and resource management tool and various reports. It has not only replaced Metronom but also a selection of SaaS (for example Pipedrive and TeamGantt) we were using previously.

The whole thing is open source because I don't want to try making agency software into a business anymore, I only want to solve the problems we have. That being said, if someone else finds it useful then that's certainly alright as well.

The license has been MIT for the first few years but I have switched to the GPL because I wanted to integrate the excellent qrbill module which is also licensed under the GPL. As I wrote elsewhere, I have released almost everything under the GPL in my first few open source years but have switched to BSD/MIT later when starting to work mainly with Python and Django because I thought that this license is a better fit³ for the ecosystem. That being said, for a product such as this the GPL is certainly an excellent fit.

Final words?

There are a few things I could write about to make this into a series. I'm putting a few ideas here, not as an announcement, just as a reminder for myself.

Better time tracking
Patterns for creating recurring invoices
Feature switches and configurability
Coffee time

Rather, in six minute increments. It's even worse. ↩
Of course it's never that simple, because the responsibility is a lot. The important point is: It would have been a lot of work anyways, the big difference is that it was sufficient to get consent from people; no consensus required. ↩
That's not meant as a criticism in any way! ↩

24 Apr 2024 5:00pm GMT

Django: Pinpoint upstream changes with Git

Django's release notes are extensive and describe nearly all changes. Still, when upgrading between Django versions, you may encounter behaviour changes that are hard to relate to any particular release note.

To understand whether a change is expected or a regression, you can use Django's Git repository to search the commits between versions. I often do this when upgrading client projects, at least the larger ones.

In this post, we'll cover Django's branching structure, determining and searching through those commits, a worked example, and advanced behavioural searching with git bisect.

Django's branching structure

Most open source projects use a single main branch, tagged for release when appropriate. However, because Django maintains support for multiple feature versions simultaneously, its branching structure is more complicated. Here's an example:

* main
|
| * stable/5.0.x
| |
⋮ ⋮
|/
|
*
⋮
* * stable/4.2.x
| |
⋮ ⋮
|/
|
*
⋮

There's a main branch, representing the future version of Django, and stable/<version>.x branches representing released versions (at least, released in alpha). When it is time for an alpha release of a new version, a new stable/<version>.x branch is created from main.

Commits are always merged to main. Then, they may be copied onto relevant stable/<version>.x branches with git cherry-pick, also known as backporting, if the merger deems relevant (mergers are typically the Django fellows). Typically, only bug fixes are backported, depending on Django's Supported Versions Policy.

(If you're particularly interested, the backporting script is hosted in Django's wiki.)

Clone and update Django's repository

Before inspecting inter-version history, ensure you have an up-to-date clone of Django's repository. If you're cloning fresh, you're fine. But if you have an existing clone, you want to update any local stable/* branches so they include all backported commits. Here's a short command using git for-each-ref and a while loop to run git pull on all local stable/ branches:

$ git for-each-ref 'refs/heads/stable/*' --format="%(refname:short)" | \
while read entry
do
  git switch $entry
  git pull
done

For example, here's what I see when I run it when all branches are up-to-date:

Switched to branch 'stable/3.0.x'
Your branch is up to date with 'upstream/stable/3.0.x'.
Already up to date.
Switched to branch 'stable/3.1.x'
Your branch is up to date with 'upstream/stable/3.1.x'.
Already up to date.
...
Switched to branch 'stable/5.0.x'
Your branch is up to date with 'upstream/stable/5.0.x'.
Already up to date.

Find changes between versions

To see the changes between versions <old> and <new>, we want commits starting at the point <old> branched from main and ending at the tip of the <new> branch. This start point is what Git calls the merge base between <old> and main, as it's the base that a merge between <old> and main would use for conflicts. Git's git merge-base command can report the merge base between two branches. For example, the point that Django 4.2 branched from main is:

$ git merge-base main stable/4.2.x
9409312eef72d1263dae4b0303523260a54010c5

We can double-check with git show:

$ git show --stat 9409312eef72d1263dae4b0303523260a54010c5
commit 9409312eef72d1263dae4b0303523260a54010c5
Author: Mariusz Felisiak <...>
Date:   Sun Jan 15 19:12:57 2023 +0100

    Updated man page for Django 4.2 alpha.

 docs/man/django-admin.1 | 434 ++++++++++++++++++++++++++++++++++++++++++++++++++++----------------------------------------------------------------------------------------------------------
 1 file changed, 142 insertions(+), 292 deletions(-)

Yup, that looks right-it's ex-Fellow Mariusz preparing for the Django 4.2 alpha release right before creating stable/4.2.x.

For the complete log between <old> and <new>, use git log with git merge-base to template the start point. Use the two-dot range syntax to select commits between two points. So, from Django 4.2 to 5.0:

$ git log --oneline $(git merge-base main stable/4.2.x)..stable/5.0.x

For the diff between versions, use git diff with its three-dot syntax between the versions. Note that Git is inconsistent here: three dots for diff mean the same as two dots for log.

$ git diff $(git merge-base main stable/4.2.x)...stable/5.0.x

Both commands give an overwhelming number of changes (go Django contributors!). They get more useful when you add some filtering options. For example, use git log -S to narrow commits to those that added or removed a given string:

$ git log -S FORMS_URLFIELD_ASSUME_HTTPS $(git merge-base main stable/4.2.x)...stable/5.0.x
commit 92af3d4d235448446e53e982275315bedcc4c204
Author: Mariusz Felisiak <...>
Date:   Tue Nov 28 20:04:21 2023 +0100

    [5.0.x] Refs #34380 -- Added FORMS_URLFIELD_ASSUME_HTTPS transitional setting.

    This allows early adoption of the new default "https".

    Backport of a4931cd75a1780923b02e43475ba5447df3adb31 from main.

Or use git diff with a pathspec to limit the diff to particular files:

$ git diff $(git merge-base main stable/4.2.x)...stable/5.0.x -- 'django/contrib/admin/*.css'
diff --git django/contrib/admin/static/admin/css/base.css django/contrib/admin/static/admin/css/base.css
index 72f4ae169b3..44f2fc8802e 100644
--- django/contrib/admin/static/admin/css/base.css
+++ django/contrib/admin/static/admin/css/base.css
@@ -22,11 +22,11 @@ :root {

     --breadcrumbs-fg: #c4dce8;
     --breadcrumbs-link-fg: var(--body-bg);
-    --breadcrumbs-bg: var(--primary);
+    --breadcrumbs-bg: #264b5d;
...

There are many other useful git log and git diff options for narrowing down changes. I cover my top picks in Boost Your Git DX.

A worked example

I've recently been working on upgrading a client project from Django 4.1 to 4.2. One change that I found was an admin unit test started failing its assertNumQueries() assertion. The test looked like this:

class BookAdminTest(TestCase):
    ...

    def test_save_change_list_view_num_of_queries(self) -> None:
        ...  # Some setup

        with self.assertNumQueries(7):
            """
            1. SAVEPOINT ...
            2. SELECT "django_session" ...
            3. SELECT "core_user" ...
            4. SELECT COUNT(*) AS "__count" FROM "library_book"
            5. SELECT COUNT(*) AS "__count" FROM "library_book"
            6. SELECT "library_book" ...
            7. RELEASE SAVEPOINT ...
            """
            self.client.post("/admin/library/book/", ...)

And the failure message looked like this:

E   AssertionError: 9 != 7 : 9 queries executed, 7 expected
E   Captured queries were:
E   1. SAVEPOINT ...
E   2. SELECT "django_session" ...
E   4. SELECT COUNT(*) AS "__count" FROM "entity_entity"
E   5. SELECT COUNT(*) AS "__count" FROM "entity_entity"
E   6. SELECT "entity_entity" ...
E   7. SAVEPOINT ...
E   8. RELEASE SAVEPOINT ...
E   9. RELEASE SAVEPOINT ...

There were two extra queries. Thankfully, the test author had diligently copied simplified versions of the queries into a comment. I could easily compare and see the new queries were #7, a SAVEPOINT, and #8, a RELEASE SAVEPOINT. These are SQL for a nested transaction, and in Django we typically use transaction.atomic() to create transactions, nested or not.

I didn't immediately spot any relevant release note for this extra transaction, so I checked Django's history. I checked Django's Git log for commits that:

Were between version 4.1 and 4.2.
Added or removed the string "atomic", with git log -S.
Affected django/contrib/admin, with pathspec limiting.

The combined command found precisely the responsible commit straight away:

$ git log $(git merge-base main stable/4.1.x)..stable/4.2.x -S atomic -- django/contrib/admin
commit 7a39a691e1e3fe13588c8885a222eaa6a4648d01
Author: Shubh1815 <...>
Date:   Sat Sep 24 15:42:28 2022 +0530

    Fixed #32603 -- Made ModelAdmin.list_editable use transactions.

Looking at the commit, it turned out it added a pretty clear release note:

$ git show 7a39a691e1e3fe13588c8885a222eaa6a4648d01
commit 7a39a691e1e3fe13588c8885a222eaa6a4648d01
Author: Shubh1815 <shubhparmar14@gmail.com>
Date:   Sat Sep 24 15:42:28 2022 +0530

    Fixed #32603 -- Made ModelAdmin.list_editable use transactions.

...
diff --git docs/releases/4.2.txt docs/releases/4.2.txt
index 5a849cbbe5..5774bfef7b 100644
--- docs/releases/4.2.txt
+++ docs/releases/4.2.txt
@@ -51,6 +51,9 @@ Minor features
 * The ``admin/base.html`` template now has a new block ``nav-breadcrumbs``
   which contains the navigation landmark and the ``breadcrumbs`` block.

+* :attr:`.ModelAdmin.list_editable` now uses atomic transactions when making
+  edits.
+
 :mod:`django.contrib.admindocs`
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

...

I must have skimmed over it, woops! Well, at least I had verified that the behaviour change was expected. I opted to update the test's expected query count and comment:

class BookAdminTest(TestCase):
    ...

    def test_save_change_list_view_num_of_queries(self) -> None:
        ...  # Some setup

        with self.assertNumQueries(9):
            """
            1. SAVEPOINT ...
            2. SELECT "django_session" ...
            3. SELECT "core_user" ...
            4. SELECT COUNT(*) AS "__count" FROM "library_book"
            5. SELECT COUNT(*) AS "__count" FROM "library_book"
            6. SELECT "library_book" ...
            7. SAVEPOINT ...
            8. RELEASE SAVEPOINT ...
            9. RELEASE SAVEPOINT ...
            """
            self.client.post("/admin/library/book/", ...)

Search by behaviour with `git bisect`

Searching through the log and diff is hard when you don't know which files to look at or strings to search for. If you're observing a behaviour change and don't know its cause, try using git bisect to find the responsible commit.

See my git bisect basics post for an introduction to the command. Here, we'll discuss the specifics of bisecting Django in your project.

First, use an editable install of your local Django repository within the target project. With Pip, use pip install -e:

$ python -m pip install -e ~/Projects/django

Replace ~/Projects with the appropriate path to the Django repository.

Second, ensure you can run your behaviour test on both Django versions. That behaviour test might be loading a page under ./manage.py runserver, a unit test with ./manage.py test or pytest, or some other command. Your project must work sufficiently on both Django versions before you can bisect between them.

Switch your Django repository to the older version:

$ git switch stable/4.2.x

Then, run the behaviour test in your project:

$ ./manage.py runserver
...
[24/Apr/2024 05:25:59] "GET / HTTP/1.1" 200 1797

Similarly, repeat on the newer version:

$ git switch stable/5.0.x

$ ./manage.py runserver
...
[24/Apr/2024 05:26:25] "GET / HTTP/1.1" 200 1761

If your test does not run smoothly on both versions, modify the project until it does. Typically, this means acting on deprecation warnings from the old version. But in the worst case, you may need to fork code between the old and new versions, especially if you're trying to upgrade many versions. Try this pattern for forking based on Django's version tuple:

import django

if django.VERSION >= (5, 0):
    # do the new thing
    ...
else:
    # do the old thing
    ...

Third, run the bisect. In the Django repository, start the bisect and label the old and new versions like so:

$ git bisect start
$ git bisect old $(git merge-base main stable/4.2.x)
$ git bisect new stable/5.0.x

Do the usual thing of iterating with the old and new subcommands until you find the responsible commit. Remember to finish up with git bisect reset.

Finally, roll back your project to use the non-editable install of Django. For example, with Pip:

$ python -m pip install -r requirements.txt

Fin

See you in the Django' Git log,

-Adam

24 Apr 2024 4:00am GMT

23 Apr 2024

Django community aggregator: Community blog posts

Django News - [Resend of #228] PyPI Expanding Trusted Publisher Support - Apr 22nd 2024

Introduction

Hi everyone,

We apologize to anyone who didn't receive the Django News Newsletter Issue #228 last Friday and to anyone who just received a duplicate edition. Issue #229 is a re-send of what everything should have received last Friday.

Last week, our newsletter provider had a hiccup, and we estimate that less than 10% of our subscribers received their weekly Friday edition of Django News. We felt terrible that you didn't receive it and that two conference CFPs will have ended before our next newsletter goes out.

We decided our best bet was to re-send everyone an updated Monday edition of Django News and apologize again for Friday's mishap.

Please note that Wagtail Space US's CFP ends today, April 22nd (hours left), and DjangoCon US's CFP ends Wednesday, April 24th (less than two days left).

Jeff and Will

Django Newsletter

News

Don't Miss Out: Last Call for DjangoCon US 2024 Talk Proposals!

Have you submitted your talk or tutorial for DjangoCon US 2024, in beautiful Durham, North Carolina, USA?

This is your last call to submit a talk or tutorial. The CFP deadline is April 24, 2024, at 12 PM EDT.

djangocon.us

PyPI: Expanding Trusted Publisher Support

PyPI added GitLab CI/CD, Google Cloud, and ActiveState as Trusted Publishing providers.

pypi.org

Django Software Foundation

DSF Board meeting minutes for April 11, 2024

Here are the DSF Board's meeting minutes for April 11, 2024.

djangoproject.com

Updates to Django

Today 'Updates to Django' is presented by Raffaella Suardini from Djangonaut Space!

Last week we had 10 pull requests merged into Django by 5 different contributors - including 1 first-time contributor! Congratulations to Aleksander Milinkevich for having their first commits merged into Django - welcome on board!

Coming in Django 5.0.5 (expected May 6th):

Resolved a compatibility issue with Python 3.11.9+ and 3.12.3+ when validating email max line lengths with content decoded using the surrogateescape error handling scheme.
Resolved a bug in Django 5.0 that makes Model.save() crash when creating a model instance with a GeneratedField and specifying a primary key.

Django Newsletter

Wagtail CMS

Wagtail Space CFP deadline is April 22

It's the last call to speak this summer at Wagtail Space US 2024.

Wagtail Space US 2024 - Call For Proposals

Wagtail Space Virtual Talks 2024 - Call For Proposals

wagtail.space

Articles

7 simple examples using Django GeneratedField

Django 5.0 added a new feature, GeneratedField, which allows us to auto-calculate database fields. This article shows seven short examples of how to use it, so the database performs calculations exceptionally quickly.

photondesigner.com

Django from first principles, part 3

In the third installment of his series on constructing a comprehensive Django project from a single file, Eric Matthes explores enhancing your homepage with templates.

mostlypython.com

Building forms with the Django admin

A look at multiple ways, including over time, to build and style forms in the Django admin.

406.ch

Styling a Django RSS Feed

A straightforward way to style your RSS feed in a Django app.

hyteck.de

Enforcing conventions in Django projects with introspection

Some code and tips to combine Python and Django introspection APIs to enforce naming conventions in your Django models.

lukeplant.me.uk

Tutorials

Building a Voice Notes App with Django and OpenAI

In this tutorial, you will learn how to build a voice notes app using Django and OpenAI for speech-to-text conversion. Additionally, AlpineJS will manage the state on the front end.

circumeo.io

Videos

Django 2024: The Latest Development Trends

Scheduled for Apr 25, 2024. Tune in to our upcoming livestream, where we'll take you through the latest Django Developers Survey results based on responses from 4,000 Django developers.

youtube.com

Understanding Wasm: How We Got Here by Chris Dickinson @ Wasm I/O 2024

Let's put Wasm and the problems it solves into historical context: what are we trying to solve, and for whom? What has been tried before? What makes this effort more likely to succeed? We'll examine the history of virtual machines, operating systems, & hypervisors from the 1960s through the 2010s.

youtube.com

Podcasts

Django Chat #161: Kraken - Çağıl Uluşahin Sönmez

Çağıl is a Lead Backend Engineer at Kraken Tech, Django Software Foundation Vice President, and Django London Meetup co-organizer. We discuss her background studying computer science in Turkey, organizing DjangoGirls and Python events in Istanbul, and her current work today.

djangochat.com

Python Test #218: Balancing test coverage with test costs

Nicole is a software engineer and writer, and recently wrote about the trade-offs we make when deciding which tests to write and how much testing is enough.

pythontest.com