15 Feb 2025

feedDjango community aggregator: Community blog posts

Bookmarklets, defaults-from-GET, and iommi

Phil Gyford wrote an article about how nice it is that the Django admin pre-populates inputs from the GET parameters if there are any. This can be used for bookmarklets as in his examples, or just general bookmarks where you can quickly go to a page with parts of a form prefilled.

Another very useful case for this pattern is to have a link on one page of your product with a link to a create form with prefilled data based on the context of the page you linked from. Like having an artist page with a link to the create album page with the artist filled in.

The Django admin does this, but Django forms do not. Because Django forms have an API that takes a dict for the data and not the request object itself, it can't be retrofitted to have this feature either. It's a nice example of where limiting the API surface area also limits future development.

In iommi, defaults-from-GET is the default for all forms. So if you build with iommi you get this feature across your product for free, not just in the admin. We even handle the edge cases for you like when you have a GET form, and you supply parameters via GET, so the form needs to know if this is from some URL or from a submit of the form. This is handled in iommi for you transparently.

15 Feb 2025 6:00am GMT

14 Feb 2025

feedPlanet Python

Kay Hayen: Nuitka this week #16

Hey Nuitka users! This started out as an idea of a weekly update, but that hasn't happened, and so we will switch it over to just writing up when something interesting happens and then push it out relatively immediately when it happens.

Nuitka Onefile Gets More Flexible: --onefile-cache-mode and {PROGRAM_DIR}

We've got a couple of exciting updates to Nuitka's onefile mode that give you more control and flexibility in how you deploy your applications. These enhancements stem from real-world needs and demonstrate Nuitka's commitment to providing powerful and adaptable solutions.

Taking Control of Onefile Unpacking: --onefile-cache-mode

Onefile mode is fantastic for creating single-file executables, but the management of the unpacking directory where the application expands has sometimes been a bit… opaque. Previously, Nuitka would decide whether to clean up this directory based on whether the path used runtime-dependent variables. This made sense in theory, but in practice, it could lead to unexpected behavior and made debugging onefile issues harder.

Now, you have complete control! The new --onefile-cache-mode option lets you explicitly specify what happens to the unpacking directory:

This gives you the power to choose the behavior that best suits your needs. No more guessing!

Relative Paths with {PROGRAM_DIR}

Another common request, particularly from users deploying applications in more restricted environments, was the ability to specify the onefile unpacking directory relative to the executable itself. Previously, you were limited to absolute paths or paths relative to the user's temporary directory space.

We've introduced a new variable, {PROGRAM_DIR}, that you can use in the --onefile-tempdir-spec option. This variable is dynamically replaced at runtime with the full path to the directory containing the onefile executable.

For example:

nuitka --onefile --onefile-tempdir-spec="{PROGRAM_DIR}/.myapp_data" my_program.py

This would create a directory named .myapp_data inside the same directory as the my_program.exe (or my_program on Linux/macOS) and unpack the application there. This is perfect for creating truly self-contained applications where all data and temporary files reside alongside the executable.

Nuitka Commercial and Open Source

These features, like many enhancements to Nuitka, originated from a request by a Nuitka commercial customer. This highlights the close relationship between the commercial offerings and the open-source core. While commercial support helps drive development and ensures the long-term sustainability of Nuitka, the vast majority of features are made freely available to all users.

Give it a Try!

This change will be in 2.7 and is currently

We encourage you to try out these new features and let us know what you think! As always, bug reports, feature requests, and contributions are welcome on GitHub.

14 Feb 2025 11:00pm GMT

Django Weblog: DjangoCongress JP 2025 Announcement and Live Streaming!

DjangoCongress JP 2025, to be held on Saturday, February 22, 2025 at 10 am (Japan Standard Time), will be broadcast live!

It will be streamed on the following YouTube Live channels:

This year there will be talks not only about Django, but also about FastAPI and other asynchronous web topics. There will also be talks on Django core development, Django Software Foundation (DSF) governance, and other topics from around the world. Simultaneous translation will be provided in both English and Japanese.

Schedule

ROOM1
ROOM2

A public viewing of the event will also be held in Tokyo. A reception will also be held, so please check the following connpass page if you plan to attend.

Registration (connpass page): DjangoCongress JP 2025パブリックビューイング

14 Feb 2025 10:12pm GMT

Eli Bendersky: Decorator JITs - Python as a DSL

Spend enough time looking at Python programs and packages for machine learning, and you'll notice that the "JIT decorator" pattern is pretty popular. For example, this JAX snippet:

import jax.numpy as jnp
import jax

@jax.jit
def add(a, b):
  return jnp.add(a, b)

# Use "add" as a regular Python function
... = add(...)

Or the Triton language for writing GPU kernels directly in Python:

import triton
import triton.language as tl

@triton.jit
def add_kernel(x_ptr,
               y_ptr,
               output_ptr,
               n_elements,
               BLOCK_SIZE: tl.constexpr):
    pid = tl.program_id(axis=0)
    block_start = pid * BLOCK_SIZE
    offsets = block_start + tl.arange(0, BLOCK_SIZE)
    mask = offsets < n_elements
    x = tl.load(x_ptr + offsets, mask=mask)
    y = tl.load(y_ptr + offsets, mask=mask)
    output = x + y
    tl.store(output_ptr + offsets, output, mask=mask)

In both cases, the function decorated with jit doesn't get executed by the Python interpreter in the normal sense. Instead, the code inside is more like a DSL (Domain Specific Language) processed by a special purpose compiler built into the library (JAX or Triton). Another way to think about it is that Python is used as a meta language to describe computations.

In this post I will describe some implementation strategies used by libraries to make this possible.

Preface - where we're going

The goal is to explain how different kinds of jit decorators work by using a simplified, educational example that implements several approaches from scratch. All the approaches featured in this post will be using this flow:

Flow of Python source --> Expr IR --> LLVM IR --> Execution Expr IR --> LLVM IR --> Execution" /> Expr IR --> LLVM IR --> Execution" class="align-center" src="https://eli.thegreenplace.net/images/2025/decjit-python.png" />

These are the steps that happen when a Python function wrapped with our educational jit decorator is called:

  1. The function is translated to an "expression IR" - Expr.
  2. This expression IR is converted to LLVM IR.
  3. Finally, the LLVM IR is JIT-executed.

Steps (2) and (3) use llvmlite; I've written about llvmlite before, see this post and also the pykaleidoscope project. For an introduction to JIT compilation, be sure to read this and maybe also the series of posts starting here.

First, let's look at the Expr IR. Here we'll make a big simplification - only supporting functions that define a single expression, e.g.:

def expr2(a, b, c, d):
    return (a + d) * (10 - c) + b + d / c

Naturally, this can be easily generalized - after all, LLVM IR can be used to express fully general computations.

Here are the Expr data structures:

class Expr:
    pass

@dataclass
class ConstantExpr(Expr):
    value: float

@dataclass
class VarExpr(Expr):
    name: str
    arg_idx: int

class Op(Enum):
    ADD = "+"
    SUB = "-"
    MUL = "*"
    DIV = "/"

@dataclass
class BinOpExpr(Expr):
    left: Expr
    right: Expr
    op: Op

To convert an Expr into LLVM IR and JIT-execute it, we'll use this function:

def llvm_jit_evaluate(expr: Expr, *args: float) -> float:
    """Use LLVM JIT to evaluate the given expression with *args.

    expr is an instance of Expr. *args are the arguments to the expression, each
    a float. The arguments must match the arguments the expression expects.

    Returns the result of evaluating the expression.
    """
    llvm.initialize()
    llvm.initialize_native_target()
    llvm.initialize_native_asmprinter()
    llvm.initialize_native_asmparser()

    cg = _LLVMCodeGenerator()
    modref = llvm.parse_assembly(str(cg.codegen(expr, len(args))))

    target = llvm.Target.from_default_triple()
    target_machine = target.create_target_machine()
    with llvm.create_mcjit_compiler(modref, target_machine) as ee:
        ee.finalize_object()
        cfptr = ee.get_function_address("func")
        cfunc = CFUNCTYPE(c_double, *([c_double] * len(args)))(cfptr)
        return cfunc(*args)

It uses the _LLVMCodeGenerator class to actually generate LLVM IR from Expr. This process is straightforward and covered extensively in the resources I linked to earlier; take a look at the full code here.

My goal with this architecture is to make things simple, but not too simple. On one hand - there are several simplifications: only single expressions are supported, very limited set of operators, etc. It's very easy to extend this! On the other hand, we could have just trivially evaluated the Expr without resorting to LLVM IR; I do want to show a more complete compilation pipeline, though, to demonstrate that an arbitrary amount of complexity can be hidden behind these simple interfaces.

With these building blocks in hand, we can review the strategies used by jit decorators to convert Python functions into Exprs.

AST-based JIT

Python comes with powerful code reflection and introspection capabilities out of the box. Here's the astjit decorator:

def astjit(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        if kwargs:
            raise ASTJITError("Keyword arguments are not supported")
        source = inspect.getsource(func)
        tree = ast.parse(source)

        emitter = _ExprCodeEmitter()
        emitter.visit(tree)
        return llvm_jit_evaluate(emitter.return_expr, *args)

    return wrapper

This is a standard Python decorator. It takes a function and returns another function that will be used in its place (functools.wraps ensures that function attributes like the name and docstring of the wrapper match the wrapped function).

Here's how it's used:

from astjit import astjit

@astjit
def some_expr(a, b, c):
    return b / (a + 2) - c * (b - a)

print(some_expr(2, 16, 3))

After astjit is applied to some_expr, what some_expr holds is the wrapper. When some_expr(2, 16, 3) is called, the wrapper is invoked with *args = [2, 16, 3].

The wrapper obtains the AST of the wrapped function, and then uses _ExprCodeEmitter to convert this AST into an Expr:

class _ExprCodeEmitter(ast.NodeVisitor):
    def __init__(self):
        self.args = []
        self.return_expr = None
        self.op_map = {
            ast.Add: Op.ADD,
            ast.Sub: Op.SUB,
            ast.Mult: Op.MUL,
            ast.Div: Op.DIV,
        }

    def visit_FunctionDef(self, node):
        self.args = [arg.arg for arg in node.args.args]
        if len(node.body) != 1 or not isinstance(node.body[0], ast.Return):
            raise ASTJITError("Function must consist of a single return statement")
        self.visit(node.body[0])

    def visit_Return(self, node):
        self.return_expr = self.visit(node.value)

    def visit_Name(self, node):
        try:
            idx = self.args.index(node.id)
        except ValueError:
            raise ASTJITError(f"Unknown variable {node.id}")
        return VarExpr(node.id, idx)

    def visit_Constant(self, node):
        return ConstantExpr(node.value)

    def visit_BinOp(self, node):
        left = self.visit(node.left)
        right = self.visit(node.right)
        try:
            op = self.op_map[type(node.op)]
            return BinOpExpr(left, right, op)
        except KeyError:
            raise ASTJITError(f"Unsupported operator {node.op}")

When _ExprCodeEmitter finishes visiting the AST it's given, its return_expr field will contain the Expr representing the function's return value. The wrapper then invokes llvm_jit_evaluate with this Expr.

Note how our decorator interjects into the regular Python execution process. When some_expr is called, instead of the standard Python compilation and execution process (code is compiled into bytecode, which is then executed by the VM), we translate its code to our own representation and emit LLVM from it, and then JIT execute the LLVM IR. While it seems kinda pointless in this artificial example, in reality this means we can execute the function's code in any way we like.

AST JIT case study: Triton

This approach is almost exactly how the Triton language works. The body of a function decorated with @triton.jit gets parsed to a Python AST, which then - through a series of internal IRs - ends up in LLVM IR; this in turn is lowered to PTX by the NVPTX LLVM backend. Then, the code runs on a GPU using a standard CUDA pipeline.

Naturally, the subset of Python that can be compiled down to a GPU is limited; but it's sufficient to run performant kernels, in a language that's much friendlier than CUDA and - more importantly - lives in the same file with the "host" part written in regular Python. For example, if you want testing and debugging, you can run Triton in "interpreter mode" which will just run the same kernels locally on a CPU.

Note that Triton lets us import names from the triton.language package and use them inside kernels; these serve as the intrinsics for the language - special calls the compiler handles directly.

Bytecode-based JIT

Python is a fairly complicated language with a lot of features. Therefore, if our JIT has to support some large portion of Python semantics, it may make sense to leverage more of Python's own compiler. Concretely, we can have it compile the wrapped function all the way to bytecode, and start our translation from there.

Here's the bytecodejit decorator that does just this [1]:

def bytecodejit(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        if kwargs:
            raise BytecodeJITError("Keyword arguments are not supported")

        expr = _emit_exprcode(func)
        return llvm_jit_evaluate(expr, *args)

    return wrapper


def _emit_exprcode(func):
    bc = func.__code__
    stack = []
    for inst in dis.get_instructions(func):
        match inst.opname:
            case "LOAD_FAST":
                idx = inst.arg
                stack.append(VarExpr(bc.co_varnames[idx], idx))
            case "LOAD_CONST":
                stack.append(ConstantExpr(inst.argval))
            case "BINARY_OP":
                right = stack.pop()
                left = stack.pop()
                match inst.argrepr:
                    case "+":
                        stack.append(BinOpExpr(left, right, Op.ADD))
                    case "-":
                        stack.append(BinOpExpr(left, right, Op.SUB))
                    case "*":
                        stack.append(BinOpExpr(left, right, Op.MUL))
                    case "/":
                        stack.append(BinOpExpr(left, right, Op.DIV))
                    case _:
                        raise BytecodeJITError(f"Unsupported operator {inst.argval}")
            case "RETURN_VALUE":
                if len(stack) != 1:
                    raise BytecodeJITError("Invalid stack state")
                return stack.pop()
            case "RESUME" | "CACHE":
                # Skip nops
                pass
            case _:
                raise BytecodeJITError(f"Unsupported opcode {inst.opname}")

The Python VM is a stack machine; so we emulate a stack to convert the function's bytecode to Expr IR (a bit like an RPN evaluator). As before, we then use our llvm_jit_evaluate utility function to lower Expr to LLVM IR and JIT execute it.

Using this JIT is as simple as the previous one - just swap astjit for bytecodejit:

from bytecodejit import bytecodejit

@bytecodejit
def some_expr(a, b, c):
    return b / (a + 2) - c * (b - a)

print(some_expr(2, 16, 3))

Bytecode JIT case study: Numba

Numba is a compiler for Python itself. The idea is that you can speed up specific functions in your code by slapping a numba.njit decorator on them. What happens next is similar in spirit to our simple bytecodejit, but of course much more complicated because it supports a very large portion of Python semantics.

Numba uses the Python compiler to emit bytecode, just as we did; it then converts it into its own IR, and then to LLVM using llvmlite [2].

By starting with the bytecode, Numba makes its life easier (no need to rewrite the entire Python compiler). On the other hand, it also makes some analyses harder, because by the time we're in bytecode, a lot of semantic information existing in higher-level representations is lost. For example, Numba has to sweat a bit to recover control flow information from the bytecode (by running it through a special interpreter first).

Tracing-based JIT

The two approaches we've seen so far are similar in many ways - both rely on Python's introspection capabilities to compile the source code of the JIT-ed function to some extent (one to AST, the other all the way to bytecode), and then work on this lowered representation.

The tracing strategy is very different. It doesn't analyze the source code of the wrapped function at all - instead, it traces its execution by means of specially-boxed arguments, leveraging overloaded operators and functions, and then works on the generated trace.

The code implementing this for our smile demo is surprisingly compact:

def tracejit(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        if kwargs:
            raise TraceJITError("Keyword arguments are not supported")

        argspec = inspect.getfullargspec(func)

        argboxes = []
        for i, arg in enumerate(args):
            if i >= len(argspec.args):
                raise TraceJITError("Too many arguments")
            argboxes.append(_Box(VarExpr(argspec.args[i], i)))

        out_box = func(*argboxes)
        return llvm_jit_evaluate(out_box.expr, *args)

    return wrapper

Each runtime argument of the wrapped function is assigned a VarExpr, and that is placed in a _Box, a placeholder class which lets us do operator overloading:

@dataclass
class _Box:
    expr: Expr

_Box.__add__ = _Box.__radd__ = _register_binary_op(Op.ADD)
_Box.__sub__ = _register_binary_op(Op.SUB)
_Box.__rsub__ = _register_binary_op(Op.SUB, reverse=True)
_Box.__mul__ = _Box.__rmul__ = _register_binary_op(Op.MUL)
_Box.__truediv__ = _register_binary_op(Op.DIV)
_Box.__rtruediv__ = _register_binary_op(Op.DIV, reverse=True)

The remaining key function is _register_binary_op:

def _register_binary_op(opcode, reverse=False):
    """Registers a binary opcode for Boxes.

    If reverse is True, the operation is registered as arg2 <op> arg1,
    instead of arg1 <op> arg2.
    """

    def _op(arg1, arg2):
        if reverse:
            arg1, arg2 = arg2, arg1
        box1 = arg1 if isinstance(arg1, _Box) else _Box(ConstantExpr(arg1))
        box2 = arg2 if isinstance(arg2, _Box) else _Box(ConstantExpr(arg2))
        return _Box(BinOpExpr(box1.expr, box2.expr, opcode))

    return _op

To understand how this works, consider this trivial example:

@tracejit
def add(a, b):
    return a + b

print(add(1, 2))

After the decorated function is defined, add holds the wrapper function defined inside tracejit. When add(1, 2) is called, the wrapper runs:

  1. For each argument of add itself (that is a and b), it creates a new _Box holding a VarExpr. This denotes a named variable in the Expr IR.
  2. It then calls the wrapped function, passing it the boxes as runtime parameters.
  3. When (the wrapped) add runs, it invokes a + b. This is caught by the overloaded __add__ operator of _Box, and it creates a new BinOpExpr with the VarExprs representing a and b as children. This BinOpExpr is then returned [3].
  4. The wrapper unboxes the returned Expr and passes it to llvm_jit_evaluate to emit LLVM IR from it and JIT execute it with the actual runtime arguments of the call: 1, 2.

This might be a little mind-bending at first, because there are two different executions that happen:

  • The first is calling the wrapped add function itself, letting the Python interpreter run it as usual, but with special arguments that build up the IR instead of doing any computations. This is the tracing step.
  • The second is lowering this IR our tracing step built into LLVM IR and then JIT executing it with the actual runtime argument values 1, 2; this is the execution step.

This tracing approach has some interesting characteristics. Since we don't have to analyze the source of the wrapped functions but only trace through the execution, we can "magically" support a much richer set of programs, e.g.:

@tracejit
def use_locals(a, b, c):
    x = a + 2
    y = b - a
    z = c * x
    return y / x - z

print(use_locals(2, 8, 11))

This just works with our basic tracejit. Since Python variables are placeholders (references) for values, our tracing step is oblivious to them - it follows the flow of values. Another example:

@tracejit
def use_loop(a, b, c):
    result = 0
    for i in range(1, 11):
        result += i
    return result + b * c

print(use_loop(10, 2, 3))

This also just works! The created Expr will be a long chain of BinExpr additions of i's runtime values through the loop, added to the BinExpr for b * c.

This last example also leads us to a limitation of the tracing approach; the loop cannot be data-dependent - it cannot depend on the function's arguments, because the tracing step has no concept of runtime values and wouldn't know how many iterations to run through; or at least, it doesn't know this unless we want to perform the tracing run for every runtime execution [4].

The tracing approach is useful in several domains, most notably automatic differentiation (AD). For a slightly deeper taste, check out my radgrad project.

Tracing JIT case study: JAX

The JAX ML framework uses a tracing approach very similar to the one described here. The first code sample in this post shows the JAX notation. JAX cleverly wraps Numpy with its own version which is traced (similar to our _Box, but JAX calls these boxes "tracers"), letting you write regular-feeling Numpy code that can be JIT optimized and executed on accelerators like GPUs and TPUs via XLA. JAX's tracer builds up an underlying IR (called jaxpr) which can then be emitted to XLA ops and passed to XLA for further lowering and execution.

For a fairly deep overview of how JAX works, I recommend reading the autodidax doc.

As mentioned earlier, JAX has some limitations with things like data-dependent control flow in native Python. This won't work, because there's control flow that depends on a runtime value (count):

import jax

@jax.jit
def sum_datadep(a, b, count):
    total = a
    for i in range(count):
        total += b
    return total

print(sum_datadep(10, 3, 3))

When sum_datadep is executed, JAX will throw an exception, saying something like:

This concrete value was not available in Python because it depends on the value of the argument count.

As a remedy, JAX has its own built-in intrinsics from the jax.lax package. Here's the example rewritten in a way that actually works:

import jax
from jax import lax

@jax.jit
def sum_datadep_fori(a, b, count):
    def body(i, total):
        return total + b

    return lax.fori_loop(0, count, body, a)

fori_loop (and many other built-ins in the lax package) is something JAX can trace through, generating a corresponding XLA operation (XLA has support for While loops, to which this lax.fori_loop can be lowered).

The tracing approach has clear benefits for JAX as well; because it only cares about the flow of values, it can handle arbitrarily complicated Python code, as long as the flow of values can be traced. Just like the local variables and data-independent loops shown earlier, but also things like closures. This makes meta-programming and templating easy [5].

Code

The full code for this post is available on GitHub.


[1] Once again, this is a very simplified example. A more realistic translator would have to support many, many more Python bytecode instructions.
[2] In fact, llvmlite itself is a Numba sub-project and is maintained by the Numba team, for which I'm grateful!
[3] For a fun exercise, try adding constant folding to the wrapped _op: when both its arguments are constants (not boxes), instead placing each in a _Box(ConstantExpr(...)), it could perform the mathematical operation on them and return a single constant box. This is a common optimization in compilers!
[4]

In all the JIT approaches showed in this post, the expectation is that compilation happens once, but the compiled function can be executed many times (perhaps in a loop). This means that the compilation step cannot depend on the runtime values of the function's arguments, because it has no access to them. You could say that it does, but that's just for the very first time the function is run (in the tracing approach); it has no way of knowing their values the next times the function will run.

JAX has some provisions for cases where a function is invoked with a small set of runtime values and we want to separately JIT each of them.

[5] A reader pointed out that TensorFlow's AutoGraph feature combines the AST and tracing approaches. TF's eager mode performs tracing, but it also uses AST analyses to rewrite Python loops and conditions into builtins like tf.cond and tf.while_loop.

14 Feb 2025 9:49pm GMT

feedDjango community aggregator: Community blog posts

Django News - DjangoCon US Call for Proposals - Feb 14th 2025

News

Python 3.14.0 alpha 5 is out

Python 3.14.0a5 is the fifth of seven planned alpha releases.

blogspot.com

DjangoCon US Call for Proposals

The CFP is now open until April 27th. The earlier you submit, the better!

pretalx.com

DSF member of the month - Lily Foote

Lily is a long-time contributor to Django core, especially on the ORM, and is currently a member of the Django 6.x Steering Council.

djangoproject.com

I'm excited to join the Sovereign Tech Fellowship

Hugo van Kemenade is now a full-time open source developer, working on Python, with a focus on CPython, including as release manager for Python 3.14 and 3.15.

hugovk.dev

Updates to Django

Today 'Updates to Django' is presented by Abigail Afi Gbadago from the DSF Board!

Last week we had 13 pull requests merged into Django by 9 different contributors - including 3 first-time contributors! Congratulations to Andrew, Brian Nettleton and Arnaldo Govene for having their first commits merged into Django - welcome on board!🎊

This Week's Highlights:

Django Newsletter

Sponsored Link 1

Hiring Jr. Web Services Engineer

This position is for someone who can bring their python software development experience to support Playdate, Game Publishing, and our Apps! You would be responsible for contributing to and maintaining our growing flock of Flask and Django web applications.

Additionally, Panic hires interns every summer via The Script, designed for college students with underrepresented backgrounds entering freshman through senior years.

panic.com

Articles

Django in government - Thibaud's blog

DSF President Thibaud Colas focuses on almost 1,000 Django websites used in governments around the world.

thib.me

Pytest, A Practical Guide

Andrew Mshar provides an introduction to key pytest features, including fixtures, flags, markers, and Django-specific tools.

programmingmylife.com

Things I'd Like to See in a DjangoCon US 2025 Talk

Former DjangoCon US program chair Drew Winstel has a detailed list of suggested topics for DjangoCon US talks this year. The CFP is now open.

winstel.dev

Prefixed Parameters for Django querystring tag

An overview of Django 5.1's new querystring tag and how to add support for prefixed parameters.

informatikamihelac.com

Jump to dev

A clever Django middleware lets developers paste production URLs into their local server, automatically stripping out the domain for quick issue reproduction.

kodare.net

Events

DjangoCongress JP 2025

This online event is coming soon! Feb, 22, 2025.

djangocongress.jp

Tutorials

Django MongoDB Backend Quickstart

Learn how to connect to a MongoDB deployment, ensure your deployment is hosted on MongoDB Atlas, and interact with the data using simple CRUD operations.

dev.to

Videos

uv and Django - Managing Django Projects & Dependencies with uv!

This video dives into using uv and Django together for project and dependency management, including how to install packages into different groups and run Django commands.

youtu.be

Sponsored Link 2

Django & Wagtail Hosting for just $9/month

New year, new site! Finally put those domain names to use and launch that side-project, blog, or personal site in 2025. Lock in this limited time pricing by Feb 28.

codered.cloud

Django News Jobs

Full-Stack engineer at Exoscale

Jr. Web Services Engineer at Panic (featured)

Senior Backend Engineer at BactoBio

Django Newsletter

Projects

safwanrahman/django-webpush

Web Push Notification Package for Django.

github.com

OmenApps/django-pint-field

Store, validate, and convert physical quantities in Django using Pint.

github.com


This RSS feed is published on https://django-news.com/. You can also subscribe via email.

14 Feb 2025 5:00pm GMT

13 Feb 2025

feedDjango community aggregator: Community blog posts

Prefixed Parameters for Django querystring tag

An overview of Django 5.1's new querystring tag and how to add support for prefixed parameters.

13 Feb 2025 8:05pm GMT

15 Jan 2025

feedPlanet Twisted

Glyph Lefkowitz: Small PINPal Update

Today on stream, I updated PINPal to fix the memorization algorithm.

If you haven't heard of PINPal before, it is a vault password memorization tool. For more detail on what that means, you can check it out the README, and why not give it a ⭐ while you're at it.


As I started writing up an update post I realized that I wanted to contextualize it a bit more, because it's a tool I really wish were more popular. It solves one of those small security problems that you can mostly ignore, right up until the point where it's a huge problem and it's too late to do anything about it.

In brief, PINPal helps you memorize new secure passcodes for things you actually have to remember and can't simply put into your password manager, like the password to your password manager, your PC user account login, your email account1, or the PIN code to your phone or debit card.

Too often, even if you're properly using a good password manager for your passwords, you'll be protecting it with a password optimized for memorability, which is to say, one that isn't random and thus isn't secure. But I have also seen folks veer too far in the other direction, trying to make a really secure password that they then forget right after switching to a password manager. Forgetting your vault password can also be a really big deal, making you do password resets across every app you've loaded into it so far, so having an opportunity to practice it periodically is important.

PINPal uses spaced repetition to ensure that you remember the codes it generates.

While periodic forced password resets are a bad idea, if (and only if!) you can actually remember the new password, it is a good idea to get rid of old passwords eventually - like, let's say, when you get a new computer or phone. Doing so reduces the risk that a password stored somewhere on a very old hard drive or darkweb data dump is still floating around out there, forever haunting your current security posture. If you do a reset every 2 years or so, you know you've never got more than 2 years of history to worry about.

PINPal is also particularly secure in the way it incrementally generates your password; the computer you install it on only ever stores the entire password in memory when you type it in. It stores even the partial fragments that you are in the process of memorizing using the secure keyring module, avoiding plain-text whenever possible.


I've been using PINPal to generate and memorize new codes for a while, just in case2, and the change I made today was because encountered a recurring problem. The problem was, I'd forget a token after it had been hidden, and there was never any going back. The moment that a token was hidden from the user, it was removed from storage, so you could never get a reminder. While I've successfully memorized about 10 different passwords with it so far, I've had to delete 3 or 4.

So, in the updated algorithm, the visual presentation now hides tokens in the prompt several memorizations before they're removed. Previously, if the password you were generating was 'hello world', you'd see hello world 5 times or so, times, then •••• world; if you ever got it wrong past that point, too bad, start over. Now, you'll see hello world, then °°°° world, then after you have gotten the prompt right without seeing the token a few times, you'll see •••• world after the backend has locked it in and it's properly erased from your computer.

If you get the prompt wrong, breaking your streak reveals the recently-hidden token until you get it right again. I also did a new release on that same livestream, so if this update sounds like it might make the memorization process more appealing, check it out via pip install pinpal today.

Right now this tool is still only extremely for a specific type of nerd - it's command-line only, and you probably need to hand-customize your shell prompt to invoke it periodically. But I'm working on making it more accessible to a broader audience. It's open source, of course, so you can feel free to contribute your own code!

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more things like it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!


  1. Your email account password can be stored in your password manager, of course, but given that email is the root-of-trust reset factor for so many things, being able to remember that password is very helpful in certain situations.

  2. Funny story: at one point, Apple had an outage which made it briefly appear as if a lot of people needed to reset their iCloud passwords, myself included. Because I'd been testing PINPal a bunch, I actually had several highly secure random passwords already memorized. It was a strange feeling to just respond to the scary password reset prompt with a new, highly secure password and just continue on with my day secure in the knowledge I wouldn't forget it.

15 Jan 2025 12:54am GMT

10 Jan 2025

feedPlanet Twisted

Glyph Lefkowitz: The “Active Enum” Pattern

Have you ever written some Python code that looks like this?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
from enum import Enum, auto

class SomeNumber(Enum):
    one = auto()
    two = auto()
    three = auto()

def behavior(number: SomeNumber) -> int:
    match number:
        case SomeNumber.one:
            print("one!")
            return 1
        case SomeNumber.two:
            print("two!")
            return 2
        case SomeNumber.three:
            print("three!")
            return 3

That is to say, have you written code that:

  1. defined an enum with several members
  2. associated custom behavior, or custom values, with each member of that enum,
  3. needed one or more match / case statements (or, if you've been programming in Python for more than a few weeks, probably a big if/elif/elif/else tree) to do that association?

In this post, I'd like to submit that this is an antipattern; let's call it the "passive enum" antipattern.

For those of you having a generally positive experience organizing your discrete values with enums, it may seem odd to call this an "antipattern", so let me first make something clear: the path to a passive enum is going in the correct direction.

Typically - particularly in legacy code that predates Python 3.4 - one begins with a value that is a bare int constant, or maybe a str with some associated values sitting beside in a few global dicts.

Starting from there, collecting all of your values into an enum at all is a great first step. Having an explicit listing of all valid values and verifying against them is great.

But, it is a mistake to stop there. There are problems with passive enums, too:

  1. The behavior can be defined somewhere far away from the data, making it difficult to:
    1. maintain an inventory of everywhere it's used,
    2. update all the consumers of the data when the list of enum values changes, and
    3. learn about the different usages as a consumer of the API
  2. Logic may be defined procedurally (via if/elif or match) or declaratively (via e.g. a dict whose keys are your enum and whose values are the required associated value).
    1. If it's defined procedurally, it can be difficult to build tools to interrogate it, because you need to parse the AST of your Python program. So it can be difficult to build interactive tools that look at the associated data without just calling the relevant functions.
    2. If it's defined declaratively, it can be difficult for existing tools that do know how to interrogate ASTs (mypy, flake8, Pyright, ruff, et. al.) to make meaningful assertions about it. Does your linter know how to check that a dict whose keys should be every value of your enum is complete?

To refactor this, I would propose a further step towards organizing one's enum-oriented code: the active enum.

An active enum is one which contains all the logic associated with the first-party provider of the enum itself.

You may recognize this as a more generalized restatement of the object-oriented lens on the principle of "separation of concerns". The responsibilities of a class ought to be implemented as methods on that class, so that you can send messages to that class via method calls, and it's up to the class internally to implement things. Enums are no different.

More specifically, you might notice it as a riff on the Active Nothing pattern described in this excellent talk by Sandi Metz, and, yeah, it's the same thing.

The first refactoring that we can make is, thus, to mechanically move the method from an external function living anywhere, to a method on SomeNumber . At least like this, we present an API to consumers externally that shows that SomeNumber has a behavior method that can be invoked.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
from enum import Enum, auto

class SomeNumber(Enum):
    one = auto()
    two = auto()
    three = auto()

    def behavior(self) -> int:
        match self:
            case SomeNumber.one:
                print("one!")
                return 1
            case SomeNumber.two:
                print("two!")
                return 2
            case SomeNumber.three:
                print("three!")
                return 3

However, this still leaves us with a match statement that repeats all the values that we just defined, with no particular guarantee of completeness. To continue the refactoring, what we can do is change the value of the enum itself into a simple dataclass to structurally, by definition, contain all the fields we need:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
from dataclasses import dataclass
from enum import Enum
from typing import Callable

@dataclass(frozen=True)
class NumberValue:
    result: int
    effect: Callable[[], None]

class SomeNumber(Enum):
    one = NumberValue(1, lambda: print("one!"))
    two = NumberValue(2, lambda: print("two!"))
    three = NumberValue(3, lambda: print("three!"))

    def behavior(self) -> int:
        self.value.effect()
        return self.value.result

Here, we give SomeNumber members a value of NumberValue, a dataclass that requires a result: int and an effect: Callable to be constructed. Mypy will properly notice that if x is a SomeNumber, that x will have the type NumberValue and we will get proper type checking on its result (a static value) and effect (some associated behaviors)1.

Note that the implementation of behavior method - still conveniently discoverable for callers, and with its signature unchanged - is now vastly simpler.

But what about...

Lookups?

You may be noticing that I have hand-waved over something important to many enum users, which is to say, by-value lookup. enum.auto will have generated int values for one, two, and three already, and by transforming those into NumberValue instances, I can no longer do SomeNumber(1).

For the simple, string-enum case, one where you might do class MyEnum: value = "value" so that you can do name lookups via MyEnum("value"), there's a simple solution: use square brackets instead of round ones. In this case, with no matching strings in sight, SomeNumber["one"] still works.

But, if we want to do integer lookups with our dataclass version here, there's a simple one-liner that will get them back for you; and, moreover, will let you do lookups on whatever attribute you want:

1
by_result = {each.value.result: each for each in SomeNumber}

enum.Flag?

You can do this with Flag more or less unchanged, but in the same way that you can't expect all your list[T] behaviors to be defined on T, the lack of a 1-to-1 correspondence between Flag instances and their values makes it more complex and out of scope for this pattern specifically.

3rd-party usage?

Sometimes an enum is defined in library L and used in application A, where L provides the data and A provides the behavior. If this is the case, then some amount of version shear is unavoidable; this is a situation where the data and behavior have different vendors, and this means that other means of abstraction are required to keep them in sync. Object-oriented modeling methods are for consolidating the responsibility for maintenance within a single vendor's scope of responsibility. Once you're not responsible for the entire model, you can't do the modeling over all of it, and that is perfectly normal and to be expected.

The goal of the Active Enum pattern is to avoid creating the additional complexity of that shear when it does not serve a purpose, not to ban it entirely.

A Case Study

I was inspired to make this post by a recent refactoring I did from a more obscure and magical2 version of this pattern into the version that I am presenting here, but if I am going to call passive enums an "antipattern" I feel like it behooves me to point at an example outside of my own solo work.

So, for a more realistic example, let's consider a package that all Python developers will recognize from their day-to-day work, python-hearthstone, the Python library for parsing the data files associated with Blizzard's popular computerized collectible card game Hearthstone.

As I'm sure you already know, there are a lot of enums in this library, but for one small case study, let's look a few of the methods in hearthstone.enums.GameType.

GameType has already taken the "step 1" in the direction of an active enum, as I described above: as_bnet is an instancemethod on GameType itself, making it at least easy to see by looking at the class definition what operations it supports. However, in the implementation of that method (among many others) we can see the worst of both worlds:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
class GameType(IntEnum):
    def as_bnet(self, format: FormatType = FormatType.FT_STANDARD):
        if self == GameType.GT_RANKED:
            if format == FormatType.FT_WILD:
                return BnetGameType.BGT_RANKED_WILD
            elif format == FormatType.FT_STANDARD:
                return BnetGameType.BGT_RANKED_STANDARD
            # ...
            else:
                raise ValueError()
        # ...
        return {
            GameType.GT_UNKNOWN: BnetGameType.BGT_UNKNOWN,
            # ...
            GameType.GT_BATTLEGROUNDS_DUO_FRIENDLY: BnetGameType.BGT_BATTLEGROUNDS_DUO_FRIENDLY,
        }[self]

We have procedural code mixed with a data lookup table; raise ValueError mixed together with value returns. Overall, it looks like this might be hard to maintain this going forward, or to see what's going on without a comprehensive understanding of the game being modeled. Of course for most python programmers that understanding can be assumed, but, still.

If GameType were refactored in the manner above3, you'd be able to look at the member definition for GT_RANKED and see a mapping of FormatType to BnetGameType, or GT_BATTLEGROUNDS_DUO_FRIENDLY to see an unconditional value of BGT_BATTLEGROUNDS_DUO_FRIENDLY. Given that this enum has 40 elements, with several renamed or removed, it seems reasonable to expect that more will be added and removed as the game is developed.

Conclusion

If you have large enums that change over time, consider placing the responsibility for the behavior of the values alongside the values directly, and any logic for processing the values as methods of the enum. This will allow you to quickly validate that you have full coverage of any data that is required among all the different members of the enum, and it will allow API clients a convenient surface to discover the capabilities associated with that enum.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!


  1. You can get even fancier than this, defining a typing.Protocol as your enum's value, but it's best to keep things simple and use a very simple dataclass container if you can.

  2. derogatory

  3. I did not submit such a refactoring as a PR before writing this post because I don't have full context for this library and I do not want to harass the maintainers or burden them with extra changes just to make a rhetorical point. If you do want to try that yourself, please file a bug first and clearly explain how you think it would benefit their project's maintainability, and make sure that such a PR would be welcome.

10 Jan 2025 10:37pm GMT

16 Dec 2024

feedPlanet Twisted

Glyph Lefkowitz: DANGIT

Over the last decade, it has become a common experience to be using a social media app, and to perceive that app as saying something specific to you. This manifests in statements like "Twitter thinks Rudy Giuliani has lost his mind", "Facebook is up in arms about DEI", "Instagram is going crazy for this new water bottle", "BlueSky loves this bigoted substack", or "Mastodon can't stop talking about Linux". Sometimes this will even be expressed with "the Internet" as a metonym for the speaker's preferred social media: "the Internet thinks that Kate Middleton is missing".

However, even the smallest of these networks comprises literal millions of human beings, speaking dozens of different languages, many of whom never interact with each other at all. The hot takes that you see from a certain excitable sub-community, on your particular timeline or "for you" page, are not necessarily representative of "the Internet" - at this point, a group that represents a significant majority of the entire human population.

If I may coin a phrase, I will refer to these as "Diffuse, Amorphous, Nebulous, Generalized Internet Takes", or DANGITs, which handily evokes the frustrating feeling of arguing against them.

A DANGIT is not really a new "internet" phenomenon: it is a specific expression of the availability heuristic.

If we look at our device and see a bunch of comments in our inbox, particularly if those comments have high salience via being recent, emotive, and repeated, we will naturally think that this is what The Internet thinks. However, just because we will naturally think this does not mean that we will accurately think it.

It is worth keeping this concept in mind when participating in public discourse because it leads to a specific type of communication breakdown. If you are arguing with a DANGIT, you will feel like you are arguing with someone with incredibly inconsistent, hypocritical, and sometimes even totally self-contradictory views. But to be self-contradictory, one needs to have a self. And if you are arguing with 9 different people from 3 different ideological factions, all making completely different points and not even taking time to agree on the facts beforehand, of course it's going to sound like cacophonous nonsense. You're arguing with the cacophony, it's just presented to you in a way that deceives you into thinking that it's one group.

There are subtle variations on this breakdown; for example, it can also make people's taste seem incoherent. If it seems like one week the Interior Designer internet loves stark Scandinavian minimalism, and the next week baroque Rococo styles are making a comeback, it might seem like The Internet has no coherent sense of taste, and these things don't go together. That's because it doesn't! Why would you expect it to?

Most likely, you are simply seeing some posts from minimalists, and then, separately, some posts from Rococo aficionados. Any particular person's feed may be dedicated to a specific, internally coherent viewpoint, aesthetic, or ideology, but if you dump them all into a blender to separate them from their context, of course they will look jumbled together.

This is what social media does. It is context collapse as a service. Even if you eliminate engagement-maximizing algorithms and view everything perfectly chronologically, even if you have the world's best trust & safety team making sure that there is nothing harmful and no disinformation, social media - like email - inherently remains that context-collapsing blender. There's no way for it not to be; if two people you follow, who do not follow and are not aware of each other, are both posting unrelated things at the same time, you're going to see them at around the same time.

Do not argue with a DANGIT. Discussions are the internet are famously Pyrrhic battles to begin with, but if you argue with a DANGIT it's not that you will achieve a Pyrrhic victory, you cannot possibly achieve any victory, because you are shadowboxing an imagined consensus where none exits.

You can't win against something that isn't there.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more things like it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor!

16 Dec 2024 10:58pm GMT

29 Nov 2024

feedPlanet Plone - Where Developers And Integrators Write

Maurits van Rees: Lightning talks Friday

Bonnie Tyler Sprint

On 12 August 2026 there is a total solar eclipse that can be seen from Valencia, Spain. So we organise a sprint there.

This conference

We had 291 participants, 234 in person and 57 online. 13 Brazilian states (that is all of them), 14 countries.

24.5 percent women, was 13% in 2013, so that has gone up, but we are not there yet. Thank you to PyLadies and Django Girls for making this happen.

We had more than 80 presenters, about 30 lightning talks, lots of talk in the hall ways.

Thanks also to the team!

Ramiro Luz: Yoga time

Yoga exercise.

Rikupekka: University case student portal

We have a student portal at the university. But mostly:

Welcome to Jyväskylä university in Finald for Plone conference 2025, October 13-19!

Jakob: Beethovensprint

26-30 mei 2025 in Bonn, Duitsland.

Afterwards, on May 30 and June 1 there will be FedCon in Bonn, a SciFi convention.

Piero/Victor: BYOUI

Add-ons first development with @plone/registry. See https://plone-registry.readthedocs.io/

It allows for development that is framework agnostic, so it is not only for Plone. It is around configuration that can be extended and injected, which is tricky in most javascript frameworks.

Imagine it.

Ana Dulce: 3D printing

For a difficult model I had trust the process, it took a week, but it worked.

Renan & Iza: Python Brasil

We organised the Python Brasil conference from 16 to 23 October this year in Rio de Janeiro.

Next year 21-27 October in São Paulo.

Erico: Python Cerrado

31 July to 2 August 2025 is the next Python Cerrado conference.

29 Nov 2024 10:25pm GMT

Maurits van Rees: Paul Roeland: The value of longevity

Link to talk information on Plone conference website.

I work for the Clean Clothes Campaign: https://cleanclothes.org/

After three large disasters in factories in 2012 and 2013 with over 1000 deaths, it took three years to get an agreement with clothes manufacturers to get 30 million dollar compensation. It does not bring lives back, but it helps the survivors.

See Open Supply Hub for open data that we collected, for checking which brands are produced in which factories.

Documenting history matters. Stories must be told.

The global closing industry is worth around 1.8 trillion dollars, in a country that would put them on the 12th place in the world. 75 million workers.

Our strongest weapon: backlinks. We have links from OECD, UN, wikipedia, school curriculum, books. Especially those last two don't change ever, so you should never change urls.

Plone: enable the sitemap, please, why not by default? Create a good robots.txt. I weekly check Google Search console, looking for broken links. Tag early, tag often, great tool, even if you have an AI do it.

Our website: started 1998 written in Notepad, 2004 Dreamweaver, 2006 Bluefish, 2010 Joomla, 2013 Plone 4, 2020 Castle CMS (opinionated distribution of Plone, but does not really exist anymore) 2024 Plone 6 with Volto Light Theme (work in progress). Thank you kitconcept for all the help, especially Jonas.

Migrations are painful. Along the years we used wget to csv to SQL to csv, Python script, "Franken-mogrifier", collective.exportimport.

Lessons learned: stable urls are awesome, migrations are painful. Please don't try to salvage CSS from your old site, just start fresh in your new system. Do not try to migrate composite pages or listings.

What if your website does not provide an export? Use wget, still works and is better than httrack. sed/awk/regex are your friend. archivebox (WARC).

Document your steps for your own sanity.

To manage json, jq or jello can be used. sq is a Swiss knife for json/sql/csv. emuto is a hybrid between jq and GraphQL.

Normalize import/export. We have `plone.exportimport` in core now.

In the future I would like a plone exporter script that accepts a regex and exports only matching pages. Switch backends: ZODB, relstorage, nick, quantum-db. Sitewide search/replace/sed. Sneakernet is useful in difficult countries where you cannot send data over the internet: so export to a usb stick.

A backup is only a backup if it regularly gets restored so you know that it works.

  • Keeping content and URL stability is a superpower.
  • Assuming that export/import/backup/restore/migration are rare occurrences, is wrong.
  • Quick export/import is very useful.

Do small migrations, treat it as maintenance. Don't be too far behind. Large migrations one every five years will be costly. Do a small migration every year. Do your part. Clients should also do their part, by budgeting this yearly. That is how budgeting works. Use every iteration to review custom code.

Make your sites live long and prosper.

29 Nov 2024 8:58pm GMT

Maurits van Rees: Fred van Dijk: Run Plone in containers on your own cluster with coolify.io

Link to talk information on Plone conference website.

Sorry, I ran out of time trying to set up https://coolify.io

So let's talk about another problem. Running applications (stacks) in containers is the future. Well: abstraction and isolation is the future, and containers is the current phase.

I am on the Plone A/I team, with Paul, Kim, Erico. All senior sysadmins, so we kept things running. In 2022 we worked on containerisation. Kubernetes was the kool kid then, but Docker Swarm was easier. Checkout Erico's training with new cookieplone templates.

Doing devops well is hard. You have a high workload, but still need to keep learning new stuff to keep up with what is changing.

I want to plug Coolify, which is a full open source product. "Self-hosting with super powers." The main developer, Andras Bacsal, believes in open source and 'hates' pay by usage cloud providers with a vengeance.

Coolify is still docker swarm. We also want Kubernetes support. But we still need sysadmins. Someone will still need to install coolify, and keep it updated.

I would like to run an online DevOps course somewhere January-March 2025. 4-6 meetings of 2 hours, maybe Friday afternoon. Talk through devops and sysadmin concepts, show docker swarm, try coolify, etc.

29 Nov 2024 7:58pm GMT