01 Jun 2026
Planet Python
death and gravity: DynamoDB crash course: part 3 – design patterns
This is the last part of a series covering core DynamoDB concepts. The goal is to help you understand idiomatic usage and trade-offs in under an hour.
In the first part, I summarized DynamoDB's main proposition to its users like so:
data modeling complexity is always preferable to complexity coming from infrastructure maintenance, availability, and scalability
Today, we're looking at the design patterns that help manage this complexity, making the most of its data model and features and working around its limits.
Contents
- Composite keys
- Single table design
- GSI overloading
- Partition key sharding
- Sparse indexes
- Base table indexes
- Optimistic locking
Composite keys #
Composite (aka synthetic) keys underpin most other patterns.
The idea is simple: keys don't have to be natural attributes of your data, they can be composed of other attributes that enable specific access patterns. This works both with table and index keys.
How do you compose keys? By string concatenation, of course! Careful with numbers though, they need padding to be useful in sort keys.
Example
To sort lexicographically by more than one attribute, you group them in a sort key, e.g. {Album}#{Song}.
Or, in single table design, you distinguish between item types by prefixing keys with the type, e.g. album#{Album}.
Or, in partition key sharding, you spread the load on a GSI partition by splitting one partition key into multiple ones, e.g. {Genre}#{shard}.
But denormalization has its trade-offs. For sort key {Album}#{Song}, should Album and Song also be separate attributes? If yes, you need to ensure they never change, but you can use them in indexes (e.g. a GSI with Album as primary key). If no, items can't become inconsistent, but you need to parse the key to get them.
This was inconvenient enough that DynamoDB finally added multi-attribute keys support to GSIs in 2025 (although not inconvenient enough to also add it to tables).
See also
Single table design #
The AWS guidance is to use as few tables as possible:
As a general rule, you should maintain as few tables as possible in a DynamoDB application. [...] A single table with inverted indexes can usually enable simple queries to create and retrieve the complex hierarchical data structures required by your application.
This culminates in single table design, where you put all entities in the same table, and tell them apart based on the key format, usually using a prefix. With this pattern, one DynamoDB table corresponds to a whole relational database.
The easiest way is to put items related to a top-level entity on the same partition. The main benefit is that joins with the top-level entity become trivial. A second one is that you can sometimes get different entity types in a single query, which can be both faster and cheaper (fewer queries; small items pack into fewer capacity units).
Example
You can group items related to an Artist on the same partition, with sort keys like artist, album#{Album}, and song#{Album}#{Song}.
# table Music (partition key: Artist, sort key: sk)
Solar Fields: !btree
'album#Leaving Home': { Genre: Electronic }
'artist': { Variations: [ Solarfields ] }
'song#Leaving Home#Air Song': { Duration: 741 }
'song#Leaving Home#Monogram': { Duration: 944 }
Besides getting items of a single type, you can also get artist details and albums in a single query (sk BETWEEN "album#" AND "artist").
But choose wisely - queries can have only one sort key condition, so you can't also get album details and songs in a single query with this schema; sort keys {Album} and {Album}#{Song} would do it, at the expense of the first query.
Sometimes, it can be useful to put some sub-entities on dedicated partitions, accepting that joins will have to be done in code.
Example
In the example above, a popular artist with lots of songs can lead to:
- throttling due to partition throughput limits
- slow list songs for artist due to sequential paginated queries
Perhaps it's better to put the songs in each album on separate partitions:
- partition key
artist#{Artist}, sort keyartistoralbum#{Album} - partition key
song#{Artist}#{Album}, sort key{Song}
# table Music (partition key: pk, sort key: sk)
'artist#Solar Fields': !btree
'album#Leaving Home': { Genre: Electronic }
'artist': { Variations: [ Solarfields ] }
'song#Solar Fields#Leaving Home': !btree
'Air Song': { Duration: 741 }
'Monogram': { Duration: 944 }
This spreads the load onto multiple partitions, which should fix throttling.
The downside is that list songs for artist is now a two-step operation: first one query for the albums, then one query per album for the songs. The upside is that the per-album queries can be done in parallel, which wasn't possible before.
A consequence of this design is that you need a GSI to list items of a specific type (otherwise, you have to do a full table scan). Of note, exceeding the GSI partition throughput limit will cause write throttling on the base table; in the absence of a natural high-cardinality GSI partition key, sharding or some other composite key can help.
A final benefit of using a single table is better utilization with provisioned mode: usage gets averaged across entities and tends to be smoother, and spikes can share the same spare capacity.
See also
- NoSQL design
- Data modeling foundations # Single table design
- Relational modeling # JOIN operations
- (blog) Single-table vs. multi-table design in Amazon DynamoDB
- (unofficial) The What, Why, and When of Single-Table Design with DynamoDB
GSI overloading #
GSI overloading is just single table design for indexes - you put different values in the GSI key attributes, depending on item type. This way you can index more attributes than the 20 GSIs per table quota, and it can be cheaper too, since, like with tables, fewer indexes make better use of spare provisioned capacity.
Example
For a table that contains both artist and album items, a single GSI can be used for entirely different purposes:
- artist: partition key
artist#{Country}- list artists by country - album: partition key
album#{Genre}- list albums by genre
# table Music (partition key: Artist, sort key: sk)
2 Bit Pie: !btree
'album#2 Pie Island': { gsi1pk: 'album#Electronic' }
'artist': { gsi1pk: 'artist#United Kingdom' }
Ishome: !btree
'album#Confession': { gsi1pk: 'album#Electronic' }
'artist': { gsi1pk: 'artist#Russia' }
# GSI GSI1 (partition key: gsi1pk, sort key: Artist)
'artist#United Kingdom': !btree
2 Bit Pie: { sk: 'artist' }
'artist#Russia': !btree
Ishome: { sk: 'artist' }
'album#Electronic': !btree
2 Bit Pie: { sk: 'album#2 Pie Island' }
Ishome: { sk: 'album#Confession' }
See also
Partition key sharding #
Sometimes, a partition key composed of multiple natural attributes is not enough to spread the load evenly across partitions; you can deal with this by putting items with the same natural attributes on multiple partitions.
So, what partition key should you use? One option is to use a random suffix from a known range; this allows you to list items for a natural attribute value by doing multiple queries, one for each suffix.
Example
For a table of songs, using Album as the partition key won't work, since not all songs are released on an album; Artist always has a value, but some artists have hundreds or even thousands of songs, which can lead to throttling.
Instead, we can use {Artist}#{randrange(10)} as partition key, which allows ten times as many items before we reach throughput limits. To list an artist's songs:
for shard in range(10):
for item in dynamodb.query(f"{artist}#{shard}"):
yield item
A downside of random suffixes is that you can't get a specific item, because you don't know what its suffix is. A better option is to calculate the suffix from an attribute that you do know, for example using its hash modulo N.
Example
With primary key {Artist}#{hash(Song) % 10)}, we can get a song like this:
def hash(s):
return int.from_bytes(sha256(s.encode()).digest())
shard = hash(song_title) % 10
dynamodb.get_item(f"{artist}#{shard}", song_title)
A lot of times you need to list items by a low-cardinality attribute, so sharding may be even more important for GSIs.
Example
Assuming dedicated album items, you can list all the albums by putting them in a single GSI partition key called albums, but this will definitely cause throttling.
To avoid it, you can use GSI partition key album#{hash(Album} % 100} if you don't care about the order, or something like album#{Album[:2].lower()} if you do (but likely more sophistication is needed - th will be a very common album title prefix, and some album titles don't contain letters at all).
Even if throttling is not an issue (e.g. single infrequent reader), sharding allows you to query multiple partitions in parallel, which can speed up getting the entire result set.
So, how many shards should you have? That depends on the number, size, and how often you access the items, and is also a trade-off - too many shards means additional queries and latency, too few shards means you still overload the partitions sometimes.
Importantly, increasing the number of shards is non-trivial. For tables, you usually need to rebalance the items in place. For indexes, it's cleaner to move to a new index, or if you just need to list items by type, you can put all new items on new shards.
Regardless, you have to support it in code, do a backfill, and orchestrate the migration, which all become more complex if downtime and inconsistencies are not acceptable (e.g. if you expose a pagination token based on LastEvaluatedKey, you may want to support both versions during the switch).
See also
Sparse indexes #
An item with missing index partition/sort key attributes won't appear in the index, and you won't pay for it. This can be used deliberately to query a subset of the items in the table, like those of a specific type or in a specific state.
Example
Assuming dedicated album items, an alternative way to list all the albums is to have a GSI with {Album} as partition key, and just scan the entire index (the primary key has to be a dedicated attribute that only albums have, so that only album items appear in the index).
Or, you can use a dedicated GSI with CoverOf as primary key to list cover songs.
See also
Base table indexes #
In some cases, GSIs won't cut it - maybe you need a strongly consistent index, or need to model a many-to-one relationship (indexes map one item in the base table to one item in the index).
Instead, you can maintain an index in the base table by having additional index items associated with the main item; to guarantee atomic updates, use transactions. You then go from the main item to the index items via a main item attribute, and from the index items to the main item via their partition key.
Example
Songs have different identifiers in external systems, such as ISRC, ISWC, or MBID. To query songs by multiple external ids, you'd structure your database like this:
- song
- partition key
song#{Artist}#{Album} - sort key
{Song} external_{type}:id, ...
- partition key
- external ids
- partition key
external#{type}#{id} - sort key
song#{Artist}#{Album}#{Song}
- partition key
(Alternatively, you could have one sparse index per external id type, but then you lose strong consistency, and risk running out of GSIs).
Note that modeling one-to-many relationships isn't this involved, since it fits neatly into the related-items-same-partition variant of single table design.
See also
- Working with item collections (modeling one-to-many relationships)
- Many-to-many relationships
Optimistic locking #
Optimistic locking is a concurrency control method useful when conflicts are rare, so instead of acquiring a lock to do changes, you check if someone else changed the data right before commiting, as part of an atomic operation.
In DynamoDB, that operation is a conditional write; items get an integer version attribute, and every time you want to update an item, you:
- read the item, including the version
- increment the version and modify the item
- update the item, using a condition expression to ensure the version matches
- if successful, you're done
- else, start over from the beginning
You can also do this in transactions to update groups of related items, like in the base table index pattern above, with only the main item needing a version.
The upside of optimistic locking is that it is faster on average, since updates usually succeed on the first try; for fewer conflicts, use strongly consistent reads.
The downside is that it requires explicit support - it must be possible to start over from the beginning, which complicates logic, especially if you need to interact with other systems besides updating the item (e.g. to send a notification).
See also
- Implementing version control via optimistic locking (Python example)
- Optimistic locking with version number (Java example)
Anyway, that's it for now.
See also
For mode details and examples, check out the official documentation:
- Data modeling
- Data modeling schemas (worked examples)
Learned something new today? Share it with others, it really helps!
Want to know when new articles come out? Subscribe here to get new stuff straight to your inbox!
01 Jun 2026 10:44am GMT
Speed Matters: Scandir Rs
layout: post title: scandir-rs tagline: Blazing-fast directory traversal for Python - up to 70× faster than os.walk. date: 2026-06-01 08:40:00 +0100 categories: posts ------
scandir-rs: High-Performance Directory Traversal for Python
File system traversal is often a hidden bottleneck.
Whether you're indexing files, collecting statistics, searching large directory trees, or building developer tools, performance matters. That's why I created scandir-rs: a Rust-powered Python library designed to be a drop-in replacement for os.walk() and os.scandir(), while delivering dramatically better performance and additional functionality.
A new version (2.9.9) is available with following changes compared to the version I've introduced here the last time (2.7.1):
- Add support for Python 3.13 and 3.14
- Add support for MacOS on ARM64
- Add support for Linux on PPC64 and S390
- Add SSE builds (for older CPUs which have no AVX) for Linux and Windows
- Add optional argument
follow_links. - Changed
skip_hiddento false by default
Why scandir-rs?
Because speed matters…
🚀 Significant Performance Improvements
Compared to Python's built-in implementations:
- Walk is up to 13× faster on Linux
- Walk is up to 70× faster on Windows
- Scandir is up to 6.5× faster on both Linux and Windows
When processing millions of files, these speedups can turn minutes into seconds.
Benchmarks results for running scandir in linux-5.9 folder
scandir-rs Walk benchmark on Linux (kernel 5.9)
scandir-rs Walk benchmark on Windows (kernel 5.9)
🔍 Richer Metadata
Beyond the standard os.walk() and os.scandir() APIs, scandir-rs can return:
- Extended file metadata
- Hardlink detection
- Additional file type classification
- Error collection without interrupting traversal
⚡ Background Processing
Long-running scans can run asynchronously in the background, allowing your application to process results while scanning is still in progress.
Installation
pip install scandir-rs
Usage Examples
Directory Statistics
Get fast statistics for an entire directory tree:
import scandir_rs as scandir
print(scandir.Count("/usr").collect())
Extended Statistics
Include additional metadata and hardlink detection:
import scandir_rs as scandir
print(
scandir.Count(
"/usr",
return_type=scandir.ReturnType.Ext
).collect()
)
Background Scanning
Process results while scanning continues in the background:
import scandir_rs as scandir
counter = scandir.Count("/usr")
with counter:
while counter.busy:
results = counter.results()
# Process intermediate results
# Final results as JSON
results = counter.to_json()
Faster os.walk()
A familiar interface with significantly better performance:
import scandir_rs as scandir
for root, dirs, files in scandir.Walk("/usr"):
# Process files
Extended Walk Information
Retrieve additional file categories and error information:
import scandir_rs as scandir
for root, dirs, files, symlinks, other, errors in scandir.Walk(
"/usr",
return_type=scandir.ReturnType.Ext
):
# Process files
On Unix systems, other includes special file types such as pipes and devices.
Faster os.scandir()
Collect all entries at once:
import scandir_rs as scandir
entries, errors = scandir.Scandir("/usr").collect()
Or iterate lazily:
import scandir_rs as scandir
for entry in scandir.Scandir("/usr"):
# Process entry
Extended Metadata
Request detailed information for each directory entry:
import scandir_rs as scandir
for entry in scandir.Scandir(
"/usr",
return_type=scandir.ReturnType.Ext
):
# Process entry
Entries are returned as DirEntryExt objects. Errors are reported as tuples containing:
(relative_path, error_message)
allowing scans to continue even when individual files cannot be accessed.
Benchmark Results
Walk Performance
| Operation | Linux | Windows |
|---|---|---|
Walk vs os.walk |
Up to 13× faster | Up to 70× faster |
Scandir Performance
| Operation | Linux | Windows |
|---|---|---|
Scandir vs os.scandir |
Up to 6.5× faster | Up to 6.5× faster |
For detailed benchmark data and methodology, see the benchmark documentation:
https://github.com/brmmm3/scandir-rs/blob/master/pyscandir/doc/benchmarks.md
Get Started
If your application spends time traversing large directory trees, scandir-rs can provide substantial performance improvements with minimal code changes.
The API is intentionally familiar, making migration from os.walk() and os.scandir() straightforward while unlocking additional capabilities and significantly faster execution.
Source code, documentation, and issue tracker:
https://github.com/brmmm3/scandir-rs
Licensed under the MIT License.
01 Jun 2026 12:00am GMT
31 May 2026
Django community aggregator: Community blog posts
Django: introducing django-integrity-policy
Back in January, Firefox's Security & Privacy Newsletter for 2025 Q4 piqued my interest with this mention:
Integrity-Policy: Firefox 145 has added support for the Integrity-Policy response header. The header allows websites to ensure that only scripts with an integrity attribute will load.
A new security header! That's right up my street: I've cared about getting security headers right since 2018, when I created django-permissions-policy to set the Permissions-Policy header. (At the time, it was called Feature-Policy: why they changed it, I can't say, people just liked it better that way.)
The new Integrity-Policy header helps with subresource integrity, a tool for securely including third-party scripts and stylesheets on your website. Browsers support the integrity attribute on <script> and <link> tags, which allows you to specify a hash of the expected content, like:
<script
src=https://cdn.jsdelivr.net/npm/htmx.org@4.0.0-beta4/dist/htmx.min.js
integrity=sha384-aWZK1NtOs/aWb/+YZdTM8q2JkWEshlMc9mgZ189numT9bwFhyAyYEoO4nO/2dTXt
crossorigin=anonymous></script>
If the content downloaded from the external source doesn't match the expected hash, the browser blocks it from loading. This is a great defense against the target URL changing its contents, executing a supply chain attack against your visitors.
(Generally, I recommend you avoid loading anything from a third-party URL, per Reasons to avoid Javascript CDNs. But sometimes, you gotta do what you gotta do, and declaring integrity is a great idea, then.)
Integrity-Policy allows you to opt in to requiring integrity attributes on your page, ensuring that you can never load potentially-compromised resources. The header is fairly simple, at least right now-here's a complete example that requires integrity for all scripts and stylesheets:
Integrity-Policy: blocked-destinations=(script style)
You can add in endpoints to tell browsers where to send violation reports to, and there's the second Integrity-Policy-Report-Only header which lets you test a policy without enforcing it.
Note there's no possibility to differentiate between first- and third-party resources in Integrity-Policy. If you set it, you'll need to add integrity attributes to all your scripts and stylesheets, including those you host yourself. This is by design: the header is being developed as part of Web Application Integrity, Consistency and Transparency (WAICT), an initiative to bring app-store-level "code signing" to the web, where users can be sure that CDNs and other intermediaries haven't tampered with served code.
Integrity-Policy is supported on Firefox 145+ and Chrome 138+.
django-integrity-policy
My new package, django-integrity-policy provides a middleware for setting the Integrity-Policy headers in a familiar Django-style way. You install it, add the middleware:
MIDDLEWARE = [
...,
"django.middleware.security.SecurityMiddleware",
"django_integrity_policy.IntegrityPolicyMiddleware",
...,
]
…and then configure the appropriate setting(s):
INTEGRITY_POLICY = {
"blocked-destinations": ["script", "style"],
}
So far, it's pretty basic, but I expect WAICT will increase the complexity of Integrity-Policy over time, and I'll add support for new options as they come along.
Once Integrity-Policy is set, the browser will block any scripts or stylesheets (depending on configuration) that lack a valid integrity attribute, including your first-party resources. That means you need to add integrity attributes to all your static files. Luckily, this has been considered before by the legendary Jake Howard, who made a package called django-sri. It provides template tags to generate appropriately hashed HTML tags. For example:
{% load sri %}
{% sri_static "app.js" %}
{% sri_static "app.css" %}
…will output:
<script src="/static/app.js" integrity="sha256-..."></script>
<link rel="stylesheet" href="/static/app.css" integrity="sha256-..."/>
These tags would be allowed under a maximally strict integrity policy.
See the example application in the django-integrity-policy repository for a full working project.
LLM generation
I built this package using just two prompts to Claude. I copied the repository for my previous security header package, django-permissions-policy, and reset its Git history. I then used this prompt to Claude, inside Zed:
The current repository is a copy of my package django-permissions-policy
It's time to turn it into django-integrity-policy, for the relevant headers per the below mdn docs
(Integrity-Policy MDN docs from https://github.com/mdn/content/blob/main/files/en-us/web/http/reference/headers/integrity-policy/index.md?plain=1)
(Integrity-Policy-Report-Only MDN docs from https://github.com/mdn/content/blob/main/files/en-us/web/http/reference/headers/integrity-policy-report-only/index.md?plain=1)
Check and edit every file to be that new package, copyright 2026, keeping the general testing infrastructure and so on.
Don't run any commands yet, just check and edit every file
I then took a 20-minute nap and woke up to a near-complete package. I reviewed it, ran the tests, made some minor edits, committed, and pushed to PyPI!
LLMs are rightfully a hot topic, with heady supporters and heavy detractors. I use them begrudgingly and somewhat sparingly, and I cannot wait for the future where I can stick to local models. (Gemma 4 can run on my M1 Mac and approaches Claude's performance on many tasks, so we're getting there.)
This task, though, was a perfect fit for LLM code generation: the existing repository acted as great context for structure, the new package was very similar in shape ("same same but different"), and the documentation provided a clear specification for what to build. The LLM could mash things up for me with minimal oversight, and I could check the work quickly.
I would guess that overall, using an LLM saved me a couple of hours of mostly grunt-work, like checking every copied-over configuration file. That's pretty valuable for me, and honestly made creating the package feasible.
The future
I'm not sure how widely used Integrity-Policy will be, and therefore how popular django-integrity-policy will end up. But this was an interesting exercise and I am interested to see how the header and other work from WAICT evolves. I will try to keep the package updated, and we'll see if it ever reaches a point where proposing support in Django itself makes sense.
31 May 2026 4:00am GMT
30 May 2026
Planet Python
Kay Hayen: Nuitka Release 4.1
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, "download now".
This release adds many new features and corrections with a focus on async code compatibility, missing generics features, and Python 3.14 compatibility and Python compilation scalability yet again.
Bug Fixes
-
Python 3.14: Fix, decorators were breaking when disabling deferred annotations. (Fixed in 4.0.1 already.)
-
Fix, nested loops could have wrong traces lead to mis-optimization. (Fixed in 4.0.1 already.)
-
Plugins: Fix, run-time check of package configuration was incorrect. (Fixed in 4.0.1 already.)
-
Compatibility: Fix,
__builtins__lacked necessary compatibility in compiled functions. (Fixed in 4.0.1 already.) -
Distutils: Fix, incorrect UTF-8 decoding was used for TOML input file parsing. (Fixed in 4.0.1 already.)
-
Fix, multiple hard value assignments could cause compile time crashes. (Fixed in 4.0.1 already.)
-
Fix, string concatenation was not properly annotating exception exits. (Fixed in 4.0.2 already.)
-
Windows: Fix,
--verbose-outputand--show-modules-outputdid not work with forward slashes. (Fixed in 4.0.2 already.) -
Python 3.14: Fix, there were various compatibility issues including dictionary watchers and inline values. (Fixed in 4.0.2 already.)
-
Python 3.14: Fix, stack pointer initialization to
localspluswas incorrect to avoid garbage collection issues. (Fixed in 4.0.2 already.) -
Python 3.12+: Fix, generic type variable scoping in classes was incorrect. (Fixed in 4.0.2 already.)
-
Python 3.12+: Fix, there were various issues with function generics. (Fixed in 4.0.2 already.)
-
Python 3.8+: Fix, names in named expressions were not mangled. (Fixed in 4.0.2 already.)
-
Plugins: Fix, module checksums were not robust against quoting style of module-name entry in YAML configurations. (Fixed in 4.0.2 already.)
-
Plugins: Fix, doing imports in queried expressions caused corruption. (Fixed in 4.0.2 already.)
-
UI: Fix, support for
uv_buildin the--projectoption was broken. (Fixed in 4.0.2 already.) -
Compatibility: Fix, names assigned in assignment expressions were not mangled. (Fixed in 4.0.2 already.)
-
Python 3.12+: Fix, there were still various issues with function generics. (Fixed in 4.0.3 already.)
-
Clang: Fix, debug mode was disabled for clang generally, but only ClangCL and macOS Clang didn't want it. (Fixed in 4.0.3 already.)
-
Zig: Fix,
--windows-console-mode=attach|disablewas not working when using Zig. (Fixed in 4.0.3 already.) -
macOS: Fix, yet another way self dependencies can look like, needed to have support added. (Fixed in 4.0.3 already.)
-
Python 3.12+: Fix, generic types in classes had bugs with multiple type variables. (Fixed in 4.0.3 already.)
-
Scons: Fix, repeated builds were not producing binary identical results. (Fixed in 4.0.3 already.)
-
Scons: Fix, compiling with newer Python versions did not fall back to Zig when the developer prompt MSVC was unusable, and error reporting could crash. (Fixed in 4.0.4 already.)
-
Zig: Fix, the workaround for Windows console mode
attachordisablewas incorrectly applied on non-Windows platforms. (Fixed in 4.0.4 already.) -
Standalone: Fix, linking with Python Build Standalone failed because
libHacl_Hash_SHA2was not filtered out unconditionally. (Fixed in 4.0.4 already.) -
Python 3.6+: Fix, exceptions like
CancelledErrorthrown into an async generator awaiting an inner awaitable could be swallowed, causing crashes. (Fixed in 4.0.4 already.) -
Fix, not all ordered set modules accepted generators for update. (Fixed in 4.0.5 already.)
-
Plugins: Disabled warning about rebuilding the
pytokensextension module. (Fixed in 4.0.5 already.) -
Standalone: Filtered
libHacl_Hash_SHA2from link libs unconditionally. (Fixed in 4.0.5 already.) -
Debugging: Disabled unusable unicode consistency checks for Python versions 3.4 to 3.6. (Fixed in 4.0.5 already.)
-
Python3.12+ Avoided cloning call nodes on class level which caused issues with generic functions in combination with decorators. (Added in 4.0.5 already.)
-
Python 3.12+: Added support for generic type variables in
async deffunctions. (Added in 4.0.5 already.) -
UI: Fix, flushing outputs for prompts was not working in all cases when progress bars were enabled. (Fixed in 4.0.6 already.)
-
UI: Fix, unused variable warnings were missing at C compile time when using
zigas a C compiler. (Fixed in 4.0.6 already.) -
Scons: Fix, forced stdout and stderr paths as a feature was broken. (Fixed in 4.0.6 already.)
-
Fix, replacing a branch did not accurately track shared active variables causing optimization crashes. (Fixed in 4.0.7 already.)
-
macOS: Fix, failed to remove extended attributes because files need to be made writable first. (Fixed in 4.0.7 already.)
-
Fix, dict
popandsetdefaultusing with:=rewrites lacked exception-exit annotations for un-hashable keys. (Fixed in 4.0.8 already.) -
Python 3.13: Fix, the
__parameters__attribute of generic classes was not working. (Fixed in 4.0.8 already.) -
Python 3.11+: Fix, starred arguments were not working as type variables. (Fixed in 4.0.8 already.)
-
Python2: Fix,
FileNotFoundErrorcompatibility fallback handling was not working properly. (Fixed in 4.0.8 already.) -
Compatibility: Fix, loop ownership check in value traces was missing, causing issues with nested loops.
-
Windows: Improved
--windows-console-mode=attachto properly handle console handles, enabling cases likeos.systemto work nicely. -
Python2: Fix, there was a compatibility issue where providing default values to the
mkdtempfunction was failing. -
Windows: Fix, there were spurious issues with C23 embedding in 32-bit MinGW64 by switching to
coff_objresource mode for it as well. -
Plugins: Fix, the
post-import-codeexecution could fail because the triggering sub-package was not yet available insys.modules. -
UI: Fix, listing package DLLs with
--list-package-dllswas broken due to recent plugin lifecycle changes. -
UI: Fix,
--list-package-exewas not working properly on non-Windows platforms failing to detect executable files correctly. -
UI: Handled paths starting with
{PROGRAM_DIR}the same as a relative path when parsing the--onefile-tempdir-specoption. -
Plugins: Followed multiprocessing
forkserverchanges for newer Python versions. -
Python 3.12+: Fix, generic class type parameters handling was incorrect.
-
Python 3.12: Fix, deferred evaluation of type aliases was failing.
-
Python 3.12+: Aligned
sumbuilt-in float summation with CPython's compensated sum for better accuracy. -
Python 3.10+: Fix, uncompiled coroutine
throw()return handling was incorrect, restoring completed coroutine results viaStopIteration.valuerather than exposing them as ordinary return values to the outer await chain. -
Python 3.13+: Fix, uncompiled coroutine
cancel()/awaitsuspension handling was incorrect, improved to ensure integration compatibility. -
macOS: Made finding
create-dmgmore robustly by also checking the Homebrew path for Intel and fromPATHproperly. -
Compatibility: Fix, class frames were not exposing frame locals.
-
UI: Detected
static-libpythonproblems, which affected some forms of Anaconda. -
Distutils: Rejected
--projectmixed with--mainarguments as it is not useful. -
macOS: Fix,
zigfromPATHor fromziglangwas not being used. -
Distutils: Fix, the wrong
module-rootconfig value was being checked foruvbuild backend. -
macOS: Fix, was attempting to change removed (rejected) DLLs, which of course failed and errored out.
-
Python 3.14: Fix, tuple reuse was not fully compatible, potentially causing crashes due to outdated hash caches.
-
Fix, fake modules were still being attempted to located when imported by other code, which could conflict with existing modules.
-
Python 3.5+: Fix, failed to send uncompiled coroutines the sent in value in
yield from. -
Fix, older
gcccompilers lacking newer intrinsic methods had compilation issues that needed to be addressed. -
Standalone: Fix, multiphase module extension modules with post-load code were not working properly.
-
Fix, Avoid using the non-inline copy of
pkg_resourceswith the inline copy of Jinja2. These could mismatch and cause errors. -
Fix, loops could make releasing of previous values very unclear, causing optimization errors.
-
Fix,
incbinresource mode was not working with oldgccC++ fallback. -
Python 3.4 to 3.6: Fix, bytecode demotion was not working properly for these versions, also bytecode only files not working.
-
Plugins: Added a check for the broken
patchelfversions 0.10 and 0.11 to prevent breaking Qt plugins. -
Android: Allowed
patchelfversion 0.18 on Android. -
Windows: Fix, the header path for self uninstalled Python was not detected correctly.
-
Release: Fix, inclusion of the
pkg_resourcesinline copy for Python 2 to source distributions was missing. -
UI: Detected the OBS versions of SUSE Linux better.
-
Suse: Allowed using
patchelf0.18.0 there too. -
Python 3.11: Fix, package and module dicts were not aligned close enough to avoid a CPython bug.
-
Fix, unbound compiled methods could crash when called without an object passed.
-
Standalone: Fix, multiphase module extension modules with postload. (Fixed in 4.0.8 already.)
-
Onefile: Fix, while waiting for the child, it may already be terminated.
-
macOS: Removed existing absolute rpaths for Homebrew and MacPorts.
-
Python 3.14: Avoided warning in CPython headers.
-
Python 3.14: Followed allocator changes more closely.
-
Compatibility: Avoided using
pkg_resourcesfor Jinja2 template location for loading. -
No-GIL: Applied some bug fixes to get basic things to work.
Package Support
-
Standalone: Add support for newer
paddleversion. (Added in 4.0.1 already.) -
Standalone: Add workaround for refcount checks of
pandas. (Fixed in 4.0.1 already.) -
Standalone: Add support for newer
h5pyversion. (Added in 4.0.2 already.) -
Standalone: Add support for newer
scipypackage. (Added in 4.0.2 already.) -
Plugins: Revert accidental
os.getenvoveros.environ.getchanges in anti-bloat configurations that stopped them from working. Affected packages arenetworkx,persistent, andtensorflow. (Fixed in 4.0.5 already.) -
Standalone: Added missing DLLs for
openvino. (Added in 4.0.7 already.) -
Enhanced the package configuration YAML schema by adding the
relative_toparameter forfrom_filenamesDLL specification, avoiding error-prone purely relative paths. -
Standalone: Fix,
flet_desktopapp assets were missing, now preserving the packaged runtime and sidecar DLLs. -
Standalone: Added support for the
tyropackage. -
Standalone: Added data files for the
perfettopackage. -
Standalone: Added support for
anyioprocess forking. -
Standalone: Added support for the
plotly.graphpackage. -
Anaconda: Fix, dependencies for the
numpyconda package on Windows were incorrect. -
Plugins: Enhanced the auto-icon hack in PySide6 to use compatible class names.
-
Standalone: Fix, Qt libraries were duplicated with
PySide6WebEngine framework support on macOS. -
Plugins: Fix, automatic detection of
mypycruntime dependencies was including all top level modules of the containing package by accident. (Fixed in 4.0.5 already.) -
Anaconda: Fix,
delvewheelplugin was not working with Python 3.8+. This enhances compatibility with installed PyPI packages that use it for their DLLs. (Fixed in 4.0.6 already.) -
Plugins: Fix, our protection workaround could confuse methods used with
PySide6.
New Features
-
UI: Added the
--recommended-python-versionoption to display recommended Python versions for supported, working, or commercial usage. -
UI: Add message to inform users about
Nuitka[onefile]if compression is not installed. (Added in 4.0.1 already.) -
UI: Add support for
uv_buildin the--projectoption. (Added in 4.0.1 already.) -
Onefile: Allow extra includes as well. (Added in 4.0.2 already.)
-
UI: Add
nuitka-project-setfeature to define project variables, checking for collisions with reserved runtime variables. (Added in 4.0.2 already.) -
Scons: Added new option to select
--reproduciblebuilds or not. (Added in 4.0.6 already.) -
Python 3.10+: Added support for
importlib.metadata.package_distributions(). (Added in 4.0.8 already.) -
Plugins: Added support for the multiprocessing
forkservercontext. (Added in 4.0.8 already, for 4.1 Python 3.6 and earlier, as well as 3.14 support were added too.) -
Reports: Added structured resource usage (
rusage) performance information to compilation reports. -
Reports: Included individual module-level C compiler caching (
ccache/clcache) statistics in compilation reports. -
Added support for detecting and correctly resolving the Python prefix for the
PyEnv on HomebrewPython flavor. -
macOS: Added support for
rusageinformation for Scons. -
UI: Added the
__compiled__.extension_filenameattribute to give the real filename of the containing extension module. -
Windows: Added support for
--clangor ARM. (Added in 4.0.8 already.) -
Windows: Added support for resources names as not just integers, important when we copy them from template files.
-
MacPorts: Added basic support for this Python flavor. More work will be needed to get it to work fully though.
Optimization
-
Avoid including
importlib._bootstrapandimportlib._bootstrap_external. (Added in 4.0.1 already.) -
Linux: Cached the
syscallused for time keeping during compilation to avoid loadinglibcfor each trace. (Added in 4.0.8 already.) -
UI: Output a warning for modules that remain unfinished after the third optimization pass.
-
Added an extra micro pass trigger when new variables are introduced or variable usage changes severely, ensuring optimizations are fully propagated, avoiding unnecessary extra full passes.
-
Provided scripts to compile Python statically with PGO tailored for Nuitka on Linux, Windows, and macOS.
-
Added support for running the Data Composer tool from a compiled Nuitka binary without spawning an uncompiled Python process.
-
Enhanced the usage of
vectorcallforPyCFunctionobjects by directly checking for its presence instead of relying purely on flags, allowing more frequent use of this faster execution path. -
Cached frequently used declarations for top-level variables to speed up C code generation.
-
Sped up trace collection merging by avoiding unnecessary set creation and using a set instead of a list for escaped traces.
-
Optimized plugin hook execution by tracking overloaded methods and added an option to show plugin usage statistics.
-
Improved performance of module location by avoiding unnecessary module name reconstruction and redundant filesystem checks for pre-loaded packages.
-
Improved the caching of distribution name lookups to effectively avoid repeated IO operations across all package types.
-
Plugins: Cached callback plugin dispatch for
onFunctionBodyParsingandonClassBodyParsingto skip argument computation when no plugin overrides them. -
Python 3.13: Handled sub-packages of
pathlibas hard modules. -
Handled hard attributes through merge traces as well.
-
Made constant blobs more compact by avoiding repeated identifiers and unnecessary fields.
-
Enhanced Python compilation scripts further. (Fixed in 4.0.8 already.)
-
Recognized late incomplete variables better. (Fixed in 4.0.8 already.)
-
Made constant blobs more compact. (Fixed in 4.0.8 already.)
-
Optimized calls with only constant keywords and variable posargs too.
Anti-Bloat
-
Fix, memory bloat occurred when C compiling
sqlalchemy. (Fixed in 4.0.2 already.) -
Avoid using
pydocinPySimpleGUI. (Added in 4.0.2 already.) -
Avoided using
doctestfromzodbpickle. (Added in 4.0.5 already.) -
Avoided inclusion of
cythonwhen usingpyav. (Added in 4.0.7 already.) -
Avoided including
typing_extensionswhen usingnumpy. (Added in 4.0.7 already.)
Organizational
-
UI: Relocated the warning about the available source code of extension modules to be evaluated at a more appropriate time.
-
Debian: Remove recommendation for
libfuse2package as it is no longer useful. -
Debian: Used
platformdirsinstead ofappdirs. -
Debugging: Removed Python 3.11+ restriction for
clang-formatas it is available everywhere, even Python 2.7, and we still want nicely formatted code when we read things. (Added in 4.0.6 already.) -
Removed no longer useful inline copy of
wax_off. We have our own stubs generator project. -
Release: Added missing package to the CI container for building Nuitka Debian packages.
-
Developer: Updated AI instructions for creating Minimal Reproducible Examples (MRE) to skip unneeded C compilation.
-
Debugging: Added an internal function for checking if a string is a valid Python identifier.
-
AI: Added a task in Visual Studio Code to export the currently selected Python interpreter path to a file, making it available as "python" and "pip" matching the selected interpreter. This makes it easier to use a specific version with no instructions needed.
-
AI: Updated the rules to instruct AI to only generate useful comments that add context not present in the code.
-
Containers: Added template rendering support for Jinja2 (
.j2) container files in our internal Podman tools. -
Projects: Clarified the current status and rationale of Python 2.6 support in the developer manual.
-
Debugging: Added experimental flag
--experimental=ignore-extra-micro-passto allow ignoring extra micro pass detection. -
Visual Code: Added integration scripts for
bashandzshautocompletion of Nuitka CLI options. These are now also integrated into Visual Studio Code terminal profiles and the Debian package. -
RPM: Included the Python compile script for Linux.
-
RPM: Removed the requirement for
distutilsin the spec.
Tests
-
Install only necessary build tools for test cases.
-
Avoided spurious failures in reference counting tests due to Python internal caching differences. (Fixed in 4.0.3 already.)
-
Fix, the parsing of the compilation report for reflected tests was incorrect.
-
Python 3.14: Ignored a syntax error message change.
-
Python 3.14: Added test execution support options to the main test runner to use this version as well.
-
Fix, the runner binary path was mishandled for the third pass of reflected compilations.
-
Removed the usage of obsolete plugins in reflected compilation tests.
-
Debugging: Prevented boolean testing of
namedtuplesto avoid unexpected bugs. -
Added the
Testsuffix to syntax test files and disabled "python" mode and spell checking for them to resolve issues reported in IDEs. -
Fix, newline handling in diff outputs from the output comparison tool was incorrect.
-
Covered
post-import-codefunctionality with a new subpackage test case. -
Prevented the program test suite from running an unnecessary variant to save execution time.
-
macOS: Ignored differences from GUI framework error traces in headless runs in output comparisons.
-
Reflected test for Nuitka, where it compiles itself and compares its operation has been restored to functional state.
-
Used the new method to clear internal caches if available for reference counts.
-
Disabled running nested loops test with Python 2.6.
-
Containers: Detected Python 2 defaulting containers in Podman tooling.
Cleanups
-
UI: Fix, there was a double space in the Windows Runtime DLLs inclusion message. (Fixed in 4.0.1 already.)
-
Onefile: Separated files and defines for extra includes for onefile boot and Python build.
-
Scons: Provided nicer errors in case of "unset" variables being used, so we can tell it.
-
Refactored the process execution results to correctly utilize our
namedtuplesvariant, that makes it easier to understand what code does with the results. -
Quality: Enabled automatic conversion of em-dashes and en-dashes in code comments to the autoformat tool. AI won't stop producing them and they can cause
SyntaxErrorfor older Python versions, nor is unnecessarily using UTF-8 welcome. -
Ensured that cloned outline nodes are assigned their correct names immediately upon creation, that avoids inconsistencies during their creation.
-
Quality: Updated to the latest versions of
blackand adopted a fasterisortexecution by caching results. -
Quality: Modified the PyLint wrapper to exit gracefully instead of raising an error when no matching files require checking.
-
Quality: Avoided checking YAML package configuration files twice, since autoformat already handles them.
-
Quality: Ensured that YAML package configuration checks output the original filename instead of the temporary one when a failure occurs.
-
Quality: Prevented pushing of tags from triggering git pre-push quality checks.
-
Quality: Silenced the output of
optipngandjpegoptimduring image optimization auto-formatting. -
Visual Code: Added the generated Python alias path file to the ignore list.
-
Quality: Enabled auto-formatting for the Nuitka devcontainer configuration file.
-
Watch: Avoided absolute paths in compilation to make reports more comparable across machines.
-
Quality: Changed
mdformatchecks to run only once and silently. -
Scons: Disabled format security errors in debug mode and moved Python-related warning disables into common build setup code.
-
Quality: Updated to the latest
deepdiffversion. -
Scons: Avoided MSVC telemetry since it can produce outputs that break CI.
-
Debugging: Enhanced non-deployment handler for importing excluded modules.
-
Split import module finding functionality into more pieces for enhanced readability.
-
Debugging: Added more assertions for constants loading and checking.
-
macOS: Dropped the
universaltarget arch. -
Debugging: Added more traces for deep hash verification.
Summary
This release builds on the scalability improvements established in 4.0, with enhanced Python 3.14 support, expanded package compatibility, and significant optimization work.
The --project option seems usable now.
Python 3.14 support remains experimental, but only barely made the cut, and probably will get there in hotfixes. Some of the corrections came in so late before the release, that it was just not possible to feel good about declaring it fully supported just yet.
30 May 2026 10:00pm GMT
29 May 2026
Django community aggregator: Community blog posts
Issue 339: Early Bird DjangoCon US Tickets Ending Soon
News
DjangoCon US 2026: Early Bird Tickets End May 31st!
Early bird ticket sales for DjangoCon US 2026 end on May 31, 2026, with discounted pricing available. The conference runs five days at Voco Chicago Downtown and includes community-selected talks plus Django contribution sprints.
Wagtail CMS News
Wagtail Space NL - June 12
A full-day conference in Rotterdam, The Netherlands on Wagtail, with talks covering a range of topics, lightning talks, hallway discussions, and more.
Updates to Django
Today, "Updates to Django" is presented by Pradhvan from Djangonaut Space! 🚀
Last week we had 16 pull requests merged into Django by 10 different contributors.
This week's Django highlights: 🦄
- Django's built-in error pages, admin, and registration templates now include the CSP nonce on
<script>,<link>, and<style>elements when available. (#36825) - Fixed
HttpResponse.reason_phraseto raiseBadHeaderErrorwhen set to a value containing control characters. (#37100) - Fixed
Query.clear_ordering()to recursively clear ordering on combined queries, preventing errors when using__inlookups on nestedunion()querysets. (#37097) - Admin change form actions now use
ModelAdmin.get_queryset(), ensuring custom annotations and filtering are consistently applied to form actions. (#37117)
If you haven't already, give Django 6.1 alpha 1 a spin and report anything suspicious to the issue tracker! 🎉
That's all for this week in Django development! 🐍🦄
Articles
Upgrade PostgreSQL from 17 to 18 on Ubuntu 26.04
After moving to Ubuntu 26.04, upgrade an existing 17/main cluster to 18 by running pg_upgradecluster 17 main -v 18, then verify the new 18/main cluster is online. Once confirmed, drop the old 17 cluster with pg_dropcluster 17 main and optionally purge postgresql-17 and postgresql-client-17 packages.
My not-so-static new static website
Jake Howard walks through his eighth website rewrite, this time ditching Wagtail for a custom "semi-static" Django setup that renders Markdown content into SQLite at startup and serves it dynamically with Jinja2 templates.
Improving First Byte and Contentful Paint on a Django Website
A look at how to use Django's StreamingHttpResponse to send the ` and above-the-fold content first, letting the browser fetch static assets and start painting while the rest of the page renders.
PyCon US 2026 Recap - Black Python Devs
A recap from from the community booth to open spaces, hallway track, and Jay Miller receiving the PSF Community Service Award.
django-removals 1.2.0 - Now with Django 6.1 deprecations
How the maintainers of django-removals shipped new warnings for the Django 6.1 deprecation wave.
Mentoring GSoC 2026: Experimental Flags - Software Crafts
Mentor and mentee are starting a GSoC 2026 project around an "Experimental Flags" framework for Django core, using the forum to gather requirements and drive early consensus. The plan balances fast iteration with faster-than-normal Django consensus, including an initial third-party package to test ideas before wider adoption.
Django Forum
GSoC 2026: Implementing a Formal Experimental API Framework for Django Core
A lively discussion around how experimental features can be merged into the main repository but remain explicitly non-stable.
Thoughts on advertising on djangoproject.com
New thoughts and comments on the age-old question.
Django Fellow Reports
Jacob Walls
Not much going on, "just" the 6.1 Feature Freeze/alpha release, a sprint at PyCon US, and a kickoff meeting with Google Summer of Code participants & mentors.
Sarah Boyce
As we had the feature freeze, focused on a few feature PRs I had prioritized for 6.1 release.
Natalia Bidart
This week was mostly about returning from PyCon, which was quite exhausting. I arrived back on Wednesday, fairly drained (and very hungry), so I worked during Thu and Fri catching up on a large backlog of email notifications and syncing with the other Fellows.
Events
Django on the Med - September 23-25 in Pescara, Italy
PyCon Italia this week has been Django members in attendance, so it is a good time to remind readers that Django on the Med will be back in Italy later in the year.
Django Job Board
Founding Engineer at MyDataValue
Projects
feincms/feincms3-cookiecontrol
Cookie banner with support for embedded media.
emfpdlzj/django-deploy-probes
HTTP deployment probes for Django applications.
29 May 2026 2:00pm GMT
27 May 2026
Django community aggregator: Community blog posts
Please add an RSS Feed to Your Site
Why syndication feeds are having a moment in 2026.
27 May 2026 9:57pm GMT
22 May 2026
Planet Twisted
Glyph Lefkowitz: Opaque Types in Python
Let's say you're writing a Python library.
In this library, you have some collection of state that represents "options" or "configuration" for a bunch of operations. Such a set of options is a bundle of potentially ever-increasing complexity. Thus, you will want it to have an extremely minimal compatibility surface, with a very carefully chosen public interface, that is either small, or perhaps nothing at all. Such an object conveys state and might have some private behavior, but all you want consumers to be able to do is build it in very constrained, specific ways, and then pass it along as a parameter to your own APIs.
By way of example, imagine that you're wrapping a library that handles shipping physical packages.
There are a zillion ways to do it ship a package. There are different carriers who can ship it for you. There's air freight, and ground freight, and sea freight. There's overnight shipping. There's the option to require a signature. There's package tracking and certified mail. Suffice it to say, lots of stuff.
If you are starting out to implement such a library, you might need an object called something like ShippingOptions that encapsulates some of this. At the core of your library you might have a function like this:
1 2 3 4 5 |
|
If you are starting out implementing such a library, you know that you're going to get the initial implementation of ShippingOptions wrong; or, at the very least, if not "wrong", then "incomplete". You should not want to commit to an expansive public API with a ton of different attributes until you really understand the problem domain pretty well.
Yet, ShippingOptions is absolutely vital to the rest of your library. You'll need to construct it and pass it to various methods like estimateShippingCost and shipPackage. So you're not going to want a ton of complexity and churn as you evolve it to be more complex.
Worse yet, this object has to hold a ton of state. It's got attributes, maybe even quite complex internal attributes that relate to different shipping services.
Right now, today, you need to add something so you can have "no rush", "standard" and "expedited" options. You can't just put off implementing that indefinitely until you can come up with the perfect shape. What to do?
The tool you want here is the opaque data type design pattern. C is lousy with such things (FILE, pthread_*_t, fd_set, etc). A typedef in a header file can easily achieve this.
But in Python, if you expose a dataclass - or any class, really - even if you keep all your fields private, the constructor is still, inherently, public. You can make it raise an exception or something, but your type checker still won't help your users; it'll still look like it's a normal class.
Luckily, Python typing provides a tool for this: typing.NewType.
Let's review our requirements:
- We need a type that our client code can use in its type annotations; it needs to be public.
- They need to be able to consruct it somehow, even if they shouldn't be able to see its attributes or its internal constructor arguments.
- To express high-level things (like "ship fast") that should stay supported as we add more nuanced and complex configurations in the future (like "ship with the fastest possible option provided by the lowest-cost carrier that supports signature verification").
In order to solve these problems respectively, we will use:
- a public
NewType, which gives us our public name... - which wraps a private class with entirely private attributes, to give us an actual data structure, while not exposing the constructor,
- a set of public constructor functions, which returns our
NewType.
When we put that all together, it looks like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
|
As a snapshot in time, this is not all that interesting; we could have just exposed _RealShipOpts as a public class and saved ourselves some time. The fact that this exposes a constructor that takes a string is not a big deal for the present moment. For an initial quick and dirty implementation, we can just do checks like if options._speed == "fast" in our shipping and estimation code.
However, the main thing we are doing here is preserving our flexibility to evolve the related APIs into the future, so let's see how we might do that. For example, let's allow the shipping options to contain a concrete and specific carrier and freight method:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
|
As a NewType, our public ShippingOptions type doesn't have a constructor. Since _RealShipOpts is private, and all its attributes are private, we can completely remove the old versions.
Anything within our shipping library can still access the private variables on ShippingOptions; as a NewType, it's the same type as its base at runtime, so it presents minimal1 overhead.
Clients outside our shipping library can still call all of our public constructors: shipFast, shipNormal, and shipSlow all still work with the same (as far as calling code knows) signature and behavior.
If you need to build and convey some state within your public API, while avoiding breakages associated with compatibility churn, hopefully this technique can help you do that!
Acknowledgments
Thanks for reading, and thank you to my patrons who are supporting my writing on this blog. If you like what you've read here and you'd like to read more of it, or you'd like to support my various open-source endeavors, you can support my work as a sponsor.
-
The overhead is minimal, but it is not completely zero. The suggested idiom for converting to a
NewTypeis to call it like a function, as I've done in these examples, but if you are wanting to use this pattern inside of a hot loop, you can use# type: ignore[return-value]comments to avoid that small cost. ↩
22 May 2026 12:33am GMT
04 Apr 2026
Planet Twisted
Donovan Preston: Using osascript with terminal agents on macOS
Here is a useful trick that is unreasonably effective for simple computer use goals using modern terminal agents. On macOS, there has been a terminal osascript command since the original release of Mac OS X. All you have to do is suggest your agent use it and it can perform any application control action available in any AppleScript dictionary for any Mac app. No MCP set up or tools required at all. Agents are much more adapt at using rod terminal commands, especially ones that haven't changed in 30 years. Having a computer control interface that hasn't changed in 30 years and has extensive examples in the Internet corpus makes modern models understand how to use these tools basically Effortlessly. macOS locks down these permissions pretty heavily nowadays though, so you will have to grant the application control permission to terminal. But once you have done that, the range of possibilities for commanding applications using natural language is quite extensive. Also, for both Safari and chrome on Mac, you are going to want to turn on JavaScript over AppleScript permission. This basically allows claude or another agent to debug your web applications live for you as you are using them.In chrome, go to the view menu, developer submenu, and choose "Allow JavaScript from Apple events". In Safari, it's under the safari menu, settings, developer, "Allow JavaScript from Apple events". Then you can do something like "Hey Claude, would you Please use osascript to navigate the front chrome tab to hacker news". Once you suggest using OSA script in a session it will figure out pretty quickly what it can do with it. Of course you can ask it to do casual things like open your mail app or whatever. Then you can figure out what other things will work like please click around my web app or check the JavaScript Console for errors. Another very important tips for using modern agents is to try to practice using speech to text. I think speaking might be something like five times faster than typing. It takes a lot of time to get used to, especially after a lifetime of programming by typing, but it's a very interesting and a different experience and once you have a lot of practice It starts to to feel effortless.
04 Apr 2026 1:31pm GMT
16 Mar 2026
Planet Twisted
Donovan Preston: "Start Drag" and "Drop" to select text with macOS Voice Control
I have been using macOS voice control for about three years. First it was a way to reduce pain from excessive computer use. It has been a real struggle. Decades of computer use habits with typing and the mouse are hard to overcome! Text selection manipulation commands work quite well on macOS native apps like apps written in swift or safari with an accessibly tagged webpage. However, many webpages and electron apps (Visual Studio Code) have serious problems manipulating the selection, not working at all when using "select foo" where foo is a word in the text box to select, or off by one errors when manipulating the cursor position or extending the selection. I only recently expanded my repertoire with the "start drag" and "drop" commands, previously having used "Click and hold mouse", "move cursor to x", and "release mouse". Well, now I have discovered that using "start drag x" and "drop x" makes a fantastic text selection method! This is really going to improve my speed. In the long run, I believe computer voice control in general is going to end up being faster than WIMP, but for now the awkwardly rigid command phrasing and the amount of times it misses commands or misunderstands commands still really holds it back. I've been learning the macOS Voice Control specific command set for years now and I still reach for the keyboard and mouse way too often.
16 Mar 2026 11:04am GMT