28 May 2025
Planet Python
Kay Hayen: Nuitka Release 1.2
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, "download now".
This release contains a large amount of new compatibility features and a few new optimization, while again consolidating what we have. Scalability should be better in many cases.
Bug Fixes
-
Standalone: Added implicit dependency of
thinc
backend. Fixed in 1.1.1 already. -
Python3.10: Fix,
match
statements with unnamed star matches could give incorrect results. Fixed in 1.1.1 already.match x: case [*_, y]: ... # y had wrong value here.
-
Python3.9+: Fix, file reader objects must convert to
str
objects. Fixed in 1.1.1 already.# This was the `repr` rather than a path value, but it must be usable # like that too. str(importlib.resources.files("package_name").joinpath("lala"))
-
Standalone: Added data file of
echopype
package. Fixed in 1.1.1 already. -
Anti-Bloat: Remove non-sense warning of compiled
pyscf
. Fixed in 1.1.1 already. -
macOS: Fix, in LTO mode using
incbin
can fail, switch to source mode for constants resources. Fixed in 1.1.2 already. -
Standalone: Add support for
sv_ttk
module. Fixed in 1.1.2 already. -
macOS: Fix, was no longer correcting
libpython
path, this was a regression preventing CPython for creating properly portable binary. Fixed in 1.1.2 already. -
macOS: Fix, main binary was not included in signing command. Fixed in 1.1.3 already.
-
Standalone: Added implicit dependency of
orjson
. Due tozoneinfo
not being automatically included anymore, this was having a segfault. Fixed in 1.1.3 already. -
Standalone: Added support for new
shapely
. Fixed in 1.1.4 already. -
macOS: Ignore extension module of non-matching architecture. Some wheels contain extension modules for only
x86_64
arch, and others contain them only forarm64
, preventing the standalone build. Fixed in 1.1.4 already. -
Standalone: Added missing
sklearn
dependencies. Fixed in 1.1.4 already. -
Fix, packages available through relative import paths could be confused with the same ones imported by absolute paths. This should be very hard to trigger, by normal users, but was seen during development. Fixed in 1.1.4 already.
-
Standalone: Apply import hacks for
pywin32
modules only on Windows, otherwise it can break e.g. macOS compilation. Fixed in 1.1.4 already. -
Windows: More robust DLL dependency caching, otherwise e.g. a Windows update can break things. Also consider plugin contribution, and Nuitka version, to be absolutely sure, much like we already do for bytecode caching. Fixed in 1.1.4 already.
-
Standalone: Fix,
seaborn
needs the same workaround asscipy
for corruption with MSVC. Fixed in 1.1.4 already. -
UI: Fix, the
options-nanny
was no longer functional and therefore failed to warn about non working options and package usages. Fixed in 1.1.5 already. -
macOS: Do not use extension modules of non-matching architecture. Fixed in 1.1.5 already.
-
Windows: Fix, resolving symlinks could fail for spaces in paths. Fixed in 1.1.6 already.
-
Standalone: Added missing DLL for
lightgbm
module. Fixed in 1.1.6 already. -
Compatibility: Respect
super
module variable. It is now possible to have a module level change ofsuper
but still get compatible behavior with Nuitka. Fixed in 1.1.6 already. -
Compatibility: Make sure we respect
super
overloads in the builtin module. Fixed in 1.1.6 already. -
Fix, the anti-bloat replacement code for
numpy.testing
was missing a required function. Fixed in 1.1.6 already. -
Fix,
importlib.import_module
static optimization was mishandling a module name of.
with a package name given. Fixed in 1.1.6 already. -
macOS: Fix, some extension modules use wrong suffixes in self references, we need to not complain about this kind of error. Fixed in 1.1.6 already.
-
Fix, do not make
ctypes.wintypes
a hard import on non-Windows. Nuitka asserted against it failing, where some code handles it failing on non-Windows platforms. Fixed in 1.1.6 already. -
Standalone: Added data files for
vedo
package. Fixed in 1.1.7 already. -
Plugins: Fix, the
gi
plugin did always setGI_TYPELIB_PATH
even if already present from user code. Also it did not handle errors to detect its value during compile time. Fixed in 1.1.7 already. -
Standalone: Added missing dependencies for
sqlalchemy
to have all SQL backends working. Fixed in 1.1.7 already. -
Added support Nixpkgs's default non-writable
HOME
directory. Fixed in 1.1.8 already. -
Fix, distribution metadata name and package name need not align, need to preserve the original looked up name from
importlib.metadata.distribution
call. Fixed in 1.1.8 already. -
Windows: Fix, catch usage of unsupported
CLCACHE_MEMCACHED
mode with MSVC compilation. It is just unsupported. -
Windows: Fix, file version was spoiled from product version if it was the only version given.
-
Windows: The default for file description in version information was not as intended.
-
Plugins: Workaround for PyQt5 as contained in Anaconda providing wrong paths from the build machine.
-
macOS: After signing a binary with a certificate, compiling the next one was crashing on a warning about initially creating an ad-hoc binary.
-
Fix, detect case of non-writable cache path, make explaining error exit rather than crashing attempting to write to the cache.
-
macOS: Added support for
pyobjc
in version 9.0 or higher.
New Features
-
Python3.11: For now prevent the execution with 3.11 and give a warning to the user for a not yet supported version. This can be overridden with
--experimental=python311
but at this times will not compile anything yet due to required and at this time missing core changes. -
macOS: Added option
--macos-sign-notarization
that signs with runtime signature, but requires a developer certificate from Apple. As its name implies, this is for use with notarization for their App store. -
DLLs used via
delvewheel
were so far only handled in thezmq
plugin, but this has been generalized to cover any package using it. With that, e.g.shapely
just works. This probably helps many other packages as well. -
Added
__compiled__
and__compiled_constant__
attributes to compiled functions.With this, it can be decided per function what it is and bridges like
pyobjc
can use it to create better code on their side for constant value returning functions. -
Added
support_info
check to Nuitka package format. Make it clear thatpyobjc
is only supported after9.0
by erroring out if it has a too low version. It will not work at all before that version added support in upstream. Also using this to make it clear thatopencv-python
is best supported in version 4.6 or higher. It seems e.g. that video capture is not working with 4.5 at this time. -
Added
--report-template
which can be used to provide Jinja2 templates to create custom reports, and refer to built-in reports, at this time e.g. a license reports.
Optimization
-
Trust the absence of a few selected hard modules and convert those to static raises of import errors.
-
For conditional nodes where only one branch exits, and the other does not, no merging of the trace collection should happen. This should enhance the scalability and leads to more static optimization being done after these kinds of branches on variables assigned in such branches.
if condition1: a = 1 else: raise KeyError if condition2: b = 1 # Here, "a" is known to be assigned, but before it was only "maybe" # assigned, like "b" would have to be since, the branch may or may # not have been taken.
-
Do not merge tried blocks that are aborting with an exception handler that is not aborting. This is very similar to the change for conditional statements, again there is a control flow branch, that may have to be merged with an optional part, but sometimes that part is not optional from the perspective of the code following.
try: ... # potentially raising, but not aborting code return something() # this aborts except Exception: a = 1 try: ... # potentially raising, but not aborting code except Exception: b = 1 # Here, "a" is known to be assigned, but before it was only "maybe" # assigned, like "b" would have to be since, the branch may or may # not have been taken.
-
Exception matches were annotating a control flow escape and an exception exit, even when it is known that no error is possible to be happening that comparison.
try: ... except ImportError: # an exception match is done here, that cannot raise ...
-
Trust
importlib.metadata.PackageNotFoundError
to exist, with this some more metadata usages are statically optimized. Added in 1.1.4 already. -
Handle constant values from trusted imports as trusted values. So far, trusted import values were on equal footing to regular variables, which on the module level could mean that no optimization was done, due to control flow escapes happening.
# Known to be False at compile time. from typing import TYPE_CHECKING ... if TYPE_CHECKING: from something import normally_unused_unless_type_checking
In this code example above, the static optimization was not done because the value may be changed on the outside. However, for trusted constants, this is no longer assumed to be happening. So far only
if typing.TYPE_CHECKING:
using code had been optimized. -
macOS: Use sections for main binary constants binary blob rather than C source code (which we started in a recent hotfix due to LTO issues with incbin) and onefile payload. The latter enables notarization of the onefile binary as well and makes it faster to unpack as well.
-
Windows: Do not include DLLs from SxS. For PyPI packages these are generally unused, and self compiled modules won't be SxS installations either. We can add it back where it turns out needed. This avoids including
comctl32
and similar DLLs, which ought to come from the OS, and might impede backward compatibility only. -
Disabled C compilation of very large
azure
modules. -
The per module usage information of other modules was only updated in first pass was used in later passes. But since they can get optimized away, we have to update every time, avoiding to still include unused modules.
-
Anti-Bloat: Fight the use of
dask
inxarray
andpint
, adding a mode controlling its use. This is however still incomplete and needs more work. -
Fix, the anti-bloat configuration for
rich.pretty
introduced aSyntaxError
that went unnoticed. In the future compilation will abort when this happens. -
Standalone: Added support for including DLLs of
llvmlite.binding
package. -
Anti-Bloat: Avoid using
pywin32
throughpkg_resources
import. This seems rather pointless and follows an optimization done for the inline copy of Nuitka already, thectypes
code path works just fine, and this may well be the only reason whypywin32
is included, which is by itself relatively large. -
Cache directory contents when scanning for modules. The
sys.path
and package directories were listed over and over, wasting time during the import analysis. -
Optimization: Was not caching not found modules, but retrying every time for each usage, potentially wasting time during import analysis.
-
Anti-Bloat: Initial work to avoid
pytest
in patsy, it is however incomplete.
Organizational
-
User Manual: Explain how to create 64/32 bits binaries on Windows, with there being no option to control it, this can otherwise be a bit unobvious that you have to just use the matching Python binary.
-
UI: Added an example for a cached onefile temporary location spec to the help output of
--onefile-tempdir-spec
to make cached more easy to achieve in the proper way. -
UI: Quote command line options with space in value better, no need to quote an affected command line option in its entirety, and it looks strange.
-
macOS: Catch user error of disabling the console without using the bundle mode, as it otherwise it has no effect.
-
macOS: Warn about not providing an icon with disabled console, otherwise the dock icon is empty, which just looks bad.
-
Debian: Also need to depend on
glob2
packages which the yaml engine expects to use when searching for DLLs. -
Debian: Pertain inline copies of modules in very old builds, there is e.g. no
glob2
for older releases, but only recent Debian releases need very pure packages, our backport doesn't have to do it right. -
macOS: More reliable detection of Homebrew based Python. Rather than checking file system via its
sitecustomize
contents. The environment variables are only present to some usages. -
Installations with pip did not include all license, README files, etc. which however was intended. Also the attempt to disable bytecode compilation for some inline copies was not effective yet.
-
Renamed
pyzmq
plugin todelvewheel
as it is now absolutely generic and covers all uses of said packaging technique. -
Caching: Use cache directory for cached downloads, rather than application directory, which is just wrong. This will cause all previously cached downloads to become unused and repeated.
-
Quality: Updated development requirements to latest
black
,isort
,yamllint
, andtqdm
. -
Visual Code: Added recommendation for extension for Debian packaging files.
-
Added warning for
PyQt5
usage, since its support is very incomplete. Made thePyQt6
warning more concrete. It seems only Qt threading does not work, which is of course still significant. Instead PySide2 and PySide6 are recommended. -
UI: Have dedicated options group for onefile, the spec for the temporary files is not a Windows option at all anymore. Also move the warnings group to the end, people need to see the inclusion related group first.
-
User Manual: Explain how to create 64/32 bits binaries on Windows, which is not too obvious.
Cleanups
-
Moved PySide plugins DLL search extra paths to the Yaml configuration. In this way it is not dependent on the plugin being active, avoiding cryptic errors on macOS when they are not found.
-
Plugins: Avoid duplicate link libraries due to casing. We are now normalizing the link library names, which avoids e.g.
Shell32
andshell32
to be in the result. -
Cleanups to prepare a PyLint update that so otherwise failed due to encountered issues.
-
Plugins: Pass so called build definitions for generically. Rather than having dedicated code for each, and plugins can now provide them and pass their index to the scons builds.
-
Onefile: Moved file handling code to common code reducing code duplication and heavily cleaning up the bootstrap code generally.
-
Onefile: The CRC32 checksum code was duplicated between constants blob and onefile, and has moved to shared code, with an actual interface to take the checksum.
-
Spelling cleanups resumed, e.g. this time more clearly distinguishing between
run time
andruntime
, the first is when the program executes, but the latter can be an environment provided by a C compiler.
Tests
-
Tests: Added test that applies anti-bloat configuration to all found modules.
-
Tests: Tests: Avoid including unused
nuitka.tools
code in reflected test, which should make it faster. The compiler itself doesn't use that code.
Summary
This release is again mainly a consolidation of previous release, as well as finishing off existing features. Optimization added in previous releases should have all regressions fixed now, again with a strong series of hotfixes.
New optimization was focused around findings with static optimization not being done, but still resulting in general improvements. Letting static optimization drive the effort is still paying off.
Scalability has seen improvements through some of the optimization, this time a lot less anti-bloat work has been done, and some things are only started. The focus will clearly now shift to making this a community effort. Expect postings that document the Yaml format we use.
For macOS specifically, with the sections work, onefile should launch faster, should be more compatible with signing, and can now be used in notarization, so for that platform, things are more round.
For Windows, a few issues with version information and onefile have been addressed. We should be able to use memory mapped view on this platform too, for faster unpacking of the payload, since it doesn't have to go through the file anymore.
28 May 2025 6:31pm GMT
Kay Hayen: Nuitka Release 1.1
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, "download now".
This release contains a large amount of new compatibility features, while consolidating what we have. Scalability should be better in some cases.
Bug Fixes
-
Standalone: Enhanced dependency scan of dependent DLLs to forward the containing package, so it can be searched in as well. This fixed at least PySide on macOS. Fixed in 1.0.1 already.
-
macOS: Enhanced dependency detection to use normalized paths and therefore to be more stable. Fixed in 1.0.1 already.
-
Standalone: Added support for the
networkx
package which uses new support for a function decorator trying to copy function default values. Fixed in 1.0.1 already. -
Standalone: Include data files for
pandas.io.format
package. This one has Jinja2 template files that will be needed when using this package. -
Python3.10: Fix, could crash in case a class was not giving
match
arguments, but the user did attempt to match them. This happened e.g. withrange
objects. Fixed in 1.0.2 already. -
Standalone: Added data files needed for
pyenchant
package. Fixed in 1.0.2 already. -
Python3.10: Fix, matching sequence with
as
assignments in them didn't check for sub-pattern given. Fixed in 1.0.2 already. -
Standalone: Fix, do not attempt to list non-existent
PATH
entries on Windows, these can crash the dependency detection otherwise. Fixed in 1.0.2 already. -
Standalone: Fix, on newer Linux,
linux-vdso.so.1
appears in output ofldd
in a way that suggests it may exist, which of course it does not, this is a kernel virtual DLL. Fixed in 1.0.3 already. -
Fix, comparison expressions could give wrong results as a regression of the new release. Fixed in 1.0.3 already.
-
Fix, on older Python (before 3.6), it could crash on data files defined in the Yaml config. Fixed in 1.0.4 already.
-
Fix, binary operations could give wrong results as a regression of the new release. Fixed in 1.0.4 already.
-
Standalone: Added support for
pyzbar
package. Fixed in 1.0.5 already. -
Standalone: Fix, empty directory structures were not working anymore due to a regression in the last release. Fixed in 1.0.5 already.
-
Windows: Fix, detected Pythons from Windows registry may of course fail to execute, because they were e.g. manually deleted. This would show e.g. in onefile compression. Fixed in 1.0.5 already.
-
Onefile: Fix, using a too old
zstandard
without finding another Python with a suitable one, lead to run time unpacking errors. Fixed in 1.0.6 already. -
Fix, the inline copy of Jinja2 imported
logging
for no good reason, which lead to errors for users who have a module of the same name, that it was then using instead. Fixed in 1.0.6 already. -
Fix, disable LTO mode for Anaconda Python, it is known to not work. Fixed in 1.0.6 already.
-
Linux: Fix, no need to insist on icon path for onefile anymore. Fixed in 1.0.6 already.
-
Standalone: Fix, the new version
certifi
was not working on Windows and 3.10 anymore. Fixed in 1.0.7 already. -
Standalone: Added support for more
rapidfuzz
implicit dependencies. Fixed in 1.0.8 already. -
Standalone: Added support for
vibora
. Fixed in 1.0.8 already. -
Fix, must not expose module name objects to Python import hooks. Fixed in 1.0.8 already.
-
Fix, calls to bound methods of string values generated incorrect calls. Fixed in 1.0.8 already.
-
Fix, do not crash in version detection on
gcc
error exit querying of its version. -
Standalone: Added back support for older versions of the
pyzmq
package. -
Standalone: Ignore
PATH
elements that fail to be listed as a directory. It appears e.g. on Windows, folders can exist, despite being unusable in fact. These can then cause errors in DLL dependency scan. Also avoid havingPATH
set when executing dependency walker, it appears to use it even if not asked to. -
Standalone: Added support for
tzlocal
package. -
Python3.10: Fix,
complex
literals were not working for mappings inmatch
statements. -
Fix,
bool
built-in expressions were not properly annotating exception raises, where the value cannot raise on truth check. -
Standalone: Added support for the
agscheduler
package. Plugins must be done manually still with explicit--include-module
calls. -
Standalone: Added support for using
shapely
in Anaconda as well. -
Debian: Fix, versioned dependency for
libzstd
should also be in package, this should restore Nuitka package builds for Debian Jessie. -
Standalone: Added support for
vtk
package. -
Windows: Fix, avoid using
pywin32
in our appdirs usage, it might be a broken installation and is optional toappdirs
anyway, which then will fallback to usingctypes
to make the lookups. -
Standalone: Added support for more
pandas
versions. -
Standalone: Adding support for
mkl
implicit DLL usage in Anaconda. -
Standalone: Added support for
jsonschema
with Python 3.10. -
Standalone: Added support for
pyfiglet
fonts data files. -
Scons: Avoid gcc linker command line length limits for module mode too.
-
Standalone: Added data file of
distributed.config
. -
Standalone: Add support for
cv2
GUI on Linux, the Qt platform plugin is now included. -
Fix, the anti-bloat configuration for
numpy.testing
tools exposed an incompletesuppress_warnings
replacement that could lead to errors in some functions ofnumpy
. -
Standalone: Fix DLL dependency caching on Windows, need to consider DLL content of course too.
-
Standalone: Added missing dependency for
torchvision
. -
Standalone: Added support for
torchvision
on Anaconda as well. -
Standalone: Added support for
panda3d
. -
Windows: Fix, need to make sure to use UTF-8 encoding for define values like company name. Otherwise the local system encoding is used, but the C compiler expects UTF-8 in wide literals. This may crash of give wrong results.
-
Standalone: Added
facenet_torch
data files. -
Anaconda: Include
libstdc++.so
on Linux or else e.g.cv2
will not work with system library. -
Windows: Fix, can have file version without a company name.
New Features
-
Python3.10: Added support for assignments in
match
alternatives|
syntax. -
Compatibility: Register Nuitka meta path based loader with
pkg_resources
such that checking resource presence withhas_resource
works too. This should also add support for usingjinja2.PackageLoader
, previously onlyjinja2.FileSystemLoader
worked. Fixed in 1.0.1 already. -
Compatibility: Make function
__defaults__
attribute size changeable. For a long time, this was a relatively big issue for some packages, but now this is supported as well. -
Compatibility: Added support for
importlib.metadata.distribution
andimportlib_metadata.distribution
functions as wellimportlib.metadata.metadata
andimportlib_metadata.metadata
functions. -
Onefile: Added support for including other binaries than the main executable in the payload. So far on non-Windows, we only made the main binary executable, hard coded, and nothing else. But Some things, e.g. Qt web engine, do require binaries to be used, and these no longer have the issue of missing x-bit on macOS and Linux now.
-
Standalone: Resolve executable path when called through symbolic link, which makes file resolutions work properly for it, for this type of installing it in
%PATH%
. -
Python3.9+: Added support for
importlib.resources.files
with compiled modules.It returns traversable objects, which can be used to opens files, checks them, etc. and this e.g. allows
jsonschema
to work with Python 3.10, despite bugs in CPython's compatibility layer. -
UI: Added interface method to specify filename patterns with package data inclusion option, making
--include-package-data
usable in many more cases, picking the only files or file types you want. You can now use--include-package-data=package_name=*.txt
and select only a subset of package data files in this way. Before this, it included everything and--noinclude-data-files
would have to be used. -
macOS: Make
runtime
signing an experimental option. -
Consistently allow
when
conditions for all package configuration elements, e.g. also DLLs. -
Plugins: Added method to overload to work on standalone binary specifically. This makes it easier to only modify that specific binary.
-
Plugins: Added support for regular expressions in anti-bloat replacements, with new
replacements_re
code.
Optimization
-
Add support for
os.path
hard module imports along with specialized nodes for file testsos.path.exists
,os.path.isfile
, andos.path.isdir
aiming at tracking used files, producing warnings about missing files in the future. -
Standalone: Do not include
concurrent
standard library package automatically. This avoids the inclusion ofmultiprocessing
which we essentially had reverted during last release cycle. -
Standalone: Do not include
zoneinfo
standard library package automatically. It has many data files and is often not used (yet). -
Standalone: Do not include
asyncio
standard library package automatically anymore. -
Avoid compilation of large generated codes in the
asyncua
package. -
Compile time optimize
pkg_resources.iter_entry_points
too, such that these can be used to resolve plugin modules, which helps with adding support foragscheduler
package plugins. Note that these still need to be manually included with--include-module
but now that works. -
For known truth values of the right hand side of
and
oror
conditions, reduce the expression as far as possible. -
Added dedicated assignment node for hard imports, which then are propagated in classes as well, allowing for more static optimization for code on the class level.
-
Added linker options to make static
--static-libpython
work with clang on Linux as well. -
macOS: Make sure
libpython
is loaded relative to the executable. This is needed for at least Anaconda Python. -
macOS: Fix, need to search environment specific DLL paths and only then global paths, otherwise mixed Python versions will not work correctly.
-
Anti-Bloat: Remove IPython usage in
rich
package. -
Anti-Bloat: Avoid
doctest
dependency when usingpyrect
. -
Anti-Bloat: Some
unittest
removals frompytorch
using libraries. -
Keep the Scons report items sorted, or else it varies for the hashing of dependencies with Python versions before 3.6, causing cache misses without need.
Organizational
-
UI: Output the
.cmd
file created (if any) on Windows, e.g. when run in a virtualenv or for uninstalled Python versions, it will otherwise not run in accelerated mode, but previously the output suggested to run the executable directly. -
UI: Enhanced command line option description of
--include-plugin-directory
which is frequently misunderstood. That option barely does what people want it to do. Point them to using the other options that are easy to use and will work. -
UI: Specified needed Python version for use in
--python-for-scons
so users can know ahead of time what versions are suitable. -
Reports: Added information about data files including, optimization times per module, active plugins.
-
Debugging: Repaired offline DLL dependency listing tool, such that it can be used during Windows DLL analysis.
-
Make
--xml
accept a filename for the node tree dump, and change it so it can be executed in addition to actual compilation. This way we need not be super-robust about keeping stdout clean, to not break XML parsing. -
Plugins: Avoid useless warning about PySide2 plugin usage if another Qt plugin is actually selected.
-
UI: Catch error of directories being used as data files where plain files are expected and point out that other options must be used.
-
User Manual: Added section about accessing files in standalone mode too, so people can make sure it works properly.
-
Onefile: Using
%TEMP%
folder should not by itself prevent cached onefile mode, only really variable paths should. People may want to have this as some kind of temporary cache still. -
UI: Catch user error of using elements, that resolve to absolute values in the middle of path specs, so using e.g.
something/%PROGRAM%
is now a mistake caught at compile time. These values can only be at the start of spec values naturally. -
Quality: Updated to newer version of
rstfmt
. -
UI: Nicer error message when a forbidden import is requested as an implicit import by a plugin.
-
Python3.11: Adapted to allocator and exception state changes, but more will be needed to compile at all.
-
Visual Code: Find
clang-format
from the recommended C++ extension of Visual Code, which makes it finally available on macOS easily too. -
UI: Quote command line argument values as necessary when stating them in the logging. Otherwise they are not directly usable on the shell and also less readable.
-
Debian: Do not list fake modules as used debian packages codes, which could e.g. happen with the pre-load code of
pkg_resources
if that is from a Debian package. Fake packages should not be mentioned for these lists though. -
Nuitka-Python: Added support to set link time flags coming from statically included packages.
-
For our
isort
trick of splitting files in two parts (mostly to setup import paths fornuitka
package), make sure the second parts starts with a new line. -
Added more usable form
--output-filename
to specify the output filename, the short form has become barely usable after we switched to enforcing no space separation for command line arguments. -
UI: Check if output filename's directory exists ahead of time, and error exit if not, otherwise compilation crashed only in the very end, trying to create the final result.
-
UI: When exiting with no error code, do not use red color or
FATAL
error annotation, that is not justified. -
Quality: Make sure the Yaml auto-format does not change effective contents.
-
Quality: Added ability to limit autoformat by file type, which can be handy when e.g. only the yaml files should be scanned.
-
UI: Removed
--windows-onefile-tempdir-spec
alias of--onefile-tempdir-spec
option, it is no longer Windows specific.
Cleanups
-
Prefer single quotes rather than double quotes in our package configuration Yaml files, otherwise esp. regular expressions with escapes become very confusing.
-
Move import hacks to general mechanism in Yaml package configuration files. This is for extra paths from package names or from directory paths relative to the package. This removes special purpose code from core code paths and allows their reuse.
-
Again more spelling cleanups have been done, to make the code cleaner to read and search.
-
Unified how plugins treat iteration over their value list, and how the
when
condition is applied for the various kinds of sections. -
Output compilation command that failed during coverage taking, which makes it unnecessary to attempt to reconstruct what happened from test modes.
Tests
-
Added coverage for comparisons that need argument swaps.
-
Allow more time in onefile keyboard signal test, otherwise it can be a race on slow machines, e.g. emulated machines.
-
Tests: Added support for running a local web server.
Summary
This release is mainly a consolidation of previous release. Optimization added in previous release did in fact introduce regressions, that needed to be addressed and were cause for relatively many hotfixes.
The Yaml Nuitka Package Configuration feature is getting ever more powerful, but is not one bit more documented, such that the community as a whole is not yet capable of adding missing dependencies, data files, DLLs, and even anti-bloat patches.
New optimization was focused around compatibility with very few exceptions, where the non-automatic standard library work is standing out, and allows for smaller binaries in many cases.
Scalability has seen improvements through a few optimization, but mainly again with anti-bloat work being done. This is owed to the fact that consolidation was the name of the game.
For Anaconda specifically, a lot more software is covered, and generally, cv2
and torch
related tools are now working better, but it seems DLL handling will remain problematic in many instances.
The compilation report contains much more information and is getting there is terms of completeness. At some point, we should ask for it in bug reports.
28 May 2025 6:31pm GMT
Kay Hayen: Nuitka Release 1.0
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, "download now".
This release contains a large amount of new features, while consolidating what we have with many bug fixes. Scalability should be dramatically better, as well as new optimization that will accelerate some code quite a bit. See the summary, how this release is paving the way forward.
Bug Fixes
-
Python3: Fix,
bytes.decode
with onlyerrors
argument given was not working. Fixed in 0.9.1 already. -
MSYS2: Fix, the accelerate mode
.cmd
file was not working correctly. Fixed in 0.9.1 already. -
Onefile: Fix, the bootstrap when waiting for the child, didn't protect against signals that interrupt this call. This only affected users of the non-public
--onefile-tempdir
option on Linux, but with that becoming the default in 1.0, this was discovered. Fixed in 0.9.1 already. -
Fix,
pkg_resources
compile time generatedDistribution
values could cause issues with code that put it into calls, or in tried blocks. Fixed in 0.9.1 already. -
Standalone: Added implicit dependencies of
Xlib
package. Fixed in 0.9.1 already. -
macOS: Fix, the package configuration for
wx
had become invalid when restructuring the Yaml with code and schema disagreeing on allowed values. Fixed in 0.9.1 already. -
Fix: The
str.format
with a single positional argument didn't generate proper code and failed to compile on the C level. Fixed in 0.9.1 already. -
Fix, the type shape of
str.count
result was wrong. Fixed in 0.9.1 already. -
UI: Fix, the warning about collision of just compiled package and original package in the same folder hiding the compiled package should not apply to packages without an
__init__.py
file, as those do not take precedence. Fixed in 0.9.2 already. -
Debugging: Fix, the fallback to
lldb
fromgdb
when using the option--debugger
was broken on anything but Windows. Fixed in 0.9.2 already. -
Python3.8: The module
importlib.metadata
was not recognized before 3.9, but actually 3.8 already has it, causing the compile time resolution of package versions to not work there. Fixed in 0.9.3 already. -
Standalone: Fix, at least on macOS we should also scan from parent folders of DLLs, since they may contain sub-directories in their names. This is mostly the case, when using frameworks. Fixed in 0.9.2 already.
-
Standalone: Added package configuration for
PyQt5
to require onefile bundle mode on macOS, and recommend to disable console for PyQt6. This is same as we already do forPySide2
andPySide6
. Fixed in 0.9.2 already. -
Standalone: Removed stray macOS onefile bundle package configuration for
pickle
module which must have been added in error. Fixed in 0.9.2 already. -
UI: Catch user error of attempting to compile the
__init__.py
rather than the package directory. Fixed in 0.9.2 already. -
Fix, hard name import nodes failed to clone, causing issues in optimization phase. Fixed in 0.9.2 already.
-
Fix, avoid warnings given with gcc 11. Fixed in 0.9.2 already.
-
Fix, dictionary nodes where the operation itself has no effect, e.g.
dict.copy
were not properly annotating that their dictionary argument could still cause a raise and have side effects, triggering an assertion violation in Nuitka. Fixed in 0.9.2 already. -
Standalone: Added
pynput
implicit dependencies on Linux. Fixed in 0.9.2 already. -
Fix, boolean condition checks on variables converted immutable constant value assignments to boolean values, leading to incorrect code execution. Fixed in 0.9.2 already.
-
Python3.9: Fix, could crash on generic aliases with non-hashable values. Fixed in 0.9.3 already.
dict[str:any]
-
Python3: Fix, an iteration over
sys.version_info
was falsely optimized into a tuple, which is not always compatible. Fixed in 0.9.3 already. -
Standalone: Added support for
xgboost
package. Fixed in 0.9.3 already. -
Standalone: Added data file for
text_unidecode
package. Fixed in 0.9.4 already. -
Standalone: Added data files for
swagger_ui_bundle
package. Fixed in 0.9.4 already. -
Standalone: Added data files for
connexion
package. Fixed in 0.9.4 already. -
Standalone: Added implicit dependencies for
sklearn.utils
andrapidfuzz
. Fixed in 0.9.4 already. -
Python3.10: Fix, the reformulation of
match
statements could create nodes that are used twice, causing code generation to assert. Fixed in 0.9.4 already. -
Fix, module objects removed from
sys.modules
but still used could lack a reference to themselves, and therefore crash due to working on a released module variables dictionary. Fixed in 0.9.5 already. -
Fix, the MSVC compiles code generated for SciPy 1.8 wrongly. Added a workaround for that code to avoid triggering it. Fixed in 0.9.6 already.
-
Fix, calls to
str.format
where the result is not used, could crash the compiler during code generation. Fixed in 0.9.6 already. -
Standalone: For DLLs on macOS and Anaconda, also consider the
lib
directory of the root environment, as some DLLs are otherwise not found. -
Fix, allow
nonlocal
andglobal
for__class__
to be used on the class level. -
Fix,
xrange
with large values didn't work on all platforms. This affected at least Python2 on macOS, but potentially others as well. -
Windows: When scanning for installed Pythons to e.g. run Scons or onefile compression, it was attempting to use installations that got deleted manually and could crash.
-
macOS: Fix, DLL conflicts are now resolved by checking the version information too, also all cases that previously errored out after a conflict was reported, will now work.
-
Fix, conditional expressions whose statically decided condition picking a branch will raise an exception could crash the compilation.
# Would previously crash Nuitka during optimization. return 1/0 if os.name == "nt" else 1/0
-
Windows: Make sure we set C level standard file handles too
At least newer subprocess was affected by this, being unable to provide working handles to child processes that pass their current handles through, and also this should help DLL code to use it as level.
-
Standalone: Added support for
pyqtgraph
data files. -
Standalone: Added support for
dipy
by anti-bloat removal of its testing framework that wants to do unsupported stuff. -
UI: Could still give warnings about modules not being followed, where that was not true.
-
Fix,
--include-module
was not working for non-automatic standard library paths.
New Features
-
Onefile: Recognize a non-changing path from
--onefile-tempdir-spec
and then use cached mode. By default a temporary folder is used in the spec value, make it delete the files afterwards.The cached mode is not necessarily faster, but it is not going to change files already there, leaving the binaries there intact. In the future it may also become faster to execute, but right now checking the validity of the file takes about as long as re-creating it, therefore no gain yet. The main point, is to not change where it runs from.
-
Standalone: Added option to exclude DLLs. You can npw use
--noinclude-dlls
to exclude DLLs by filename patterns.The may e.g. come from Qt plugins, where you know, or experimented, that it is not going to be used in your specific application. Use with care, removing DLLs will lead to very hard to recognize errors.
-
Anaconda: Use
CondaCC
from environment variables for Linux and macOS, in case it is installed. This can be done with e.g.conda install gcc_linux-64
on Linux orconda install clang_osx-64
on macOS. -
Added new option
--nowarn-mnemonic
to disable warnings that use mnemonics, there is currently not that many yet, but it's going to expand. You can use this to acknowledge the ones you accept, and not get that warning with the information pointer anymore. -
Added method for resolving DLL conflicts on macOS too. This is using version information and picks the newer one where possible.
-
Added option
--user-package-configuration-file
for user provided Yaml files, which can be used to provide package configuration to Nuitka, to e.g. add DLLs, data files, do some anti-bloat work, or add missing dependencies locally. The documentation for this does not yet exist though, but Nuitka contains a Yaml schema in themisc/nuitka-package-config-schema.json
file. -
Added
nuitka-project-else
to avoid repeating conditions in Nuitka project configuration, this can e.g. be used like this:# nuitka-project-if: os.getenv("TEST_VARIANT", "pyside2") == "pyside2": # nuitka-project: --enable-plugin=no-qt # nuitka-project-else: # nuitka-project: --enable-plugin=no-qt # nuitka-project: --noinclude-data-file=*.svg
Previously, the inverted condition had to be used in another
nuitka-project-if
which is no big deal, but less readable. -
Added support for deep copying uncompiled functions. There is now a section in the User Manual that explains how to clone compiled functions. This allows a workaround like this:
def binder(func, name): try: result = func.clone() except AttributeError: result = types.FunctionType(func.__code__, func.__globals__, name=func.__name__, argdefs=func.__defaults__, closure=func.__closure__) result = functools.update_wrapper(result, func) result.__kwdefaults__ = func.__kwdefaults__ result.__name__ = name return result
-
Plugins: Added explicit deprecation status of a plugin. We now have a few that do nothing, and are just there for compatibility with existing users, and this now informs the user properly rather than just saying it is not relevant.
-
Fix, some Python installations crash when attempting to import modules, such as
os
with aModuleName
object, because we limit string operations done, and e.g. refuse to do.startswith
which of course, other loaders that your installation has added, might still use. -
Windows: In case of not found DLLs, we can still examine the run time of the currently compiling Python process of Nuitka, and locate them that way, which helps for some Python configurations to support standalone, esp. to find CPython DLL in unusual spots.
-
Debian: Workaround for
lib2to3
data files. These are from stdlib and therefore the patched code from Debian needs to be undone, to make these portable again.
Optimization
-
Scalability: Avoid merge traces of initial variable versions, which came into play when merging a variable used in only one branch. These are useless and only made other optimization slower or impossible.
-
Scalability: Also avoid merge traces of merge traces, instead flatten merge traces and avoid the duplication doing so. There were pathological cases, where this reduced optimization time for functions from infinite to instant.
-
For comparison helpers, switch comparison where possible, such that there are only 3 variants, rather than 6. Instead the boolean result is inverted, e.g. changing
>=
intonot <
effectively. Of course this can only be done for types, where we know that nothing special, i.e. no method overloads of__gte__
is going on. -
For binary operations that are commutative with the selected types, in mixed type cases, swap the arguments during code generation, such that e.g.
long_a + float_b
is actually computed asfloat_b + long_a
. This again avoids many helpers. It also can be done for*
with integers and container types. -
In cases, where a comparison (or one of the few binary operation where we consider it useful), is used in a boolean context, but we know it is impossible to raise an exception, a C boolean result type is used rather than a
nuitka_bool
which is now only used when necessary, because it can indicate the exception result. -
Anti-Bloat: More anti-bloat work was done for popular packages, covering also uses of
setuptools_scm
,nose
andnose2
package removals and warnings. There was also a focus on makingmmvc
,tensorflow
andtifffile
compile well, removing e.g. the uses of the tensorflow testing framework. -
Faster comparison of
int
values with constant values, this uses helpers that work with Clong
values that represent a single "digit" of a value, or ones that use the full value space of Clong
. -
Faster comparison of
float
values with constant values, this uses helpers that work with Cfloat
values, avoiding the useless Python level constant objects. -
Python2: Comparison of
int
andlong
now has specialized helpers that avoids converting theint
to along
through coercion. This takes advantage of code to compare Clong
values (which are at the core of Python2int
objects, withlong
objects. -
For binary operation on mixed types, e.g.
int * bytes
the slot of the first function was still considered, and called to give aPy_NotImplemented
return value for no good reason. This also applies to mixed operations ofint
,long
, andfloat
types, and forstr
andunicode
values on Python2. -
Added missing helper for
**
operation with floats, this had been overlooked so far. -
Added dedicated nodes for
ctypes.CDLL
which aims to allow us to detect used DLLs at compile time in the future, and to move closer to support its bindings more efficiently. -
Added specialized nodes for
dict.popitem
as well. With this, now all of the dictionary methods are specialized. -
Added specialized nodes for
str.expandtabs
,str.translate
,str.ljust
,str.rjust
,str.center
,str.zfill
, andstr.splitlines
. While these are barely performance relevant, this completes allstr
methods, exceptremoveprefix
andremovesuffix
that are Python3.9 or higher. -
Added type shape for result of
str.index
operation as well, this was missing so far. -
Optimize
str
,bytes
anddict
method calls through variables. -
Optimize calls through variables containing e.g. mutable constant values, these will be rare, because they all become exceptions.
-
Optimize calls through variables containing built-in values, unlocking optimization of such calls, where it is assigned to a local variable.
-
For generated attribute nodes, avoid local doing import statements on the function level. While these were easier to generate, they can only be slow at run time.
-
For the
str
built-in annotate its value as derived fromstr
, which unfortunately does not allow much optimization, since that can still change many things, but it was still a missing attribute. -
For variable value release nodes, specialize them by value type as well, enhancing the scalability, because e.g. parameter variable specific tests, need not be considered for all other variable types as well.
Organizational
-
Plugins: Major changes to the Yaml file content, cleaning up some of the DLL configuration to more easy to use.
The DLL configuration has two flavors, one from code and one from filename matching, and these got separated into distinct items in the Yaml configuration. Also how source and dest paths get provided got simplified, with a relative path now being used consistently and with sane defaults, deriving the destination path from where the module lives. Also what we called patterns, are actually prefixes, as there is still the platform specific DLL file naming appended.
-
Plugins: Move mode checks to dedicated plugin called
options-nanny
that is always enabled, giving also much cleaner Yaml configuration with a new section added specifically for these. It controls advice on the optional or required use of--disable-console
and the like. Some packages, e.g.wx
are known to crash on macOS when the console is enabled, so this advice is now done with saner configuration. -
Plugins: Also for all Yaml configuration sub-items where is now a consistent
when
field, that allows checking Python version, OS, Nuitka modes such as standalone, and only apply configuration when matching this criterion, with that the anti-bloat options to allow certain bloat, should now have proper effect as well. -
The use of
AppImage
on Linux is no more. The performance for startup was always slower, while having lost the main benefit of avoiding IO at startup, due to new cached mode, so now we always use the same bootstrap binary as on macOS and Windows. -
UI: Do not display implicit reports reported by plugins by default anymore. These have become far too many, esp. with the recent stdlib work, and often do not add any value. The compilation report will become where to turn to find out why a module in included.
-
UI: Ask the user to install the ordered set package that will actually work for the specific Python version, rather than making him try one of two, where sometimes only one can work, esp. with Python 3.10 allowing only one.
-
GitHub: More clear wording in the issue template that
python -m nuitka --version
output is really required for support to given. -
Attempt to use Anaconda
ccache
binary if installed on non-Windows. This is esp. handy on macOS, where it is harder to get it. -
Windows: Avoid byte-compiling the inline copy of Scons that uses Python3 when installing for Python2.
-
Added experimental switches to disable certain optimization in order to try out their impact, e.g. on corruption bugs.
-
Reports: Added included DLLs for standalone mode to compilation report.
-
Reports: Added control tags influencing plugin decisions to the compilation report.
-
Plugins: Make the
implicit-imports
dependency section in the Yaml package configuration a list, for consistency with other blocks. -
Plugins: Added checking of tags such from the package configuration, so that for things dependent on python version (e.g.
python39_or_higher
,before_python39
), the usage of Anaconda (anaconda
) or certain OS (e.g.macos
), or modes (e.g.standalone
), expressions inwhen
can limit a configuration item. -
Quality: Re-enabled string normalization from black, the issues with changes that are breaking to Python2 have been worked around.
-
User Manual: Describe using a minimal virtualenv as a possible help low memory situations as well.
-
Quality: The yaml auto-format now properly preserves comments, being based on
ruamel.yaml
. -
Nuitka-Python: Added support for the Linux build with Nuitka-Python for our own CPython fork as well, previously only Windows was working, amd macOS will follow later.
-
The commit hook when installed from git bash was working, but doing so from
cmd.exe
didn't find a proper path for shell from thegit
location. -
Debugging: A lot of experimental toggles were added, that allow control over the use of certain optimization, e.g. use of dict, list, iterators, subscripts, etc. internals, to aid in debugging in situations where it's not clear, if these are causing the issue or not.
-
Added support for Fedora 36, which requires some specific linker options, also recognize Fedora based distributions as such.
-
Removed long deprecated option
--noinclude-matplotlib
from numpy plugin, as it hasn't had an effect for a long time now. -
Visual Code: Added extension for editing Jinja2 templates. This one even detects that we are editing C or Python and properly highlights accordingly.
Cleanups
-
Standalone: Major cleanup of the dependency analysis for standalone. There is no longer a distinction between entry points (main binary, extension modules) and DLLs that they depend on. The OS specific parts got broken out into dedicated modules as well and decisions are now taken immediately.
-
Plugins: Split the Yaml package configuration files into 3 files. One contains now Python2 only stdlib configuration, and another one general stdlib.
-
Plugins: Also cleanup the
zmq
plugin, which was one the last holdouts of now removed plugin method, moving parts to the Yaml configuration. We therefore no longer haveconsiderExtraDlls
which used to work on the standalone folder, but instead only plugin code that provides included DLL or binary objects fromgetExtraDlls
which gives Nuitka much needed control over DLL copying. This was a long lasting battle finally won, and will allow many new features to come. -
UI: Avoid changing whitespace in warnings, where we have intended line breaks, e.g. in case of duplicate DLLs. Went over all warnings and made sure to either avoid new-lines or have them, depending on wanted output.
-
Iterator end check code now uses the same code as rich comparison expressions and can benefit from optimization being done there as well.
-
Solved TODO item about code generation time C types to specify if they have error checking or not, rather than hard coding it.
-
Production of binary helper function set was cleaned up massively, but still needs more work, comparison helper function set was also redesigned.
-
Changing the spelling of our container package to become more clear.
-
Used
namedtuple
objects for storing used DLL information for more clear code. -
Added spellchecker ignores for all attribute and argument names of generated fixed attribute nodes.
-
In auto-format make sure the imports float to the top. That very much cleans up generated attribute nodes code, allowing also to combine the many ones it makes, but also cleans up some of our existing code.
-
The package configuration Yaml files are now sorted according to module names. This will help to avoid merge conflicts during hotfixes merge back to develop and automatically group related entries in a sane way.
-
Moved large amounts of code producing implicit imports to Yaml configuration files.
-
Changed the
tensorflow
plugin to Yaml based configuration, making it a deprecated do nothing plugin, that only remains there for a few releases, to not crash existing build scripts. -
Lots of spelling cleanups, e.g. renaming
nuitka.codegen
tonuitka.code_generation
for clarity.
Tests
-
Added generated test to cover
bytes
method. This would have found the issue withdecode
potentially. -
Enhanced standalone test for
ctypes
on Linux to actually have something to test.
Summary
This release improves on many things at once. A lot of work has been put into polishing the Yaml configuration that now only lacks documentation and examples, such that the community as a whole should become capable of adding missing dependencies, data files, DLLs, and even anti-bloat patches.
Then a lot of new optimization has been done, to close the missing gaps with dict
and str
methods, but before completing list
which is already a work in progress pull request, and bytes
, we want to start and generate the node classes that form the link or basis of dedicated nodes. This will be an area to work on more.
The many improvements to existing code helpers, and them being able to pick target types for the arguments of comparisons and binary operations, is a pre-cursor to universal optimization of this kind. What is currently only done for constant values, will in the future be interesting for picking specific C types for use. That will then be a huge difference from what we are doing now, where most things still have to use PyObject *
based types.
Scalability has again seen very real improvements, memory usage of Nuitka itself, as well as compile time inside Nuitka are down by a lot for some cases, very noticeably. There is never enough of this, but it appears, in many cases now, large compilations run much faster.
For macOS specifically, the new DLL dependency analysis, is much more capable or resolving conflicts all by itself. Many of the more complex packages with some variants of Python, specifically Anaconda will now be working a lot better.
And then, of course there is the big improvement for Onefile, that allows to use cached paths. This will make it more usable in the general case, e.g. where the firewall of Windows hate binaries that change their path each time they run.
Future directions will aim to make the compilation report more concise, and given reasons and dependencies as they are known on the inside more clearly, such that is can be a major tool for testing, bug reporting and analysis of the compilation result.
28 May 2025 6:31pm GMT
Kay Hayen: Nuitka Release 0.9
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, "download now".
This release has a many optimization improvements, and scalability improvements, while also adding new features, with also some important bug fixes.
Bug Fixes
-
Fix, hard module name lookups leaked a reference to that object. Fixed in 0.8.1 already.
-
Python2: Fix,
str.decode
witherrors
as the only argument wasn't working. Fixed in 0.8.1 already. -
Fix, could corrupt created uncompiled class objects
__init__
functions in case of descriptors being used. -
Standalone: Added support for newer
torch
. Fixed in 0.8.1 already. -
Standalone: Added support for newer
torchvision
. Fixed in 0.8.1 already. -
Fix, could compile time crash during initial parsing phase on constant dictionary literals with non-hashable keys.
{ {}:1, }
-
Fix, hard imported sub-modules of hard imports were falsely resolved to their parent. Fixed in 0.8.3 already.
importlib.resources.read_bytes() # gave importlib has no attribute...
-
Windows: Fix, outputs with
--force-stdout-spec
or--force-stderr-spec
were created with the file system encoding on Python3, but they nee to beutf-8
. -
Fix, didn't allow zero spaces in Nuitka project options, which is not expected.
-
Modules: Fix, the
del __file__
in the top level module in module mode caused crashes at run time, when trying to restore the original__file__
value, after the loading CPython corrupted it. -
Python2.6: Fixes for installations without
pkg_resources
. -
Windows: Fix for very old Python 2.6 versions, these didn't have a language assigned that could be used.
-
Security: Fix for CVE-2022-2054 where environment variables used for transfer of information between Nuitka restarting itself, could be used to execute arbitrary code at compile time.
-
Anaconda: Fix, the torch plugin was not working on Linux due to missing DLL dependencies.
-
Fix, static optimization of
importlib.import_module
with a package given, for an absolute import was optimized into the wrong import, package was not ignored as it should be. -
Windows: Installed Python scan could crash on trying installation paths from registry that were manually removed in the mean time, but not through an uninstaller.
-
Standalone: Added missing implicit dependency for
pyreadstat
because parts of standard library it uses are no more automatically included. -
Windows: Could still crash when no
powershell
is available with symlinks, handle this more gracefully. -
Standalone: Added more missing Plotly dependencies, but more work will be needed to complete this.
-
Standalone: Add missing stdlib dependency on
multiprocessing
byconcurrent.futures.process
. -
Standalone: Fix, implicit dependencies assigned to
imageio
on PIL plugins should actually be assigned toPIL.Image
that actually loads them, so it works outside ofimageio
too.
New Features
-
UI: Added new option
--user-package-configuration-file
to allow users to provide extra Yaml configuration files for the Nuitka plugin mechanism to add hidden dependencies, anti-bloat, or data files, for packages. This will be useful for developing PRs to the standard file of Nuitka. Currently the schema is available, but it is not documented very well yet, so not really ready for end users just yet. -
Standalone: Added new
no-qt
plugin as an easy way to prevent all of the Qt bindings from being included in a compilation. -
Include module search path in compilation report.
Optimization
-
Faster dictionary iteration with our own replacement for
PyDict_Next
that avoids the DLL call overhead (in case of non-static libpython) and does less unnecessary checks. -
Added optimization for
str.count
andstr.format
methods as well, this should help in some cases with compile time optimization. -
The node for
dict.update
with only an iterable argument, but no keyword arguments, was in fact unused due to wrongly generated code. Also the form with no arguments wasn't yet handled properly. -
Scalability: Use specialized nodes for pair values, i.e. the representation of
x = y
in e.g. dictionary creations. With constant keys, and values, these avoid full constant value nodes, and therefore save memory and compile time for a lot of code. -
Anti-Bloat: Added more scalability work to avoid including modules that make compilation unnecessarily big.
-
Python3.9+: Faster calls in case of mixed code, i.e. compiled code calling uncompiled code.
-
Removing duplicates and non-existent entries from modules search path should improve performance when locating modules.
-
Optimize calls through variables as well. These are needed for the package resource nodes to properly resolve at compile time from their hard imports to the called function.
-
Hard imported names should also be considered very trusted themselves, so they are e.g. also optimized in calls.
-
Anti-Bloat: Avoid more useless imports in Pandas, Numba, Plotly, and other packages, improving the scalability some more.
-
Added dedicated nodes for
pkg_resources.require
,pkg_resources.get_distribution
,importlib.metadata.version
, andimportlib_metadata.version
, so we can use compile time optimization to resolve their argument values where possible. -
Avoid annotating control flow escape for all release statements. Sometimes we can tell that
__del__
will not execute outside code ever, so this then avoids marking values as escaped, and taking the time to do so. -
Calls of methods through variables on
str
,dict
,bytes
that have dedicated nodes are now also optimized through variables. -
Boolean tests through variables now also are optimized when the original assignment is a compile time constant that is not mutable. This is only basic, but will allow tests on
TYPE_CHECKING
coming from afrom typing import TYPE_CHECKING
statement to be optimized, avoiding this overhead.
Cleanups
-
Changed to
torch
plugin to Yaml based configuration, making it obsolete, it only remains there for a few releases, to not crash existing build scripts. -
Moved older package specific hacks to the Yaml file. Some of these were from hotfixes where the Yaml file wasn't yet used by default, but now there is no need for them anymore.
-
Removed most of the
pkg-resources
plugin work. This is now done during optimization phase and rather than being based on source code matches, it uses actual value tracing, so it immediately covers many more cases. -
Continued spelling improvements, renaming identifiers used in the source that the cspell based extension doesn't like. This aims at producing more readable and searchable code.
-
Generated attribute nodes no longer do local imports of the operation nodes they refer to. This also avoids compile time penalties during optimization that are not necessary.
-
Windows: Avoid useless bytecode of inline copy used by Python3 when installing for Python2, this spams just a lot of errors.
Organizational
-
Removed MSI installers from the download page. The MSI installers are discontinued as Python has deprecated their support for them, as well as Windows 10 is making it harder for users to install them. Using the PyPI installation is recommended on Windows.
-
Merged our Yaml files into one and added schema description, for completion and checking in Visual Code while editing. Also check the schema in
check-nuitka-with-yamllint
which is now slightly misnamed. The schema is in no way final and will see improvements in future releases. -
UI: Nicer progress bar layout that avoids flicker when optimizing modules.
-
UI: When linking, output the total number of object files used, to have that knowledge after the progress bar for C compilation is gone.
-
Quality: Auto-format the package configuration Yaml file for anti-bloat, implicit dependencies, etc.
-
GitHub: Point out the commit hook in the PR template.
-
UI: Nicer output in case of no commercial version is used.
-
Updated the MinGW64 winlibs download used on Windows to the latest version based on gcc 11, the gcc 12 is not yet ready.
-
Git: Make sure we are not affected by
core.autocrlf
setting, as it interferes with auto-format enforcing Unix newlines. -
Removed the MSI downloads. Windows 10 has made them harder to install and Python itself is discontinuing support for them, while often it was only used by beginners, for which it was not intended.
-
Anaconda: Make it more clear how to install static libpython with precise command.
-
UI: Warn about using module from Debian packages. These can be made non-portable to other OSes. Read more on the info page for detailed information.
-
Quality: The auto-format now floats imports to the top for consistency. With few exceptions, it was already done like this. But it makes things easier for generated code.
Tests
-
The reflected test was adapted to preserve
PYTHONPATH
now that module presence influences optimization.
Summary
This release marks a point, that will allow us to open up the compatibility work for implicit dependencies and anti-bloat stuff even further. The Yaml format will need documentation and potentially more refinement, but will open up a future, where latest packages can be supported with just updating this configuration.
The scalability improvements really make a difference for many libraries and are a welcome improvement on both memory usage and compile time. They are achieved by an accord of static optimization of
One optimization aimed at optimizing tuple unpacking, was not finished in time for this release, but will be subject of a future release. It has driven many other improvements though.
Generally, also from the UI, this is a huge step forward. With links to the website for complex topics being started, and the progress bar flicker being removed, the tool has yet again become more user friendly.
28 May 2025 6:31pm GMT
Kay Hayen: Nuitka Release 0.8
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, "download now".
This release has a massive amount of bug fixes, builds on existing features, and adds new ones.
Bug Fixes
-
Windows: Fix, need to ignore cases where shorting in path for external use during compilation gives an permission error. Fixed in 0.7.1 already.
-
Compatibility: Added workaround for
scipy.stats
function copying. Fixed in 0.7.1 already. -
Windows: Fix, detect ARM64 arch of MSVC properly, such that we can give a proper mismatch for the Python architecture. Fixed in 0.7.1 already.
-
Standalone: Fix, using
importlib.metadata
module failed to includeemail
from standard library parts no longer included by default. Fixed in 0.7.1 already. -
macOS: Fix, the dependency parser was using normalized paths where original paths must be used. Fixed in 0.7.1 already.
-
Standalone: Fix, using
shiboken6
module (mostly due toPySide6
) failed to includeargparse
from the standard library from standard library parts no longer included by default. Fixed in 0.7.1 already. -
Onefile: Fix, the detection of a usable Python for compression could crash. Fixed in 0.7.2 already.
-
Onefile: Adding the payload on Windows could run into locks still being held, need to wait in that case. Fixed in 0.7.2 already.
-
Fix, need to include
pkg_resources
as well, we need it for when we use Jinja2, which is more often now. For Python3 this was fixed in 0.7.3 already. Later a version to use with Python2 was added as well. -
Release: The wheels built for Nuitka when installed through URLs were not version specific, but due to different inline copies per OS and Python version, they must not be reused. Therefore we now pretend to contain an extension module, which handles that. Fixed in 0.7.3 already.
-
Standalone: Fix, using
urllib.requests
module failed to includehttp.client
from standard library parts no longer included by default. Fixed in 0.7.3 already. Laterhttp.cookiejar
was added too. -
Standalone: Do not compress MSVC run time library when using
upx
plugin, that is not going to work. Fixed in 0.7.4 already. -
Standalone: Fix, on Windows more files should be included for TkInter to work with all software. Fixed in 0.7.5 already.
-
Distutils: Added support for
package_dir
directive to specify where source code lives. Fixed in 0.7.6 already. -
Standalone: Fix, using
shelve
module failed to includedbm
from standard library parts no longer included by default. Fixed in 0.7.6 already. -
Standalone: Added support for
arcade
data files. Fixed in 0.7.7 already. -
Standalone: Fix, bytecode demotions should use relative filenames rather than original ones. Fixed in 0.7.7 already.
-
Standalone: Fix, must remove extension module objects from
sys.modules
before executing an extension module that will create it. This fixes cases of cyclic dependencies from modules loaded by the extension module. -
Windows: In case of an exception,
clcache
was itself triggering one during its handling, hiding the real exception behind aTypeError
. -
Windows: Improved
clcache
locking behavior, avoiding a race. Also increase characters used for key from 2 to 3 chars, making collisions far more rare. -
Standalone: Added support for
persistent
package. -
Standalone: Added support for newer
tensorflow
package. -
Module: Fix, need to restore the
__file__
and__spec__
values of top level module. It is changed by CPython after import to an incompatible file name, and not our loader, preventing package resources to be found. -
Standalone: Added support for
Crpytodome.Cipher.PKCS1_v1_5
. -
Fix,
pkgutil.iter_modules
without arguments was not working properly with our meta path based loader. -
Windows: Fix, could crash on when the Scons report was written due to directories in
PATH
that failed to encode. -
Compatibility: Fix, positive
divmod
and modulo%
with negative remainders of positive floats was not correct. -
Fix,
str.encode
with onlyerrors
value, but default value for encoding was crashing the compilation. -
Python3.10+: Fix,
match
statement sliced values must be lists, not tuples. -
Standalone: Added support for newer
glfw
andOpenGL
packages. -
Python3: Fix, failed to read bytecode only stdlib files. This affect mostly Fedora Python which does this for encodings.
-
Python3.5+: Fix, two phase loading of modules could release it immediately.
-
Standalone: Added missing dependency for
pydantic
. -
Fix, the
str.split
rejected defaultsep
value with onlymaxsplit
given as a keyword argument. -
Standalone: Added missing dependency of
wsgiref
module. -
Standalone: Added support for
falcon
module. -
Standalone: Added support for
eliot
module. -
Fix, need to mark assigned from variables as escaped. Without it, some aliased loop variables could be misunderstood and falsely statically optimized.
-
Standalone: Added support for newer
uvicorn
package. -
Standalone: Added data files for the
accessible_output2
,babel
,frozendict
, andsound_lib
package. -
Standalone: Added support for newer
sklearn
package. -
Standalone: Added support for
tkinterdnd2
package. -
Python3.7+: Fix, the error message wasn't fully compatible for unsubscriptable type exception messages.
-
Standalone: Fix,
idlelib
from stdlib was always ignored. -
Python3.4+: Fix, the
__spec__.origin
as produced byfind_spec
of our meta path based loader, didn't have the correctorigin
attribute value. -
Standalone: Disable QtPDF plugin of PySide 6.3.0, because it's failing dependency checks. On macOS this was blocking, we will change it to detection if that is necessary in a future release.
-
Standalone: Added support for
orderedmultidict
. -
Standalone: Added support for
clr
module. -
Standalone: Added support for newer
cv2
package.
New Features
-
Added new option
--module-name-choice
to select what value__name__
and__package__
are going to be. With--module-name-choice=runtime
(default for--module
mode), the created module uses the parent package to deduce the value of__package__
, to be fully compatible. The value--module-name-choice=original
(default for other modes) allows for more static optimization to happen. -
Added support for
get_resource_reader
to our meta path based loader. This allows to avoid useless temporary files in caseimportlib.resources.path
is used, due to a bad interaction with the fallback implementation used without it. -
Added support for
--force-stdout-spec```and ``--force-stderr-spec
on all platforms, this was previously limited to Windows. -
Added support for requiring and suggesting modes. In part this was added to 0.7.3 already, and is used e.g. to enforce that on macOS the
wx
will only work as a GUI program and crash unless--disable-console
is specified. These will warn the user or outright error the compilation if something is known to be needed or useful. -
Debian: Detect version information for "Debian Sid". Added in 0.7.4 already, and also improved how Debian/Ubuntu versions are output.
-
Added new option
--noinclude-data-files
to instruct Nuitka to not include data files matching patterns given. Also attached loggers and tags to included data file and include them in the compilation report. -
Distutils: When using
pyproject.toml
withoutsetup.py
so far it was not possible to pass arguments. This is now possible by adding a section like this.[tool.nuitka] # options without an argument are passed as boolean value show-scons = true # options with single values, e.g. enable a plugin of Nuitka enable-plugin = "pyside2" # options with several values, e.g. avoiding including modules, accepts # list argument. nofollow-import-to = ["*.tests", "*.distutils"]
The option names are the same, but without leading dashes. Lists are only needed when passing multiple values with the same option.
-
macOS: Add support for specifying signing identity with
--macos-sign-identity
and access to protected resources--macos-app-protected-resource
. -
Included data files are now reported in the compilation report XML as well.
-
Accept absolute paths for searching paths of binaries. This allows e.g. the
upx
plugin to accept both a folder path and the full path including the binary name to work when you specify the binary location with--upx-binary
making it more user friendly. -
Python3.10: Added support for positional matching of classes, so far only keyword matching was working.
-
Added support for path spec values
%CACHE_DIR
,%COMPANY%
,%PRODUCT%
,%VERSION%
, and%HOME
in preparation of onefile once again being able to be cached and not unpacked repeatedly for each execution. -
Standalone: Detect missing
tk-inter
plugin at run time. When TCL fails to load, it then outputs a more helpful error. This ought to be done for all plugins, where it's not clear if they are needed. -
Anti-Bloat: Added support for plain replacements in the
anti-bloat.yml
file. Before withreplacement`
, the new value had to be produced by aneval
, which makes for less readable values due to extra quoting. for plain values.
Optimization
-
Python3.10+: Added support for
ordered-set
PyPI package to speed up compilation on these versions too, adding a warning if no accelerated form ofOrderedSet
is used, but believed to be usable. -
Optimization: Added
bytes.decode
operations. This is only a start and we needed this for internal usage, more should follow later. -
Much more
anti-bloat
work was added. Avoidingipython
,unittest
, and sometimes evendoctest
usage for some more packages. -
The
ccache
was not always used, sometimes it believed to catch a potential race, that we need to tell it to ignore. This will speed up re-compilation of the C side in many cases. -
Do not compile the meta path based loader separate, which allows us to not expose functions and values only used by it. Also spares the C compiler one file.
-
Added various dedicated nodes for querying package resources data, e.g.
pkgutil.get_data
. This will make it easier to detect cases of missing data files in the future. -
Added more hard imports, some of which help scalability in the compilation, because these are then known to exist in standalone mode, others are used for package resource specific operations.
-
Onefile: Releasing decompression buffers avoiding unnecessary high memory usage of bootstrap binary.
-
Standalone: Avoid proving directories with no DLLs (e.g. from packages) towards
ldd
, this should avoid exceeding command line limits. -
For
clcache
remove writing of the stats file before Scons has completed, which avoids IO and locking churn. -
Standalone: Avoid including
wsgiref
from stdlib by default.
Cleanups
-
Removed references to
chrpath
and dead code around it, it was still listed as a dependency, although we stopped using it a while ago. -
Removed uses of
anti-bloat
in examples and tests, it is now enabled by default. -
Made standard plugin file naming consistent, their name should be
*Plugin.py
. -
Cleaned up
tensorflow
plugin. The source modification was moved toanti-bloat
where it is easy to do. The implicit dependencies are now in the config file ofimplicit-imports
plugin. -
Massive cleanups of data file handling in plugins. Adding methods for producing the now required objects.
-
The Scons file handling got further cleaned up and unified, doing more things in common code.
-
Avoid
#ifdefs
usages with new helper functionNuitka_String_FromFormat
that implies them for more readable code. -
Split the allowance check from the encountering. Allow plugins and options all to say if an import should be followed, and only when that is decided, to complain about it. Previously the attempt was causing an error, even if another plugin were to decide against it later.
-
Python2.6: Avoid warnings from MSVC for out specialized
long
code. In testing it worked correctly, but this is more explicit and doesn't rely on C implementation specific behavior, although it appears to be universal.
Organizational
-
UI: Warning tests are now wrapped to multiple lines if necessary. That makes it more accessible for larger messages that contain more guiding information.
-
Documented how to use local Nuitka checkout with
pyproject.toml
files, this makes debugging Nuitka straightforward in these setups. -
Added instructions on how to pass extra C and linker flags and to the User Manual.
-
Made our auto-format usable for the Nuitka website code too.
-
Removed dependencies on
chrpath
and the now dead code that would use it, we are happy withpatchelf
. -
Updated to latest versions of requirements for development, esp.
black
andpylint
. -
Renamed
--macos-onefile-icon
to--macos-app-icon
because that is what it is really used for. -
Unified how dependencies are installed in GitHub actions.
-
Updated man page contents for option name changes from last releases.
-
Updated the MinGW64 winlibs download used on Windows to the latest version.
-
Updated the
ccache
binary used on Windows with MinGW64. This is in preparation of using it potentially for MSVC as well. -
Updated Visual Code C config to use Python3.10 and MSVC 2022 include paths.
Tests
-
Better outputs from standalone library compilation test, esp. when finding a problem, present the script to reproduce it immediately.
-
Enhanced generated tests to cover
str
methods to use keyword arguments. -
Added automatic execution of
pyproject.toml
driven test case. -
Enhanced output in case of
optimization
test failures, dumping what value is there that has not become a compile time constant.
Summary
This release has seen a lot of consolidation. The plugins layer for data files is now all powerful, allowing much nicer handling of them by the plugins, they are better reported in normal output, and they are also part of the report file that Nuitka can create. You may now also inhibit their inclusion from the command line, if you decide otherwise.
The pyproject.toml
now supporting Nuitka arguments is closing an important gap.
Generally many features got more polished, e.g. non-automatic inclusion of stdlib modules has most problems fixed up.
An important area of improvement, are the hard imports. These will be used to replace the source based resolution of package requirements with ones that are proper nodes in the tree. Getting these hard imports to still retain full compatibility with existing imports, that are more or less __import__
uses only, was revealing quite a bit of technical debt, that has been addressed with this release.
For onefile, the cached mode is being prepared with the variables added, but will only be in a later release.
Also a bunch of new or upgraded packages are working now, and the push for anti-bloat
work has increased, making many compilations even more lean, but scalability is still an issue.
28 May 2025 6:31pm GMT
Kay Hayen: Nuitka Release 0.7
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, "download now".
This release is massively improving macOS support, esp. for M1 and the latest OS releases, but it also has massive improvements for usability and bug fixes in all areas.
Bug Fixes
-
Fix,
set
creation wasn't annotating its possible exception exit from hashing values and is not as free of side effects aslist
andtuple
creations are. Fixed in 0.6.19.1 already. -
Windows: Fix,
--experimental
option values got lost for the C compilation when switching from MSVC to MinGW64, making them have no effect. Fixed in 0.6.19.1 already. -
Windows: Fix, Clang from MinGW64 doesn't support LTO at this time, therefore default to
no
for it. Fixed in 0.6.19.1 already. -
Debian: Fix, failed to detect Debian unstable as suitable for linking, it doesn't have the release number. Fixed in 0.6.19.1 already.
-
Standalone: Added data files for
pygsheets
package. Fixed in 0.6.19.2 already. -
Fix, paths from plugin related file paths need to be made absolute before used internally, otherwise the cache can fail to deduplicate them. Fixed in 0.6.19.2 already.
-
Python3: With gcc before version 5, e.g. on CentOS 7, where we switch to using
g++
instead, the gcc version checks could crash. Fixed in 0.6.19.2 already. -
Windows: Disable MinGW64 wildcard expansion for command line arguments. This was breaking command lines with arguments like
--filename *.txt
, which undercmd.exe
are left alone by the shell, and are to be expanded by the program. Fixed in 0.6.19.2 already. -
Standalone: Added missing implicit dependency needed for
--follow-stdlib
with Python for some uses of thelocale
module. Fixed in 0.6.19.2 already. -
Standalone: Added workarounds for newest
numpy
that wants to set__code__
objects and required improvements for macOS library handling. Fixed in 0.6.19.3 already. -
Windows: Caching of DLL dependencies for the main programs was not really working, requiring to detect them anew for every standalone compilation for no good reason. Fixed in 0.6.19.3 already.
-
Windows: Fix, CTRL-C from a terminal was not propagated to child processes on Windows. Fixed in 0.6.19.4 already.
-
Standalone: With
certifi
and Python3.10 theimportlib.resource
could trigger Virus scanner inflicted file access errors. Fixed in 0.6.19.4 already. -
Python3.10: Reverted error back iteration past end of generator change for Python 3.10.2 or higher to become compatible with that too. Fixed in 0.6.19.5 already.
-
Standalone: Added support for
anyio
and by proxy for Solana. Fixed in 0.6.19.5 already. -
Fix, compilation with resource mode
incbin
and--debugger
was not working together. Fixed in 0.6.19.5 already. -
Fix, format optimization of known
str
objects was not properly annotating an exception exit when being optimized away, causing consistency checks to complain. Fixed in 0.6.19.5 already. -
Windows: Fix,
clcache
didn't work for non-standard encoding source paths due to using th direct mode, where wrong filenames are output by MSVC. Fixed in 0.6.19.5 already. -
Windows: Fix,
ccache
cannot handle source code paths for non-standard encoding source paths. Fixed in 0.6.19.5 already. -
Python2.6: Fix, calls to
iteritems
anditerkeys
on known dictionary values could give wrong values. Fixed in 0.6.19.5 already. -
Fix, the value of
__module__
if set by the metaclass was overwritten when creating types. Fixed in 0.6.19.6 already. -
Plugins: Add support for the latest version of
pkg_resources
that has "vendored" even more packages. Fixed in 0.6.19.6 already. -
Onefile: The onefile binary was locked during run time and could not be renamed, preventing in-place updates. This has been resolved and now on Windows, the standard trick for updating a running binary of renaming it, then placing the new file works.
-
Fix, wasn't checking the
zstandard
version and as a result could crash if too old versions of it. This is now checked. -
macOS: Large amounts of bug fixes for the dependency scanner. It got cleaned up and now handles many more cases correctly.
-
Windows: Fix, was not properly detecting wrong ClangCL architecture mismatch with the Python architecture. This could result in strange errors during C compilation in this setup.
-
Standalone: Added implicit dependencies for the
asyncpg
module. -
Linux: Detect Debian or Ubuntu base and distribution name more reliably. This helps esp. with static libpython optimization being recognized automatically.
New Features
-
We now disallow options that take arguments to be provided without using
=
. -
Previously
--lto no
worked just as well as--lto=no
did. And that was the cause of problems when--lto
first became a choice.Recently similar, but worse problems were observed, where e.g.
--include-module
could swallow trailing other arguments when users forgot to specify the name by accident. Therefore this style of giving options is now explicitly rejected. -
Compiled types of Nuitka now inherit from uncompiled types. This should allow easier and more complete compatibility, making even code in extension modules that uses
PyObject_IsInstance
work, e.g.pydantic
. -
macOS: Added signing of application bundles and standalone binaries for deployment to newer macOS platforms and esp. M1 where these are mandatory for execution.
-
macOS: Added support for selecting the single macOS target arch to create a binary for. The
universal
architecture is not yet supported though, but will be added in a future release. -
Added support for compression in onefile mode through the use of an other Python installation, that has the
zstandard
module installed. With this it will work with 2.6 or higher, but require a 3.5 or higher Python with it installed in eitherPATH
or on Windows in the registry alternatively. -
Added UPX plugin to compress created extension modules and binaries and for standalone mode, the included DLLs. For onefile, the compression is not useful since it has the payload already compressed.
-
Added a more explicit way to list usable MSVC versions with
--msvc=list
rather than requiring an invalid value. Check values given in the same way that Scons will do. -
Added support for
--python-flag=-u
which disabled outputs buffers, so that these outputs are written immediately. -
Plugins: Always on plugins now can have command line options. We want this for the
anti-bloat
plugin that is enabled by default in this release. -
Plugins: Added ability for plugin to provide fake dependencies for a module. We want the this for the
multiprocessing
plugin, that is now enabled by default in this release too. -
Plugins: Added ability for plugins to modify DLLs after copy for standalone. We will be using this in the new
upx
plugin. -
Added retry for file copies that fail due to still running program. This can happen on Windows with DLLs in standalone mode. For interactive compilation, this allows a retry to happen after prompting the user.
-
UI: Added ability to list MSVC versions with
--msvc=list
, and detect illegal values given to--msvc=
before Scons sees them, it also crashes with a relative unhelpful error message. -
UI: When linking, close the C compilation progress bar and state that that linking is going on. For some very large LTO compilations, it was otherwise at 100% and still taking a long time, confusing users.
-
Plugins: Added new plugin that is designed to handle DLL dependencies through a configuration file that can both handle filename patterns as well as code provided DLL locations.
-
Optimization: Exclude parts of the standard library by default. This allows for much smaller standalone distributions on modules, that can be expected to never be an implicit dependency of anything, e.g.
argparse
orpydoc
.
Optimization
-
Standalone: Do not include
encodings.bz2_codec
andencodings.idna
anymore, these are not file system encodings, but require extension modules. -
Make sure we use proper
(void)
arguments for C functions without arguments, as for C functions, that makes a real difference, they are variable args functions and more expensive to call otherwise. -
For standalone, default to using
--python-flag=no_site
to avoid the overhead that the typically unusedsite
module incurs. It often includes large parts of the standard library, which we now want to be more selective about. There is new Python flag added called--python-flag=site
that restores the inclusion ofsite
module. -
Standalone: Exclude non-critical codec modules from being technical, i.e. have to be available at program startup. This removes the need for e.g.
bz2
related extension modules previously included. -
In reformulations, use dictionary methods directly, we have since introduced dictionary specific methods, and avoid the unnecessary churn during optimization.
-
The complex call helper could trigger unnecessary passes in some cases. The pure functions were immediately optimized, but usages in other modules inside loops sometimes left them in incomplete states.
-
Windows: Avoid repeated hashing of the same files over and over for
clcache
. -
Cache dependencies of bytecode demoted modules in first compile and reuse that information in subsequent compilations.
-
Linux: Added option for switching compression method for onefile created with
AppImage
. The default is also nowgzip
and notxz
which has been observed to cause much slower startup for little size gains. -
Standalone: For failed relative imports, during compiled time absolute imports were attempted still and included if successful, the imports would not be use them at run time, but lead to more modules being included than necessary.
Organizational
-
There is now a Discord server for Nuitka community where you can hang out with the developers and ask questions. It is mirrored with the Gitter community chat, but offers more features.
-
The
anti-bloat
is now on by default. It helps scalability by changing popular packages to not provide test frameworks, installation tools etc. in the resulting binary. This oftentimes reduces the compilation by thousands of modules. -
Also the
multiprocessing
plugin is now on by default. Detecting its need automatically removes a source of problems for first time users, that didn't know to enable it, but had all kinds of strange crashes from multiprocessing malfunctioning. This should enhance the out of the box experience by a lot. -
With this release, the version numbering scheme will be changed. For a long time we have used 4 digits, where one is a leading zero. That was initially done to indicate that it's not yet ready. However, that is just untrue these days. Therefore, we switch to 3 digits, and a first hotfix with now be 0.7.1 rather than 0.6.19.1, which is too long.
It has been observed that people disregard differences in the third digit, but actually for Nuitka these have oftentimes been very important updates. This change is to rectify it, and a new release will be
0.8
, and there will be a1.0
release after0.9
. -
Added a new section to User Manual that explains how to manually load files, such that it is cleaner and compatible code. Using paths relative to current directory is not the right way, but there are nice helpers that make it very simple and correct with all kinds of contexts.
-
Report the MSVC version in Scons output during compilation. The 2022 version is required, but we support everything back to 2008, to work on very old systems as well. This will help identifying differences that arise from there.
-
Quality: Find Clang format from MSVC 2022 too. We use in auto format of Nuitka source code, but need to also search that as a new path.
-
Added a spellchecker extension for Visual Code, resulting in many spelling fixes in all kinds of documentation and code. This finds more things than
codespell
, but also has a lot of false alarms. -
Check value of
--onefile-tempdir-spec
for typical user errors. It cannot be.
as that would require to overwrite the onefile binary on Windows, and will generally behave very confusing. Warn about absolute or relative paths going outside of where the binary lives. Can be useful in controlled setups, but not generally. Also warn about using no variables, making non-unique paths. -
macOS: Flavor detection was largely expanded. The
Apple
flavor is recognized on more systems.Homebrew
was newly added, and we actually can detectCPython
reliably as a first. -
Added a tool from leo project to create better
.pyi
files for modules. We will make use of it in the future to enhance the files created by Nuitka to not only contain hidden dependencies, but optionally also module signatures. -
Plugins: Clearer information from
pyside2
that patched wheels might be mandatory and workarounds only patches cannot be done for older Python. -
Added progress bars for DLL dependency detection and DLL copying for standalone. These both can end up using take a fair bit of time depending on project size, and it's nice to know what's going on.
-
macOS: Added support for using both
--onefile
and--macos-create-app-bundle
as it is needed for PySide2 due to issues with signing code now. -
Added warning when attempting to include extension modules in an accelerated compilation.
-
Modules: Catch the user error of following all imports when creating a module. This is very unlikely to produce usable results.
-
Start integrating Sourcery for improved Nuitka code. It will comment on PRs and automatically improve Nuitka code as we develop it.
-
Debugging: Added command line tool
find-module
that outputs how Nuitka locates a module in the Python environment it's ran with. That removes the need to use Python prompt to dump__file__
of imported modules. Some modules even hide parts of their namespace actively during run-time, the tool will not be affected by that.
Cleanups
-
Refactored Python scan previously used for Scons usage on versions that need to run in with another Python to be more generally usable.
-
Use explicit
nuitka.utils.Hashing
module that allows the core to perform these operations with simpler code. -
macOS: Use
isPathBelow
for checking if something is a system library for enhanced robustness and code clarity. -
macOS: Make sure to use our proper error checking wrappers for command execution when using tools like
otool
orcodesign
. -
Standalone: Avoid a temporary file with a script during technical import detection. These have been observed to potentially become corrupted, and this avoids any chance of that happening, while also being simpler code.
-
Avoid naming things
shlib
and call themextension
instead. Inspired by the spell checker disliking the former term, which is also less precise. -
Removed the dead architecture selection option for Windows, it was unused for a long time.
-
Moved Windows
SxS
handling of DLLs to a more general place where also macOS specific tasks are applied, to host standard modification of DLLs during their copying.
Tests
-
Better matching of relative filenames for search modes of the individual test suite runners.
-
Debugger outputs on segfaults were no longer visible and have been restored.
Summary
This release is tremendous progress for macOS. Finally biting the bullet and paying obscene amounts of money to rent an M1 machine, it was possible to enhance the support for this platform. Currently typical packages for macOS are being made compatible as well, so it can now be expected to perform equally well.
On the quality side, the spell checker has had some positive effects, finding typos and generally misspelled code, that codespell
does not, due to it being very conservative.
The trend to enhance plugins has continued. The copying of DLLs is very nearly finalized. Making more plugins enabled by default is seeing a lot of progress, with 2 important ones addressed.
Work on the size of distributions has seen a lot of positive results, in that now standalone distributions are often very minimal, with many extension modules from standard library no longer being present.
28 May 2025 6:31pm GMT
Kay Hayen: Nuitka Release 0.6.9
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, "download now".
This releases contains important bug fixes for regressions of the 0.6.8 series which had relatively many problems. Not all of these could be addressed as hotfixes, and other issues were even very involved, causing many changes to be necessary.
There are also many general improvements and performance work for tracing and loops, but the full potential of this will not be unlocked with this release yet.
Bug Fixes
-
Fix, loop optimization sometimes didn't determinate, effectively making Nuitka run forever, with no indication why. This has been fixed and a mechanism to give up after too many attempts has been added.
-
Fix, closure taking object allowed a brief period where the garbage collector was exposed to uninitialized objects. Fixed in 0.6.8.1 already.
-
Python3.6+: Fix corruption for exceptions thrown into asyncgen. Fixed in 0.6.8.1 already.
-
Fix, deleting variables detected as C type bool could raise an
UnboundLocalError
that was wrong. Fixed in 0.6.8.1 already. -
Python3.8.3+: Fix, future annotations parsing was using hard coded values that were changed in CPython, leading to errors.
-
Windows: Avoid encoding issues for Python3 on more systems, by going from wide characters to unicode strings more directly, avoiding an encoding as UTF-8 in the middle. Fixed in 0.6.8.2 already.
-
Windows: Do not crash when warning about uninstalled MSVC using Python3. This is a Scons bug that we fixed. Fixed in 0.6.8.3 already.
-
Standalone: The output of dependency walker should be considered as "latin1" rather than UTF-8. Fixed in 0.6.8.3 already.
-
Standalone: Added missing hidden dependencies for
flask
. Fixed in 0.6.8.1 already. -
Standalone: Fixed
win32com.client
on Windows. Fixed in 0.6.8.1 already. -
Standalone: Use
pkgutil
to scan encoding modules, properly ignoring the same files as Python does in case of garbage files being there. Fixed in 0.6.8.2 already. -
Plugins: Enabling a plugin after the filename to compile was given, didn't allow for arguments to the passed, causing problems. Fixed in 0.6.8.3 already.
-
Standalone: The
certifi
data file is now supported for all modules using it and not only some. -
Standalone: The bytecode for the standard library had filenames pointing to the original installation attached. While these were not used, but replaced at run time, they increased the size of the binary, and leaked information.
-
Standalone: The path of
sys.executable
was not None, but pointing to the original executable, which could also point to some temporary virtualenv directory and therefore not exist, also it was leaking information about the original install. -
Windows: With the MSVC compiler, elimination of duplicate strings was not active, causing even unused strings to be present in the binary, some of which contained file paths of the Nuitka installation.
-
Standalone: Added support for pyglet.
-
Plugins: The command line handling for Pmw plugin was using wrong defaults, making it include more code than necessary, and to crash if it was not there.
New Features
-
Windows: Added support for using Python 2.7 through a symlink too. This was already working for Python3, but a scons problem prevented this from working.
-
Caching of compiled C files is now checked with ccache and clcache, and added automatically where possible, plus a report of the success is made. This can accelerate the re-compile very much, even if you have to go through Nuitka compilation itself, which is not (yet) cached.
-
Added new
--quiet
option that will disable informational traces that are going to become more. -
The Clang from MSVC installation is now picked up for both 32 and 64 bits and follows the new location in latest Visual Studio 2019.
-
Windows: The
ccache
from Anaconda is now supported as well as the one from msys64.
Optimization
-
The value tracing has become more correct with loops and in general less often inhibits optimization. Escaping of value traces is now a separate trace state allowing for more appropriate handling of actual unknowns.
-
Memory used for value tracing has been lowered by removing unnecessary states for traces, that we don't use anymore.
-
Windows: Prevent scons from scanning for MSVC when asked to use MinGW64. This avoids a performance loss doing something that will then end up being unused.
-
Windows: Use function level linking with MSVC, this will allow for smaller binaries to be created, that don't have to include unused helper functions.
Cleanups
-
The scons file now uses Nuitka utils functions and is itself split up into several modules for enhanced readability.
-
Plugin interfaces for providing extra entry points have been cleaned up and now named tuples are used. Backward compatibility is maintained though.
Organizational
-
The use of the logging module was replaced with more of our custom tracing and we now have the ability to write the optimization log to a separate file.
-
Old style plugin options are now detected and reported as a usage error rather than unknown plugin.
-
Changed submodules to use git over https, so as to not require ssh which requires a key registered and causes problems with firewalls too.
-
More correct Debian copyright file, made formatting of emails in source code consistent.
-
Added repository for Ubuntu focal.
Summary
The main focus of this release has been bug fixes with only a little performance work due to the large amount of regressions and other findings from the last release.
The new constants loading for removes a major scalability problem. The checked and now consistently possible use of ccache
and clcache
allows for much quicker recompilation. Nuitka itself can still be slow in some cases, but should have seen some improvements too. Scalability will have to remain a focus for the next releases too.
The other focus, was to make the binaries contain no original path location, which is interesting for standalone mode. Nuitka should be very good in this area now.
For optimization, the new loop code is again better. But it was also very time consuming, to redo it, yet again. This has prevented other optimization to be added.
And then for correctness, the locals scope work, while very invasive, was necessary, to handle the usage of locals inside of contractions, but also will be instrumental for function inlining to become generally available.
So, ultimately, this release is a necessary intermediate step. Upcoming releases will be able to focus more clearly on run time performance again as well as on scalability for generated C code.
28 May 2025 6:31pm GMT
Kay Hayen: Nuitka Release 0.6.8
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, "download now".
This releases contains important general improvements and performance improvements and enhanced optimization as well as many bug fixes that enhance the Python 3.8 compatibility.
Bug Fixes
-
Python3.5+: Fix, coroutines and asyncgen could continue iteration of awaited functions, even after their return, leading to wrong behaviour.
-
Python3.5+: Fix, absolute imports of names might also refer to modules and need to be handled for module loading as well.
-
Fix, the
fromlist
of imports could loose references, potentially leading to corruption of contained strings. -
Python3.8: Fix, positional only arguments were not enforced to actually be that way.
-
Python3.8: Fix, complex calls with star arguments that yielded the same value twice, were not yet caught.
-
Python3.8: Fix, evaluation order for nested dictionary contractions was not followed yet.
-
Windows: Use short paths, these work much better to load extension modules and TCL parts of TkInter cannot handle Unicode paths at all. This makes Nuitka work in locations, where normal Python cannot.
-
Windows: Fixup dependency walker in unicode input directories.
-
Standalone: Use frozen module loader only at
libpython
initialisation and switch to built-in bytecode loader that is more compatible afterwards, increasing compatibility. -
Standalone: Fix for
pydantic
support. -
Standalone: Added missing hidden dependency of uvicorn.
-
Fix, the parser for
.pyi
files couldn't handle multiline imports. -
Windows: Derive linker arch of Python from running binary, since it can happen that the Python binary is actually a script.
-
Fixup static linking with
libpython.a
that containsmain.o
by making our colliding symbols forPy_GetArgcArgv
weak. -
Python3.7: Fix misdetection as asyncgen for a normal generator, if the iterated value is async.
-
Distutils: Fix
build_nuitka
for modules under nested namespaces. -
OpenBSD: Follow usage of clang and other corrections to make accelerated mode work.
-
macOS: Fixup for standalone mode library scan.
-
Fix, the logging of
--show-modules
was broken. -
Windows: Enable
/bigobj
mode for MSVC for large compilations to work. -
Windows: Fixup crash in warning with pefile dependency manager.
-
Windows: Fixup
win32com
standalone detection of other Python versionwin32com
is in systemPATH
. -
Fix, the python flag for static hashes didn't have the intended effect.
-
Fix, generators may be resurrected in the cause of their destruction, and then must not be released.
-
Fix, method objects didn't implement the methods
__reduce__
and__reduce_ex__
necessary for pickling them. -
Windows: Fix, using a Python installation through a symlink was not working.
-
Windows: Fix, icon paths that were relative were not working anymore.
-
Python3.8: Detect duplicate keywords yielded from star arguments.
-
Fix, methods could not be pickled.
-
Fix, generators, coroutines and asyncgen might be resurrected during their release, allow for that.
-
Fix, frames need to traverse their attached locals to be released in some cases.
New Features
-
Plugin command line handling now allows for proper
optparse
options to be used, doing away with special parameter code for plugins. The arguments now also become automatically passed to the instantiations of plugins.Loading and creation of plugins are now two separate phases. They are loaded when they appear on the command line and can add options in their own group, even required ones, but also with default values.
-
Started using logging with name-spaces. Applying logging per plugin to make it easier to recognize which plugin said what. Warnings are now colored in red.
-
Python3.5+: Added support for two step module loading, making Nuitka loading even more compatible.
-
Enhanced import tracing to work on standalone binaries in a useful manner, allow to compare with normal binaries.
-
Fix, the
setattr
built-in was leaking a reference to theNone
value.
Optimization
-
Proper loop SSA capable of detecting shapes with an incremental initial phase and a final result of alternatives for variables written in the loop. This detects shapes of manual integer incrementing loops correctly now, it doesn't see through iterators yet, but this will come too.
-
Added type shapes for all operations and all important built-in types to allow more compile time optimization and better target type selection.
-
Target type code generation was expanded from manual usage with conditions to all operations allowing to get at bool target values more directly.
-
For in-place operations, there is the infrastructure to generate them for improved performance, but so far it's only used for Python2 int, and not for the many types normal operations are supported.
-
Force usage of C boolean type for all indicator variables from the re-formulation. In some cases, we are not yet there with detections, and this gives instant benefit.
-
Complex constants didn't annotate their type shape, preventing compile time optimization for them.
-
Python3.8: Also support vectorcall for compiled method objects. These are rarely used in new Python, but can make a difference.
-
Remove loops that have only a final break. This happens in static optimization in some cases, and allows more optimization to be done.
-
Avoid using a preparing a constant tuple value for calls with only constant arguments.
-
Avoid using
PyErr_Format
where it's not necessary by adding specialized helpers for common cases. -
Detect
del
statements that will raise an exception and replace with that. -
Exception matching is boolean shape, allowing for faster code generation.
-
Disable recursion checks outside of full compat mode.
-
Avoid large blocks for conditional statements that only need to enclose the condition evaluation.
-
Added shortcuts for interactions between compiled generator variants, to avoid calls to their C methods with argument passing, etc.
Organizational
-
Updated Developer Manual with changes that happened, removing the obsolete language choice section.
-
Added 3.8 support mentions in even more places.
-
The mailing list has been deleted. We now prefer Gitter chat and GitHub issues for discussions.
-
Visual Code recommended extensions are now defined as such in the project configuration and you will be prompted to install them.
-
Visual Code environments for
Py38
andPy27
were added for easier switch. -
Catch usage of Python from the Microsoft App Store, it is not supported and seems to limit access to the Python installation for security reasons that make support impossible.
-
Make it clear that
--full-compat
should not be used in help output. -
Added instructions for MSVC runtimes and standalone compilation to support Windows 7.
-
More complete listing of copyright holders for Debian.
-
Updated to newer black and PyLint.
-
Enhanced gcc version check, properly works with gcc 10 and higher.
Tests
-
Pylint cleanups for some of the tests.
-
Added test for loading of user plugins.
-
Removed useless outputs for
search
mode skipping non-matches.
Cleanups
-
Limit command line handling for multiprocessing module to when the plugin is actually used, avoiding useless code of Windows binaries.
-
Pylint cleanup also foreign code like
oset
andodict
. -
In preparation of deprecating the alternative,
--enable-plugin
has become the only form used in documentation and tests. -
Avoid numeric pylint symbols more often.
-
Distutils: Cleanup module name for distutils commands, these are not actually enforced by distutils, but very ugly in our coding conventions.
-
The "cannot get here" code to mark unreachable code has been improved and no longer needs an identifier passed, but uses the standard C mechanism for that.
-
Removed accessors for lookup sources from nodes, allowing for faster usage and making sure, lookups are only done where needed.
Summary
This release is huge in terms of bugs fixed, but also extremely important, because the new loop SSA and type tracing, allows for many more specialized code usages. We now can trace the type for some loops to be specifically an integer or long value only, and will become able to generate code that avoids using Python objects, in these cases.
Once that happens, the performance will make a big jump. Future releases will have to consolidate the current state, but it is expected that at least an experimental addition of C type float
or C long
can be added, add to that iterator
type shape and value analsis, and an actual jump in performance can be expected.
28 May 2025 6:31pm GMT
Kay Hayen: Nuitka Release 0.6.3
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, "download now".
This has a focus on organizational improvements. With more and more people joining Nuitka, normal developers as well as many GSoC 2019 students, the main focus was to open up the development tools and processes, and to improve documentation.
That said, an impressive amount of bug fixes was contributed, but optimization was on hold.
Bug Fixes
-
Windows: Added support for running compiled binaries in Unicode path names.
-
Standalone: Added support for crytodomex and pycparser packages.
-
Standalone: Added support for OpenSSL support in PyQt on Windows.
-
Standalone: Added support for OpenGL support with QML in PyQt on Windows.
-
Standalone: Added support for SciPy and extended the NumPy plugin to also handle it.
-
UI: The option
--plugin-list
still needed a positional argument to work. -
Make sure
sys.base_prefix
is set correctly too. -
Python3: Also make sure
sys.exec_prefix
andsys.base_exec_prefix
are set correctly. -
Standalone: Added platform plugins for PyQt to the default list of sensible plugins to include.
-
Fix detection of standard library paths that include
..
path elements.
Optimization
-
Avoid using static C++ runtime library when using MinGW64.
New Features
-
Plugins: A plugin may now also generate data files on the fly for a given module.
-
Added support for FreeBSD/PowerPC arch which still uses
gcc
and notclang
.
Organizational
-
Nuitka is participating in the GSoC 2019.
-
Added documentation on how to create or use Nuitka plugins.
-
Added more API doc to functions that were missing them as part of the ongoing effort to complete it.
-
Updated to latest PyLint 2.3.1 for checking the code.
-
Scons: Using newer Scons inline copy with Python 2.7 as, the old one remains only used with Python 2.6, making it easier to know the relevant code.
-
Auto-format was very much enhanced and handles C and ReST files too now. For Python code it does pylint comment formatting, import statement sorting, and blackening.
-
Added script
misc/install-git-hooks.py
that adds a commit hook that runs auto-format on commit. Currently it commits unstaged content and therefore is not yet ready for prime time. -
Moved adapted CPython test suites to GitHub repository under Nuitka Organisation.
-
Moved Nuitka-website repository to GitHub repository under Nuitka Organisation.
-
Moved Nuitka-speedcenter repository to GitHub repository under Nuitka Organisation.
-
There is now a Gitter chat for Nuitka community.
-
Many typo and spelling corrections on all the documentation.
-
Added short installation guide for Nuitka on Windows.
Cleanups
-
Moved commandline parsing helper functions from common code helpers to the main program where of course their only usage is.
-
Moved post-processing of the created standalone binary from main control to the freezer code.
-
Avoid using
chmod
binary to remove executable bit from created extension modules. -
Windows: Avoid using
rt.exe
andmt.exe
to deal with copying the manifest from thepython.exe
to created binaries. Instead use new code that extracts and adds Windows resources. -
Fixed many
ResourceWarnings
on Python3 by improved ways of handling files. -
Fixed deprecation warnings related to not using
collections.abc
. -
The runners in
bin
directory are now formatted withblack
too.
Tests
-
Detect Windows permission errors for two step execution of Nuitka as well, leading to retries should they occur.
-
The salt value for CPython cached results was improved to take more things into account.
-
Tests: Added more trick assignments and generally added more tests that were so far missing.
Summary
With the many organizational changes in place, my normal work is expected to resume for after and yield quicker improvements now.
It is also important that people are now enabled to contribute to the Nuitka web site and the Nuitka speedcenter. Hope is to see more improvements on this otherwise neglected areas.
And generally, it's great to see that a community of people is now looking at this release in excitement and pride. Thanks to everybody who contributed!
28 May 2025 6:31pm GMT
Kay Hayen: Nuitka Release 0.6.2
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, "download now".
This release has a huge focus on organizational things. Nuitka is growing in terms of contributors and supported platforms.
Bug Fixes
-
Fix, the Python flag
--python-flag=-O
was removing doc strings, but that should only be done with--python-flag=-OO
which was added too. -
Fix, accelerated binaries failed to load packages from the
virtualenv
(notvenv
) that they were created and ran with, due to not propagatingsys.prefix
. -
Standalone: Do not include
plat-*
directories as frozen code, and also on some platforms they can also contain code that fails to import without error. -
Standalone: Added missing implicit dependency needed for newer NumPy versions.
New Features
-
Added support for Alpine Linux.
-
Added support for MSYS2 based Python on Windows.
-
Added support for Python flag
--python flag=-OO
, which allows to remove doc strings. -
Added experimental support for
pefile
based dependency scans on Windows, thanks to Orsiris for this contribution. -
Added plugin for proper Tkinter standalone support on Windows, thanks to Jorj for this contribution.
-
There is now a
__compiled__
attribute for each module that Nuitka has compiled. Should be like this now, and contains Nuitka version information for you to use, similar to whatsys.version_info
gives as anamedtuple
for your checks.__nuitka_version__(major=0, minor=6, micro=2, releaselevel="release")
Optimization
-
Experimental code for variant types for
int
andlong
values, that can be plain C value, as well as thePyObject *
. This is not yet completed though. -
Minor refinements of specialized code variants reducing them more often the actual needed code.
Organizational
-
The Nuitka GitHub Organisation that was created a while ago and owns the Nuitka repo now, has gained members. Check out https://github.com/orgs/Nuitka/people for their list. This is an exciting transformation for Nuitka.
-
Nuitka is participating in the GSoC 2019 under the PSF umbrella. We hope to grow even further. Thanks to the mentors who volunteered for this important task. Check out the GSoC 2019 page and thanks to the students that are already helping out.
-
Added Nuitka internal API documentation that will receive more love in the future. It got some for this release, but a lot is missing.
-
The Nuitka code has been
black
-ened and is formatted with an automatic tool now all the way, which makes contributors lives easier. -
Added documentation for questions received as part of the GSoC applications and ideas work.
-
Some proof reading pull requests were merged for the documentation, thanks to everybody who addresses these kinds of errors. Sometimes typos, sometimes broken links, etc.
-
Updated inline copy of Scons used for Python3 to 3.0.4, which hopefully means more bugs are fixed.
Summary
This release is a sign of increasing adoption of Nuitka. The GSoC 2019 is showing early effects, as is more developers joining the effort. These are great times for Nuitka.
This release has not much on the optimization side that is user visible, but the work that has begun is capable of producing glorious benchmarks once it will be finished.
The focus on this and coming releases is definitely to open up the Nuitka development now that people are coming in as permanent or temporary contributors in (relatively) high numbers.
28 May 2025 6:31pm GMT
Kay Hayen: Nuitka Release 0.6.19
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, "download now".
This release adds support for 3.10 while also adding very many new optimization, and doing a lot of bug fixes.
Bug Fixes
-
Calls to
importlib.import_module
with expressions that need releases, i.e. are not constant values, could crash the compilation. Fixed in 0.6.18.1 already. -
After a fix for the previous release, modules that fail to import are attempted again when another import is executed. However, during this initialization for top level module in
--module
mode, this was was done repeatedly, and could cause issues. Fixed in 0.6.18.1 already. -
Standalone: Ignore warning given by
patchelf
on Linux with at least newer OpenSUSE. Fixed in 0.6.18.1 already. -
Fix, need to avoid computing large values out of
<<
operation as well. Fixed in 0.6.18.2 already.# This large value was computed at run time and then if used, also # converted to a string and potentially hashed, taking a long time. 1 << sys.maxint
-
Standalone: Ignore warning given by
patchelf
on Linux about a workaround being applied. -
Fix, calls to
importlib.import_module
were not correctly creating code for dynamic argument values that need to be released, causing the compilation to report the error. Fixed in 0.6.18.1 already. -
MSYS2: Fix, the console scripts are actually good for it as opposed to CPython, and the batch scripts should not be installed. Fixed in 0.6.18.2 already.
-
Setuptools: Added support older version of
setuptools
in metabuild
integration of Nuitka. -
Fix, calls to
importlib.import_module
with 2 arguments that are dynamic, were not working at all. Fixed in 0.6.18.2 already. -
Windows: Compiling with MinGW64 without
ccache
was not working due to issues in Scons. Fixed in 0.6.18.2 already. -
Fix, the
repr
built-in was falsely annotated as producing astr
value, but it can be also derived orunicode
in Python2. -
Fix, attribute nodes were not considering the value they are looking up on. Now that more values will know to have the attributes, that was causing errors. Fixed in 0.6.18.2 already.
-
Fix, left shifting can also produce large values and needs to be avoided in that case, similar to what we do for multiplications already. Fixed in 0.6.18.2 already.
-
UI: The new option
--disable-ccache
didn't really have the intended effect. Fixed in 0.6.18.3 already. -
UI: The progress bar was causing tearing and corrupted outputs, when outputs were made, now using proper
tqdm
API for doing it, this has been solved. Fixed in 0.6.18.4 already. -
Fix, the constant value
sys.version_info
didn't yet have support for its type to be also a compile time constant in e.g. tuples. Fixed in 0.6.18.4 already. -
Onefile: Assertions were not disabled, and on Windows with MinGW64 this lead to including the C filenames of the
zstd
inline copy files and obviously less optimal code. Fixed in 0.6.18.4 already. -
Standalone: Added support for
bottle.ext
loading extensions to resolve at compile time. Fixed in 0.6.18.5 already. -
Standalone: Added support for
seedir
required data file. Fixed in 0.6.18.5 already. -
MSYS2: Failed to link when using the static libpython, which is also now the default for MSYS2. Fixed in 0.6.18.5 already.
-
Python3.6+: Fix, the intended finalizer of compiled
asyncgen
was not present and in fact associated to help type. This could have caused corruption, but that was also very unlikely. Fixed in 0.6.18.5 already. -
Python3: Fix, need to set
__file__
before executing modules, as some modules, e.g. newer PyWin32 use them to locate things during their initialization already. -
Standalone: Handle all PyWin32 modules that need the special DLLs and not just a few.
-
Fix, some
.pth
files create module namespaces with__path__
that does not exist, ignore these in module importing. -
Python2.6-3.4: Fix, modules with an error could use their module name after it was released.
-
Distutils: When providing arguments, the method suggested in the docs is not compatible with all other systems, e.g. not
setuptools_rust
for which a two elements tuple form needs to be used for values. Added support for that and documented its use as well in the User Manual. -
Python3.7+: Do no longer allow deleting cell values, this can lead to corruption and should be avoided, it seems unlikely outside of tests anyway.
-
Standalone: Added support for more ciphers and hashes with
pycryptodome
andpycryptodomex
, while also only including Ciphers when needed. -
Distutils: Was not including modules or packages only referenced in the entry point definition, but not in the list of packages. That is not compatible and has been fixed.
-
Fix, must not expose the constants blob from extension modules, as loading these into a compiled binary can cause issues in this case.
-
Standalone: Added support for including OpenGL and SSL libraries with
PySide2
andPySide6
packages. -
Windows: Fix, the
cmd
files created for uninstalled Python and accelerated programs to find the Python installation were not passing command line arguments. -
Windows: Executing modules with
--run
was not working properly due to missing escaping of file paths. -
Fix, parsing
.pyi
files that make relative imports was not resolving them correctly. -
Python3: Fix, when disabling the console on Windows, make sure the file handles still work and are not
None
. -
Windows: Fix, need to claim all OS versions of Windows as supported, otherwise e.g. high DPI features are not available.
New Features
-
Programs that are to be executed with the
-m
flag, can now be compiled with--python-flag=-m
and will then behave in a compatible way, i.e. load the containing package first, and have a proper__package__
value at run time. -
We now can write XML reports with information about the compilation. This is initially for use in PGO tests, to decide if the expected forms of inclusions have happened and should grow into a proper reporting tool over time. At this point, the report is not very useful yet.
-
Added support for Python 3.10, only
match
statements are not completely supported. Variations with|
matches that also assign are not allowed currently. -
Windows: Allow using
--clang
with--mingw64
to e.g. use theclang.exe
that is contained in the Nuitka automatic download rather thangcc.exe
. -
Added support for Kivy. Works through a plugin that is automatically enabled and needs no other inputs, detecting everything from using Kivy at compile time.
-
Added initial support for Haiku OS, a clone of BeOS with a few differences in their Python installation.
-
Added experimental plugin
trio
that works around issues with that package.
Optimization
-
Also trust hard imports made on the module level in function level code, this unlocks many more static optimization e.g. with
sys.version_info
when the import and the use are not on the same level. -
For the built-in type method calls with generic implementation, we now do faster method descriptor calls. These avoid creating a temporary
PyCFunction
object, that the normal call slot would, this should make these calls faster. Checking them for compiled function, etc. was only wasteful, so this makes it more direct. -
Loop and normal merge traces were keeping assignments made before the loop or inside a branch, that was otherwise unused alive. This should enable more optimization for code with branches and loops. Also unused loop traces are now recognized and removed as well.
-
Avoiding merges of escaped traces with the unescaped trace, there is no point in them. This was actually happening a lot and should mean a scalability improvement and unlock new optimization as well.
-
Avoid escaping un-init traces. Unset values need not be considered as potentially modified as that cannot be done.
-
The
str
shape is now detected through variables, this enables many optimization on the function level. -
Added many
str
operation nodes.These are specifically all methods with no arguments, as these are very generic to add, introduced a base class for them, where we know they all have no effect or raise, as these functions are all guaranteed to succeed and can be served by a common base class.
This covers the
str.capitalize
,str.upper
,str.lower
,str.swapcase
,str.title
,str.isalnum
,str.isalpha
,str.isdigit
,str.islower
,str.isupper
,str.isspace
, andstr.istitle
functions.For static optimization
str.find
andstr.rfind
were added, as they are e.g. used in asys.version.find(...)
style in theos
module, helping to decide to not considerOS/2
only modules.Then, support for
str.index
andstr.rindex
was added, as these are very similar tostr.find
forms, only that these may raise an exception.Also add support for
str.split
andstr.rsplit
which will be used sometimes for code needed to be compile time computed, to e.g. detect imports.Same goes for
endswith
andstartswith
, the later is e.g. popular withsys.platform
checks, and can remove a lot of code from compilation with them now being decided at compile time.Note
A few
str
methods are still missing, with time we will achieve all of them, but this will take time. -
Added trust for
sys.builtin_module_names
as well. Theos
module is using it to make platform determinations. -
When writing constant values, esp.
tuple
,list
, ordict
values, an encoding of "last value" has been added, avoiding the need to repeat the same value again, making many values more compact. -
When starting Nuitka, it usually restarts itself with information collected in a mode without the
site
module loaded, and with hash randomization disabled, for deterministic behaviour. There is a option to prevent this from happening, where the goal is to avoid it, e.g. in testing, say for the coverage taking, but that meant to parse the options twice, which also loads a lot of code.Now only a minimal amount of code is used, and the options are parsed only on the restart, and then an error is raised when it notices, it was not allowed to do so. This also makes code a lot cleaner.
-
Specialized comparison code for Python2
long
and Python3int
code, making these operations much faster to use. -
Specialized comparison code for Python2
unicode
and Python3str
code, making these operations much faster to use, currently only==
and!=
are fully accelerated, the other comparisons will follow. -
Enable static libpython with Python3 Debian packages too. As with Python2, this will improve the performance of the created binary a lot and reduce size for standalone distribution.
-
Comparisons with
in
andnot in
also consider value traces and go through variables as well where possible. So far only the rich comparisons andis
andis not
did that. -
Create fixed import nodes in re-formulations rather than
__import__
nodes, avoiding later optimization doing that, and of course that's simpler code too. -
Python 3.10: Added support for
union
types as compiled time constants. -
Modules are now fully optimized before considering which modules they are in turn using, this avoids temporary dependencies, that later turn out unused, and can shorten the compilation in some cases by a lot of time.
-
On platforms without a static link library, in LTO mode, and with gcc, we can use the
-O3
mode, which doesn't work forlibpython
, but that's not used there. This also includes fake static libpython, as used by MinGW64 and Anaconda on Windows. -
The
anti-bloat
plugin now also handles newersklearn
and knows more about the standard library, and its runners which it will exclude from compilation if use for it. Currently that is not the default, but it should become that.
Organizational
-
Migrated the Nuitka blog from Nikola to Sphinx based ABlog and made the whole site render with Sphinx, making it a lot more usable.
-
Added a small presentation about Nuitka on the Download page, to make sure people are aware of core features.
-
The
gi
plugin is now always on. The copying of thetypelib
whengi
is imported is harmless and people can disable the plugin if that's not needed. -
The
matplotlib
plugin is new and also always on. It previously was part of thenumpy
plugin, which is doing too many unrelated things. Moving this one out is part of a plan to split it up and have it on by default without causing issues. -
MSYS2: Detecting
MinGW
andPOSIX
flavors of this Python. For theMinGW
flavor of MSYS2, the option--mingw64
is now the default, before it could attempt to use MSVC, which is not going to work for it. And also the Tcl and Tk installations of it are being detected automatically for thetk-inter
plugin. -
Added Windows version to Nuitka version output, so we have this for bug reports.
-
User Manual: Added example explaining how to access values from your code in Nuitka project options.
-
UI: For Python flavors where we expect a static libpython, the error message will now point out how to achieve it for each flavor.
-
UI: Disable progress bar when
--show-scons
is used, it makes capturing the output from the terminal only harder. -
UI: Catch error of specifying both
--msvc=
and--mingw64
options. -
Distutils: Improved error messages when using
setuptools
orbuild
integration and failing to provide packages to compile. -
Plugins: Removed now unused feature to rename modules on import, as it was only making the code more complex, while being no more needed after recently adding a place for meta path based importers to be accounted for.
-
Twitter: Use embedded Tweet in Credits, and regular follow button in User Manual.
-
Warnings about imports not done, are now only given when optimization can not remove the usage, and no options related to following have been given.
-
Added Windows version to
--version
output of Nuitka. This is to more clearly recognize Windows 10 from Windows 11 report, and also the odd Windows 7 report, where tool chain will be different. -
In Visual Code, the default Python used is now 3.9 in the "Linux" C configuration. This matches Debian Bullseye.
-
Nicer outputs from check mode of the auto-format as run for CI testing, displays problematic files more clearly.
-
Remove broken links to old bug tracker that is no longer online from the Changelog.
-
UI: When hitting CTRL-C during initial technical import detection, no longer ask to submit a bug report with the exception stack, instead exit cleanly.
-
Windows: Enable LTO mode for MinGW64 and other gcc by default. We require a version that can do it, so take advantage of that.
-
For cases, where code generation of a module takes long, make sure its name is output when CTRL-C is hit.
-
Windows: Splash screen only works with MSVC, added error indicator for MinGW64 that states that and asks for porting help.
Cleanups
-
Generate all existing C code for generic builtin type method calls automatically, and use those for method attribute lookups, making it easier to add more.
-
Changed
TkInter
module to data file providing interface, yielding the 2 directories in question, with a filter fordemos
. -
The importing code got a major overhaul and no longer works with relative filenames, or filenames combined with package names, and module names, but always only with module names and absolute filenames. This cleans up some of the oldest and most complex code in Nuitka, that had grown to address various requirements discovered over time.
-
Major cleanup of Jinja2 template organisation.
Renamed all C templates from
.j2
to.c.j2
for clarity, this was not done fully consistent before. Also move all C templates tonuitka.codegen
package data, it will be confusing to make a difference between ones used during compile time and for the static generation, and the lines are going to become blurry.Added Jinja2 new macro
CHECK_OBJECTS
to avoid branches on argument count in the call code templates. More of these things should be added.Cleanup of code that generates header declarations, there was some duplication going on, that made it hard to generate consistent code.
-
Removed
nuitka.finalizatios.FinalizationBase
, we only have one final visitor that does everything, and that of course makes a lot of sense for its performance. -
Major cleanup of the Scons C compiler configuration setup. Moved things to the dedicate function, and harmonized it more.
-
Resolved deprecation warnings given by with
--python-debug
for Nuitka.
Tests
-
Started test suite for Python PGO, not yet completely working though, it's not yet doing what is needed though.
-
Added generated test that exercises str methods in multiple variations.
-
Revived
reflected
test suite, that had been removed, because of Nuitka special needs. This one is not yet passing again though, due to a few details not yet being as compatible as needed. -
Added test suite for CPython 3.10 and enable execution of tests with this version on GitHub actions.
Summary
This release is another big step forward.
The amount of optimization added is again very large, some of which yet again unlocks more static optimization of module imports, that previously would have to be considered implicit. Now analyzing these on the function level as well, we can start searching for cases, where it could be done, but is not done yet.
After starting with dict
, method optimization has focused on str
which is esp. important for static optimization of imports. The next goal will here be to cover list
which are important for run time performance and currently not yet optimized. Future releases will progress there, and also add more types.
The C type specialization for Python3 has finally progressed, such that it is also covering the long
and unicode
and as such not limited to Python2 as much. The focus now needs to turn back to not working with PyObject *
for these types, but e.g. with += 1
to make it directly work with CLONG
rather than LONG
for which structural changes in code generation will be needed.
For scalability, the anti-bloat
work has not yet progressed as much as to be able to enable it by default. It needs to be more possible to disable it where it causes problems, e.g. when somebody really wants to include pytest
and test frameworks generally, that's something that needs to be doable. Compiling without anti-bloat
plugin is something that is immediately noticeable in exploding module amounts. It is very urgently recommended to enable it for your compilations.
The support for Windows has been further refined, actually fixing a few important issues, esp. for the Qt bindings too.
This release adds support for 3.10 outside of very special match
statements, bringing Nuitka back to where it works great with recent Python. Unfortunately orderedset
is not available for it yet, which means it will be slower than 3.9 during compilation.
Overall, Nuitka is closing many open lines of action with this. The setuptools
support has yet again improved and at this point should be very good.
28 May 2025 6:31pm GMT
Kay Hayen: Nuitka Release 0.6.18
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, "download now".
This release has a focus on new features of all kinds, and then also new kinds of performance improvements, some of which enable static optimization of what normally would be dynamic imports, while also polishing plugins and adding also many new features and a huge amount of organizational changes.
Bug Fixes
-
Python3.6+: Fixes to asyncgen, need to raise
StopAsyncIteration
rather thanStopIteration
in some situations to be fully compatible. -
Onefile: Fix, LTO mode was always enabled for onefile compilation, but not all compilers support it yet, e.g. MinGW64 did not. Fixed in 0.6.17.1 already.
-
Fix,
type
calls with 3 arguments didn't annotate their potential exception exit. Fixed in 0.6.17.2 already. -
Fix, trusted module constants were not working properly in all cases. Fixed in 0.6.17.2 already.
-
Fix,
pkg-resources
exiting with error at compile time for unresolved requirements in compiled code, but these can of course still be optional, i.e. that code would never run. Instead give only a warning, and run time fail on these. Fixed in 0.6.17.2 already. -
Standalone: Prevent the inclusion of
drm
libraries on Linux, they have to come from the target OS at run time. Fixed in 0.6.17.2 already. -
Standalone: Added missing implicit dependency for
ipcqueue
module. Fixed in 0.6.17.3 already. -
Fix, Qt webengine support for everything but
PySide2
wasn't working properly. Partially fixed in 0.6.17.3 already. -
Windows: Fix, bootstrap splash screen code for Windows was missing in release packages. Fixed in 0.6.17.3 already.
-
Fix, could crash on known implicit data directories not present. Fixed in 0.6.17.3 already.
-
macOS: Disable download of
ccache
binary for M1 architecture and systems before macOS 10.14 as it doesn't work on these. Fixed in 0.6.17.3 already. -
Standalone: The
pendulum.locals
handling for Python 3.6 was regressed. Fixed in 0.6.17.4 already. -
Onefile: Make sure the child process is cleaned up even after its successful exit. Fixed in 0.6.17.4 already.
-
Standalone: Added support for
xmlschema
. Fixed in 0.6.17.4 already. -
Standalone: Added support for
curses
on Windows. Fixed in 0.6.17.4 already. -
Standalone: Added support for
coincurve
module. Fixed in 0.6.17.5 already. -
Python3.4+: Up until Python3.7 inclusive, a workaround for stream encoding (was ASCII), causing crashes on output of non-ASCII, other Python versions are not affected. Fixed in 0.6.17.5 already.
-
Python2: Workaround for LTO error messages from older gcc versions. Fixed in 0.6.17.5 already.
-
Standalone: Added support for
win32print
. Fixed in 0.6.17.6 already. -
Fix, need to prevent usage of static
libpython
in module mode or else on some Python versions, linker errors can happen. Fixed in 0.6.17.6 already. -
Standalone: Do not load
site
module early anymore. This might have caused issues in some configurations, but really only would be needed for loadinginspect
which doesn`t depend on it in standalone mode. Fixed in 0.6.17.6 already. -
Fix, could crash with generator expressions in finally blocks of tried blocks that return. Fixed in 0.6.17.7 already.
try: return 9 finally: "".join(x for x in b"some_iterable")
-
Python3.5+: Compatibility of comparisons with
types.CoroutineType
andtypes.AsyncGeneratorType
types was not yet implemented. Fixed in 0.6.17.7 already.# These already worked: assert isinstance(compiledCoroutine(), types.CoroutineType) is True assert isinstance(compiledAsyncgen(), types.AsyncGeneratorType) is True # These now work too: assert type(compiledCoroutine()) == types.CoroutineType assert type(compiledAsyncgen()) == types.AsyncGeneratorType
-
Standalone: Added support for
ruamel.yaml
. Fixed in 0.6.17.7 already. -
Distutils: Fix, when building more than one package, things could go wrong. Fixed in 0.6.17.7 already.
-
Fix, for module mode filenames are used, and for packages, you can specify a directory, however, a trailing slash was not working. Fixed in 0.6.17.7 already.
-
Compatibility: Fix, when locating modules, a package directory and an extension module of the same name were not used according to priority. Fixed in 0.6.17.7 already.
-
Standalone: Added workaround
importlib_resources
insisting on Python source files to exist to be able to load datafiles. Fixed in 0.6.17.7 already. -
Standalone: Properly detect usage of hard imports from standard library in
--follow-stdlib
mode. -
Standalone: Added data files for
opensapi_spec_validator
. -
MSYS2: Fix, need to normalize compiler paths before comparing.
-
Anaconda: For accelerated binaries, the created
.cmd
file wasn't containing all needed environment. -
macOS: Set minimum OS version derived from the Python executable used, this should make it work on all supported platforms (of that Python).
-
Standalone: Added support for automatic inclusion of
xmlschema
package datafiles. -
Standalone: Added support for automatic inclusion of
eel
package datafiles. -
Standalone: Added support for
h5py
package. -
Standalone: Added support for
phonenumbers
package. -
Standalone: Added support for
feedparser
package, this currently depends on theanti-bloat
plugin to be enabled, which will become enabled by default in the future. -
Standalone: Added
gi
plugin for said package that copiestypelib
files and sets the search path for them in standalone mode. -
Standalone: Added necessary datafiles for
eel
package. -
Standalone: Added support for
QtWebEngine
to all Qt bindings and also make it work on Linux. Before only PySide2 on Windows was supported. -
Python3: Fix, the
all
built-in was wrongly assuming that bytes values could not be false, but in fact they are if they contain\0
which is actually false. The same does not happen for string values, but that's a difference to be considered. -
Windows: The LTO was supposed to be used automatically on with MSVC 14.2 or higher, but that was regressed and has been repaired now.
-
Standalone: Extension modules contained in packages, depending on their mode of loading had the
__package__
value set to a wrong value, which at least impacted new matplotlib detection of Qt backend. -
Windows: The
python setup.py install
was installing binaries for no good reason.
New Features
-
Setuptools support. Documented
bdist_nuitka
andbdist_wheel
integration and added support for Nuitka as abuild
package backend inpyproject.toml
files. Using Nuitka to build your wheels is supposed to be easy now. -
Added experimental support for Python 3.10, there are however still important issues with compatibility with the CPython 3.9 test suite with at least asyncgen and coroutines.
-
macOS: For app bundles, version information can be provided with the new option
--macos-app-version
. -
Added Python vendor detection of
Anaconda
,pyenv
,Apple Python
, andpyenv
and output the result in version output, this should make it easiert to analyse reported issues. -
Plugins: Also handle the usage of
__name__
for metadata version resolution of thepkg-resources
standard plugin. -
Plugins: The
data-files
standard plugin now reads configuration from a Yaml file thatdata-files.yml
making it more accessible for contributions. -
Windows: Allow enforcing usage of MSVC with
--msvc=latest
. This allows you to prevent accidental usage of MinGW64 on Windows, when MSVC is intended, but achieves that without fixing the version to use. -
Windows: Added support for LTO with MinGW64 on Windows, this was previously limited to the MSVC compiler only.
-
Windows: Added support for using
--debugger
with the downloaded MinGW64 providedgdb.exe
.Note
It doesn`t work when executed from a Git bash prompt, but e.g. from a standard command prompt.
-
Added new experimental flag for compiled types to inherit from uncompiled types. This should allow easier and more complete compatibility, making even code in extension modules that uses
PyObject_IsInstance
work, providing support for packages likepydantic
. -
Plugins: The Qt binding plugins now resolve
pyqtgraph
selection of binding by hard codingQT_LIB
. This will allow to resolve its own dynamic imports depending on that variable at compile time. At this time, the compile time analysis is not covering all cases yet, but we hope to get there. -
macOS: Provide
minOS
for standalone builds, derived from the setting of the Python used to create it. -
UI: Added new option
--disable-ccache
to prevent Nuitka from injectingccache
(Clang, gcc) andclcache
(MSVC) for caching the C results of the compilation. -
Plugins: Added experimental support for
PyQt6
. While usingPySide2
orPySide6
is very much recommended with Nuitka, this allows its use. -
UI: Added option
--low-memory
to allow the user to specify that the compilation should attempt to use less memory where possible, this increases compile times, but might enable compilation on some weaker machines.
Optimization
-
Added dedicated attribute nodes for attribute values that match names of dictionary operations. These are optimized into dedicate nodes for methods of dictionaries should their expression have an exact dictionary shape. These in turn optimize calls on them statically into dictionary operations. This is done for all methods of
dict
for both Python2 and Python3, namelyget
,items
,iteritems
,itervalues
,iterkeys
,viewvalues
,viewkeys
,pop
,setdefault
,has_key
,clear
,copy
,update
.The new operation nodes also add compile time optimization for being used on constant values where possible.
-
Also added dedicated attribute nodes for string operations. For operations, currently only part of the methods are done. These are currently only
join
,strip
,lstrip
,rstrip
,partition
,rpartition
. Besides performance, this subset was enough to cover compile time evaluation of module name computation forimportlib.import_module
as done by SWIG bindings, allowing these implicit dependencies to be discovered at compile time without any help, marking a significant improvement for standalone usage. -
Annotate type shape for dictionary
in
/not in
nodes, this was missing unlike in the genericin
/not in
nodes. -
Faster processing of "expression only" statement nodes. These are nodes, where a value is computed, but then not used, it still needs to be accounted for though, representing the value release.
something() # ignores return value, means statement only node
-
Windows: Enabled LTO by default with MinGW64, which makes it produce much faster results. It now yield faster binaries than MSVC 2019 with pystone.
-
Windows: Added support for C level PGO (Profile Guided Optimization) with MSVC and MinGW64, allowing extra speed boosts from the C compilation on Windows as well.
-
Standalone: Better handling of
requests.packages
andsix.moves
. The old handling could duplicate their code. Now uses a new mechanism to resolve metapath based importer effects at compile time. -
Avoid useless exception checks in our dictionary helpers, as these could only occur when working with dictionary overloads, which we know to not be the case.
-
For nodes, have dedicated child mixin classes for nodes with a single child value and for nodes with a tuple of children, so that these common kind of nodes operate faster and don't have to check at run time what type they are during access.
-
Actually make use of the egg cache. Nuitka was unpacking eggs in every compilation, but in wheel installs, these can be quite common and should be faster.
-
Star arguments annotated their type shape, but the methods to check for dictionary exactly were not affected by this preventing optimization in some cases.
-
Added
anti-bloat
configuration for main programs present in the modules of the standard library, these can be removed from the compilation and should lower dependencies detected. -
Using static libpython with
pyenv
automatically. This should give both smaller (standalone mode) and faster results as is the case when using this feature.. -
Plugins: Added improvements to the
anti-bloat
plugin forgevent
to avoid including its testing framework. -
Python3.9+: Faster calls into uncompiled functions from compiled code using newly introduced API of that version.
-
Statically optimize
importlib.import_module
calls with constant args into fixed name imports. -
Added support for
sys.version_info
to be used as a compile time constant. This should enable many checks to be done at compile time. -
Added hard import and static optimization for
typing.TYPE_CHECKING
. -
Also compute named import lookup through variables, expanding their use to more cases, e.g. like this:
import sys ... if sys.version_info.major >= 3: ...
-
Also optimize compile time comparisons through variable names if possible, i.e. the value cannot have changed.
-
Faster calls of uncompiled code with Python3.9 or higher avoiding DLL call overhead.
Organizational
-
Commercial: There are
Buy Now
buttons available now for the direct purchase of the Nuitka Commercial offering. Finally Credit Card, Google Pay, and Apple Pay are all possible. This is using Stripe. Get in touch with me if you want to use bank transfer, which is of course still best for me. -
The main script runners for Python2 have been renamed to
nuitka2
andnuitka2-run
, which is consistent with what we do for Python3, and avoids issues wherebin
folder ends up insys.path
and prevents the loading ofnuitka
package. -
Windows: Added support for Visual Studio 2022 by updating the inline copy of Scons used for Windows to version 4.3.0, on non Windows, the other ones will keep being used.
-
Windows: Requiring latest MinGW64 with version 11.2 as released by winlibs, because this is known to allow LTO, where previous releases were missing needed binaries.
-
Reject standalone mode usage with Apple Python, as it works only with the other supported Pythons, avoiding pitfalls in attempting to distribute it.
-
Move hosting of documentation to Sphinx, added Changelog and some early parts of API documentation there too. This gives much more readable results than what we have done so far with Nikola. More things will move there.
-
User Manual: Add description how to access code attributes in
nuitka-project
style options. -
User Manual: Added commands used to generate performance numbers for Python.
-
User Manual: List other Python's for which static linking is supposed to work.
-
Improved help for
--include-package
with a hint how to exclude some of the subpackages. -
Started using Jinja2 in code templates with a few types, adding basic infrastructure to do that. This will be expanded in the future.
-
Updated plugin documentation with more recent information.
-
Added Python flavor as detected to the
--version
output for improved bug reports. -
Linux: Added distribution name to
--version
output for improved bug reports. -
Always enable the
gevent
plugin, we want to achieve this for all plugins, and this is only a step in that direction. -
Added project URLs for PyPI, so people looking at it from there have some immediate places to checkout.
-
Debian: Use common code for included PDF files, which have page styles and automatic corrections for
rst2pdf
applied. -
Updated to latest
black
,isort
,pylint
versions. -
The binary names for Python2 changed from
nuitka
andnuitka-run
tonuitka2
andnuitka2-run
. This harmonizes it with Python2 and avoids issues, where thebin
folder insys.path
can cause issues with re-execution of Nuitka finding those to import.Note
You ought to be using
python -m nuitka
style of calling Nuitka anyway, as it gives you best control over what Python is used to run Nuitka, you can pickpython2
there if you want it to run with that, even with full path. Check the relevant section in the User Manual too. -
Added support for Fedora 34 and Fedora 35.
Cleanups
-
In a change of mind
--enable-plugin
has become the only form to enable a plugin used in documentation and tests. -
Massive cleanup of
numpy
and Qt binding plugins, e.g.pyside2
. Data files and DLLs are now provided through proper declarative objects rather than copied manually. The handling of PyQt5 from the plugin should have improved as a side effect. -
Massive cleanups of all documentation in ReST format. Plenty of formatting errors were resolved. Many typos were identified and globally fixed. Spellings e.g. of "Developer Manual" are now enforced with automatic replacements. Also missing or wrong quotes were turned to proper methods. Also enforce code language for shell scripts to be the same everywhere.
-
Removed last usages of
getPythonFlags()
and made the function private, replacing their use with dedicated function to check for individual flags. -
Avoid string comparison with
nuitka.utils.getOS()
and instead add accessors that are more readable, e.g.nuitka.utils.isMacOS()
and put them to use where it makes sense. -
Replaced usages of string tests in list of python flags specified, with functions that check for a specific name with a speaking function name.
-
Added mixin for expressions that have no side effect outside of their value, providing common method implementation more consistently.
-
Remove code geared to using old PyLint and on Python2, we no longer use that. Also removed annotations only used for overriding Python2 builtins from Nuitka code.
-
The PDF specific annotations were moved into being applied only in the PDF building step, avoiding errors for raw PDF directives.
-
Apply Visual Code auto-format to our Yaml files. This is unfortunately not and automatic formatting yet.
-
Introduce dedicated
nuitka.utils.Json
module, as we intend to expand its usage, e.g. for caching. -
Replacing remaining usages of
print
functions with uses ofnuitka.Tracing
instead. -
Massive cleanup of the
gevent
plugin, user proper method to execute code after module load, rather than source patching without need. The plugin no longer messes with inclusions that other code already provides for standalone. -
Using own helper to update
sys
module attributes, to avoid errors from old C compilers, and also cleaning up using code to not have to cast on string constants. -
More consistent naming of plugin classes, and enforce a relationship of detector class names to the names of detected plugins. The new naming consistency is now enforced.
Tests
-
Added CPython 3.10 test suite, it needs more work though.
-
Added generated test that exercises dictionary methods in multiple variations.
-
Test suite names were specified wrongly in a few of them.
Summary
This release is again a huge step forward. It refines on PGO and LTO for C level to work with all relevant compilers. Internally Python level PGO is prepared, but only a future release will feature it. With that, scalability improvements as well as even more performance improvements will be unlocked.
The amount of optimization added this time is even bigger, some of which unlocks static optimization of module imports, that previously would have to be considered implicit. This work will need one extra step, namely to also trace hard imports on the function level, then this will be an extremely powerful tool to solve these kinds of issues in the future. The next release will have this and go even further in this area.
With the dictionary methods, and some string methods, also a whole new kind of optimization has been started. These will make working with dict
containers faster, but obviously a lot of ground is to cover there still, e.g. list
values are a natural target not yet started. Future releases will progress here.
Type specialization for Python3 has not progressed though, and will have to be featured in a future releases though.
For scalability, the anti-bloat
work has continued, and this should be the last release, where this is not on by default. Compiling without it is something that is immediately noticeable in exploding module amounts. It is very urgently recommended to enable it for your compilations.
The support for macOS has been refined, with version information being possible to add, and adding information to the binary about which OSes are supported, as well as rejecting Apple Python, which is only a trap if you want to deploy to other OS versions. More work will be needed to support pyenv
or even Homebrew there too, for now CPython is still the recommended platform to use.
This release achieves major compatibility improvements. And of course, the experimental support for 3.10 is not the least. The next release will strive to complete the support for it fully, but this should be usable at least, for now please stay on 3.9 if you can.
28 May 2025 6:31pm GMT
Kay Hayen: Nuitka Release 0.6.17
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, "download now".
This release has a focus on performance improvements, while also polishing plugins and adding many new features.
Bug Fixes
-
Fix, plugins were not catching being used on packages not installed. Fixed in 0.6.16.2 already.
-
macOS: Fix weaknesses in the
otool
parsing to determine DLL dependency parsing. Fixed in 0.6.16.2 already. -
Linux: Allow onefile program args with spaces contained to be properly passed. Fixed in 0.6.16.3 already.
-
Windows: Avoid using less portable C function for
%PID%
formatting, which restores compilation on Windows 7 with old toolchains. Fixed in 0.6.16.3 already. -
Standalone: Added support for
fstrings
package. Fixed in 0.6.16.3 already. -
Compatibility: Fix, need to import
.pth
files aftersite
module, not before. This was causing crashes on CentOS7 with Python2. Fixed in 0.6.16.3 already. -
Compatibility: Fix, when extension modules failed to load, in some cases the
ImportError
was lost to aKeyError
. Fixed in 0.6.16.3 already. -
Fix, linker resource modes
code
andlinker
were not working anymore, but are needed with LTO mode at least. Fixed in 0.6.16.3 already. -
Standalone: Bytecode modules with null bytes in standard library, typically from disk corruption, were not handled properly. Fixed in 0.6.16.3 already.
-
Fix, failed
.throw()
into generators could cause corruption. Fixed in 0.6.16.4 already. -
Python2: Fix, the bytecode compilation didn't respect the
--python-flag=no_asserts
mode. Fixed in 0.6.16.4 already. -
Fix, calls were not annotating their arguments as escaped, causing corruption of mutable in static optimization. Fixed in 0.6.16.5 already.
-
Fix, some sequence objects, e.g.
numpy.array
actually implement in-place add operations that need to be called. Fixed in 0.6.16.5 already. -
Windows: Fix, onefile binaries were not working after being signed. This now works.
-
Standalone: Added missing implicit dependency for
sklearn
. -
Compatibility: Modules giving
SyntaxError
from source were not properly handled, giving run timeImportError
. Now they are givingSyntaxError
. -
Fix, the LTO mode has issues with
incbin
usage on older gcc, so uselinker
mode when it is enabled. -
Python3: Fix, locals dict codes were not properly checking errors that the mapping might raise when setting values.
-
Fix, modules named
entry
were causing compile time errors in the C stage. -
macOS: Never include files from OS private frameworks in standalone mode.
-
Fix, the python flag
--python-flag=no_warning
wasn't working on all platforms. -
Compatibility: Fix, the main code of the
site
module wasn't executing, so that its added builtins were not there. Of course, you ought to use--python-flag=no_site
to not have it in the normal case. -
Python2: Added code path to handle edited standard library source code which then has no valid bytecode file.
-
Anaconda: In module mode, the CondaCC wasn't recognized as form of gcc.
-
Fix, bytecode modules could shadow compiled modules of the same name.
-
Onefile: Fix, expansion of
%PID%
wasn't working properly on non-Windows, making temp paths less unique. The time stamp is not necessarily enough. -
Fix,
multiprocessing
error exits from slave processes were not reporting tracebacks. -
Standalone: Added
xcbglintegrations
to the list of sensible Qt plugins to include by default, otherwise rendering will be inferior. -
Standalone: Added
platformthemes
to the list of sensible Qt plugins to include by default, otherwise file dialogs on non-Windows would be inferior. -
Fix, created
.pyi
files were not ordered deterministically. -
Standalone: Added support for
win32file
. -
Fix, namespace packages were not using run time values for their
__path__
value. -
Python3.7+: Fix, was leaking
AttributeError
exceptions during name imports. -
Fix, standard library detection could fail for relative paths.
New Features
-
Added experimental support for C level PGO (Profile Guided Optimization), which runs your program and then uses feedback from the execution. At this time only gcc is supported, and only C compiler is collecting feedback. Check the User Manual for a table with current results.
-
macOS: Added experimental support for creating application bundles. For these, icons can be specified and console can be disabled. But at this time, onefile and accelerated mode are not yet usable with it, only standalone mode works.
-
Plugins: Add support for
pkg_resources.require
calls to be resolved at compile time. These are not working at run time, but this avoids the issue very nicely. -
Plugins: Massive improvements to the
anti-bloat
plugin, it can now makenumpy
,scipy
,skimage
,pywt
, andmatplotlib
use much less packages and has better error handling. -
Plugins: Added
anti-bloat
ability ability to append code to a module, which might get used in the future by other plugins that need some sort of post load changes to be applied. -
Plugins: Added ability to replace code of functions at parse time, and use this in
anti-bloat
plugin to replace functions that do unnecessary stuff with variants that often just do nothing. This is illustrated here.gevent._util: description: "remove gevent release framework" change_function: "prereleaser_middle": "'(lambda data: None)'" "postreleaser_before": "'(lambda data: None)'"
This example is removing
gevent
code that loads dependencies used for their CI release process, that need not be part of normal programs. -
Added ability to persist source code changes done by plugins in the Python installation. This is considered experimental and needs write access to the Python installation, so this is best done in a virtualenv and it may confuse plugins.
-
Added support for
multiprocessing.tracker
and spawn mode for all platforms. For non-default modes outside of Windows, you need to--enable-plugin=multiprocessing
to use these. -
Plugins: Allow multiple entry points to be provided by one or several plugins for the same modules. These are now merged into one automatically.
-
Standalone: Fix for numpy not working when compiling with
--python-flag=no_docstrings
. -
Fix, method calls were not respecting descriptors provided by types with non-generic attribute lookups.
-
Windows: Add support for using self-compiled Python3 from the build folder too.
-
Added support for Nuitka-Python 2.7, which will be our faster Python fork.
-
Colorized output for error outputs encountered in Scons, these are now yellow for better recognition.
Optimization
-
Faster threading code was used for Python3.8 or higher, and this has been extended to 3.7 on Windows, but we won't be able to have it other platforms and not on earlier Python3 versions.
-
Faster calls esp. with keyword arguments. Call with keywords no longer create dictionaries if the call target supports that, and with 3.8 or higher, non-compiled code that allows vectorcall is taken advantage of.
-
Faster class creation that avoids creation of argument tuples and dictionaries.
-
Faster attribute check code in case of non-present attributes.
-
Faster unbound method calls, unlike bound methods calls these were not optimized as well yet.
-
Type shapes for star arguments are now known and used in optimization.
def f(*args, **kwargs): type(args) # Statically known to be tuple type(kwargs) # Statically known to be dict
-
Python2: Faster old-style class creation. These are classes that do not explicitly inherit from
object
. -
Python2: Faster string comparisons for Python by specializing for the
str
type as well. -
Python3: Added specialization for
bytes
comparisons too. These are naturally very much the same asstr
comparisons in Python2. -
Added specialization for
list
comparisons too. We had them fortuples
only so far. -
Faster method calls when called from Python core, our
tp_call
slot wasn't as good as it can be. -
Optimization: Faster deep copies of constants. This can speed up constant calls with mutable types. Before it was checking the type too often to be fast.
-
Allow using static linking with Debian Python giving much better performance with the system Python. This is actually a huge improvement as it makes things much faster. So far it's only automatically enabled for Python2, but it seems to work for Python3 on Debian too. Needs more tweaking in the future.
-
Optimization: Added
functools
module to the list of hard imports in preparation of optimizingfunctools.partial
to work better with compiled functions. -
Python2: Demote to
xrange
when iterating overrange
calls, even for small ranges, they are always faster. Previously this was only done for values with at least 256 values. -
Enable LTO automatically for Debian Python, this also allows more optimization.
-
Enable LTO automatically for Anaconda with CondaCC on non-Windows, also allowing more optimization.
Organizational
-
Added section in the User Manual on how to deal with memory issues and C compiler bugs. This is a frequent topic and should serve as a pointer for this kind of issue.
-
The
--lto
option was changed to require an argument, so that it can also be disabled. The default isauto
which is the old behaviour where it's enabled if possible. -
Changed
--no-progress
to--no-progressbar
in order to make it more clear what it's about. Previously it was possible to relate it to--show-progress
. -
No longer require specific versions of dependencies in our
requirements.txt
and relegate those to only being inrequirements-devel.txt
such that by default Nuitka doesn't collide with user requirements on those same packages which absolutely all the time don't really make a difference. -
Added ability to check all unpushed changes with pylint with a new
./bin/check-nuitka-with-pylint --unpushed
option. Before it was only possible to make the check (quickly) with--diff
, but that stopped working after commits are made. -
Revived support for
vmprof
based analysis of compiled programs, but it requires a fork of it now. -
Make Windows specific compiler options visible on all platforms. There is no point in them being errors, instead warnings are given when they are specified on non-Windows.
-
Added project variable
Commercial
for use in Nuitka project syntax. -
Consistent use of metavars for nicer help output should make it more readable.
-
Avoid
ast
tree dumps in case ofKeyboardInterrupt
exceptions, they are just very noisy. Also not annotate where Nuitka was in optimization when a plugin is asking tosysexit
.
Cleanups
-
Encoding names for UTF-8 in calls to
.encode()
were used inconsistent with and without dashes in the source code, added cleanup to auto-format that picks the one blessed. -
Cleanup taking of run time traces of DLLs used in preparation for using it in main code eventually, moving it to a dedicated module.
-
Avoid special names for Nuitka options in test runner, this only adds a level of confusion. Needs more work in future release.
-
Unify implementation to create modules into single function. We had 3 forms, one in recursion, one for main module, and one for plugin generated code. This makes it much easier to understand and use in plugins.
-
Further reduced code duplication between the two Scons files, but more work will be needed there.
-
Escaped variables are still known to be assigned/unassigned rather than unknown, allowing for many optimizations to still work on them., esp. for immutable value
-
Enhanced auto-format for rest documents, bullet list spacing is now consistent and spelling of organizational is unified automatically.
-
Moved icon conversion functionality to separate module, so it can be reused for other platforms more easily.
Tests
-
Removed
reflected
test, because of Nuitka special needs to restart with variable Python flags. This could be reverted though, since Nuitka no longer needs anything outside inline copies, and therefore no longer loads from site packages. -
Use
anti-bloat
plugin in standalone tests of Numpy, Pandas and tests to reduce their compile times, these have become much more manageable now. -
Enhanced checks for used files to use proper below path checks for their ignoring.
-
Remove reflected test, compiling Nuitka with Nuitka has gotten too difficult.
-
Verify constants integrity at program end in debug mode again, so we catch corruption of them in tests.
Summary
This release is one of the most important ones in a long time. The PGO and LTO, and static libpython work make a big different for performance of created binaries.
The amount of optimization added is also huge, calls are much faster now, and object creations too. These avoiding to go through actual dictionaries and tuples in most cases when compiled code interacts gives very significant gains. This can be seen in the increase of pystone performance.
The new type specializations allow many operations to be much faster. More work will follow in this area and important types, str
and int
do not have specialized comparisons for Python3, holding it back somewhat to where our Python2 performance is for these things.
For scalability, the anti-bloat
work is extremely valuable, and this plugin should become active by default in the future, for now it must be strongly recommended. It needs more control over what parts you want to deactivate from it, in case of it causing problems, then we can and should do it.
The support for macOS has been enhanced a lot, and will become perfect in the next release (currently develop). The bundle mode is needed for all kinds of GUI programs to not need a console. This platform is becoming as well supported as the others now.
Generally this release marks a huge step forward. We hope to add Python level PGO in the coming releases, for type knowledge retrofitted without any annotations used. Benchmarks will become more fun clearly.
28 May 2025 6:31pm GMT
Kay Hayen: Nuitka Release 0.6.15
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, "download now".
This release polished previous work with bug fixes, but there are also important new things that help make Nuitka more usable, with one important performance improvement.
Bug Fixes
-
Fix, hard imports were not automatically used in code generation leading to errors when used. Fixed in 0.6.14.2 already.
-
Windows: Fix,
clcache
was disabled by mistake. Fixed in 0.6.14.2 already. -
Standalone: Added data files for
jsonschema
to be copied automatically. -
Standalone: Support for
pendulum
wasn't working anymore with the last release due to plugin interface issues. -
Retry downloads without SSL if that fails, as some Python do not have working SSL. Fixed in 0.6.14.5 already.
-
Fix, the
ccache
path wasn't working if it contained spaces. Fixed in 0.6.14.5 already. -
Onefile: For Linux and ARM using proper download off appimage. Fixed in 0.6.14.5 already.
-
Standalone: Added support for
pyreadstat
. Fixed in 0.6.14.5 already. -
Standalone: Added missing dependencies for
pandas
. Fixed in 0.6.14.6 already. -
Standalone: Some preloaded packages from
.pth
do not have a__path__
, these can and must be ignored. -
Onefile: On Linux, the
sys.argv[0]
was not the original file as advertised. -
Standalone: Do not consider
.mesh
and.frag
files as DLls in the Qt bindings when including the qml support. This was causing errors on Linux, but was generally wasteful. -
Fix, project options could be injected twice, which could lead to errors with options that were only allowed once, e.g.
--linux-onefile-icon
. -
Windows: When updating the resources in created binaries, treat all kinds of
OSError
with information output. -
Onefile: Remove onefile target binary path at startup as well, so it cannot cause confusion after error exit.
-
Onefile: In case of error exit from
AppImage
, preserve its outputs and attempt to detect if there was a locking issue. -
Standalone: Scan package folders on Linux for DLLs too. This is necessary to properly handle
PyQt5
in case of Qt installed in the system as well. -
Standalone: On Linux, standard QML files were not found.
-
Standalone: Enforce C locale when detecting DLLs on Linux, otherwise whitelisting messages didn't work properly on newer Linux.
-
Standalone: Added support for
tzdata
package data files. -
Standalone: Added support for
exchangelib
. -
Python3.9: Fix, type subscripts could cause optimization errors.
-
UI: Project options didn't properly handle quoting of arguments, these are normally removed by the shell.
-
Linux: The default icon for Python in the system is now found with more version specific names and should work on more systems.
-
Standalone: Do not include
libstdc++
as it should come from the system rather.
New Features
-
Added plugin
anti-bloat
plugin, intended to fight bloat. For now it can make including certain modules an error, a warning, or forceImportError
, e.g.--noinclude-setuptools-mode=nofollow
is very much recommended to limit compilation size. -
The
pkg-resources
builtin now coversmetadata
and importlib_metadata packages for compile time version resolution as well. -
Added support for
PySide2
on Python version before 3.6, because the patched code needs no workarounds. Fixed in 0.6.14.3 already. -
Windows: Convert images to icon files on the fly. So now you can specify multiple PNG files, and Nuitka will create an icon out of that automatically.
-
macOS: Automatically download
ccache
binary if not present. -
Plugins: New interface to query the main script path. This allows plugins to look at its directory.
-
UI: Output the versions of Nuitka and Python during compilation.
-
UI: Added option to control static linking. So far this had been enabled only automatically for cases where we are certain, but this allows to force enable or disable it. Now an info is given, if Nuitka thinks it might be possible to enable it, but doesn't do it automatically.
-
UI: Added
--no-onefile
to disable--onefile
from project options.
Optimization
-
Much enhanced GIL interaction with Python3.9 giving a big speed boost and better threading behaviour.
-
Faster conversion of iterables to
list
, if size can be know, allocation ahead saves a lot of effort. -
Added support for
GenericAlias
objects as compile time constants.
Organizational
-
Enhanced GitHub issue raising instructions.
-
Apply
rstfmt
to all documentation and make it part of the commit hook. -
Make sure to check Scons files as well. This would have caught the code used to disable
clcache
temporarily. -
Do not mention Travis in PR template anymore, we have stopped using it.
-
Updated requirements to latest versions.
-
Bump requirements for development to 3.7 at least, toosl like black do not work with 3.6 anymore.
-
Started work on Nuitka Python, a CPython fork intended for enhanced performance and standalone support with Nuitka.
Cleanups
-
Determine system prefix without virtualenv outside of Scons, such that plugins can share the code. There was duplication with the
numpy
plugin, and this will only be more complete using all approaches. This also removes a lot of noise from the scons file moving it to shared code. -
The Qt plugins now collect QML files with cleaner code.
Tests
-
Nicer error message if a wrong search mode is given.
-
Windows: Added timeout for determining run time traces with dependency walker, sometimes this hangs.
-
Added test to cover the zip importer.
-
Making use of project options in onefile tests, making it easier to execute them manually.
Summary
For Windows, it's now easier than ever to create an icon for your deployment, because you can use PNG files, and need not produce ICO files anymore, with Nuitka doing that for you.
The onefile for Linux had some more or less severe problems that got addressed, esp. also when it came to QML applications with PySide.
On the side, we are preparing to greatly improve the caching of Nuitka, starting with retaining modules that were demoted to bytecode. There are changes in this release, to support that, but it's not yet complete. We expect that scalability will then be possible to improve even further.
Generally this is mostly a maintenance release, which outside of the threading performance improvement has very little to offer for faster execution, but that actually does a lot. Unfortunately right now it's limited to 3.9, but some of the newer Python's will also be supported in later releases.
28 May 2025 6:31pm GMT
Kay Hayen: Nuitka Release 0.6.13
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, "download now".
This release follows up with yet again massive improvement in many ways with lots of bug fixes and new features.
Bug Fixes
-
Windows: Icon group entries were not still not working properly in some cases, leading to no icon or too small icons being displayed. Fixed in 0.6.12.2 already.
-
Windows: Icons and version information were copied from the standalone executable to the onefile executable, but that failed due to race situations, sometimes reproducible. Instead we now apply things to both independently. Fixed in 0.6.12.2 already.
-
Standalone: Fixup scanning for DLLs with
ldconfig
on Linux and newer versions making unexpected outputs. Fixed in 0.6.12.2 already. -
UI: When there is no standard input provided, prompts were crashing with
EOFError
when--assume-yes-for-downloads
is not given, change that to defaulting to"no"
instead. Fixed in 0.6.12.2 already. -
Windows: Detect empty strings for company name, product name, product and file versions rather than crashing on them later. Them being empty rather than not there can cause a lot of issues in other places. Fixed in 0.6.12.2 already.
-
Scons: Pass on exceptions during execution in worker threads and abort compilation immediately. Fixed in 0.6.12.2 already.
-
Python3.9: Still not fully compatible with typing subclasses, the enhanced check is now closely matching the CPython code. Fixed in 0.6.12.2 already.
-
Linux: Nicer error message for missing
libfuse
requirement. -
Compatibility: Lookups on dictionaries with
None
value giving aKeyError
exception, but with no value, which is not what CPython does. -
Python3.9: Fix, future annotations were crashing in debug mode. Fixed in 0.6.12.3 already.
-
Standalone: Corrections to the
glfw
were made. Fixed in 0.6.12.3 already. -
Standalone: Added missing ìmplicit dependency for
py.test
. Fixed in 0.6.12.3 already. -
Standalone: Adding missing implicit dependency for
pyreadstat
. -
Windows: Added workaround for common clcache locking problems. Since we use it only inside a single Scons process, we can avoiding using Windows mutexes, and use a process level lock instead.
-
Plugins: Fix plugin for support for
eventlet
. Fixed in 0.6.12.3 already. -
Standalone: Added support for latest
zmq
on Windows. -
Scons: the
--quiet
flag was not fully honored yet, with Scons still making a few outputs. -
Standalone: Added support for alternative DLL name for newer
PyGTK3
on Windows. Fixed in 0.6.12.4 already. -
Plugins: Fix plugin for support for
gevent
. Fixed in 0.6.12.4 already. -
Standalone: Added yet another missing implicit dependency for
pandas
. -
Plugins: Fix, the
qt-plugins
plugin could stumble over.mesh
files. -
Windows: Fix, dependency walker wasn't properly working with unicode
%PATH%
which could e.g. happen with a virtualenv in a home directory that requires them. -
Python3: Fixed a few Python debug mode warnings about unclosed files that have sneaked into the codebase.
New Features
-
Added new options
--windows-force-stdout-spec
and--windows-force-stderr-spec
to force output to files. The paths provided at compile time can resolve symbolic paths, and are intended to e.g. place these files near the executable. Check the User Manual for a table of the currently supported values. At this time, the feature is limited to Windows, where the need arose first, but it will be ported to other supported OSes eventually. These are most useful for programs run as--windows-disable-console
or with--enable-plugin=windows-service
.Note
These options have since been renamed to
--force-stdout
and--force-stderr
and have been made to work on all OSes. -
Windows: Added option
--windows-onefile-tempdir-spec
(since renamed to--onefile-tempdir-spec
) to provide the temporary directory used with--windows-onefile-tempdir
in onefile mode, you can now select your own pattern, and e.g. hardcode a base directory of your choice rather than%TEMP
. -
Added experimental support for
PySide2
with workarounds for compiled methods not being accepted by its core. There are known issues withPySide2
still, but it's working fine for some people now. Upstream patches will have to be created to remove the need for workarounds and full support.
Optimization
-
Use binary operation code for their in-place variants too, giving substantial performance improvements in all cases that were not dealt with manually already, but were covered in standard binary operations. Until now only some string, some float operations were caught sped up, most often due to findings of Nuitka being terribly slower, e.g. not reusing string memory for inplace concatenation, but now all operations have code that avoids a generic code path, that is also very slow on Windows due calling to using the embedded Python via API being slow.
-
For mixed type operations, there was only one direction provided, which caused fallbacks to slower forms, e.g. with
long
andfloat
values leading to inconsistent results, such thata - 1
and1 - a
being accelerated or not. -
Added C boolean optimization for a few operations that didn't have it, as these allow to avoid doing full computation of what the object result would have to do. This is not exhausted fully yet.
-
Python3: Faster
+
/-
/+=
/-=
binary and in-place operations withint
values providing specialized code helpers that are much faster, esp. in cases where no new storage is allocated for in-place results and where not a lot of digits are involved. -
Python2: The Python3
int
code is the Python2long
type and benefits from the optimization of+
/-
/+=
/-=
code as well, but of course its use is relatively rare. -
Improved
__future__
imports to become hard imports, so more efficient code is generated for them. -
Counting of instances had a run time impact by providing a
__del__
that was still needed to be executed and limits garbage collection on types with older Python versions. -
UI: Avoid loading
tqdm
module before it's actually used if at all (it may get disabled by the user), speeding up the start of Nuitka. -
Make sure to optimize internal helpers only once and immediately, avoiding extra global passes that were slowing down Python level compilation by of large programs by a lot.
-
Make sure to recognize the case where a module optimization can provide no immediate change, but only after a next run, avoiding extra global passes originating from these, that were slowing down compilation of large programs by a lot. Together with the other change, this can improve scalability by a lot.
-
Plugins: Remove implicit dependencies for
pkg_resources.extern
and use aliases instead. Using one of the packages, was causing all that might be used, to be considered as used, with some being relatively large. This was kind of a mistake in how we supported this so far. -
Plugins: Revamped the
eventlet
plugin, include needed DNS modules as bytecode rather than as source code, scanning them withpkgutil
rather than filesystem, with much cleaner code in the plugin. The plugin is also now enabled by default.
Organizational
-
Removed support for
pefile
dependency walker choice and inline copy of the code. It was never as good giving incomplete results, and after later improvements, slower, and therefore has lost the original benefit over using Dependency Walker that is faster and more correct. -
Added example for onefile on Windows with the version information and with the temporary directory mode.
-
Describe difference in file access with onefile on Windows, where
sys.argv[0]
andos.path.dirname(__file__)
will be different things. -
Added inline copy of
tqdm
to make sure it's available for progress bar output for 2.7 or higher. Recommend having it in the Debian package. -
Added inline copy of
colorama
for use on Windows, where on some terminals it will give better results with the progress bar. -
Stop using old PyLint for Python2, while it would be nice to catch errors, the burden of false alarms seems to high now.
-
UI: Added even more checks on options that make no sense, made sure to do this only after a possible restart in proper environment, so warnings are not duplicated.
-
For Linux onefile, keep appimage outputs in case of an error, that should help debugging it in case of issues.
-
UI: Added traces for plugin provided implicit dependencies leading to inclusions.
-
Added inline copy of
zstd
C code for use in decompression for the Windows onefile bootstrap, not yet used though. -
Added checks to options that accept package names for obvious mistakes, such that
--include-package-data --mingw64
will not swallow an option, as that is clearly not a package name, that would hide that option, while also not having any intended effect. -
Added ignore list for decision to recompile extension modules with available source too. For now, Nuitka will not propose to recompile
Cython
modules that are very likely not used by the program anyway, and also not forlxml
until it's clear if there's any benefit in that. More will be added in the future, this is mostly for cases, where Cython causes incompatibilities. -
Added support for using abstract base classes in plugins. These are not considered for loading, and allow nicer implementation of shared code, e.g. between
PyQt5
andPySide2
plugins, but allow e.g. to enforce the provision of certain overloads. -
User Manual: Remove the instruction to install
clcache
, since it's an inline copy, this makes no sense anymore and that was obsolete. -
Updated PyLint to latest versions, and our requirements in general.
Cleanups
-
Started removal of PyLint annotations used for old Python2 only. This will be a continuous action to remove these.
-
Jinja2 based static code generation for operations was cleaned up, to avoid code for static mismatches in the result C, avoiding language constructs like
if (1 && 0)
with sometimes larger branches, replacing it with Jinja2 branches of the{% if ... %}
form. -
Jinja2 based Python2
int
code was pioniering the use of macros, but this was expanded to allow kinds of types for binary operations, allow their reuse for in-place operation, with these macros making returns via goto exits rather than return statements in a function. Landing pads for these exits can then assign target values for in-place different from what those for binary operation result return do. -
Changed the interfacing of plugins with DLL dependency detection, cleaning up the interactions considerably with more unified code, and faster executing due to cached plugin decisions.
-
Integrate manually provided slot function for
unicode
andstr
into the standard static code generation. Previously parts were generated and parts could be generated, but also provided with manual code. The later is now all gone. -
Use a less verbose progress bar format with less useless infos, making it less likely to overflow.
-
Cleanup how payload data is accessed in Windows onefile bootstrap, preparing the addition of decompression, doing the reading from the file in only one dedicated function.
-
When Jinja2 generated exceptions in the static code, it is now done via proper Jinja2 macros rather than Python code, and these now avoid useless Python version branches where possible, e.g. because a type like
bytes
is already Python version specific, with the goal to get rid ofPyErr_Format
usage in our generated static code. That goal is future work though. -
Move safe strings helpers (cannot overflow) to a dedicated file, and remove the partial duplication on the Windows onefile bootstrap code.
-
The Jinja2 static code generation was enhanced to track the usage of labels used as goto targets, so that error exits, and value typed exits from operations code no longer emitted when not used, and therefore labels that are not used are not present.
-
For implicit dependencies, the parsing of the
.pyi
file of a module no longer emits a dependency on the module itself. Also from plugins, these are now filtered away.
Tests
-
Detect if onefile mode has required downloads and if there is user consent, otherwise skip onefile tests in the test runner.
-
Need to also allow accesses to files via short paths on Windows if these are allowed long paths.
-
The standalone tests on Windows didn't actually take run time traces and therefore were ineffective.
-
Added standalone test for
glfw
coverage. -
construct-based tests for in-place operations are now using a value for the first time, and then a couple more times, allowing for real in-place usage, so we are sure we measure correctly if that's happening.
Summary
Where the big change of the last release were optimization changes to reduce the global passes, this release addresses remaining causes for extra passes, that could cause these to still happen. That makes sure, Nuitka scalability is very much enhanced in this field again.
The new features for forced outputs are at this time Windows only and make a huge difference when it comes to providing a way to debug Windows Services or programs in general without a console, i.e. a GUI program. These will need even more specifiers, e.g. to cover program directory, rather than exe filename only, but it's a very good start.
On the tooling side, not a lot has happened, with the clcache fix, it seems that locking issues on Windows are gone.
The plugin changes from previous releases had left a few of them in a state where they were not working, but this should be restored. Interaction with the plugins is being refined constantly, and this releases improved again on their interfaces. It will be a while until this becomes stable.
Adding support for PySide2 is a headline feature actually, but not as perfect as we are used to in other fields. More work will be needed, also in part with upstream changes, to get this to be fully supported.
For the performance side of things, the in-place work and the binary operations work has taken proof of concept stuff done for Python2 and applied it more universally to Python3. Until we cover all long operations, esp. *
seems extremely important and is lacking, this cannot be considered complete, but it gives amazing speedups in some cases now.
Future releases will revisit the type tracing to make sure, we know more about loop variables, to apply specific code helpers more often, so we can achieve the near C speed we are looking for in the field of int
performance.
28 May 2025 6:31pm GMT
Kay Hayen: Nuitka Release 0.6.11
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, "download now".
This release is a massive improvement in many ways with lots of bug fixes and new features.
Bug Fixes
-
Fix, the
.pyi
file parser didn't handle relative imports. Fixed in 0.6.10.1 already. -
Windows: Fix, multiprocessing plugin was not working reliable following of imports from the additional entry point. Fixed in 0.6.10.1 already.
-
Pipenv: Workaround parsing issue with our
setup.py
to allow installation from GitHub. Fixed in 0.6.10.1 already. -
Merging of branches in optimization could give nondeterministic results leading to more iterations than necessary. Fixed in 0.6.10.1 already.
-
Windows: Avoid profile powershell when attempting to resolve symlinks. Fixed in 0.6.10.1 already.
-
Windows: Fix, always check for stdin, stdout, and stderr presence. This was so far restricted to gui mode applications, but it seems to be necessary in other situations too. Fixed in 0.6.10.1 already.
-
Python2: Fix,
--trace-execution
was not working for standalone mode but can be useful for debugging. Fixed in 0.6.10.1 already. -
Windows: Onefile could run into path length limits. Fixed in 0.6.10.3 already.
-
Windows: The winlib gcc download link became broken and was updated. Fixed in 0.6.10.3 already.
-
Plugins: The "__main__" module was not triggering all plugin hooks, but it needs to for completeness.
-
Standalone: Fix, symlinked Python installations on Windows were not working, with dependency walker being unable to look into these. Fixed in 0.6.10.4 already.
-
Standalone: Fix support for numpy on Windows and macOS, the plugin failed to copy important DLLs. Fixed in 0.6.10.4 already.
-
Python3: For versions before 3.7, the symlink resolution also needs to be done, but wasn't handling the bytes output yet. Fixed in 0.6.10.4 already.
-
Fix, folder based inclusion would both pick up namespace folders and modules of the same name, crashing the compilation due to conflicts. Fixed in 0.6.10.4 already.
-
Fix, the
--lto
wasn't used for clang on non-Windows yet. -
Fix, the order of locals dict releases wasn't enforced, which could lead to differences that break caching of C files potentially. Fixed in 0.6.10.5 already.
-
Fix,
hash
nodes didn't consider if their argument was raising, even if the type of the argument wasstr
and therefore the operation should not. Fixed in 0.6.10.5 already. -
Fix, need to copy type shape and escape description for the replacement inverted comparisons when used with
not
, otherwise the compilation can crash as these are expected to be present at all times. Fixed in 0.6.10.5 already. -
Fix, some complex constant values could be confused, e.g.
-0j
and0j
. These corner cases were not properly considered in the constant loading code, only forfloat
so far. -
Standalone: Fix, bytecode only standard library modules were not working. This is at least used with Fedora 33.
-
Linux: Fix, extension modules compiled with
--lto
were not working. -
Windows: Retry if updating resources fails due to Virus checkers keeping files locked.
-
Plugins: Pre- and postload code of modules should not be allowed to cause
ImportError
, as these will be invisible to the other parts of optimization, instead make them unraisable error traces. -
Standalone: Adding missing import for SciPy 1.6 support.
-
Windows: Fix, only export required symbols when using MinGW64 in module mode.
New Features
-
Python3.9: Added official support for this version.
-
Onefile: Added command line options to include data files. These are
--include-package-data
which will copy all non-DLLs and non-Python files of package names matching the pattern given. And--include-data-file
takes source and relative target file paths and copies them. For onefile this is the only way to include files, for standalone mode they are mostly a convenience function. -
Onefile: Added mode where the file is unpacked to a temporary folder before running instead of doing it to appdata.
-
Onefile: Added linux specific options
--linux-onefile-icon
to allow provision of an icon to use in onefile mode on Linux, so far this was only available as the hard coded path to a Python icon, which also didn't exist on all platforms. -
UI: Major logging cleanup. Everything is now using our tracing classes and even error exits go through there and are therefore colored if possible.
-
Plugins: Make it easier to integrate commercial plugins, now only an environment variable needs to point to them.
-
UI: Enhanced option parsing gives notes. This complains about options that conflict or that are implied in others. Trying to catch more usage errors sooner.
-
Plugins: Ignore exceptions in buggy plugin code, only warn about them unless in debug mode, where they still crash Nuitka.
-
Scons: More complete scons report files, includes list values as well and more modes used.
-
Windows: The
clcache
is now included and no longer used from the system. -
Output for
clcache
andccache
results got improved. -
Enhanced support for
clang
, on Windows if present near agcc.exe
like it is the case for some winlibs downloads, it will be used. To use it provide--mingw64 --clang
both. Without the first one, it will meanclangcl.exe
which uses the MSVC compiler as a host.
Optimization
-
Some modules had very slow load times, e.g. if they used many list objects due to linear searches for memory deduplication of objects. We now have dictionaries of practically all constant objects loaded, making these more instant.
-
Use less memory at compile time due using
__slots__
for all node types, finally figured out, how to achieve this with multiple inheritance. -
Use hedley for compiler macros like
unlikely
as they know best how to do these. -
Special case the merging of 2 branches avoiding generic code and being much faster.
-
Hard imports have better code generated, and are being optimized into for the few standard library modules and builtin modules we handle, they also now annotate the type shape to be module.
-
No longer annotate hard module import attribute lookups as control flow escapes. Not present attributes are changed into static raises. Trust for values is configured for a few values, and experimental.
-
Avoid preloaded packages for modules that have no side effects and are in the standard library, typically
.pth
files will use e.g.os
but that's not needed to be preserved. -
Use
incbin
for including binary data through inline assembly of the C compiler. This covers many more platforms than our previous linker option hacks, and the fallback to generated C code. In fact everything but Windows uses this now.
Organizational
-
Windows: For Scons we now require a Python 3.5 or higher to be installed to use it.
-
Windows: Removed support for gcc older than version 8. This specifically affects CondaCC and older MinGW64 installations. Since Nuitka can now download the MinGW64 10, there is no point in having these and they cause issues.
-
We took over the maintenance of clcache as Nuitka/clcache which is not yet ready for public consumption, but should become the new source of clache in the future.
-
Include an inline copy of clcache in Nuitka and use it on Windows for MSVC and ClangCL.
-
Removed compatibility older aliases of follow option,
--recurse-*
and require--follow-*
options to be used instead. -
For pylint checking, the tool now supports a
--diff
mode where only the changed files get checked. This is much faster and allows to do it more often before commit. -
Check the versions of isort and black when doing the auto-format to avoid using outdated versions.
-
Handling missing pylint more gracefully when checking source code quality.
-
Make sure to use the codespell tool with Python3 and make sure to error exit when spelling problems were found, so we can use this in GitHub actions too.
-
Removed Travis config, we now only use GitHub actions.
-
Removed landscape config, it doesn't really exist anymore.
-
Bumped all PyPI dependnecies to their latest versions.
-
Recommend ccache on Debian, as we now consider the absence of ccache something to warn about.
-
Plugins: The DLLs asked for by plugins that are not found are no longer warned about.
-
Allow our checker and format tools to run on outside of tree code. We are using that for Nuitka/clcache.
-
Added support for Fedora 33 and openSUSE 15.3, as well as Ubuntu Groovy.
-
Windows: Check if Windows SDK is installed for MSVC and ClangCL.
-
Windows: Enhanced wording in case no compiler was found. No longer tell people how to manually install MinGW64, that is no longer necessary and
pywin32
is not needed to detect MSVC, so it's not installed if not found. -
Detect "embeddable Python" by missing include files, and reject it with proper error message.
-
Added onefile and standalone as a use case to the manual and put also the DLL and data files problems as typically issues.
Cleanups
-
Avoid decimal and string comparisons for Python versions checks, these were lazy and are going to break once 3.10 surfaces. In testing we now use tuples, in Nuitka core hexacimal values much like CPython itself does.
-
Stop using subnode child getters and setters, and instead only use subnode attributes. This was gradually changed so far, but in this release all remaining uses have migrated. This should also make the optimization stage go faster.
-
Change node constructors to not use a decorator to resolve conflicts with builtin names, rather handle these with manual call changes, the decorator only made it difficult to read and less performant.
-
Move safe string helpers to their own dedicated helper file, allowing for reuse in plugin code that doesn't want to use all of Nuitka C helpers.
-
Added utils code for inline copy imports, as we use that for quite a few things now.
-
Further restructured the Scons files to use more common code.
-
Plugins: The module name objects now reject many
str
specific APIs that ought to not be used, and the code got changed to use these instead, leading to cleaner and more correct usages. -
Using named tuples to specify included data files and entry points.
-
Use
pkgutil
in plugins to scan for modules rather than listing directories.
Tests
-
New option to display executed commands during comparisons.
-
Added test suite for onefile testing.
Summary
This release has seen Python3.9 and Onefile both being completed. The later needs compression added on Windows, but that can be added in a coming release, for now it's fully functional.
The focus clearly has been on massive cleanups, some of which will affect compile time performance. There is relatively little new optimization otherwise.
The adoption of clcache enables a very fast caching, as it's now loaded directly into the Scons process, avoiding a separate process fork.
Generally a lot of polishing has been applied with many cleanups lowering the technical debt. It will be interesting to see where the hard module imports can lead us in terms of more optimization. Static optimization of the Python version comparisons and checks is needed to lower the amount of imports to be processed.
Important fixes are also included, e.g. the constants loading performance was too slow in some cases. The multiprocessing
on Windows and numpy
plugins were regressed and finally everything ought to be back to working fine.
Future work will have to aim at enhanced scalability. In some cases, Nuitka still takes too much time to compile if projects like Pandas include virtually everything installed as an option for it to use.
28 May 2025 6:31pm GMT