JIT digest differs based on filesystem order

[hroncok]

Bug report

Bug description:

Similar to #138061 in Fedora, we have found out that the digest in JIT stencils in 3.15.0a2 differs. (Clarification edit: This is not a new bug in a2, we merely have not discovered it before.)

Due to the way we are trying to use prebuilt stencils, we assert that if we build them, they are the same as if we use the prebuilt ones. The check recently failed in one of our build systems¹. Like this:

--- /builddir/build/SOURCES/Python-3.15.0a2-x86_64-debug-jit_stencils.h	2025-11-19 00:00:00.000000000 +0000
+++ build/debug/jit_stencils-x86_64-unknown-linux-gnu.h	2025-12-07 21:16:08.857507778 +0000
@@ -1,4 +1,4 @@
-// 9df56c4e6479c166bfae5419d30792b4e139844b5e435bfee44e8a457dda422f
+// 5a195eb685aae6c83b23ec93bad4ef2c3754581224248419a33a41050b68335e
 // $ python3.15 /builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit/build.py x86_64-redhat-linux-gnu --output-dir . --pyconfig-dir . --cflags= --llvm-version= --debug
 
 void

With @befeleme we were debugging the problem. Why is the digest different when all the rest of the file is bit-by-bit identical.

I added some additional debug prints:

--- a/Tools/jit/_targets.py
+++ b/Tools/jit/_targets.py
@@ -66,17 +66,23 @@ class _Target(typing.Generic[_S, _R]):
     def _compute_digest(self) -> str:
         hasher = hashlib.sha256()
         hasher.update(self.triple.encode())
+        print(f"{self.triple=}")
         hasher.update(self.debug.to_bytes())
+        print(f"{self.debug=}")
         hasher.update(self.cflags.encode())
+        print(f"{self.cflags=}")
         # These dependencies are also reflected in _JITSources in regen.targets:
         hasher.update(PYTHON_EXECUTOR_CASES_C_H.read_bytes())
+        print(f"{PYTHON_EXECUTOR_CASES_C_H=}")
         hasher.update((self.pyconfig_dir / "pyconfig.h").read_bytes())
+        print(f"{(self.pyconfig_dir / "pyconfig.h")=}")
         for dirpath, _, filenames in sorted(os.walk(TOOLS_JIT)):
             # Exclude cache files from digest computation to ensure reproducible builds.
             if dirpath.endswith("__pycache__"):
                 continue
             for filename in filenames:
                 hasher.update(pathlib.Path(dirpath, filename).read_bytes())
+                print(dirpath, filename)
         return hasher.hexdigest()
 
     async def _parse(self, path: pathlib.Path) -> _stencils.StencilGroup:

This inadvertently also changed the generated hash. However still different in one of the Fedora build systems.

However, the output revealed the problem. This is what I got in the environment where it previously succeeded:

===============================================================
JIT support for x86_64-unknown-linux-gnu is still experimental!
            Please report any issues you encounter.            
===============================================================

self.triple='x86_64-unknown-linux-gnu'
self.debug=True
self.cflags=''
PYTHON_EXECUTOR_CASES_C_H=PosixPath('/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Python/executor_cases.c.h')
(self.pyconfig_dir / "pyconfig.h")=PosixPath('/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/build/debug/pyconfig.h')
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit README.md
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit _llvm.py
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit _optimizers.py
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit _schema.py
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit _stencils.py
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit _writer.py
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit build.py
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit jit.h
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit jit_infra.md
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit mypy.ini
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit template.c
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit trampoline.c
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit _targets.py

And this is what I get in the environment where it did not match:

===============================================================
JIT support for x86_64-unknown-linux-gnu is still experimental!
            Please report any issues you encounter.            
===============================================================

self.triple='x86_64-unknown-linux-gnu'
self.debug=True
self.cflags=''
PYTHON_EXECUTOR_CASES_C_H=PosixPath('/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Python/executor_cases.c.h')
(self.pyconfig_dir / "pyconfig.h")=PosixPath('/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/build/debug/pyconfig.h')
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit _targets.py
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit trampoline.c
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit template.c
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit mypy.ini
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit jit_infra.md
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit jit.h
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit build.py
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit _writer.py
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit _stencils.py
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit _schema.py
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit _optimizers.py
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit _llvm.py
/builddir/build/BUILD/python3.15-3.15.0_a2-build/Python-3.15.0a2/Tools/jit README.md

See, the order of the files is reversed.

I believe the function that computes the digest...

https://github.com/python/cpython/blob/v3.15.0a2/Tools/jit/_targets.py#L66-L80

Kinda wanted to avoid this by calling sorted() on os.walk(), however, there needs to be one more sorted at:

https://github.com/python/cpython/blob/v3.15.0a2/Tools/jit/_targets.py#L78

That way, files within one folder are also sorted (and deterministic, regardless of the filesystem). I will send a PR.

CPython versions tested on:

3.14, 3.15, CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-142454: Make the JIT digest more deterministic by sorting the files in Tools/jit #142455
• [3.13] gh-142454: Make the JIT digest more deterministic by sorting the files in Tools/jit (GH-142455) #142484
• [3.14] gh-142454: Make the JIT digest more deterministic by sorting the files in Tools/jit (GH-142455) #142485

Footnotes

1. Copr ↩

[vstinner]

cc @brandtbucher @savannahostrowski