Python 3.14 stack overflow detection is incompatible with C++ Boost make_fcontext() coroutines

[vstinner]

Python 3.14 introduced a new stack overflow detection mecanism: InternalDocs/stack_protection.md (#130396).

The KiCad application uses C++ Boost make_fcontext() coroutines which runs coroutine in their own stack.

Code example from fcontext doc:

// context-function
void f(intptr);

// creates a new stack
std::size_t size = 8192;
void* sp(std::malloc(size));

// context fc uses f() as context function
// fcontext_t is placed on top of context stack
// a pointer to fcontext_t is returned
fcontext_t fc(make_fcontext(sp,size,f));

_Py_InitializeRecursionLimits() is called in the main thread, whereas _Py_CheckRecursiveCall() is called for the first time in a coroutine (make_fcontext()).

Problem: Python detects a stack overflow because it's not aware that the stack base address and size changed when make_fcontext() was called.

pthread functions such as pthread_attr_getguardsize() are incompatible with make_fcontext().

cc @markshannon

Linked PRs

• gh-139653: If platform API doesn't give the current stack, use generic fallback #139667
• gh-139653: Add PyUnstable_ThreadState_SetStackProtection() #139668
• gh-139653: Remove assertions in _Py_InitializeRecursionLimits() #141551
• [3.14] gh-139653: Add PyUnstable_ThreadState_SetStackProtection() (GH-139668)  #141661
• GH-139653: Only raise an exception (or fatal error) when the stack pointer is about to overflow the stack. #141711
• [3.14] GH-139653: Only raise an exception (or fatal error) when the stack pointer is about to overflow the stack. (GH-141711) #141892
• [3.14] GH-139653: Only raise an exception (or fatal error) when the stack pointer is about to overflow the stack. (GH-141711) #141944

[vstinner]

@hugovk: I used the release-blocker label to raise awareness of this issue, but I'm not sure that it should hold Python 3.14.0 final release. We might be able to develop a workaround in Python 3.14.1.

[vstinner]

KiCad issue: https://gitlab.com/kicad/code/kicad/-/issues/21890#note_2800807662

[encukou]

cc @markshannon

[penguin42]

To me the simplest partial fix would be to reread the stack limits before throwing the abort; that would stop
spurious aborts (and be low cost), however there's also the opposite case, where if the new stack is higher than the old one, the check won't trigger at all.

[zooba]

Do we need a new (internal) API that can be called after jump_fcontext? Presumably it only occurs when user code explicitly requests it, which should mean that it can also call a CPython API after switching if needed, and it looks like the members of fcontext_t are public, so we can accept them as arguments.

[encukou]

Do these coroutines each use their own Python thread state?

reread the stack limits before throwing the abort

Unfortunately on some platforms where we can't read stack limits, so we approximate stack top using the current address. In that case, stack limits will be different each time.

[markshannon]

We can (mostly) do this without a new API, using a variant of @penguin42's suggestion.

What we would do is:

• Change the recursion check from here_addr < tstate->c_stack_soft_limit to here_addr - state->c_stack_window_base < WINDOW_SIZE
• If the check fails, check if there is an actual stack overflow, then update the window if we are within the stack.

The actual window size doesn't matter, but it should be a constant to keep the cost low. It will a bit slower, but probably not measurably so.

Unfortunately on some platforms where we can't read stack limits, so we approximate stack top using the current address. In that case, stack limits will be different each time.

True, but this bug has only surfaced on linux (so far).

[markshannon]

Having said that, new unstable ABI to notify of stack changes is probably the best long term option.

[vstinner]

Do these coroutines each use their own Python thread state?

Coroutines reuse the existing Python thread state.

Do we need a new (internal) API that can be called after jump_fcontext?

It sounds like a good idea, but this function would need a stack base address and stack size parameters, since pthread_attr_getguardsize() doesn't work in this case. make_fcontext() has a known stack base address and size: its the first and second arguments.